Questions Size of Snapshot

Hi silifuzz,
I use Silifuzz as part of my research and after walking a while through the codebase I wanted to ask you some questions.

Is there a way to define the number of instructions that the snapshot will contain (e.g after fuzzing unicorn the number of x86 instructions that a snapshot will contain ) ? Is it possible to create bigger snapshots (e.g containing 1000 instructions)? The paper mentions : "Our typical snapshot contains less than 100 bytes of code and runs in microseconds, but it can be arbitrarily large."
Is there a way to check the size of a snapshot and more specifically get the number of instructions executed?
Thank you in advance

Hi @ChrisLaspias

The sample proxies SiliFuzz provides do not have an option to define the exact number of instructions. Instead, there's an upper limit that is currently set to 100 for x86 and 4096  for aarch64. Both numbers are "more of a guideline" i.e. there's no guarantee this limit will ever be reached. Generally, snapshots can be arbitrary large both in terms of the number of dynamic instruction and in terms of how much memory they map. We have experimented with both larger and smaller snapshots, and found that smaller snapshots work better for our use case.
You can use snap_tool trace command to produce the trace (currently, x86-only). This will execute the snapshot and print every instruction.

Hi @ksteuck ,
Thank you for the quick answer.

the answer depends on what you mean by "dynamic linking". The runner binary is always linked fully statically.
$ file bazel-bin/runner/reading_runner_main_nolibc
bazel-bin/runner/reading_runner_main_nolibc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[xxHash]=394d49c23eedd37a, not stripped

There are then two ways to load the corpus file into the runner. The default is to read corpus from a file (aka reading runner).
There's also an option to codegen Snapshots => C++ that can then be compiled into a self-contained (baked-in) runner binary. This latter option doesn't have good tooling around it and may be deprecated in the future. You can see an example of how this is done here.  But the TL;DR is genrule() => C++ => cc_library() => cc_binary()
HTH

@ksteuck
Thank you very much for helping. That was exactly what I was looking for.


cmd/pack: gc linkobjs not recognized in 1.16b1

$ cat test.sh
#!/bin/bash
echo "package main" > main.go
echo "func main() {}" >> main.go
$GO tool compile -p main -complete -linkobj  pkg.a -o pkg-i.a main.go
$GO tool pack c link.a pkg.a
echo "packed contents:"
$GO tool pack tv link.a
echo "unpacked member head:"
$GO tool pack x link.a
head -n1 pkg.a
$ GO=go1.16b1/bin/go ./test.sh
packed contents:
-rw-rw-r--      0/0              1258 Dec 31 18:00 1969 pkg.a
unpacked member head:
!<arch>
$ GO=go1.15.6/bin/go ./test.sh
packed contents:
-rw-r--r--      0/0               852 Dec 31 18:00 1969 pkg.a
unpacked member head
go object linux amd64 go1.15.6 X:framepointer

CC @thanm @jeremyfaller @cherrymui

Marking as a release blocker to make sure we figure out what is going on.

Sorry, I'm not sure what you wanted to do here.
pkg.a is already an archive file. You pack it again to link.a, then unpack it, but you are not expecting to get pkg.a back. Why do you want to pack an already-packed archive file?

This situation arises when creating a package archive containing gc output and native object files. go simply appends native objects to the archive that gc emits using its knowledge of the archive format. Historically (since CL 102236), one could achieve the same end with cmd/pack because pack "saw through" gc output archives.
(I'm not at all wedded to using cmd/pack. If there was some way to pass native object files to the final link command, less copying would have to happen.)

I think you can use go tool pack r to append native objects to Go archive file, instead of go tool pack c.

If there was some way to pass native object files to the final link command, less copying would have to happen.

If you use go build, if you put a native object file named .syso in the same directory as Go source files in a package, when building the package the go command will pack it automatically. See the runtime/race package as an example.

I think you can use go tool pack r to append native objects to Go archive file, instead of go tool pack c.

c is a bit more convenient for me because I'm making a separate file from the gc output and used to work fine.

If there was some way to pass native object files to the final link command, less copying would have to happen.

If you use go build, if you put a native object file named .syso in the same directory as Go source files in a package, when building the package the go command will pack it automatically. See the runtime/race package as an example.

I just meant it would be nicer from a scalability perspective to not have to duplicate native objects into an archive in order to pass them to link. (I.e., the same reason thin archives and later gold/lld's --start-lib & --end-lib exist.)

The CL above restores the Go 1.15 behavior.
But I wanted to mention that this behavior is somewhat accidental. With Go 1.15's cmd/pack, isGoCompilerObjFile is supposed to return true when the input has exactly two entries __.PKGDEF and _go_.o. This is what I did when I rewrote cmd/pack in Go 1.16. However, when the input has one single entry _go_.o, it actually returns true. When name is _go_.o, the first iteration of the inner loop at https://go.googlesource.com/go/+/refs/tags/go1.15.6/src/cmd/pack/pack.go#537 will simply skip match[0] and the second iteration returns true. Looking at the code it doesn't seem to be intended.
If you want the append semantics, I would recommend using the "r" command, instead of this peculiar undocumented "see through" behavior of the "c" command.

The CL above restores the Go 1.15 behavior.

Thank you.

But I wanted to mention that this behavior is somewhat accidental. With Go 1.15's cmd/pack, isGoCompilerObjFile is supposed to return true when the input has exactly two entries __.PKGDEF and _go_.o. This is what I did when I rewrote cmd/pack in Go 1.16. However, when the input has one single entry _go_.o, it actually returns true. When name is _go_.o, the first iteration of the inner loop at https://go.googlesource.com/go/+/refs/tags/go1.15.6/src/cmd/pack/pack.go#537 will simply skip match[0] and the second iteration returns true. Looking at the code it doesn't seem to be intended.
If you want the append semantics, I would recommend using the "r" command, instead of this peculiar undocumented "see through" behavior of the "c" command.

I thought the commit message of CL 102236 meant the behavior was not accidental. Before the compiler always emitted an archive, one could use pack c to put compiler output into an archive. The intention of the CL was to preserve that behavior after archives became the only output format.

Before the compiler always emitted an archive, one could use pack c to put compiler output into an archive. The intention of the CL was to preserve that behavior after archives became the only output format.

It is not guaranteed that the tool's (e.g. pack) behavior not change, especially undocumented behavior. Use the "r" command if you want to append.

With Go 1.15's cmd/pack, isGoCompilerObjFile is supposed to return true when the input has exactly two entries __.PKGDEF and _go_.o.

The comment certainly says that. But judging by the nested for loops, I think I did specifically intend for it to match archive files that contained (1) only __.PKGDEF, (2) only _go_.o, or (3) only __.PKGDEF and _go_.o in that order. (Though I notice now that it also matches empty archive files, which I think was not intended.)
I think if it was really my intention to require that both files were present, I would have used a single for loop.


Ionic 4 - StackController Page reuse uses old Page data

Bug Report
Ionic Info
Ionic:

   ionic (Ionic CLI)             : 4.5.0 (/usr/local/lib/node_modules/ionic)
   Ionic Framework               : @ionic/angular 4.0.0-beta.19
   @angular-devkit/build-angular : 0.10.7
   @angular-devkit/schematics    : 7.0.7
   @angular/cli                  : 7.0.7
   @ionic/angular-toolkit        : 1.2.0

System:

   NodeJS : v10.13.0 (/usr/local/bin/node)
   npm    : 6.4.1
   OS     : macOS Mojave

Describe the Bug
(There are arguably two interrelated but separate issues here. If I should create separate issues then please let me know! 🙂)
Forward navigation to a Page with the same URL (excluding query parameters) will modify the navigation stack history (not the browser history stack but the StackController stack, which are different). It will then re-use a previous Page instance that does not contain the latest navigation data from the forward navigation to the same URL.
Steps to Reproduce
Steps to reproduce the behavior:

Clone this repo on your machine: https://github.com/KevinKelchen/ionic4-stack-page-reuse . It is a slightly modified version of the sidemenu Angular stater template.
In the terminal, path into the repo root and run npm i.
Run ionic serve.
Click the Menu icon and then List.
Click Item 1 to forward navigate to the detail screen for Item 1.

You will notice there's a timestamp that displays. I chose a timestamp in this example so that there's an easy way to see on-screen what navigation data is being used.
This timestamp data was stored in a shared service prior to navigating from the source Page. It was then retrieved from the target Page from the same service. This pattern was outlined by Josh Morony here. The data in the service is keyed in a dictionary using the navId UUID URL query parameter.
A shared service is being used because we are migrating from Ionic 3 and the use of NavParams which allowed passing complex data like callback functions and objects which cannot be serialized in a lossless way in the URL or @ionic/storage.
A UUID was chosen so that navigating to what's otherwise the same URL will use the navigation parameters specific to the current navigation. This will also preserve previous navigation parameters so they can be used when revisiting a Page in the browser history.

Click the pencil/create button in the header. This will navigate you to a different Page type called List Detail More.
From this Page, click Item 1 to forward navigate to the detail screen for Item 1. Note that the timestamp is the old value that was displayed when you initially navigated to Item 1. This is because the StackController rolled back the stack and re-used the old instance of the Page for Item 1. The new navId URL query parameter was not used to retrieve the data from the shared service and thus is not using fresh data on this forward navigation.
To observe the StackController navigation stack modification, click the ion-back-button and it will take you to the initial List Page instead of List Detail More which is the screen you were previously on before revisiting Item 1.

Related Code
The steps to reproduce above link to this repository which is set up to reproduce the issue.
Expected Behavior
I expected forward navigation to a Page with the same URL to use the latest, current navigation data from the forward navigation to the same URL and not modify the StackController navigation stack history.
Additional Context
Here is some additional context from a message I posted in #16367:

Note: My definition of "reusing" a Page is when "forward" navigating to a URL and instead of pushing a new Page onto the StackController's "stack" it re-uses an instance of the Page that is already present in the stack.
I've been having issues with StackController as well but on Beta 15. I am using the Angular router imperatively in TypeScript but I believe I was experiencing them as well using the new NavController (which uses the Angular router).
Here's a related behavior I've observed:
When you use the router to "forward" navigate to a Page, if the URL (excluding query parameters) is the same as a Page that is already in the "stack," the StackController will remove all of the previous Pages in the stack back to the first occurrence of the matching URL/Page. It then re-uses that first occurrence of the Page in the stack. And so then when you use ion-back-button, which seems to ignore browser history and only uses the stack, the Page you go back to is not what you would expect--it's the Page prior to the first occurrence of the Page in the stack.

Note: Since the Android and browser back button seems to ignore the StackController's "stack," I've tried working around this issue to some success by creating my own custom back button component that just uses Angular's Location.back() when you click it. This tends to result in a lot of destroyed/re-initializing of Pages whose state you can attempt to preserve by storing off data in a shared service that is keyed with a unique ID (such as a UUID) present in the URL.

How StackController re-uses Pages has actually been tripping me up in another way:
If I use the Angular router to "forward" navigate to a Page with the same URL as one in the StackController's stack and I have navigation parameters that I store in a shared service (as opposed to the URL) and key the parameter storage in the service off of a URL query parameter such as a UUID (a query parameter--not using the matrix URL notation as noted at the end of this section), because the Page is being re-used due to a matching URL instead of being considered as different, the new query parameter value is not being used and the Page pulls the previous parameters out of the shared service due to using the old query parameter value.
Perhaps the solve is to work around it by just putting the UUID in as a matrix URL notation parameter and effectively opt-out of any StackController reuse of Pages.

Upsides

"Forward" navigating to the same Page/URL would use the correct parameters.
With all of the Pages in the stack remaining wired up (never destroyed or reused), when you navigate back to them they are exactly the way you left them when you navigated away, which is how Ionic 3 worked.

Downsides

It's non-default behavior that differs from how Ionic seems to think it should work in Ionic 4.
Although it's like Ionic 3, you lose an optimization in which all Pages stay wired-up in the DOM as you continue to forward navigate through the app which may potentially increase the amount of memory that's used by the app; I would have concerns of how this might scale as users keep forward navigating through the app.

Thank you for all that you do and for making a great framework! 🙂

Behavior is likewise the same with Beta 17.
Ionic Team: I can update the repo for reproducing the issue to Beta 17 if you wish. Please just let me know. 🙂
Thank you!

@bryce13950, the repository for reproducing the issue has now been updated to Beta 17. Please let me know if there's anything else that you would like.
Thank you! 😃

I'm having the same page caching issue for several weeks now, which is starting to affect critical areas of my applictation for different situations. I am refreshing this thread everyday in hope a solution comes soon. I think it's a fine solution that pages pulled from cached if the user is pressing back/going back in history, but I don't think pages should ever be pulled from cache if navigating to a url (going forward) - even if the page has previously been visited.
Example 1:
User A is logged into the application and starts on the 'Home' page with info relevant to user A. User A logs out and navigates to 'Log In'.
User B logs in from the same application and navigates to 'Home' page. Ionic see's that Home was previously visited (by user A) and instead of instantiating 'Home', the view is pulled from cache and displays everything that User A was seing.
Example 2) A user clicks a recieved notification about a new post was created. This navigates to the post wall which will load the new post and show it in the wall. But if the user is already on the post wall or the post wall have been visited already, the component/page is pulled from cache and always display the old data without being able to hook into normal Angular behavior for ngOnInit.
--
Note:
I tried to remove the following as suggestedby @bryce13950 without any change:

{ provide: RouteReuseStrategy, useClass: IonicRouteStrategy }

I tried to use navigateRoot() as suggested by @manucorporat without success:
navController.navigateRoot()

@Moelgaard85 Very strange. Could you provide a repo for this, so we have a couple of examples to look at?
@KevinKelchen Did removing that provider help you at all?

@bryce13950, thank you for looking into this and for your response! Sorry it's taken several days to respond--I was sick last week and then spent time trying to investigate this issue further.
The suggestion to remove Ionic's implementation of RouteReuseStrategy does make the sample I provided appear to work as expected. I then tried it in our real application and it does not work, however. 😕 I'm still not certain why. It seems like the ActivatedRoute.queryParamMap Observable in the Page that's being re-used is not getting the updated value, but I'm not sure. I will continue looking into it. It does strike me as curious that we'd have to deviate from Ionic's default behavior with removing IonicRouteStrategy--I wonder what down the road may be trickier if we do that. At any rate, thus far it seems that the problem lies elsewhere.
@Moelgaard85, I'm glad to see that I'm not alone! I've been wondering if I'm just doing something unconventional in our application because no one else has commented on this issue experiencing the same thing. If anyone else is experiencing these challenges, please chime in! Hopefully we can then work together towards a solution. 😀
A thought:
As far as I know, Angular by default destroys and re-inits "Page" components between navigations such that only one "Page" component is in the DOM/wired-up/exists at a time. Ionic Beta 16 introduced this behavior which gave many users significant difficulties because they were depending on Pages in the navigation stack history to still exist. I was amongst these users. Beta 17 changed the behavior to work more like Beta 15 again.
Yesterday I pinged Mike Hartington on Slack asking,

In Ionic 3 the "stack" seemed to work such that every forward navigation added a Page to the DOM. With Ionic 4 and Angular routing it seems that the StackController on a "forward" navigation will try to pop Pages off the stack and re-use a Page instance earlier in the stack that has a matching URL. This has been an interesting change in our app coming from Ionic 3 as Pages get destroyed/re-initialized and even re-used. Do you know what the rationale is for that new behavior? Trying to be more memory efficient? Something else?

He graciously responded with,

mmm, more that we're working with the angular router and it's ideas of how things should work

I did not ask for more details. Perhaps the thought was to find a middle-ground between Angular and Ionic 3 where there are still Pages in the DOM (the "stack") but they can eventually still be destroyed?
What if it could work such that when you use the Angular router to "forward" navigate to a Page that it will always add a new Page instead of rolling back the StackController stack and re-using an existing Page with matching URL? At the same time, perhaps it could actually remove the existing instances of the same Page with matching URL from the stack and still have some of that Page clean-up take place? If you then navigate back through history it would then re-create the destroyed Page when it needed to (similar to when using Angular's Location.back() or the browser/Android back button to go back to a Page where the Page was deleted from the StackController stack so it re-inits it)? Might have to think about this more.
Thank you!

@bryce13950, I've updated the sample project which reproduces the issue as well as the steps to reproduce in the original post. I left the providing of IonicRouteStrategy in place since that's the default; you will once again need to remove it in order to test using Angular's default RouteReuseStrategy.
After debugging through Angular and Ionic navigation code I was able to understand the code path differences between the sample project and our actual application. Turns out I needed to add navigating to a different Page before forward navigating to the same Page/URL again (like our actual application).

Thanks, @Moelgaard85. That’s a good article from Paul on clearing up misconceptions regarding the existence of Ionic lifecycle hooks in v4.
Using something like ionViewWillEnter instead of ngOnInit doesn't fix my issue with outdated routing data, however.
The post also refers to the vanilla Angular router outlet. This router outlet sounds like it would function more like Beta 16 in which there is no stack at all and only one Page exists in the DOM at a time. I imagine you would also lose the routing animations Ionic provides and perhaps other things. Like Beta 16, I’m not sure these characteristics would be the best fit for us as, like others who had issues with Beta 16, we depend on prior Pages still existing in the DOM. Even if Angular’s outlet seemed to otherwise “work” it would probably be an unconventional configuration that may not be well supported or tested.

So there's a lot here right now, but if you're looking to not create new instances of pages in V4, just disable the ionic route reuse strategy.
This was added as a way to maintain the same behavior as V3.

Hi @mhartington! 👋
My hope kind of is to create new instances of Pages when forward navigating like v3 did--not to use cached Pages like v4 has introduced. Like @bryce13950 also suggested, I did indeed remove { provide: RouteReuseStrategy, useClass: IonicRouteStrategy } in hopes that cached Pages would not be used, but that only helped when navigating directly from one Page to another Page of the same Page type; navigating to a different Page type in between still results in a cached Page in the stack being used instead of instantiating a new Page. I've also tried creating a custom RouteReuseStrategy where shouldReuseRoute() simply returns false. No dice--cached Pages are still being re-used when I would have expected a new Page to be created. Again, the steps to reproduce will demonstrate the behavior; I just updated the referenced sample project to Beta 19 as well.
For me it's almost like: I want to have a stack of Pages still in the DOM so when I back-navigate the Page instance is still there. However, when I forward navigate I want a fresh Page that's not re-using stuff. At the same time, it's probably not great to keep adding Page after Page to the DOM from a resource consumption standpoint like v3 did. Perhaps there's a middle ground where Pages further back in the history can be cleaned up as you use the application, such as when you forward navigate to the same URL it doesn't pop the stack back to, and re-use, the prior instance of the Page, but rather silently deletes the prior instance of the Page from the DOM. Then when you go back in history to that Page it will be re-initialized. Or something like that.
Perhaps this new behavior of using a cached Page in the StackController stack could work for me, but one of the main blockers to that is this very issue which has also been reported more specifically as #16736. Per that issue, if the routing observables fire appropriately we may be able to get by with using cached Pages.
Thank you for all of your time and assistance! 😀

The same happens to my project as well. I can't seem to find a single way to make it work properly.
It's either the login and home pages that are re-used or all of the other pages.
I deleted the Ionic reuse strategy to no avail.
All my in page links using the routerDirection="root" will navigate and destroy any pages but the new one. But pages navigated using the imperative Router.navigate() or navigateByUrl() will always stay in memory. replaceUrl: true will not fix this.

My solution to this issue was using event emitter Obvservable Subjects to notify cached views to reload the data.
In a service:
private loggedInEvent: Subject<any> = new Subject();

 emitDataUpdatedEvent() {
   this.loggedInEvent.next();
 }

 getDataUpdatedEvent$(): Observable<boolean> {
   return this.loggedInEvent.asObservable();
 }

In the view that has cache problems:
  dataUpdatedEvent: Subscription;

  ngOnInit() {
    this.dataUpdatedEvent = this.myService.getDataUpdatedEvent$()
      .subscribe(() => {
        this.loadData();
      });
  }

  ngOnDestroy(): void {
    this.dataUpdatedEvent.unsubscribe();
  }

  ionViewWillEnter() {
    this.loadData();
  }

If the view is cached when I call emitDataUpdatedEvent() the data will be reloaded in the view.

Thinking that this may be fixed at some point in the future I took an approach to re-use the resolvers on existing route configs to load data for the ActivatedRoute:
https://gist.github.com/zdennis/2f73f60b4bfbab6b98ca3d2b9bfbc1fd
I've attached a basic component to show how I'm using it.
This will call your resolver twice when navigating back to a cached page. This is not a concern for me as we'll use caching in the service layer during navigation transitions.

@KevinKelchen could you try the latest nightly and see if you're still having issues here?

Hi @KevinKelchen,
I tried out the repository you provided, and it looks like this issue has been resolved by #17888. Would you be able to try it out using the latest Ionic dev build to ensure that everything is working as you would expect? You can install the nightly using npm i @ionic/core@dev @ionic/angular@dev.
Thanks!

@daem0ndev and @liamdebeasi,
I will give it a shot!
Thank you! 😀

I am experiencing the same behavior as before.
Installing the nightly changed my package.json to use the following versions:
"@ionic/angular": "^4.2.0-dev.201903291804.8ba82fc",
"@ionic/core": "^4.2.0-dev.201903291804.8ba82fc",

@liamdebeasi, is running that command to install the nightly and following the steps to reproduce what you did to check if it resolved the issue? Did you make any other changes?
Thanks!

@daem0ndev reached out to me in effort to reproduce the issue. 😀
I'm not sure why it wasn't appearing during my initial attempts, but after blowing away node_modules and trying again I began seeing TypeError: this.router.getCurrentNavigation is not a function in the console. It looks like that method was introduced in Angular 7.2.0-beta.1. My sample app repo's lockfile was using an older version of Angular 7.
After updating to Angular 7.2.x, it works as expected! 🎉 Thank you, @daem0ndev!

As I've learned more, I think there are two related "issues" in this single GitHub issue:

ActivatedRoute Observables are not firing when using a cached Page.

This seems to be fixed now in the nightly version of Ionic thanks to @daem0ndev.

Coming from Ionic 3 and not using ion-tabs I was expecting a linear navigation stack.

When forward navigating to a cached Page, preceding Pages in the Ionic stack are popped off/destroyed. Using ion-back-button to go back doesn't take you to the screen you were previously looking at but rather the screen that's before the cached Page instance. Using a custom back button that uses Angular's Location.back() or the browser back button uses browser history and thus works differently and works as I would have expected. Downsides though include wondering if I'm going against Ionic's intended non-linear navigation design as well as Page instances having to be re-initialized (not a huge deal; works closer to vanilla Angular). Perhaps a separate feature request "issue" could be used for that discussion.

I'm going to close this issue since the primary issue is resolved. 🙂
Thank you!

@KevinKelchen I have tried npm i @ionic/core@dev @ionic/angular@dev and my package.json changed as below:
    "@ionic/angular": "^4.3.0-dev.201904171418.07e739a",
    "@ionic/core": "^4.3.0-dev.201904171418.07e739a",

and the Angular version is "@angular/common": "^7.2.2",
But still I am facing the problem like, when I navigate back to last page of stack, page reuses the old data.
Can you please suggest me on this?

Hi @Topiya and @karvannan-thi, 👋
This was fixed in Ionic Angular 4.2.0.
https://github.com/ionic-team/ionic/releases/tag/v4.2.0
@Topiya,
Are all of your Angular packages upgraded? Not just @angular/common?

My package.json
"@angular/common": "^7.2.2",
"@angular/core": "^7.2.2",
"@angular/forms": "^7.2.2",
"@angular/http": "^7.2.2",
"@angular/router": "^7.2.2",
"@ionic-native/core": "^5.0.0",
"@ionic/angular": "^4.2.0",
"cordova-android": "8.0.0",
mee too facing the same problem


Ensure hydrators config is honored in non-zend-mvc contexts

Per https://discourse.zendframework.com/t/validatormanager-not-calling-custom-validator-factory/109/5?u=matthew the hydrators config key is not honored currently unless the application is within a zend-mvc context. This is due to the fact that Zend\Hydrator\Module wires configuration for the Zend\ModuleManager\Listener\ServiceListener in order to push merged service configuration into the plugin during bootstrap; no similar logic is available when not in a zend-mvc context, however.
This patch fixes that situation by modifying the HydratorPluginManagerFactory to do the following:

If a ServiceListener service exists, it returns the plugin manager immediately (old behavior).
Otherwise, it checks for the config service, and, if found, a hydrators key with an array value. When found, it feeds that value to a Zend\ServiceManager\Config instance and uses that to configure the plugin manager before returning it.

Failing tests are against latest target, and are due to PHPUnit version mismatch issues; failures only affect HydratorPluginManagerCompatibilityTest; I've run the new tests manually with latest dependencies, and they pass as expected.


gstDecoder -- failed to set pipeline state to PLAYING - Orin AGX

I am trying to get Tao person detection running on my Jetson Orin AGX, but I'm hitting an apparent issue with GST at the capture stage. Please assist. (Also, thank you @dusty-nv for all of the great containers and work! Would be totally stuck without you)
Hardware: Jetson Orin AGX
Container: dustynv/jetson-inference:r36.2.0 (launched using the run.sh command from the jetson_containers repo)
Code:
from jetson_inference import detectNet
from jetson_utils import videoSource, videoOutput, Log

# RTSP URL
rtsp_url = "rtsp://camera:password@<IP_ADDRESS>:554/stream1"

# create video source
input = videoSource(rtsp_url)
# input = videoSource('/jetson-inference/pedestrians.mp4')

img = input.Capture()

I get the folllowing output if I try to run the code above. If I use the pedestrians video instead, the output is a bit different but the error is the same. I have confirmed that I can pull a frame from the camera using OpenCV, and I have confirmed that the pedstrians.mp4 is valid.
[gstreamer] initialized gstreamer, version <IP_ADDRESS>
[gstreamer] gstDecoder -- creating decoder for<EMAIL_ADDRESS>libEGL warning: DRI3: Screen seems not DRI3 capable
libEGL warning: DRI2: failed to authenticate
sh: 1: lsmod: not found
sh: 1: modprobe: not found
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
[gstreamer] gstDecoder -- discovered video resolution: 2560x1440  (framerate 15.000000 Hz)
[gstreamer] gstDecoder -- discovered video caps:  video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)5, profile=(string)main, width=(int)2560, height=(int)1440, framerate=(fraction)15/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
[gstreamer] gstDecoder -- pipeline string:
[gstreamer] rtspsrc location=rtsp://orintapo:nvidia@<IP_ADDRESS>:554/stream1 latency=10 ! queue ! rtph264depay ! nvv4l2decoder name=decoder enable-max-performance=1 ! video/x-raw(memory:NVMM) ! nvvidconv name=vidconv ! video/x-raw ! appsink name=mysink sync=false
[video]  created gstDecoder from rtsp://orintapo:nvidia@<IP_ADDRESS>:554/stream1
------------------------------------------------
gstDecoder video options:
------------------------------------------------
  -- URI: rtsp://orintapo:nvidia@<IP_ADDRESS>:554/stream1
     - protocol:  rtsp
     - location<EMAIL_ADDRESS>     - port:      554
  -- deviceType: ip
  -- ioType:     input
  -- codec:      H264
  -- codecType:  v4l2
  -- width:      2560
  -- height:     1440
  -- frameRate:  15
  -- numBuffers: 4
  -- zeroCopy:   true
  -- flipMethod: none
  -- loop:       0
  -- latency     10
------------------------------------------------
[gstreamer] opening gstDecoder for streaming, transitioning pipeline to GST_STATE_PLAYING
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
[gstreamer] gstDecoder -- failed to set pipeline state to PLAYING (error 0)
Traceback (most recent call last):
  File "/jetson-inference/test.py", line 12, in <module>
    img = input.Capture()
Exception: jetson.utils -- videoSource failed to capture image

Thanks @dusty-nv! Seems like the same issue:
root@orin-desktop:/jetson-inference# video-viewer pedestrians.mp4 pedestrians_2.mp4
[gstreamer] initialized gstreamer, version <IP_ADDRESS>
[gstreamer] gstDecoder -- creating decoder for pedestrians.mp4
libEGL warning: DRI3: Screen seems not DRI3 capable
libEGL warning: DRI2: failed to authenticate
sh: 1: lsmod: not found
sh: 1: modprobe: not found
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
[gstreamer] gstDecoder -- discovered video resolution: 960x540  (framerate 29.970030 Hz)
[gstreamer] gstDecoder -- discovered video caps:  video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)high, width=(int)960, height=(int)540, framerate=(fraction)30000/1001, pixel-aspect-ratio=(fraction)1/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
[gstreamer] gstDecoder -- pipeline string:
[gstreamer] filesrc location=pedestrians.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder name=decoder enable-max-performance=1 ! video/x-raw(memory:NVMM) ! nvvidconv name=vidconv ! video/x-raw ! appsink name=mysink
[video]  created gstDecoder from file:///opt/jetson-inference/pedestrians.mp4
------------------------------------------------
gstDecoder video options:
------------------------------------------------
  -- URI: file:///opt/jetson-inference/pedestrians.mp4
     - protocol:  file
     - location:  pedestrians.mp4
     - extension: mp4
  -- deviceType: file
  -- ioType:     input
  -- codec:      H264
  -- codecType:  v4l2
  -- width:      960
  -- height:     540
  -- frameRate:  29.97
  -- numBuffers: 4
  -- zeroCopy:   true
  -- flipMethod: none
  -- loop:       0
------------------------------------------------
[gstreamer] gstEncoder -- codec not specified, defaulting to H.264
failed to find/open file /proc/device-tree/model
[gstreamer] gstEncoder -- detected board 'Jetson AGX Orin Developer Kit'
[gstreamer] gstEncoder -- pipeline launch string:
[gstreamer] appsrc name=mysource is-live=true do-timestamp=true format=3 ! nvvidconv name=vidconv ! video/x-raw(memory:NVMM) ! nvv4l2h264enc name=encoder bitrate=4000000 maxperf-enable=1 ! video/x-h264 ! h264parse ! qtmux ! filesink location=pedestrians_2.mp4
[video]  created gstEncoder from file:///opt/jetson-inference/pedestrians_2.mp4
------------------------------------------------
gstEncoder video options:
------------------------------------------------
  -- URI: file:///opt/jetson-inference/pedestrians_2.mp4
     - protocol:  file
     - location:  pedestrians_2.mp4
     - extension: mp4
  -- deviceType: file
  -- ioType:     output
  -- codec:      H264
  -- codecType:  v4l2
  -- frameRate:  30
  -- bitRate:    4000000
  -- numBuffers: 4
  -- zeroCopy:   true
------------------------------------------------
[OpenGL] glDisplay -- X screen 0 resolution:  3440x1440
[OpenGL] glDisplay -- X window resolution:    3440x1440
[OpenGL] glDisplay -- display device initialized (3440x1440)
[video]  created glDisplay from display://0
------------------------------------------------
glDisplay video options:
------------------------------------------------
  -- URI: display://0
     - protocol:  display
     - location:  0
  -- deviceType: display
  -- ioType:     output
  -- width:      3440
  -- height:     1440
  -- frameRate:  0
  -- numBuffers: 4
  -- zeroCopy:   true
------------------------------------------------
[gstreamer] opening gstDecoder for streaming, transitioning pipeline to GST_STATE_PLAYING
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
[gstreamer] gstDecoder -- failed to set pipeline state to PLAYING (error 0)
video-viewer:  shutting down...
video-viewer:  shutdown complete

Hmm, can you run it like this, and see if GStreamer prints additional relevant information right around the gstDecoder -- failed to set pipeline state to PLAYING error?
GST_DEBUG=5 video-viewer --debug /mount/videos/pedestrians.mp4 /mount/videos/pedestrians_4.mp4

I can't really find much about that Inappropriate ioctl for device message (which I don't receive), so I think it may be worth testing GStreamer outside the container (with gst-launch-1.0 and a standalone pipeline) to confirm that your drivers and OS are functioning correctly...
https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide/text/SD/Multimedia/AcceleratedGstreamer.html

Here is the result of what you suggested, run within the container
root@orin-desktop:/opt/jetson-inference# GST_DEBUG=5 vide
[TRUNCATED THINGS THAT LOOK FINE]
0:00:03.115263457 1916809 0xaaaad9230a00 DEBUG     GST_PLUGIN_LOADING gstplugin.c:1696:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin157> g_dir_open(/usr/lib/aarch64-linux-gnu/lv2) failed: Error opening directory “/usr/lib/aarch64-linux-gnu/lv2”: No such file or directory
0:00:03.116351852 1916809 0xaaaad9230a00 DEBUG           GST_REGISTRY gstregistry.c:1414:gst_registry_scan_path_internal:<registry0> registry changed in path /usr/lib/aarch64-linux-gnu/gstreamer-1.0: 0
0:00:03.116365580 1916809 0xaaaad9230a00 DEBUG           GST_REGISTRY gstregistry.c:1640:gst_registry_remove_cache_plugins:<registry0> removing cached plugins
0:00:03.116473613 1916809 0xaaaad9230a00 INFO            GST_REGISTRY gstregistry.c:1785:scan_and_update_registry: Registry cache has not changed
0:00:03.116485550 1916809 0xaaaad9230a00 INFO            GST_REGISTRY gstregistry.c:1862:ensure_current_registry: registry reading and updating done, result = 1
0:00:03.116502190 1916809 0xaaaad9230a00 INFO                GST_INIT gst.c:833:init_post: GLib runtime version: 2.72.4
0:00:03.196776325 1916809 0xaaaad9230a00 INFO                GST_INIT gst.c:835:init_post: GLib headers version: 2.72.1
0:00:03.196923430 1916809 0xaaaad9230a00 INFO                GST_INIT gst.c:837:init_post: initialized GStreamer successfully
0:00:03.196954375 1916809 0xaaaad9230a00 DEBUG                default gsttracerutils.c:77:_priv_gst_tracing_init: Initializing GstTracer
[gstreamer] initialized gstreamer, version <IP_ADDRESS>
[gstreamer] gstDecoder -- couldn't find file '/mount/videos/pedestrians.mp4'
[gstreamer] gstDecoder -- failed to create decoder for file:///mount/videos/pedestrians.mp4
video-viewer:  failed to create input stream

Now, doing as you suggest outside the container

Ok, I installed GSTstreamer outside the container per the doc you shared, and tried my best to follow test instructions:
orin@orin-desktop:~/jetson-containers$ gst-launch-1.0 filesrc location=<pedestrians.mp4> ! \
>      qtdemux ! queue ! h264parse ! nvv4l2decoder ! nv3dsink -e
No protocol specified
No protocol specified
No protocol specified
No protocol specified
No protocol specified
ERROR: Pipeline doesn't want to pause.
ERROR: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.
Additional debug info:
gstbasesink.c(5367): gst_base_sink_change_state (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:
Failed to start

I should note that i am ssh'd into this machine. Perhaps there's some issue with X11? I can try doing all of this directly on the machine...

[gstreamer] gstDecoder -- couldn't find file '/mount/videos/pedestrians.mp4'
[gstreamer] gstDecoder -- failed to create decoder for file:///mount/videos/pedestrians.mp4

You need to change the filenames in that command to reflect where you have pedestrians.mp4 at

I should note that i am ssh'd into this machine. Perhaps there's some issue with X11? I can try doing all of this directly on the machine...

Yes, using nv3dsink without display attached would be problematic for that particular pipeline you tried running.  You can either run it directly on machine with display attached, or use another pipeline (replacing nv3dsink with fakesink may help)

Hey sorry for my delay in getting back on this and thanks for your help so far!  I tried running your test command directly on the machine with the display attached:
`root@orin-desktop:/jetson-inference# video-viewer pedestrians.mp4
[gstreamer] initialized gstreamer, version <IP_ADDRESS>
[gstreamer] gstDecoder -- creating decoder for pedestrians.mp4
libEGL warning: DRI3: Screen seems not DRI3 capable
libEGL warning: DRI2: failed to authenticate
sh: 1: lsmod: not found
sh: 1: modprobe: not found
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
[gstreamer] gstDecoder -- discovered video resolution: 960x540  (framerate 29.970030 Hz)
[gstreamer] gstDecoder -- discovered video caps:  video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)high, width=(int)960, height=(int)540, framerate=(fraction)30000/1001, pixel-aspect-ratio=(fraction)1/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
[gstreamer] gstDecoder -- pipeline string:
[gstreamer] filesrc location=pedestrians.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder name=decoder enable-max-performance=1 ! video/x-raw(memory:NVMM) ! nvvidconv name=vidconv ! video/x-raw ! appsink name=mysink
[video]  created gstDecoder from file:///opt/jetson-inference/pedestrians.mp4
gstDecoder video options:
-- URI: file:///opt/jetson-inference/pedestrians.mp4
- protocol:  file
- location:  pedestrians.mp4
- extension: mp4
-- deviceType: file
-- ioType:     input
-- codec:      H264
-- codecType:  v4l2
-- width:      960
-- height:     540
-- frameRate:  29.97
-- numBuffers: 4
-- zeroCopy:   true
-- flipMethod: none
-- loop:       0
[OpenGL] glDisplay -- X screen 0 resolution:  3440x1440
[OpenGL] glDisplay -- X window resolution:    3440x1440
[OpenGL] glDisplay -- display device initialized (3440x1440)
[video]  created glDisplay from display://0
glDisplay video options:
-- URI: display://0
- protocol:  display
- location:  0
-- deviceType: display
-- ioType:     output
-- width:      3440
-- height:     1440
-- frameRate:  0
-- numBuffers: 4
-- zeroCopy:   true
[gstreamer] opening gstDecoder for streaming, transitioning pipeline to GST_STATE_PLAYING
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
[gstreamer] gstDecoder -- failed to set pipeline state to PLAYING (error 0)
video-viewer:  shutting down...
video-viewer:  shutdown complete`

Hello everyone,
I am facing the same issue! trying to get TAO person detection running on my Jetson Orin Nano. I have followed instructions in the Jetson AI Fundamentals - S3E4 - Object Detection Inference There is no display attached to it. The module works fine without exporting to an output file detectnet --model=peoplenet pedestrians.mp4.
[OpenGL] failed to create X11 Window.

detectNet -- loading detection network model from:
          -- prototxt
          -- model        networks/peoplenet_deployable_quantized_v2.6.1/resnet34_peoplenet_int8.etlt.engine
          -- input_blob   'input_1'
          -- output_cvg   'output_cov/Sigmoid'
          -- output_bbox  'output_bbox/BiasAdd'
          -- mean_pixel   0.000000
          -- class_labels networks/peoplenet_deployable_quantized_v2.6.1/labels.txt
          -- class_colors networks/peoplenet_deployable_quantized_v2.6.1/colors.txt
          -- threshold    0.500000
          -- batch_size   1

[TRT]    TensorRT version 8.5.2
[TRT]    loading NVIDIA plugins...
[TRT]    Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[TRT]    Registered plugin creator - ::BatchedNMS_TRT version 1
[TRT]    Registered plugin creator - ::BatchTilePlugin_TRT version 1
[TRT]    Registered plugin creator - ::Clip_TRT version 1
[TRT]    Registered plugin creator - ::CoordConvAC version 1
[TRT]    Registered plugin creator - ::CropAndResizeDynamic version 1
[TRT]    Registered plugin creator - ::CropAndResize version 1
[TRT]    Registered plugin creator - ::DecodeBbox3DPlugin version 1
[TRT]    Registered plugin creator - ::DetectionLayer_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_TRT version 1
[TRT]    Could not register plugin creator -  ::FlattenConcat_TRT version 1
[TRT]    Registered plugin creator - ::GenerateDetection_TRT version 1
[TRT]    Registered plugin creator - ::GridAnchor_TRT version 1
[TRT]    Registered plugin creator - ::GridAnchorRect_TRT version 1
[TRT]    Registered plugin creator - ::GroupNorm version 1
[TRT]    Registered plugin creator - ::InstanceNormalization_TRT version 1
[TRT]    Registered plugin creator - ::InstanceNormalization_TRT version 2
[TRT]    Registered plugin creator - ::LayerNorm version 1
[TRT]    Registered plugin creator - ::LReLU_TRT version 1
[TRT]    Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[TRT]    Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[TRT]    Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[TRT]    Registered plugin creator - ::NMSDynamic_TRT version 1
[TRT]    Registered plugin creator - ::NMS_TRT version 1
[TRT]    Registered plugin creator - ::Normalize_TRT version 1
[TRT]    Registered plugin creator - ::PillarScatterPlugin version 1
[TRT]    Registered plugin creator - ::PriorBox_TRT version 1
[TRT]    Registered plugin creator - ::ProposalDynamic version 1
[TRT]    Registered plugin creator - ::ProposalLayer_TRT version 1
[TRT]    Registered plugin creator - ::Proposal version 1
[TRT]    Registered plugin creator - ::PyramidROIAlign_TRT version 1
[TRT]    Registered plugin creator - ::Region_TRT version 1
[TRT]    Registered plugin creator - ::Reorg_TRT version 1
[TRT]    Registered plugin creator - ::ResizeNearest_TRT version 1
[TRT]    Registered plugin creator - ::ROIAlign_TRT version 1
[TRT]    Registered plugin creator - ::RPROI_TRT version 1
[TRT]    Registered plugin creator - ::ScatterND version 1
[TRT]    Registered plugin creator - ::SeqLen2Spatial version 1
[TRT]    Registered plugin creator - ::SpecialSlice_TRT version 1
[TRT]    Registered plugin creator - ::SplitGeLU version 1
[TRT]    Registered plugin creator - ::Split version 1
[TRT]    Registered plugin creator - ::VoxelGeneratorPlugin version 1
[TRT]    detected model format - engine  (extension '.engine')
[TRT]    loading network plan from engine cache... /usr/local/bin/networks/peoplenet_deployable_quantized_v2.6.1/resnet34_peoplenet_int8.etlt.engine
[TRT]    Loaded engine size: 21 MiB
[TRT]    Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[TRT]    Deserialization required 31542 microseconds.
[TRT]    [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +21, now: CPU 0, GPU 21 (MiB)
[TRT]    Total per-runner device persistent memory is 0
[TRT]    Total per-runner host persistent memory is 132288
[TRT]    Allocated activation device memory of size 12533760
[TRT]    [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 33 (MiB)
[TRT]
[TRT]    CUDA engine context initialized on device GPU:
[TRT]       -- layers       53
[TRT]       -- maxBatchSize 1
[TRT]       -- deviceMemory 12533760
[TRT]       -- bindings     3
[TRT]       binding 0
                -- index   0
                -- name    'input_1'
                -- type    FP32
                -- in/out  INPUT
                -- # dims  3
                -- dim #0  3
                -- dim #1  544
                -- dim #2  960
[TRT]       binding 1
                -- index   1
                -- name    'output_bbox/BiasAdd'
                -- type    FP32
                -- in/out  OUTPUT
                -- # dims  3
                -- dim #0  12
                -- dim #1  34
                -- dim #2  60
[TRT]       binding 2
                -- index   2
                -- name    'output_cov/Sigmoid'
                -- type    FP32
                -- in/out  OUTPUT
                -- # dims  3
                -- dim #0  3
                -- dim #1  34
                -- dim #2  60
[TRT]
[TRT]    binding to input 0 input_1  binding index:  0
[TRT]    binding to input 0 input_1  dims (b=1 c=3 h=544 w=960) size=6266880
[TRT]    binding to output 0 output_cov/Sigmoid  binding index:  2
[TRT]    binding to output 0 output_cov/Sigmoid  dims (b=1 c=3 h=34 w=60) size=24480
[TRT]    binding to output 1 output_bbox/BiasAdd  binding index:  1
[TRT]    binding to output 1 output_bbox/BiasAdd  dims (b=1 c=12 h=34 w=60) size=97920
[TRT]    device GPU, initialized /usr/local/bin/networks/peoplenet_deployable_quantized_v2.6.1/resnet34_peoplenet_int8.etlt.engine
[TRT]    detectNet -- number of object classes: 3
[TRT]    detectNet -- maximum bounding boxes:   6120
[TRT]    loaded 4 class labels
[TRT]    didn't load expected number of class descriptions  (4 of 3)
[TRT]    detectNet -- number of object classes:  3
[TRT]    class 00  color 0.000000 128.000000 0.000000 120.000000
[TRT]    class 01  color 0.000000 0.000000 128.000000 120.000000
[TRT]    class 02  color 225.000000 150.000000 0.000000 120.000000
[TRT]    loaded 3 class colors
[gstreamer] opening gstDecoder for streaming, transitioning pipeline to GST_STATE_PLAYING
Opening in BLOCKING MODE
[gstreamer] gstreamer changed state from NULL to READY ==> mysink
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter2
[gstreamer] gstreamer changed state from NULL to READY ==> vidconv
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter1
[gstreamer] gstreamer changed state from NULL to READY ==> decoder
[gstreamer] gstreamer changed state from NULL to READY ==> h264parse1
[gstreamer] gstreamer changed state from NULL to READY ==> queue0
[gstreamer] gstreamer changed state from NULL to READY ==> qtdemux1
[gstreamer] gstreamer changed state from NULL to READY ==> filesrc0
[gstreamer] gstreamer changed state from NULL to READY ==> pipeline0
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter2
[gstreamer] gstreamer changed state from READY to PAUSED ==> vidconv
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter1
[gstreamer] gstreamer changed state from READY to PAUSED ==> decoder
[gstreamer] gstreamer changed state from READY to PAUSED ==> h264parse1
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> queue0
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer stream status CREATE ==> sink
[gstreamer] gstreamer changed state from READY to PAUSED ==> qtdemux1
[gstreamer] gstreamer changed state from READY to PAUSED ==> filesrc0
[gstreamer] gstreamer stream status ENTER ==> sink
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
[gstreamer] gstreamer message stream-start ==> pipeline0
[gstreamer] gstreamer message duration-changed ==> h264parse1
[gstreamer] gstDecoder -- onPreroll()
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer mysink taglist, video-codec=(string)"H.264\ /\ AVC", bitrate=(uint)720629;
[gstreamer] gstreamer mysink taglist, encoder=(string)Lavf54.63.104, container-format=(string)"ISO\ MP4/M4A";
[gstreamer] gstreamer mysink taglist, video-codec=(string)"H.264\ \(High\ Profile\)", bitrate=(uint)720629;
[gstreamer] gstBufferManager -- map buffer size was less than max size (777600 vs 777607)
[gstreamer] gstBufferManager recieve caps:  video/x-raw, width=(int)960, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30000/1001, nvbuf-memory-type=(string)nvbuf-mem-surface-array, gpu-id=(int)0, format=(string)NV12
[gstreamer] gstBufferManager -- recieved first frame, codec=H264 format=nv12 width=960 height=540 size=777607
[cuda]   allocated 4 ring buffers (777607 bytes each, 3110428 bytes total)
[cuda]   allocated 4 ring buffers (8 bytes each, 32 bytes total)
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysink
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline0
[gstreamer] gstreamer message async-done ==> pipeline0
[gstreamer] gstreamer message new-clock ==> pipeline0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mysink
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter2
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> vidconv
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> decoder
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> h264parse1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> queue0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> qtdemux1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> filesrc0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> pipeline0
[cuda]   allocated 4 ring buffers (1555200 bytes each, 6220800 bytes total)
19 objects detected

detected obj 0  class #0 (person)  confidence=0.970148
bounding box 0  (268.10, 243.61)  (324.58, 383.77)  w=56.48  h=140.15

detected obj 1  class #0 (person)  confidence=0.817762
bounding box 1  (213.25, 217.14)  (266.94, 334.27)  w=53.69  h=117.13

detected obj 2  class #0 (person)  confidence=0.966212
bounding box 2  (221.36, 309.33)  (272.39, 414.61)  w=51.03  h=105.28

detected obj 3  class #0 (person)  confidence=0.982390
bounding box 3  (676.46, 69.43)  (709.68, 159.27)  w=33.22  h=89.84

detected obj 4  class #0 (person)  confidence=0.962530
bounding box 4  (641.42, 50.81)  (675.26, 134.70)  w=33.84  h=83.88

detected obj 5  class #0 (person)  confidence=0.966748
bounding box 5  (590.69, 76.59)  (619.97, 153.99)  w=29.28  h=77.40

detected obj 6  class #2 (face)  confidence=0.984941
bounding box 6  (229.18, 317.33)  (242.03, 334.01)  w=12.84  h=16.69

detected obj 7  class #2 (face)  confidence=0.681861
bounding box 7  (244.38, 222.31)  (255.75, 239.52)  w=11.37  h=17.21

detected obj 8  class #2 (face)  confidence=0.913575
bounding box 8  (244.99, 221.53)  (254.58, 237.62)  w=9.59  h=16.09

detected obj 9  class #2 (face)  confidence=0.902219
bounding box 9  (246.34, 222.67)  (253.39, 236.94)  w=7.05  h=14.27

detected obj 10  class #2 (face)  confidence=0.620184
bounding box 10  (693.25, 72.14)  (701.43, 81.82)  w=8.18  h=9.68

detected obj 11  class #2 (face)  confidence=0.636066
bounding box 11  (694.71, 72.66)  (700.50, 82.49)  w=5.79  h=9.84

detected obj 12  class #2 (face)  confidence=0.756589
bounding box 12  (656.43, 54.56)  (663.68, 62.25)  w=7.26  h=7.69

detected obj 13  class #2 (face)  confidence=0.968845
bounding box 13  (694.40, 72.01)  (700.52, 80.62)  w=6.12  h=8.61

detected obj 14  class #2 (face)  confidence=0.680877
bounding box 14  (693.44, 72.29)  (699.59, 80.36)  w=6.15  h=8.07

detected obj 15  class #2 (face)  confidence=0.688705
bounding box 15  (656.31, 54.19)  (661.39, 61.11)  w=5.08  h=6.93

detected obj 16  class #2 (face)  confidence=0.898147
bounding box 16  (657.40, 55.57)  (661.82, 62.60)  w=4.42  h=7.03

detected obj 17  class #2 (face)  confidence=0.984040
bounding box 17  (694.51, 73.12)  (698.40, 81.07)  w=3.88  h=7.94

detected obj 18  class #2 (face)  confidence=0.583265
bounding box 18  (657.94, 54.64)  (661.21, 61.72)  w=3.27  h=7.07
[cuda]   allocated 2 ring buffers (777600 bytes each, 1555200 bytes total)
[gstreamer] gstEncoder -- starting pipeline, transitioning to GST_STATE_PLAYING
[gstreamer] gstEncoder -- failed to set pipeline state to PLAYING (error 0)
detectnet:  shutting down...
[gstreamer] gstDecoder -- stopping pipeline, transitioning to GST_STATE_NULL
[gstreamer] gstDecoder -- onPreroll()
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> capsfilter2
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> vidconv
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> capsfilter1
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> decoder
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> h264parse1
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> queue0
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> qtdemux1
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> filesrc0
[gstreamer] gstreamer changed state from PLAYING to PAUSED ==> pipeline0
[gstreamer] gstDecoder -- pipeline stopped
detectnet:  shutdown complete.

Looking forward to hearing from you @dusty-nv!
Thank you in advance!


Fix replacment of lower cased "Unreleased" link references

When a "Unreleased" link reference was lower cased, then it was not replaced. In Markdown, link references are case insensitive. Prettier for example will make them all lower case. Therefore the identifier rather than the label must be compared to find the "Unreleased" link reference.

As an example you can see this commit: https://github.com/markuspoerschke/extractum/commit/ce4bd666340957e2c16785e51b33a8d02d6b40cd

@thomaseizinger Yes, I created the dist.js, because the Git Commit Hook warned me about doing so.


[ADR] Support for external authorization providers

Support for External Authorization Providers
Status
Proposed
Context
We are extending Istio CR by adding support for external authorization providers. This proposal is for authorization services that operate only over HTTP.
Decision

We define the API specification in such a way that it supports the current Istio mesh configuration for EnvoyExternalAuthorizationHttpProvider, which is one of the Istio's extension providers. We exclude some properties that we think will not be used in Kyma scenarios to simplify the customer experience and reduce complexity in the UI. As a result, when defining an external authorizer, users only need to configure three required properties - name, service, and port - while having the optional headers property defined.

We add support only for HTTP external authorization providers. This is because we expect the usage of external authentication providers to be a rare use case, and gRPC usage would be even more uncommon. Currently, in production, we rarely see customers using oauth2_introspection. That is why we do not consider other protocols (like gRPC) relevant at this moment.

We support multiple external authorization providers as there might be use cases for this, such as different providers having unique capabilities, differentiating by authentication flows (needing different Deployments), multi-tenancy cases where an application needs to support multiple identity providers having a different provider per tenant or user pool. We cannot assume that an authorizer will work with multiple identity providers (for example, oauth2-proxy only supports this in alpha right now). Istio currently supports configuration for multiple extension providers, which can be identified by a unique name.

We do not include oauth2-proxy Deployment with Kyma-managed Istio installation. After analysis of the current project state, we did not see many recent contributions. Very long release cycles will make the handling of security vulnerabilities very difficult and, from past experiences managing ORY Hydra/Oathkeeper Deployments, we want to avoid handling similar external components in the future.

For the optional headers, we see use cases where the user may need to adapt headers in the auth flow. Istio guides do include examples with headers (for example oauth2-proxy) and different identity providers may require specific headers.

For the sake of simplicity, we do not include configuration properties for Timeout, PathPrefix, FailOpen, StatusOnError, and IncludeRequestBodyIncheck as in Istio's mesh configuration for EnvoyExternalAuthorizationHttpProvider. We don't see a use case for them and will rely on their default values.

User Scenario / Flow (Example)

The customer installs Kyma with Istio and API Gateway.
Manual installation of oauth2-proxy. For example, see the installation documentation.
Configure Istio CR with an external authorization provider for oauth2-proxy (see the example below).
The Istio module operator configures Istio with the additional mesh configuration and creates ServiceEntry (networking.istio.io) for it to be accessible within the service mesh (see example below).
The user may need to create manually DestinationRule (networking.istio.io) for an external OIDC provider (for example, SAP IAS).
Additional VirtualService (networking.istio.io) resources must be created by the customer for the auth code flow or client credentials use cases.
Expose the service via APIRule specifying a new authorization handler which the API Gateway module operator will reconcile and create an AuthorizationPolicy (security.istio.io) with CUSTOM action (see example below).

Examples

Istio CR with external provider configuration for oauth2-proxy:

apiVersion: operator.kyma-project.io/v1alpha1
kind: Istio
metadata:
  name: default
  namespace: kyma-system
spec:
  config:
    numTrustedProxies: 1
    authorizers:
    - name: "oauth2-proxy" # required, unique identifier
      service: "oauth2-proxy.oauth2-proxy.svc.cluster.local" # required
      port: "4180" # required
      headers: # optional
        inCheck:
          include: ["authorization", "cookie"] # headers sent to the oauth2-proxy in the check request.
          add:
            x-auth-request-redirect: "https://%REQ(:authority)%%REQ(x-envoy-original-path?:path)
        toUpstream:
          onAllow: ["authorization", "path", "x-auth-request-user", "x-auth-request-email", "x-auth-request-access-token"] # headers sent to backend application when request is allowed.
        toDownstream:
          onAllow: ["set-cookie"] # headers sent back to the client when request is allowed.
          onDeny: ["content-type", "set-cookie"] # headers sent back to the client when request is denied.

ServiceEntry resource created during Istio module operator reconciliation:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
name: oauth2-proxy
namespace: oauth2-proxy
spec:
  hosts:
  - oauth2-proxy.oauth2-proxy.svc.cluster.local
  ports:
  - name: http
    number: 4180
    protocol: http
  resolution: STATIC

APIRule with new authorization handler:

apiVersion: gateway.kyma-project.io/v1beta1
kind: APIRule
metadata:
  name: httpbin
spec:
  host: httpbin.xxx.shoot.canary.k8s-hana.ondemand.com
  gateway: kyma-system/kyma-gateway
  service:
    name: httpbin
    port: 8000
  rules:
    - path: /ip
      methods: ["GET"]
      accessStrategies:
        - handler: extern
          config:
            authorizerName: oauth2-proxy

AuthorizationPolicy resource created during API Gateway module operator reconciliation for the APIRule:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
name: httpbin-abcde
spec:
  action: CUSTOM
  provider:
    name: oauth2-proxy
  rules:
  - to:
    - operation:
        methods: ["GET"]
        paths: ["/ip"]
  selector:
    matchLabels:
      app: httpbin

API golang Structure Changes:
type Config struct {
  // OPTIONAL. Defines the number of trusted proxies deployed in front of the Istio gateway proxy.
  // +kubebuilder:validation:Minimum=0
  // +kubebuilder:validation:Maximum=4294967295
  NumTrustedProxies *int `json:"numTrustedProxies,omitempty"`

  // OPTIONAL. Defines a list of external authorization providers.
  Authorizers []*Authorizer `json:"authorizers,omitempty"`
}

type Authorizer struct {
  // REQUIRED. A unique name identifying the extension authorizationprovider.
  Name string `json:"name,omitempty"`

  // REQUIRED. Specifies the service that implements the Envoy ext_authz HTTP authorization service.
  // The format is `[<Namespace>/]<Hostname>`. The specification of `<Namespace>` is required only when it is insufficient to unambiguously resolve a service in the service registry. The `<Hostname>` is a fully qualified host name of a service defined by the Kubernetes service or ServiceEntry.
  //
  // Example: "my-ext-authz.foo.svc.cluster.local" or "bar/my-ext-authz.example.com".
  Service string `json:"service,omitempty"`

  // REQUIRED. Specifies the port of the service.
  Port uint32 `json:"port,omitempty"`

  // OPTIONAL. Specifies headers to be included, added or forwarded during authorazation.
  Headers *Headers `json:"headers,omitempty"`
}

// Exact, prefix and suffix matches are supported (similar to the authorization policy rule syntax except the presence match
// https://istio.io/latest/docs/reference/config/security/authorization-policy/#Rule):
// - Exact match: "abc" will match on value "abc".
// - Prefix match: "abc*" will match on value "abc" and "abcd".
// - Suffix match: "*abc" will match on value "abc" and "xabc".

type Headers struct {
  // OPTIONAL. Defines headers to be included or added in check authorization request.
  InCheck *InCheck `json:"inCheck,omitempty"`

  // OPTIONAL. Defines headers to be forwarded to the upstream.
  ToUpstream *ToUpstream `json:"toUpstream,omitempty"`

  // OPTIONAL. Defines headers to be forwarded to the downstream.
  ToDownstream *ToDownstream `json:"toDownstream,omitempty"`
}

type InCheck struct {
  // OPTIONAL. List of client request headers that should be included in the authorization request sent to the authorization service.
  // Note that in addition to the headers specified here following headers are included by default:
  // 1. *Host*, *Method*, *Path* and *Content-Length* are automatically sent.
  // 2. *Content-Length* will be set to 0 and the request will not have a message body. However, the authorization request can include the buffered client request body (controlled by include_request_body_in_check setting), consequently the value of Content-Length of the authorization request reflects the size of its payload size.
  Include []string `json:"include,omitempty"`

  // OPTIONAL. Set of additional fixed headers that should be included in the authorization request sent to the authorization service.
  // Key is the header name and value is the header value.
  // Note that client request of the same key or headers specified in `Include` will be overridden.
  Add map[string]string `json:"add,omitempty"`
}

type ToUpstream struct {
  // OPTIONAL. List of headers from the authorization service that should be added or overridden in the original request and forwarded to the upstream when the authorization check result is allowed (HTTP code 200).
  // If not specified, the original request will not be modified and forwarded to backend as-is.
  // Note, any existing headers will be overridden.
  OnAllow []string `json:"onAllow,omitempty"`
}

type ToDownstream struct {
  // OPTIONAL. List of headers from the authorization service that should be forwarded to downstream when the authorization check result is allowed (HTTP code 200).
  // If not specified, the original response will not be modified and forwarded to downstream as-is.
  // Note, any existing headers will be overridden.
  OnAllow []string `json:"onAllow,omitempty"`

  // OPTIONAL. List of headers from the authorization service that should be forwarded to downstream when the authorization check result is not allowed (HTTP code other than 200).
  // If not specified, all the authorization response headers, except *Authority (Host)* will be in the response to the downstream.
  // When a header is included in this list, *Path*, *Status*, *Content-Length*, *WWWAuthenticate* and *Location* are automatically added.
  // Note, the body from the authorization service is always included in the response to downstream.
  OnDeny []string `json:"onDeny,omitempty"`
}

Consequences
We add support for external authorization providers with Kyma Istio and do not introduce any breaking changes.

I have few questions / clarification requests:

If more than one authorizer is defined how it is defined in APIRule? I don't see connection
ServiceEntry example is missing kind field

@pbochynski

All authorizers as in Istio's extension providers have unique names. In the APIRule an unique authorizer will be referenced per name and it will be in the upcoming APIRule extern handler configuration as required field, e.g.:

      accessStrategies:
        - handler: extern
          config:
            authorizerName: oauth2-proxy

since Istio external authorizers proposal precedes APIRule proposal, exact details like names or properties placement may change a bit. We also may restrict to use single unique authorizer per APIRule or on service / path level. To be decided.

ServiceEntry kind in example fixed.

/lgtm

During implementation we discovered that we do not need to create a ServiceEntry for the external authoriser. Having Kubernetes service specified for the external authorizer as required in here we do not have per default need to describe special service resolving via ServiceEntry.
However the user may utilize this in the future, because we map directly the service field in our configuration with the extension provider's service field in the Istio's mesh config without validation. See examples in here.


menyoki: init at 1.5.3

Motivation for this change
menyoki is a screencast and screenshot utility that can also perform various image related operations such as making/splitting GIFs and modifying/analyzing/viewing image files
https://github.com/orhun/menyoki
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 137230 run on x86_64-linux 1

1 package built:

menyoki

darwin build timed out, maybe ofborg just had a hiccup so im triggering a rebuild
@ofborg build menyoki


Object properties do not preserve order

I would like my object definitions to appear in the order found in the schema files.
It makes sense to have the definitions for fields to be in the same order as the example object.
I notice the current CF documentation has this issue.
It looks like a theme issue from my limited understanding.
@handrews I know objects are unordered, so is this actually a JSON Hyper Schema issue? If we want JHS to be used to generate documentation, then do we need a field order array?
I'm investigating the posibility of using https://facebook.github.io/immutable-js/docs/#/OrderedMap, but really I'm not even sure how to create my own theme... Yes the docs say "fork this project" but then what? I don't want to publish it to npm... It's like I should already know what to do after I fork it.

I notice that the example object preserves the order of the properties. I believe it's down to https://github.com/cloudflare/json-schema-example-loader/blob/35d0b38488c6f34a34031cb61e3dc868c0450396/lib/example-data-extractor.js#L68, right?

+1 to this, we should be loading into ordered maps or whatever the most convenient implementation is (I'm doing this on the Python side with OrderedDict, but that's not helping for documentation :-)

This issue may need to be ignored... the map might actually already be ordered by default...
It seems that by adding examples, my example and properties are coming out in the order found in the schema.json file. Which is good, but strange given that the properties list for example objects on the actual CF API docs do not match in terms of ordering.

Hm, I am not sure if we can actually guarantee the order based on properties since JS objects don't guarantee the order.

4.3.3 Object
An object is a member of the type Object. It is an unordered collection of properties each of which contains a primitive value, object, or function. A function stored in a property of an object is called a method. - ECMAScript specs

@tajo at least in python this is handled by reading the JSON or YAML into a special OrderedDict class in place of regular dict (python dict == javascript object more or less).  The load methods in both libraries have hooks to substitute a different object for the standard one- I imagine someone must have done this for JavaScript by now.

An object is an unordered set of name/value pairs.
http://json.org/

If you need to preserve the order in JSONs (or JS), you should use an array.

We're not talking about the data model here, we're talking about presentation.  If I list object members in a schema in a particular order, I want the documentation to consistently show those object members in that same order.
Native JS objects don't do that, but there are these things called libraries, you know :-)
Example (I have not even vaguely checked that this is suitable for us, just pointing out that such libraries exist):
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map

I agree that an object is an ordered list of key value pairs. It's usually phrased that you should not expect the order to be preserved, especially when looping. I would guess this is for speed. However as @handrews points out, this is a presentation issue, and there IS support for preserving order of JSON objects.
As I said in my earlier comment, one of the packages is already doing this by looping through them using a different function. Also, it seems that when the examples are included, the order is preserved... at least for me, consistently.

I've confirmed multiple times. If there is an example object given, all the key / value pairing is the same as the original json document, and consiently displayed. SO, it's just ouputting the json document in the right order to resolve this issue (for me).
I'm not sure if you expect someone to use doca without having an example object, but yet list the properties. If you might, then this is still an issue that should be addressed.

Well, I did a bit research. I haven't found any JS library that would solve this. Just a lot of negative answers.
Let's say we found a library that is parsing json files and preserves the order of objects. (It shouldn't be technically called "JSON parser" since doesn't follow specs). Our json-schema-loader is just a wapper around json-schema-ref-parser, so we don't actually have access to the parsing logic right now. We would have to find a different json-schema-ref-solver library. I'm pretty sure that none of them is using advanced datastructures as Immutable.OrderedMap. In other words, we would have to rewrite every single library that will touch the schema because they all use plain JS objects.
So, we would have to implement our own JSON parser (which is really hard and will be slow compared to the native v8 parser) and then we would have to write our own JSON schema resolver library (which I already tried once and really don't want to do it again). That's like weeks of work.
I understand that it can be important to set the specific order but the only reasonable technical solution is to add an extra array that will list that order.

Let's put this on hold.  At some point we will be looking at which JSON Schema libraries we use as new drafts come out.  It may be easier to deal with then.
There are other options we can consider, such as reading the file once and building up that ordering array automatically, but that is more effort than I want to spend on this right now.

Thanks for digging a little.
Having a further think on this, maybe the only reasonable solution would be to extend the functionality of the doca config.js file.
An object with keys of the schema files, with an array of the order of the properties keys.
schema_name => array [ order of properties ]
I mean, given I'm generating the schemas, and I have the information, this doesn't seem like an unreasonable (or as hard to implement a) solution.

OK. So I have spent a LOT of time working on this...
I have a solution for if you include in your json an array of properties in the order you want them.
https://gist.github.com/Relequestual/95a74a0a81aa20e8536fcc9e083ef0f2
I'm not sure if this is the best solution, and I feel it probably wants to be solved upstream (Possibly in the example loader? That's where the morphing and setting all_props, required_props, optional_props, is done right?)
Also I haven't catered for if the properties_order field is omitted, and obivoulsy it still needs to function in that situation.

Yea, that's the right direction if you want to solve this issue. Agree that is should be more upstream and I think it can be simplified too.

I'm still really strongly against maintaining a separate array for ordering.  If we need to work out an ordering list somewhere, then at the point where we read the file, we should read the field ordering and construct any necessary ordering array.
If no one has solved this problem (which is baffling to me- it's not hard and it's been solved in Python for many, many years now) then we should solve it.  If necessary, we can read the JSON into Python, use Python to generate the necessary arrays, and then pipe it into JavaScript.  That's a horrible kluge but it's orders of magnitude better than manually maintaining a separate array.
I'm fine with using an array within JavaScript code, but I am absolutely not going to support requiring end-users to maintain such arrays.  Think about the larger schemas with many levels of nesting.  You'd need to maintain arrays all over the place for each object.  We're not doing that.

@tajo Great. I'm going to use this solution in our theme for now (even if it's not optimal, because I need to move on for now), and change it when a solution is implemented more up stream.

@handrews At first, I dissagreed with you, but after reading, I partially agree. It would indeed be nuts to maintain many many arrays nested all over the place.
However, with Perl, it's a LOT easier to include an array of ordered field names than to order the JSON output. It's not impossible, but pretty tricky.
I see no harm in supporting it as an option, to override the given order, as I expect in other langauges this may also be preferable.
If it presents significant overhead development wise (upstream), then I'll be happy to reconsider.

From a conversation a while back with @epoberezkin, most JavaScript implementations default to preserving the order of objects unless they are very large (larger than we need to worry about right now).  This is not any sort of standards conformance thing, just a de-facto behavior.
The problem here is that json-schema-example-loader transforms structures in various ways, and the ordering gets lost in the process.  It may be possible to avoid that, in which case we should do so.
I'm not going to remove "blocked" just yet as this needs some investigation- if the problem is that we're actually running afoul of fundamental lack of support for remembering order, this will still face the same dilemma it does now.

Apprecaite the update on this. I'm happy with our solution for now - It works for what we need but wouldn't work for everyone. Really hoped to be pushing our API docs sooner, but it's delayed for various reasons.

Sorry to revive this, but was curious of the current state of the issue.  Which would be most accurate?

Still looking into preserving order internally.
Waiting for possible schema extension for view/ui, where order could be specified.
Not really of interest currently.

Thanks..

@Codelica I expect that when we follow through with the ROADMAP.md work, this will be better.  Part of it is that we will do less transforming of the schema, so JavaScript's tendency to iterate at least smaller objects in the order they were read will make this less likely to be a problem.
It will also give us an easier way to hook in extension keywords, whether they are proposed standards or just something that we want for doca.
I don't have a timeline on this.  There is related work going on- a few things have been published here, and I have some other things in flight, but there aren't set milestones, I'm afraid.

Thanks for the info.

@Codelica
This set of packages is being deprecated in favor of the JSON Schema Tools monorepo.  Contributions are still welcome here but most work will be done on the new packages.
The new back-end does a lot less re-arranging, so it's more likely to preserve order.  It depends a lot on how much you use allOf, I think (and of course, it's dependent on the JavaScript implementation's handling of object ordering).  I've filed cloudflare/json-schema-tools#4 to track further necessary work.

Thanks for the update.  In the end we just had too many custom needs, so I ended up rolling my own doc output with the help of json-schema-ref-parser, which actually worked quite well and gave us more control over ordering and other things -- at a cost of course ;)  Will check out your new collection though...  thanks!

@Codelica no worries- I knew that we weren't moving fast enough here to keep everyone.  Hopefully with the new codebase sticking more closely to the JSON Hyper-Schema spec we can build up a more active community.


[BEAM-5467] Remove createProcessWorker from docker execution path.

Please add a meaningful description for your change here

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

It causes the Docker target to fail with:
Writing /tmp/easy_install-ynAUYZ/fastavro-0.21.9/setup.cfg
Running fastavro-0.21.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ynAUYZ/fastavro-0.21.9/egg-dist-tmp-t_6442
warning: no files found matching '*' under directory 'conda.recipe'
x86_64-linux-gnu-gcc: error: fastavro/_six.c: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: [Errno 2] No such file or directory <built-in function listdir> /tmp/easy_install-ynAUYZ/fastavro-0.21.9/egg-dist-tmp-t_6442

> Task :beam-sdks-python:createProcessWorker FAILED
:beam-sdks-python:createProcessWorker (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 20.63 secs.

Run Python Flink ValidatesRunner


Upgrade MariaDB to the latest 10.6.x release

Impact of the new feature
MariaDB
Is your feature request related to a problem? Please describe.
Right now we are using MariaDB 10.6.5, dating from Nov/2021, while 10.6.19 has been released in Aug/2024.
Given the issue reported in this ticket: https://github.com/dmwm/WMCore/issues/12136 , I think we should have a conservative upgrade of MariaDB ASAP, potentially for the upcoming WMAgent release. With time and in a different ticket, we can track a major upgrade of MariaDB to the 11.x series.
Describe the solution you'd like
Upgrade the MariaDB version used by WMAgent to the latest 10.6.x minor release - at the moment of this ticket, it is 10.6.19.
Describe alternatives you've considered
We could wait for a major upgrade, but if this turns out to be real little effort, we would be running a more solid version of MariaDB.
Additional context
Release notes are: https://mariadb.com/kb/en/release-notes-mariadb-106-series/

BTW the latest stable version of MariaDB is: 11.4.3 and the whole 11.4 series  is supposed to be the long term support for MariaDB which should span at least until 2029, as mentioned to the release notes here: https://mariadb.com/kb/en/mariadb-11-4-3-release-notes/
So it would have been better if we could make the effort and push a little bit harder and upgrade to the latest version. I know it might seem a big jump, but we  should give it a try.  The change of aliases from mysql-* to mariadb-* should supposedly not be that big of a problem. There should be few places here and there, mostly in the CMSKuberenetes repository.

As discussed in our team meeting on Monday, I only wanted to have a minor upgrade of MariaDB, the reasons are:
a) so that we can use a release that will not be affected by that potential bug
b) a safe and effortlessly upgrade
c) and a task that can be done right away (as Operations task).
For the major MariaDB upgrade, it will have to go through more changes and we will have to evaluate its performance as well. To me, that has to be performed as a planned development activity.


Possible memory leak

https://github.com/stanfordnlp/GloVe/blob/master/src/glove.c#L288
It's looks like memory leak here

And maybe here:
https://github.com/stanfordnlp/GloVe/blob/master/src/glove.c#L280
https://github.com/stanfordnlp/GloVe/blob/master/src/glove.c#L281

Yea, the one in the for loop is worth fixing. The others are fairly benign, but I fixed them and a few others in the pull request


BUG/TST: added curate_streams to zip + other

* fixed curate_streams to work in a nested fashion
* added a test for curate_streams intended behaviour
* added curate_streams before emitting on zip

Discussion from #45
Also, see tests as a point of discussion as to how curate_streams should behave. I made a slight line change to allow for curate_streams(curate_streams(nds)) to work properly.

(rebased)

Codecov Report

Merging #52 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #52      +/-   ##
==========================================
+ Coverage   98.09%   98.12%   +0.03%
==========================================
  Files           6        6
  Lines        1050     1069      +19
==========================================
+ Hits         1030     1049      +19
  Misses         20       20

Impacted Files
Coverage Δ

shed/event_streams.py
95.77% <100%> (+0.01%)
:arrow_up:

shed/tests/test_streams.py
99.32% <100%> (+0.02%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b71f4c6...3ec931d. Read the comment docs.

Superseded by #55


Error: Permission denied (os error 13)

Describe the bug
I am trying to upgrade from version 0.18.1 to 0.23.1 but I'm running into an issue with os error 13.
Here's the version information for the currently running version:
$ curl -s -X GET localhost:7700/version | jq .
{
  "commitSha": "bf3f36b46ecddb6240092193d578eb221fde2a65",
  "buildDate": "2021-01-14T13:54:16.031701922+00:00",
  "pkgVersion": "0.18.1"
}

Here is the dump file I'm trying to import. It's very small compared to the database that currently contains my indexed documents:
$ du -hcd2 dumps
19M     dumps
19M     total

$ du -hcd2 meilifiles
1.4G    meilifiles/main
8.4M    meilifiles/update
1.4G    meilifiles
1.4G    total

To Reproduce

Create a dump from version 0.18.1 with a large database.
Import the dump into version 0.23.1.

Expected behavior
Dump should be imported without any errors.
Screenshots

Meilisearch version: v0.23.1
Additional context
Running on Ubuntu 20.04.4 and the resulting data directory is very small compared to the dump and the original database
$ du -hcd1 meilifiles-23-db/
2.2M    meilifiles-23-db/.tmpjUCz76
2.2M    meilifiles-23-db/
2.2M    total

Here's the output from strace (I ran the rename command from the shell and it worked fine so it's not clear to me why there is a permission error):
statx(AT_FDCWD, "./meilifiles-23-db/db.ms", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fff6b579980) = -1 ENOENT (No such file or directory)
rename("/path/./meilifiles-23-db/.tmpcFspQl", "./meilifiles-23-db/db.ms") = -1 EACCES (Permission denied)
close(10)                               = 0
statx(AT_FDCWD, "/path/./meilifiles-23-db/.tmp2egDQi", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_BASIC_STATS, stx_attributes=0, stx_mode=S_IFDIR|0777, stx_size=512, ...}) = 0
openat(AT_FDCWD, "/path/./meilifiles-23-db/.tmp2egDQi", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 10
fstat(10, {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
getdents64(10, /* 4 entries */, 32768)  = 112
unlink("/path/./meilifiles-23-db/.tmp2egDQi/metadata.json") = 0
openat(AT_FDCWD, "/path/./meilifiles-23-db/.tmp2egDQi/test", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 14
fstat(14, {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
getdents64(14, /* 5 entries */, 32768)  = 168
unlink("/path/./meilifiles-23-db/.tmp2egDQi/test/documents.jsonl") = 0
unlink("/path/./meilifiles-23-db/.tmp2egDQi/test/settings.json") = 0
unlink("/path/./meilifiles-23-db/.tmp2egDQi/test/updates.jsonl") = 0
getdents64(14, /* 0 entries */, 32768)  = 0
close(14)                               = 0
rmdir("/path/./meilifiles-23-db/.tmp2egDQi/test") = 0
getdents64(10, /* 0 entries */, 32768)  = 0
close(10)                               = 0
rmdir("/path/./meilifiles-23-db/.tmp2egDQi") = 0
close(3)                                = 0
close(5)                                = 0
madvise(0x7f9e20980000, 2625536, MADV_DONTNEED) = 0
close(4)                                = 0
close(9)                                = 0
write(2, "Error: ", 7)                  = 7
write(2, "Permission denied", 17)       = 17
write(2, " (os error ", 11)             = 11
write(2, "13", 2)                       = 2
write(2, ")", 1)                        = 1
write(2, "\n", 1)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7fb799f89000, 12288)           = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Hi @davidk01 👋
Thank you for getting in touch with us.
I am not able to reproduce the error and I believe this might not be a bug. From my end, creating a dump in v0.18.1 and importing it into v0.23.1 works great, the screenshot is below. I used the Mac OS binaries and a somewhat small dataset. Is this an issue that only occurs with huge datasets?

I close this issue due to a lack of answers. Nothing is definitive of course, feel free to ask to re-open if needed.


WIP: redoing changes to clean up & debug upcoming PR [DO NOT MERGE]

Incrementally cleaning up and testing changes towards multiple primary files.

@swift-ci please smoke test

@swift-ci Please smoke test Linux platform


Lost required packages in v3.4.1

In v3.4.1 the required packages cultuurnet/search: ~1.0 and cultuurnet/culturefeed-php: ~1.1 are lost.
Because of this the culturefeed_search module doesn't work anymore. Is this the expected behavior after updating?

The tag did not pointed to the right commit, excuses. Can you check if it's ok now?
https://github.com/cultuurnet/culturefeed/releases/tag/v3.4.1

Problem solved. Thank you.


Exercise order?

Looking through the Go exercises, I noticed that Hello World is number 4 on the list. Is there a reason why it would be after what I would consider to be much more difficult exercises?

It is in preparation (#825) for the next version of Exercism. The tracks will no longer be linear but look more like a tree. Completing certain exercises will open up other ones for  completion. You won't have to do all the exercises in order any more. So the JSON ordering does not really matter much only for the core exercises core == true && unlocked_by == null. Any exercise with core == false && unlocked_by == null is a bonus exercise which is available at the start so users have options. In our case that would be those first three exercises before hello-world. It can be complicated to look at, so there are handy visualizers!

Thank you, this explains it. I wish I had the skill set necessary to assist, I'm only just getting started with exercism, let alone Go. Appreciate the in-depth response though.

The modifications of that file are in JSON so you may be able to give it a try. Everyone starts somewhere :smile:! If you do want to reorder you can do it via a PR and reference this, i'll close it for now.

Closing this issue per @tleen 's comment


Organize outputs in app-config module

app-config/outputs.tf is getting big. Maybe organize the outputs alphabetically

@lorenyu I think I'll organize the outputs everywhere, not just in app-config if that's alright with you?

that sounds great, thanks! feel free to update the ticket


port conflict: consider changing default port 9200

9200 and 9300 is commonly used by elasticsearch, consider if changing the default port would be possible
so users will less-likely run into conflicts.

You can change that by config

I know.. this is about the "out of the box experience"

I do not see a real conflict here. Ocis is not supposed to run on that port in productive systems, but behind a reverse proxy like traefik or nginx which will result in remapping this port to 443 anyway. For evaluation purposes I think it is fine to run on 9200, if you evaluate a more complex setup changing the port to your likes should be done via config as @micbar mentioned.


fix: code robustness

What type of PR is this?

fix
Check the PR title.

[ ] This PR title match the format: <type>(optional scope): <description>
[ ] The description of this PR title is user-oriented and clear enough for others to understand.
[ ] Attach the PR updating the user documentation if the current PR requires user awareness at the usage level. User docs repo

(Optional) Translate the PR title into Chinese.
(Optional) More detailed description for this PR(en: English/zh: Chinese).

en:
zh(optional):
修复部分代码
(Optional) Which issue(s) this PR fixes:

(optional) The PR that updates user documentation:

@EZ4Jam1n check this


KAFKA-13614: Don't apply leader replication quota to consumer fetches

What
In the fetch path, we check shouldLeaderThrottle regardless of whether the read is coming from a consumer or follower broker. This results in replication quota being applied to consumer fetches.
Testing
Added unit tests for when fetch requests can be immediately satisfied and for purgatory fetches.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@dajac do you mind reviewing?