Slow repository browsing in 1.14.x

Gitea version (or commit ref): 1.14.x
Git version: 2.31.1
Operating system: FreeBSD 13
Gitea built using ports collection (www/gitea)
Gitea started by startup script provided by www/gitea port

Database (use [x]):

[x] PostgreSQL 12.6
[ ] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[x] Yes (https://try.gitea.io/tsowa/FreeBSD_ports)
[ ] No

Log gist: https://www.ttmath.org/gitea.log

Description

I saw a similar thread but there is "windows" in the title so I create a new issue.  Gitea 1.14.x is much slower in repository browsing than Gitea 1.13.
Sample repo running with 1.14.1:
https://gitea.ttmath.org/FreeBSD/ports
Try to open any directory, for example:
https://gitea.ttmath.org/FreeBSD/ports/src/branch/main/audio
It takes between 50-150 seconds to open a page.
The same repo running with 1.13.7:
https://giteaold.ttmath.org/FreeBSD/ports
Try to open similar directory, for example:
https://giteaold.ttmath.org/FreeBSD/ports/src/branch/main/audio
I takes about 5 seconds.
You can see the same problem on try.gitea.io:
https://try.gitea.io/tsowa/FreeBSD_ports
But you have a cache so you have to find a directory which was not open before. Opening such a page takes 100-300 seconds.
Let me know if more info is needed.

This is because the algorithm was changed in 1.14 due to a problem with go-git causing significant memory issues. Thank you for the test cases though because they will provide tests to improve the current algorithm.
If you are suffering significant slow downs here you can switch back to the gogit build by adding gogit to your TAGS during building.
We would otherwise appreciate help in improving the performance of the algorithm for the pure git version.

It was enabled by default but the 'adapter' option was set to 'memory'. Now I have installed memcached and changed adapter to 'memcache' and a difference is visible.
Opening https://gitea.ttmath.org/FreeBSD/ports for the first time took 79766ms and for the second time only 3063ms. Opening https://gitea.ttmath.org/FreeBSD/ports/src/branch/main/audio for the first time 141221ms and later 37205ms.
But I see that you are calling a lot of git processes, I have created a small git wrapper in such a way:
#include <unistd.h>
#include <fstream>
#include <iostream>

int main(int argc, char * argv[], char * envp[])
{
    std::ofstream file("/home/tomek/git.log", std::ios_base::out | std::ios_base::app);

    if( file )
    {
        file << "git ";

        for(size_t i=0 ; argv[i] ; ++i)
        {
            file << argv[i] << " ";
        }

        file << std::endl;
        file.close();
    }

    return execve("/usr/local/bin/git.org", argv, envp);
}

I have moved original /usr/local/bin/git to /usr/local/bin/git.org and have compiled above program as /usr/local/bin/git. And it gives me git.log with all git operations and I see that sometimes you are calling the git binary 300 times in one request:
cat ~/git.log | wc -l
     335

So it cannot be fast, this remains me of the old days when we were using cgi scripts. Is there a reason you are calling git directly instead of using a git library such as libgit2?

Could you also count what's the git command Gitea invoked in these 335 commonds?
This is because when browsing, Gitea will get last commit message for every dir/file on the ui. For v1.13.0, we use go-git which is a pure go git library, for v1.14.x, we have two versions for windows because the library have some memory problems. And maybe you could try to compile the go git version yourself to check what's the different between them.

Hello there, looking at this from Codeberg's perspective (issue)
As you can see, we're also suffering from the slow repository browsing which affects the overall performance of our machine. While setting up a Redis cache works well for us, we would like to improve the initial generation on cache misses, too.
Today, we tried to find some more information about the bottleneck, I hope this is useful for you:
We suspect especially this command where each folder is checked for the latest commit, executing /usr/bin/git -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --format=%T <commit> which has a terrible performance (loading the entire commit history).
The idea makes sense to us: getting all commits, checking if the folder or file was touched. But the logic without gogit doesn't appear to stop after all necessary information was loaded, but rather continue serving up the entire history, even if all files or subfolders in a folder have already been hit by a recent commit.
While we didn't completely understand the gogit logic yet, it appears to be a little smarter at this point and only looking so far back in history as necessary to retrieve the information it's looking for.
We assume that the process should be stopped before it ends if all information was provided.
It looks like there's a lot of stuff to be improved with the native git backend, and some actions will probably always be slower because they cannot directly interface with git operations (e. g. directly working on git results while they are fetched instead of the piped input). It might a good idea to turn back to gogit for all systems when the memory issues are resolved, or, look for another git library (that is more native and maybe faster than gogit), but offers some better interface (go-bindings for libgit2?)
Please let us know if we can provide further assistance in improving this performance issue.
Some other random observations that might be interesting to you:

multiple requests for the same resource try to generate it concurrently on cache miss, the operation doesn't get queued (thus, requesting the same page twice results in generating it twice until the cache is filled)
git processes aren't stopped when the requesting TCP connection is closed

if a connection times out (proxy), the process is still running in the backend
a user reloading the page easily doubles the resources for this operation
it's possible to DoS a huge server by simply spamming F5 (reload) on a page with a cache miss with minimum cost at the attackers side (no need to keep the connection open)

initial generation with this method will always be very slow if some files (e.g. a README , LICENCE, .gitignore, gitattributes, dockerignore etc) weren't touched for a long time
pushing to a repo invalidates the full redis cache, even if only parts of the information changed (e.g. a subfolder was updated), thus active repos won't profit very much from the cache
timed out git commands (by adjusting the timeout in Gitea) keeps them listed as running in the Gitea admin monitoring section, although the processes don't exist in the system any more

Take a look at #15891

@fnetX thanks for your long comment.

It's worth remembering that the issue precipitating the pure git backend was memory use. I've submitted a patch to go-git which should cause much lower memory load. Until that is in and working correctly, go-git will happily load in huge blobs in to memory - storing them in caches even when you want to check the size of the object. It's really worth being clear that that is an absolutely intolerable situation.
Further the issues you are highlighting in the last section are not new to the native git backend. They're present in a different way in the go-git backend just in a way you can't track. I've long advocated for changing to a more gitlab approach for this and/or for passing in request contexts to terminate things - I'm really happy to work on this - but I haven't had a chance to do this - and to be honest none of you are paying me.

Now going on to the get last commit algorithm.
All algorithms have a balance between memory and time. The current algorithm is highly optimised against memory use. If we are happy to use more memory that can be improved. It's worth remembering

We suspect especially this command where each folder is checked for the latest commit, executing /usr/bin/git -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --format=%T <commit> which has a terrible performance (loading the entire commit history).
The idea makes sense to us: getting all commits, checking if the folder or file was touched. But the logic without gogit doesn't appear to stop after all necessary information was loaded, but rather continue serving up the entire history, even if all files or subfolders in a folder have already been hit by a recent commit.
While we didn't completely understand the gogit logic yet, it appears to be a little smarter at this point and only looking so far back in history as necessary to retrieve the information it's looking for.

Looking at the length of time the rev-list process is running a bit of a distractor. Yes the go-git process can stop once it's finished looking at all the parents and the paths, but it's a question of memory and time spent tracking the parents. git rev-list avoided tracking those parents - and grabbing the root tree saved a lot of time - but we could add in %P to the format to allow tracking of parents and could allow termination once all appropriate parents are determined - but I don't think it's the primary cause of delays.
The greatest speed up in #15891 is actually preemptive passing the next tree ID to the git cat-file process as soon as we know what it's going to be. A large proportion of time appears to be spent waiting for go to fill the read buffer from the other process. This is where the go-git algorithm can be quicker as it avoids that by reading files in directly.

Some other random observations that might be interesting to you:

multiple requests for the same resource try to generate it concurrently on cache miss, the operation doesn't get queued (thus, requesting the same page twice results in generating it twice until the cache is filled)

git processes aren't stopped when the requesting TCP connection is closed

if a connection times out (proxy), the process is still running in the backend
a user reloading the page easily doubles the resources for this operation
it's possible to DoS a huge server by simply spamming F5 (reload) on a page with a cache miss with minimum cost at the attackers side (no need to keep the connection open)

initial generation with this method will always be very slow if some files (e.g. a README , LICENCE, .gitignore, gitattributes, dockerignore etc) weren't touched for a long time

pushing to a repo invalidates the full redis cache, even if only parts of the information changed (e.g. a subfolder was updated), thus active repos won't profit very much from the cache

timed out git commands (by adjusting the timeout in Gitea) keeps them listed as running in the Gitea admin monitoring section, although the processes don't exist in the system any more

These are all longstanding issues and I am aware of them. I would love to spend time fixing these but I am limited in my time and availability.

Honestly I wish you'd just talked to me directly. I'm always on Discord and could have told you and kept you abreast of what was going on and my progress in trying to speed this up.

oh my - I think I know how to seriously improve this. I think I've been way too distracted by the way it was done in the go-git implementation and there's genuinely a much quicker way to do this.

Hey, thank you very much for the explanation.

I've submitted a patch to go-git which should cause much lower memory load. Until that is in and working correctly ...

I somehow thought this was already in and just needed some further improvements, my bad.

and to be honest none of you are paying me.

Yes, we can mainly offer to being thankful as long as we aren't paid for anything either. :heart:
But let's see if we can figure something out.

Honestly I wish you'd just talked to me directly. I'm always on Discord

Yeah, the others told me that, too. But since Discord is a proprietary app that kept crashing my computer back when I last used it, I decided against this and went for dumping our findings somewhere, hoping they are of any use. Chose this issue over the thread on Codeberg as it seemed to better fit in this topic here.

I think I know how to seriously improve this

Sounds like good news. Please tell us if there's anything we can do.

oh my - I think I know how to seriously improve this. I think I've been way too distracted by the way it was done in the go-git implementation and there's genuinely a much quicker way to do this.

Unfortunately this doesn't work.
The idea was to use git log --format=%H --raw -t --no-abbrev --reverse COMMIT_ID  -- paths but I can't come up with a way to stop it from listing the contents of the trees - meaning that it takes even longer.
If I could figure out a way to not list the contents of the trees this would be definitely faster than the go-git version.

What about adding -n 1? Correct me if I am completely mistaken, because I neither fully understand the Gitea backend yet, nor do I know how git works, but this seems to give you the latest commit of a path and gives the same result as Gitea currently gives.

Oh, you probably still want to have the full list of the folder you're looking at, just not of all the subfolders?

yeah - I mean if we could just do that n times then it would be easy and fine but it's not like that.
Also it's not quite -n1 consider the following tree:
        H
       /   \
     D    E
      |     |
     C    F
      |
     B
      |
     A

If the wibble becomes the object with SHA  deadbeef at B and at E. The correct commit to report is B not E.
So -n1 is still not right. git describe will give the correct answer but it's too slow to be run n times.

Could you test #15891? In my limited testing this is faster for the root directory than the go-git native version. There is a still a slowdown problem in the subdirectories.

@fnetX - I've just made another improvement in #15891 which should solve the sub directories problem

Thank you. We haven't yet been able to properly backport it to our fork // rebase our patches to this pull. We'll look into it and test then.

@zeripath Thanks, now testing bd1455aa from your repo (cache is disabled):
https://giteanew.ttmath.org/FreeBSD/ports
The speed up is visible, browsing directories is about 5 times faster than 1.14.x. Not as fast as cgit but much better than before. Good job.

@tsowa does cgit even attempt to provide last commit information?

I have a backport of the latest get-lastcommit-cache performance improvements on to 1.14 if people would like them.

We have tested the backport you provided on codeberg-test.org and it has ~x3 loadtimes as go-git (15 to 17 seconds your pull vs. ~ 5 seconds go-git). We're using git version 2.29.2 - do you know if a more recent version might have a better performance or if there are other constraints that might decrease performance?
It's a single-core 2GB RAM VPS.

well that's interesting - as my timings appear to be similar to those on go-git.
are you sure you've built from the backport-improve-get-lastcommit branch?
The version should be 1.14.2+33-g57d45e1c2 as in SHA 57d45e1c247eaafb3a3a92ab593c31356b472d6f
Do you have commit graphs enabled for your repos?

We deployed this branch which has your commits on top of our 1.14 patches cleanly: https://codeberg.org/Codeberg/gitea/src/branch/codeberg-try-puregit-improvements (just confirmed once more that the commit matches:  1.14.2+49-g7e9e3f364)
Yes, you can browse commit graphs on Codeberg.

Hmm... I am very confused as this is now just as fast as gogit for me and possibly faster in places. Tell me there's at least some improvement here for you?
I'm almost at my limit for what I can do to speed this up any further. The main slowdowns in my testing were in filling the buffers between the pipes & adjusting when the subsequent reads occurred seemed to fix this for me - perhaps my processor is just fast enough that the earlier writes provide me just enough time to prevent the fill lock whereas on your processor that's not quite enough time. I just don't think there's any way to avoid it - I mean we could try an os.pipe instead of an io.pipe? I tried an nio.Pipe but it was just as slow. Certainly we can't switch to the same algorithm as the go-git variant as that would require even more communication and waiting for the cat-file-batch pipes to respond and fill.

By commit graph I meant the git core.commitGraph functionality. It should be enabled by default but ... I've certainly seen repos on my system that don't have a commit graph even though they would clearly benefit.

Ok I guess we're at a point of diminishing returns - and I might be better off looking at solving the problems in go-git and making last commit info stop slowing down rendering.

@zeripath
I did tests with the linux kernel and nixpkgs repos after a gitea restart with cold cache. It seems to be factor 3 on both in favor of gogit. I have no idea why the optimizations do nothing on those repos.
linux 1.14 gogit 0:21
linux 1.14 git 1:03
linux 1.14 git(new) 1:01
nixpkgs 1.14 gogit 0:06
nixpkgs 1.14 git 0:17
nixpkgs 1.14 git(new) 0:17
I also backported your optimization to 1.14 before with same results, but I blamed by lack of understanding of the code and a bad backport.

@ashimokawa you'd need to backport a few other PRs to see the improvement - it's not just #15891 that is needed. I'm happy to give you a link to that backport.

gogit has a commitgraph optimization, ref #7314, but of course, git version should also read commitgraph(#7313) if that file has been updated.

@ashimokawa could you double check that these repos actually have commit graphs? The basic graph would be in .git/objects/info/commit-graph
If they don't git commit-graph write will write one.
I suspect we might need to do more in gitea to forcibly write these graphs. (They're of benefit even when/if we switch back to gogit.)

Yes, we can update the commit-graph file once push for big repositories. And we also need to use modules/git/repo_commitgraph.go when build tag is non go-git which now is only a build tag of go-git.

@lunny It looks like I might need to be spending more time deep in go-git to come up with ways to improve it and/or avoid memory issues there. For example even if the repo level interface can't help but read objects totally into memory when stating them - we could crawl packs instead of repos to read headers to find sizes of objects at least allowing us to sidestep that issue.
Similarly at least adding in some control about the size of the caches could be useful.
(It'll need to help prep it for sha256 anyway I suspect.)

A hybrid approach of just using gogit for this last commit stuff might be reasonable too. That has less of the memory worries - although it's still potentially considerable. Every commit and it's full tree (but fortunately not the blobs in this case) that are crawled is stored completely in the unlimited cache.

At the gitea level I think we need to make rendering less dependent on getting these results - e.g. gitlab's deferred results here. So I'm gonna look at that next. The step of passing in a cancelable context is the first step for this - the next step is making an unique queue for things.

In terms of the current native approach - although I had a go commit graph reader written and setup - but on my testing the slow down was still at getting data back from the cat-file --batch. Again pointing to issues filling the buffers. It could be that os.pipe is better here or that some approach without a bufio.Reader is needed.
To be honest though I think I'm hitting a bit of wall here and I might be at the limit of what can be done using native git.

Commit graph:
root@codeberg-test:/data/git/gitea-repositories/bigrepos# ls nixpkgs.git/objects/info/
commit-graph  packs
root@codeberg-test:/data/git/gitea-repositories/bigrepos# ls linux.git/objects/info/
root@codeberg-test:/data/git/gitea-repositories/bigrepos# ls freebsd-src.git/objects/info/
packs
root@codeberg-test:/data/git/gitea-repositories/bigrepos#

present for nixpkgs, but not for the other two testing repos. But it doesn't seem to do much good here, or at least not noticable.

you'd need to backport a few other PRs to see the improvement

I took the branch you sent me, pushed it to codeberg and ashimokawa cherry-picked the last four commits onto our branch as the rest appeared to be plain Gitea 1.14, hope this was okay? We can test exactly your branch without codeberg patches once more, but we didn't touch the backend (at least not knowingly).

Have you enable last_commit_cache? I think that will resolve all non-first visit problems.

I sent a PR to add a new cache provider ledis which is a local disk cache. That means it will not use much memory but still accelerate non-first homepage visit speed for big repositories. ref #16035.
The PR is another aspect of effort and will not affect the above discussion.

Have you enable last_commit_cache?

Sometimes disabled in the experiments, but generally yes. Also we are looking into rolling out Redis on prod. We are still interested in improving the initial generation of the last commits, since even with enabled memory caching the generation slowed down the whole instance and left it unresponsive at times. And since cache is purged after every push and also not present for all the subfolders, branches etc, it sounds important to us to speed up it's generation.

@zeripath
This is what I used for the test, four commits added by you on 1.14 (codeberg branch)
https://codeberg.org/Codeberg/gitea/src/branch/codeberg-try-puregit-improvements
If there is more to cherry-pick, I am happy to re-run the "benchmarks" ASAP.

Ok thinking on I think the only answer is just to use repeated calls to git log -n1 once the number of commits reaches some high level.
If I couple that with the (in progress) deferred commit info generation pr (https://github.com/zeripath/gitea/tree/defer-last-commit-info) then we'll have a workable low memory option.
Yes this will mean that the two backends can slightly different results - but it's ultimately better than the current status.

OK - here's one more attempt at improving the original algorithm: #16042
we some slight parent tracking to assert if we are in a single parent branch - and if we can use path tracking in git rev-list to do some history simplification.

@zeripath
Thanks!, right now it would be much easier if we had a backport.
I will see if we can somehow test gitea main with this and have a proper comparison.

What about copying the testing database into one for Gitea 1.14 and one for 1.15 on codeberg-test.org? This way, we can more simply test these upstream experiments without always asking for a backport? And simply switch to the 1.14 database again for staging tests until we have this set-up on the new server?
@zeripath the issue for not using git log -n1 were the repeated calls and that it might be a bit flaky with multi-parent commits (e.g. merges)?

@ashimokawa go to my private gitea - branch backport-16042
@fnetx as far as I understand git log -n1 won't necessarily give the correct answer + it's n calls.

@zeripath
Hmm cannot find the branch you mention....

I've pushed it up to github now - it was on my private gitea that I gave the details to fnetX by email, but it's up at:
https://github.com/zeripath/gitea/tree/backport-16042

@zeripath
This ones leads to an error 500
 routers/repo/view.go:149:renderDirectory() [E] GetCommitsInfo: strconv.ParseInt: parsing ".gitignore\x00\xb1f\xa7\x8d}p㲸\xf3\x87\xac:x(ޕA\x12L100644 .version\x00b\xc3k\

ugh that's what I get for not testing the backport. Currently hacking on some code in go-git give me a few more minutes.

OK done.

@zeripath
Big improvement,
browsing nixpkgs from our slow test instance:
pure git: 17.6s
go-git 7.7s
Your backport: 6.7s

Cool. Glad to see I've finally got somewhere!

and I think I've also finally got a PR to go-git that will prevent it reading huge things into memory: go-git/go-git#330

@zeripath
Yes, thank you! This really seems good so far! Of course we have to do more tests before we can deploy it.
Do you have any idea what we should test except for browsing a repo?

browse repos and compare results I think.
Try a few subdirectories.
Unfortunately you'd need to try the directories that are quite slow too.

Confirmed, found a difference:
in https://codeberg-test.org/bigrepos/nixpkgs/src/branch/master/pkgs/build-support file https://codeberg-test.org/bigrepos/nixpkgs/src/branch/master/pkgs/build-support/build-pecl.nix is reported to being modified by https://codeberg-test.org/bigrepos/nixpkgs/commit/9f09253e521bbddabd6a7faf5803fffceba7b03c (1 year ago) via your native-git version and by https://codeberg-test.org/bigrepos/nixpkgs/commit/8c26eaa144897e9031093e9f56cc15c86f259772 (3 months ago) with go-git.
If I understand you correctly, there isn't much you can do about this with this approach, right?
My ottomated comparison approaches were quite flaky, being confused by CSRF token differences and the like, so I'm glad I spotted this one. Can't say much to performance, this was where @ashimokawa has had some good comparison :-)
Just as a sidenote, I wanted to know what this gives:
git log --format=%H --raw -t --no-abbrev -n1 --reverse -- pkgs/build-support/build-pecl.nix
8c26eaa144897e9031093e9f56cc15c86f259772
...

So at least this appears to be correct. But, it's quite slow at this point.

@ashimokawa take a look at:
https://github.com/zeripath/gitea/tree/backport-use-git-log-raw

@fnetX too.

OK so assuming that #16059 is giving the correct answers - the root page of ports.git is now rendering in 4759ms with that (on 1.15). But the most major result is "devel/" which is rendering in 2808ms down from 55s or more.

unfortunately this PR is slightly more pathological for git.git - potentially the restarting semantics aren't actually helping here.

Ok having had a look at GH I think we don't have to do anything cleverer than just use the order that comes out from git log.

I tired
https://github.com/zeripath/gitea/tree/backport-use-git-log-raw
Above: >55s (!)
pure git: 17.6s
go-git 7.7s
Your (previous) backport: 6.7s

What is going on? Why is git log raw so much faster for me on 1.15 but so much slower for you? Maybe it's something to do with nixpkgs as I haven't tested with that repo.
Ok. I guess I have to reopen the last pr and leave this one as a WIP.
I think however the shortcut that we've discovered should highlight a way to get some further improvements.

@ashimokawa are you able to retest #16059 and tell me which repo it fails on. No rush. I'll push for #16042 to be merged in the meantime

@zeripath
It did not fail, it was just extremely slow, slower that anything I have ever tested in this context.
What should I retest, there are no new commits added to this PR, right?

@ashimokawa did you actually test from 67a1aa9fc55a36364b6c2e5bd2e230410a63e3fe ? or was it from 2cfd269688c56839ddc53b0563ad7aefd3a4da2a ?
What was the repo that was so slow btw?

The repo I was testing with was https://codeberg-test.org/bigrepos/nixpkgs with cold cache
The branch I was testing was https://github.com/zeripath/gitea/tree/backport-use-git-log-raw
It took 55 seconds (!) compared to plain 1.14 with pure git which only took ~17s
go-git took 7.7s, your (previous) backport was the fastest with 6.7s so far but led to wrong data in some subdirs as @fnetX pointed out.

@zeripath some test repos, cache is disabled
zeripath:improve-get-lastcommit (cc6d732b):
https://giteanew.ttmath.org/FreeBSD/ports
~20 seconds
https://giteanew.ttmath.org/FreeBSD/ports/src/branch/main/audio
~17 seconds
https://giteanew.ttmath.org/FreeBSD/ports/src/branch/main/devel
~82 seconds
https://giteanew.ttmath.org/FreeBSD/ports/src/branch/main/polish
~17 seconds
zeripath:use-git-log-raw (67a1aa9f):
https://giteanew2.ttmath.org/FreeBSD/ports
~15 seconds
https://giteanew2.ttmath.org/FreeBSD/ports/src/branch/main/audio
~19 seconds
https://giteanew2.ttmath.org/FreeBSD/ports/src/branch/main/devel
~125 seconds
https://giteanew2.ttmath.org/FreeBSD/ports/src/branch/main/polish
~18 seconds
gitea 1.13.7
https://giteaold.ttmath.org/FreeBSD/ports
~28 seconds
https://giteaold.ttmath.org/FreeBSD/ports/src/branch/main/audio
~10 seconds
https://giteaold.ttmath.org/FreeBSD/ports/src/branch/main/devel
~48 seconds
https://giteaold.ttmath.org/FreeBSD/ports/src/branch/main/polish
~7 seconds

Thanks @tsowa
So I've discovered something terrible with git log --raw and --name-only and --name-status that seems insurmountable.
If there is a merge commit it appears that the files created in that merge don't appear.
This might simply be a bug with git but it's kinda annoying. From within the git repository:
$ git --version
git version 2.31.1

$ git log --raw --parents faefdd61ec7c7f6f3c8c9907891465ac9a2a1475 -- gitk-git/.gitignore
commit 9a6c84e6e9078b0ef4fd2c50b200e8552a28c6fa
Author: Junio C Hamano<EMAIL_ADDRESS>Date:   Wed Jan 30 13:52:44 2013 -0800

    Merge git://ozlabs.org/~paulus/gitk

    * git://ozlabs.org/~paulus/gitk:
      gitk: Ignore gitk-wish buildproduct

:000000<PHONE_NUMBER>000000 d7ebcaf366 A  gitk-git/.gitignore

$ git log --raw --parents faefdd61ec7c7f6f3c8c9907891465ac9a2a1475 -- gitk-git | grep gitk-git/.gitignore

$ echo $?
1

This might be the cause of slowdowns people are seeing.

OK I've figured that out! -c is the trick

OK I've pushed up another version of #16059  and its backport on to 1.14 to backport-use-git-log-raw.
These are radically quicker for me on all of these repositories and examples.

@zeripath I have updated https://giteanew2.ttmath.org to 5a90343b and can confirm:
ports ~14 s
ports/devel ~9.5 s
ports/audio ~5.6 s
ports/polish ~5.5 s
The times probably depend on the kind of disks on the server,  I have got old hdds 7200RPMs.

Yeah my development laptop is a bit of a beast.
The slow down appears to be with dealing with huge numbers of unseen parents.
I'll take another look though.

OK I've adjusted the heuristic slightly which appears to improve those three pathological cases.

For me the performance is sufficient, thank you for your work. Any plan to put it to 1.14.3?

well if you could comment on #16059 and say that you think this is now ready. If the codeberg peeps think it's better that would be good too.

use-git-log-raw (283959931) takes 6-7 second to render beauty-line-icon-theme:
https://giteanew2.ttmath.org/NixOS/nixpkgs/src/branch/master/pkgs/data/icons/beauty-line-icon-theme

@zeripath
Unfortunanely no change for nixpkgs/src/branch/master/pkgs/data/icons/beauty-line-icon-theme
measured 89s now.
It is good to improve pure git but I see that for this one it is even a bis regression from current pure git code (not to mention go-git which only takes less than 2 seconds)

I think the next direction is use commitgraph file or asynchronous load last commit information.
Reuse go-git and improve the memory model is also a possible case.

I was missing out here a bit, didn't yet try to get the exact changes in the different algorithms, but I'm just thinking about the first one:
That one was only slow, because it scanned all the commits for file changes and that could take a while. What about limiting this search for, let's say, 5000 commits (repos of this size seemed acceptable to browse) and render the page then. The rest is rendered asynchronously on demand, some commit info might then appear sooner than another.
With caching enabled (or at least a persistent cache), the information could already be generated for the full file tree. So the first access on repo shows some last commits and slowly loads all other information, but in the background the cache is already filled for the other files so that the whole commit log doesn't need to be scanned for each new subfolder access.
I'm still sceptic with too much async JS rendering, I mainly don't use GitLab because I know it sometimes hangs my browser when loading all the frontend stuff. Not sure if it's related to the commit info, but using the GitLab frontend is a pain for me.

The other problem with all those pure git implementation also is that if a user cancels a request the git process runs till completion and does not get killed. So if a user is impatient and presses reload, he can easily OOM an instance.

@zeripath didn't you do some work to pass the context down to git exec commans :point_up: ?

@ashimokawa:

The other problem with all those pure git implementation also is that if a user cancels a request the git process runs till completion and does not get killed. So if a user is impatient and presses reload, he can easily OOM an instance.

You're wrong that this is unique to the native git backend - it's present in the gogit backend where it is absolutely uncancelable and will happily eat all of your memory until it's done. The native git backend at least has cancellable "processes" on the process page.
#16032, which is already in 1.15 and is the PR @6543 is talking about, passes down the request context into GetLastCommitPaths for both backends making them absolutely cancellable.
However, if I could convince other maintainers to review #16063 - this PR pushes the generation of commit info out of the render loop defering it on to an uniqued queue structure which is clearly the best option.
Two further additions would be good:

Make the commit info work process manageable - this would be relatively easily
Pass in the cache into GetLastCommitInfo so that as soon as a path value is determined it is cached.

@fnetX

was missing out here a bit, didn't yet try to get the exact changes in the different algorithms, but I'm just thinking about the first one:
That one was only slow, because it scanned all the commits for file changes and that could take a while. What about limiting this search for, let's say, 5000 commits (repos of this size seemed acceptable to browse) and render the page then. The rest is rendered asynchronously on demand, some commit info might then appear sooner than another.

There are two PRs up for discussion here #16042 & #16059.
        I
       / \
      A   B
      |   |
      C   D
      |  /
      E
      |
      F
      |

Assume our file of interest changed in C and B, A < B in terms of time, but had the same SHA in F too.

Name
Method
Result

Github
Difficult to be completely sure of their algorithm but I think it's essentially git log -n1
B

1.14 GoGit
Walks I A C the commit tree oldest-parent first looking for changes to the file
C

1.14 git
Walks I A B C D E F ...  looking for the earliest time our file had the SHA
F

#16042
Walks I A B C D E for the earliest time our file had the SHA, but we stops at E as we have reached a single branch and have found that sha
C

#16049
Walks I A B with git log --name-status ... which gives the diff for each commit, stops at B as the file is changed there I B

One of the problems with 1.14 git and #16042 is that git rev-list returns only the treeID for the commit so we have to traverse the tree using git cat-file --batch. We don't repeatedly call git ls-tree for each commit as that is slower.

With caching enabled (or at least a persistent cache), the information could already be generated for the full file tree. So the first access on repo shows some last commits and slowly loads all other information, but in the background the cache is already filled for the other files so that the whole commit log doesn't need to be scanned for each new subfolder access.

#16063 could easily be changed to pre cache the full repository but gitea will already attempt to precache large repos on pushes.

I'm still sceptic with too much async JS rendering, I mainly don't use GitLab because I know it sometimes hangs my browser when loading all the frontend stuff. Not sure if it's related to the commit info, but using the GitLab frontend is a pain for me.

#16063 as currently constituted simply does setTimeout(500ms, reload); it's not doing anything cleverer than that. It would be possible to do a simple div replacement trick as used in the graph but the code that renders directories is a hellscape of spaghetti which I think could be done as a follow up to #16063 if it was to get in. We would not need to do cleverer rendering than simply replacing the div I think.

I came across this GH issue while looking for information on the same problem I've been having.  After briefly looking over the comments here I understand that the slowness is due to Gitea trying to load the last commit message on a per folder basis so it can display it on the rendered page.
It looks like even though there is an option to cache the last commit, that doesn't really do much in all cases.  However I have noticed some improvements be settings these options:
Probably could see even more improvement by increase the TTL more at a trade off of possibly having wrong info.
[cache]
ENABLED	= true
ADAPTER	= memcache
HOST	= localhost:11211
ITEM_TTL = 24h

[cache.last_commit]
ENABLED	= true
ITEM_TTL = 8860h
COMMITS_COUNT = 1500

I have one suggestion, especially for large repositories such as the FreeBSD ports repo.  Maybe offer an option to disable the display of the commit message on a per-repo basis.  Although it's a nice feature, it's kinda pointless on large repos IMHO.  I don't think I even pay attention to it.
Or you could have those commit messages indexed in the background to Elasticsearch instead of generating them on the fly with git log

I will say there is an improvement after upgrading to 1.15.0.  Though it doesn't look like additional page loads after the initial are any 'faster'.  Taking the ongoing example of the official FreeBSD ports git repository as a benchmark.
When loaded into Gitea it takes about 2 seconds to render the sub directory www:

Reloading the same page subsequent times take only slightly less time:

It's a much better experience than 1.14.x.  Thanks for the work towards improving this.  Hopefully the ideas mentioned in #16063 and in my previous comment are still on the table.

@tuaris if second loads are not any faster then it's likely you do not have the cache set up.
Regarding the last commit info - #16467 would essentially do that.


[Mesheryctl] Fix the meshery file path issues

[Mesheryctl] Fix the MesheryFolder file path issues
"mesheryctl system config" command is appending HOME directory twice.
Make other system commands OS agnostic in SetFileLocation
Signed-off-by: Hussaina Begum Nandyala<EMAIL_ADDRESS>Description
This PR fixes #2426
Notes for Reviewers
Signed commits

[x] Yes, I signed my commits.

Thank you, @hexxdump 😄


cli: ddgs searches (news, text) hang after returning 1 result

Before you open an issue:

[Y ] I have the latest version. (Check: ddgs version. Update: pip install -U duckduckgo_search)
[ Y] I tried reinstalling the library. (pip install -I duckduckgo_search)
[ Y] https://duckduckgo.com is accessible in my browser
[ N] I tried using a proxy (the site may block ip for a while)

Describe the bug
What the bug is.
command line ddgs returns 1 result then hangs until CTL-C to abort. I have waited as long as 25 minutes for additional results
Steps to reproduce the problem:
Please provide the steps to reproduce this problem.
from macos terminal:
ddgs text -k "open source ai agent" -r en-us -s off -m 5
Actual keywords do not change the outcome nor does using text or news.
Specify this information

OS: macOS 14.4.1
environment: conda ?
duckduckgo_search version: tried 5.3.1 and 5.3.1b1

It has been some time since I last used ddg cli tool so I cannot say when this started. I have uninstall ddg and installed 5.3.0 - same 1 result and hang behavior.
I am uncertain if this is an environment problem or a bug in ddg.

@jonny7737
CLI version prints the result in the terminal 1 result at a time. Just press "Enter" to see the rest of the results.

HAHAHAHAHA!
Thanks!


fix fix invalid getPartitionNames() methods

Some of getPartitionNames() is invalid can not used.
Remove these override methods to avoid unexpected behavior

run buildall


flaky tests

Describe the bug
tests flaky locally
To Reproduce
run tests multiple times locally
Runtime environment
standard ubuntu

creating issue to link several prs together

this is not really so necessary as the majority of blocking level concerns have been addressed - i think a review of the waiting strategies is not going to be forgotten - its a theme in the issues - lets just do things the way other tc libs implement them - an audit is necessary but this issue staying open doesn't help it

one of the other flaky ones was elastic - to set the memory, we should implement transferable to write the config file (rather than mount) - again, kind of questionable to do it under the banner of this ticket, but reminding myself so it doesn't get lost


[UI] EKSv2: display import command on create if cluster only has private access enabled

What kind of request is this (question/bug/enhancement/feature request):
enhancement
Description:
If an EKS cluster is created with only private access enabled, display the import commands shown for other clusters. This is because it will not be possible for rancher to communicate with the cluster and deploy the agent when public access is not enabled unless rancher is privately networked to the cluster somehow.
This import command should also be displayed when a user registers a cluster with only private access enabled.

The UI will do the following:
On cluster create when the user saves, if the eks cluster has privateAccess enabled, the user will be shown the cluster import command. This is the same command we show for the regular import clusters.
A new action has been exposed for EKS clusters with private access enabled, Import Host. Selecting this option will show the cluster registration modal with the import command. The same action exists at the cluster dashboard level.
On EKS cluster import, the user will be redirected as normal. We can't redirect the user to the import command because we don't know yet if the cluster has private access enabled. We don't know this until the configs have been synced on the backend. The user will have to use the Import Host action after the cluster specs sync between EKS and Steve. There is no way around this unless the configs can be synced as soon as the cluster is "created/imported".
Edit actions occur as normal, I dont think we should show the import command on every edit since we have the Import Host action available now.

This feature is verified with Rancher server rancher\rancher:master-head version a31eecf and rancher\rancher:v2.5-head version e2e530c
Steps:
Create an EKS cluster in rancher
Deselect public network access
Select private network access

Create cluster
Result:
Import command is displayed:

User is given the option to view the same command in cluster view during provisioning:


Multiple primary keys with one auto increment

Laravel Version: 5.3.26
PHP Version: 7.0
Database Driver & Version: MySQL

Description:
This is the target query:
mysql> CREATE TABLE mytable (
    ->     table_id MEDIUMINT NOT NULL AUTO_INCREMENT,
    ->     database_id MEDIUMINT NOT NULL,
    ->     other_column CHAR(30) NOT NULL,
    ->     PRIMARY KEY (database_id,table_id)
    -> ) ENGINE=MyISAM;

In that way you can have some scopes which do:
mysql> INSERT INTO mytable (database_id, other_column) VALUES
    ->     (1,'Foo'),(1,'Bar'),(2,'Baz'),(1,'Bam'),(2,'Zam'),(3,'Zoo');
Query OK, 6 rows affected (0.00 sec)
Records: 6  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM mytable ORDER BY database_id,table_id;
+----------+-------------+--------------+
| table_id | database_id | other_column |
+----------+-------------+--------------+
|        1 |           1 | Foo          |
|        2 |           1 | Bar          |
|        3 |           1 | Bam          |
|        1 |           2 | Baz          |
|        2 |           2 | Zam          |
|        1 |           3 | Zoo          |
+----------+-------------+--------------+
6 rows in set (0.00 sec)

Steps To Reproduce:
Try it using migrations. I didn't manage to make it work.
I am stuck. Laravel / Eloquent does not put the primary key and auto increment in one query.
I tried it like this:
$table->integer('wallet_id')->unsigned();
            $table->bigInteger('reference_id', true, true); // auto increment here
            $table->string('title');
            $table->integer('coins');
            $table->timestamps();
            $table->softDeletes();
            $table->primary(['wallet_id', 'reference_id']);

Leads to 2 queries which end up in this error:
SQLSTATE[42000]: Syntax error or access violation: 1068 Multiple pr
  imary key defined

Because the second query tries to add a primary key.

As an example
Schema::create('product', function (Blueprint $table) {
            $table->engine = 'MyISAM';
            $table->unsignedInteger('store_id');
            $table->unsignedInteger('id');
            $table->primary(['store_id', 'id']);
            // ...
        });
        Schema::table('product', function (Blueprint $table) {
            $table->integer('id', true, true)->change();
        });

Dang it. Thanks @vlsoprun . This works as planned.
Not in one query but it works.

Glad you could get it to work.

@vlsoprun thank you! it worked!

@soprun Its not work on InnoDB engine


Add "Cumulative Equivalent KLIMA " indicator so users can easily see their "KLIMA power"

There has been a request from a community member to add an indicator on the app of how much KLIMA-equivalents are present total in the user's wallet
Since the unit of account for wsKLIMA isn't the same as for sKLIMA and KLIMA, it can be confusing and difficult for the user to assess how many "units" of KLIMA are present in their wallet

Basically add a "total" line to this type of display that converts wsKLIMA into its KLIMA equivalent if it were to be unwrapped

@0xAurelius @biwano These cards don't look great as they fill up with more and more items.
Also, since the /stake view can only interact with sklima <> klima, I don't think it even makes sense to show wsKLIMA here.
If they want to see their wsklima balance they can navigate to the /wrap view where we show the unwrapped equivalent balance already.
This hints at a larger need for a general "dashboard" or "desk" view as I've teased in some mockups that I shared internally.
Closing this in favor of https://github.com/KlimaDAO/klimadao/issues/497


[executor] use enum Transaction instead of SignedTransaction

Motivation
Another step towards using enum Transaction everywhere instead of SignedTransaction.
part of the effort around #1173
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Existing coverage.
Related PRs
based on #1271

:v: @msmouse can now approve this pull request

@bors-libra r=wqfish

:pushpin: Commit c47d6f6 has been approved by wqfish

:hourglass: Testing commit c47d6f6556c200d913203091876a4c52c360b5e1 with merge beaac1ac35dbc187b3a5ad922059f2c7869725aa...

:sunny: Test successful - checks-circle_commit_workflow
Approved by: wqfish
Pushing beaac1ac35dbc187b3a5ad922059f2c7869725aa to master...


feat: export utils and types

Export utilities and types from index.
Also extend window object with BeeJs.
Later these could be provided via other packages like @types/bee-js and @ethersphere/bee-js-utils and removed from compiled code to reduce output file size.
manual tests needed from reviewers
Resolves #102, resolves: #103

@nugaon can you show to me, how to import and use anything from Types? I can only use the two constants with:
import { Types } from "@ethersphere/bee-js"

// Nothing outside of the following works
Types.ENCRYPTED_REFERENCE_LENGTH
Types.REFERENCE_LENGTH

@nugaon can you show to me, how to import and use anything from Types?

like this:
import { Types } from '@ethersphere/bee-js'
const vmi: Types.FlavoredType<string, 'ownType'> = 'value'

Also I would personally prefer for the types to be exported directly without this Types wrapper.

I uploaded this approach


Fix issue with trying to create image with incorrect datetime

Existing code can fail image handling in unusual cases - e.g. if it has to go through filename-based rule and file's name is something like BAD_NAME_20210230_121212.mp4 (i.e. nonexisting day of month). Similarly could happen with Whats app rule (though you probably have to create such a bad file name on purpose in either case). This is fixed by catching ValueError in datetime creation rule. Even though it is not expected to affect a lot of people it is better to have it fixed.

Looks good to me 👍


[AIRFLOW-1330] Add conn_type argument to CLI of adding connection

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Airflow JIRA

https://issues.apache.org/jira/browse/AIRFLOW-1330

Description

[x] Here are some details about my PR, including screenshots of any UI changes:
CLI doesn't create a connection when the scheme of conn_uri contains "_". This PR allows passing scheme as argument conn_type.

Tests

[x] My PR modifies the following unit tests

tests.core.CliTests.test_cli_connections_add_delete
tests.core.CliTests.test_cli_connections_list

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov is stuck.
https://codecov.io/gh/apache/incubator-airflow/commit/551c83b6bc2ca97d0c820bc86d028df930fd4290/build

@mrkm4ntr Thank you for the contribution. That's a good PR, but personally I think we should discuss how we offer options for a connection type which doesn't follow the URL schema rule more to determine the policy. Please join us at the JIRA issue.

@yu-iskw You're right. I commented in JIRA.

@mrkm4ntr Thanks for the comment in JIRA. I have commented a summarization of our discussion with pros and cons of each option. If my proposal is approved by @criccomini, let's go ahead the PR! Again, thanks for the great PR.

Codecov Report

Merging #2525 into master will decrease coverage by 0.07%.
The diff coverage is 84%.

@@            Coverage Diff             @@
##           master    #2525      +/-   ##
==========================================
- Coverage   71.22%   71.15%   -0.08%
==========================================
  Files         150      150
  Lines       11741    11758      +17
==========================================
+ Hits         8363     8366       +3
- Misses       3378     3392      +14

Impacted Files
Coverage Δ

airflow/models.py
87.16% <100%> (+0.05%)
:arrow_up:

airflow/bin/cli.py
53.91% <80.95%> (+0.45%)
:arrow_up:

airflow/task_runner/bash_task_runner.py
93.33% <0%> (-6.67%)
:arrow_down:

airflow/utils/helpers.py
53.44% <0%> (-2.88%)
:arrow_down:

airflow/jobs.py
77.61% <0%> (-0.46%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06bd4d3...0f6ce1b. Read the comment docs.

@criccomini Duplicate pull requests have arrived. Can you review this?

@fenglu-g can you do a review on this PR and LMK your thoughts? We discussed on AIRFLOW-1330 and arrived at this conclusion.

@mrkm4ntr, is it possible to get this in within the next few days? Alternatively, if you are busy and don't have bandwidth, I am OK with submitting this PR as it is. I can create another PR to clean it up a bit. Either case, LMK, thanks! (It's blocking airflow cli GCP connection creation, we want to get it fixed soon.)
@criccomini - fyi

Thank you for the review. I will clean codes up immediately.

@mrkm4ntr thank you for the prompt response ;)

Done.

+1 thank you @mrkm4ntr!

+1 LGTM
Want to confirm with @jgao54, then will merge.

Merged!

@mrkm4ntr Thanks for the great work!

@criccomini Thank you for merging. But it seems not be merged into 1.9-test.

Cherry picked into v1-9-test


Add glTF support for DRACOLoader and KTX2Loader

This increases the JS bundle size from 0.93 MiB to ~2.67 MiB~ 2.03 MiB.

This change is

+@SeanCurtis-TRI I'd appreciate your thoughts on the two open design questions posted in Reviewable.

My plan is to come up with a new test/*.html file before we merge this.  (I expect it to be pretty easy.)  The plan will be to have python3 -m httpserver in the loop to serve the unbundled files.

test/meshfile_object_gltf_optimized.html line 11 at r4 (raw file):
Previously, SeanCurtis-TRI (Sean Curtis) wrote…
BTW Can we think up a click javascrpt mechanism to recognize whether or not the user has loaded this page the right way an provide in-browser correction?

s/click/slick


fix the generic parameter nullable issue

Fixes https://github.com/microsoft/typespec/issues/3490

API change check
API changes are not detected in this pull request.


docs: AI copilot implementation

Summary
Hey team,
When trying to implement some Jest functionality @bernard-rr and I stumbled into some questions. So, we ended up ingesting the Jest documentation into Mendable.ai (my project) and used it to debug some code and also answer some questions.
It worked great so @bernard-rr and I thought it would be very useful to have a copilot assistant right into the Jest docs that could answer questions about Jest's documentation and help users debug their code. We ended up implementing a small button on the screen's bottom right which unveils an AI-powered chat interface when clicked. There, users can ask questions about Jest's docs. The bot will learn with the documentation daily so it will be always updated with the recent doc changes. (100% free btw)
We would love to hear your feedback on the Jest copilot assistant! Hopefully, this can help your users and decrease some of the incoming support that yall provide. Let us know and we can also email account details to you so you access analytics and use them to improve the documentation and experience around Jest.
More background: I am one of the co-founders of Mendable.ai. We are supporters of open-source projects and we'd love to offer Mendable for free, with the goal of helping yall out teach developers about your product and answer users' queries.

Hey @SimenB @cpojer I would love to hear your feedback if possible! Happy to iterate on it as well. Thank you!

@SimenB @cpojer
Hello guys. We would still love to get this PR reviewed.
Thank you.

Let's not do this at this time. I'm worried about both the branding aspect as well as the likelihood for mistakes by LLMs. Maybe in the future, and without branding. Thanks for proposing this change though.

Let's not do this at this time. I'm worried about both the branding aspect as well as the likelihood for mistakes by LLMs. Maybe in the future, and without branding. Thanks for proposing this change though.

Alright @cpojer Thanks for your input. Let's see what the future holds 🚀.


[Documentation] update documentation as per API changes

Purpose

update documentation

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[x] Documentation content changes
[ ] Other... Please describe:

Also @iaulakh  can we trying adding the tags for code:
https://www.markdownguide.org/cheat-sheet#:~:text=Extended Syntax    Element  ,sentence with a footnote. 7 more rows


[test] RV_DM access after reset test

This is a draft of a test to cover https://github.com/lowRISC/opentitan/issues/14152

It writes to the debug modules' program buffer through JTAG.
Puts the device into a normal sleep and wakes it up again.
Reads from the program buffer and checks it still has the expected value.

The test currently fails when checking the program buffer's value (step 3), the actual value is 0x0 when expecting 0xDEAF.
The problem appears to be the write not taking effect. It persists when the read and check happens straight after the write (with no sleep between them).
I can't workout why the write isn't taking effect.

The test is now passing for normal sleep (I think https://github.com/lowRISC/opentitan/pull/15725#discussion_r1019557508),
but I need to add a check after deep sleep as clarified in the updated test plan https://github.com/lowRISC/opentitan/pull/15769.
I'll do this and then turn this from a draft to a normal PR.

@HU90m i made some minor tweaks and the test is now passing for me.
We basically needed to reset the agent side because the design side had fully reset, so some of the assumptions were no longer the same.

Thank you @tjaychen  and @sriyerg! The test was working on my end as well.
I was hoping to get the test merged today, but after rebasing onto https://github.com/lowRISC/opentitan/commit/d9c8b180c1ca1267e410716a104b126569bae9d6 (https://github.com/HU90m/opentitan/tree/rv_dm_access_after_wakeup_H4), the software no longer runs. (It timeouts waiting for the "Software Setup." log.) It looks like it is caused by a  otp_ctrl_fatal_check_error, but I haven't found the cause of this yet.

@sriyerg has suggested a great way to improve the test, in https://github.com/lowRISC/opentitan/pull/16383#discussion_r1023558499 , but this'll do for now.

Thank you @tjaychen and @sriyerg! The test was working on my end as well.
I was hoping to get the test merged today, but after rebasing onto d9c8b18 (https://github.com/HU90m/opentitan/tree/rv_dm_access_after_wakeup_H4), the software no longer runs. (It timeouts waiting for the "Software Setup." log.) It looks like it is caused by a otp_ctrl_fatal_check_error, but I haven't found the cause of this yet.

Hm we should not be getting otp_ctrl_fatal_check_error at this point (this happens when the digests of the OTP partitions are incorrect, typically happens if the OTP image has not been loaded correctly, or if a backdoor write to OTP did not update the associated digests). However, if this is passing again, let's proceed and just monitor this via the nightlies. If it happens again, we can investigate further.

Thank you @tjaychen and @sriyerg! The test was working on my end as well.
I was hoping to get the test merged today, but after rebasing onto d9c8b18 (https://github.com/HU90m/opentitan/tree/rv_dm_access_after_wakeup_H4), the software no longer runs. (It timeouts waiting for the "Software Setup." log.) It looks like it is caused by a otp_ctrl_fatal_check_error, but I haven't found the cause of this yet.

Hm we should not be getting otp_ctrl_fatal_check_error at this point. This happens when any of the OTP partition digests is incorrect - typically happens if the OTP image has not been loaded correctly, or if a backdoor write to OTP did not update the associated digests. However, if this is passing again, let's proceed and just monitor this via the nightlies. If it happens again, we can investigate further.

@HU90m, please let us know if this test is passing. I think it is ready to be merged.

Thanks! Thankfully the test is passing again. I've also changed to use LOG_ERROR for unexpected reset sources.

I believe this is good to go.


Typo Rectified

DRILL-XXXX: PR Title
(Please replace PR Title with actual PR Title)
Description
(Please describe the change. If more than one ticket is fixed, include a reference to those tickets.)
Documentation
(Please describe user-visible changes similar to what should appear in the Drill documentation.)
Testing
(Please describe how this PR has been tested.)

Here are few more suggestions
--- 060-querying-the-information-schema.md.old	2020-06-18 09:51:39.640215255 +0200
+++ 060-querying-the-information-schema.md	2020-06-18 09:51:50.553274730 +0200
@@ -74,11 +74,11 @@
 The VIEWS table returns the name and definition for each view in your
 databases. Note that file schemas are the canonical repository for views in
 Drill. Depending on how you create a view, they may only be displayed in Drill
-after it has been used.
+after being used.

 ## PARTITIONS

-The PARTITIONS table returns information about table partitions including partition columns, partition keys, etc. for each table.
+The PARTITIONS table returns information about table partitions, including partition columns, partition keys, etc. for each table.

 ## FILES

@@ -142,7 +142,7 @@
 	| DRILL         | s3_years_bucket.root     | <owner>       | file  | NO          |
 	+---------------+--------------------------+---------------+-------+-------------+

-Querying the FILES table and filtering on the SCHEMA_NAME provides information about the files that exist within a workspace:
+Querying the FILES table and filtering on the SCHEMA_NAME provides information about the files within a workspace:

 **Note:** The word “files” is a reserved word in Drill and requires backticks (``).

@@ -235,7 +235,7 @@
 ### TABLES Queries

 Run an ``INFORMATION_SCHEMA.`TABLES` ``query to view all of the tables and views
-
+within a database. TABLES is a reserved word in Drill and requires backticks
 (`).

 For example, the following query identifies all of the tables and views that

@dzamo Are you ok with committing this?

@dzamo Are you ok with committing this?

Thank you @swattu and @sanel, I've incorporated your contributions.  Some notes for future contributors follow.

We don't require Drill's JIRA-based PR template for documentation destined for gh-pages.
Sending documentation as commits helps us more than sending it as Github comments.
If you can cast around for 15 minutes and send a handful of fixes or additions in your PR that helps more than a PR to fix a single typo.  We do know that time is scarce and we are grateful for any you can spare.

Thank you @swattu and @sanel, I've incorporated your contributions.  Some notes for future contributors follow.

We don't require Drill's JIRA-based PR template for documentation destined for gh-pages.
Sending documentation as commits helps us more than sending it as Github comments.
If you can cast around for 15 minutes and send a handful of fixes or additions in your PR that helps more than a PR to fix a single typo.  We do know that time is scarce and we are grateful for any you can spare.


Bug/issue#57

I changed the TVLoss computation to be based on only predicted image, as it's used as a regularizer applied to the predicted images during NN training.
As the functions _adjust_dimensions and _validate_inputs from utils module only support two input tensors, I added their counterparts _adjust_tensor_dimensions and _validate_input directly in tv module. In the future, they may be combined with the corresponding functions from utils.
The README file and tests are changed to be up to the fix.

@zakajd @denproc can you have a look and comment?


[DO NOT MERGE] Demo without security questions

I'm doing a demo tomorrow and I want to show this version of the service.

This demo is over and this PR is no longer needed.


Loop request fails for a certain waypoint

OS version: 20.04, ros foxy
Branch: Master of rmf demos
commands used: ros2 launch demos imda.launch.xml
When i launch my file, the gazebo and rviz panels work fine. The tinyrobot is able to navigate to most waypoints. Attached is the screenshot of my rviz for reference

However there is a certain point that I named waiting1 that the tinyRobot is unable to navigate to. Attached is the error message for reference.

The line referencing the point in the building.yaml file is - [353.253, 77.66800000000001, 0, waiting1], and the corresponding nav file is - - 26.782523101542488
- -6.092391183051954
- {name: waiting1}
I have also tried adding the is_passthrough_point attribute on the building.yaml and nav file, but the result is the same.
Please advise. Thank you.

Thank you.
Is it possible to share the map file with us, so that we can run it with a debugger attached when it crashes?

https://drive.google.com/drive/folders/1vKuHr03cxMLLe23md65AHZEK8pbA1amf?usp=sharing

Thank you.
Is it possible to share the map file with us, so that we can run it with a debugger attached when it crashes?

https://drive.google.com/drive/folders/1vKuHr03cxMLLe23md65AHZEK8pbA1amf?usp=sharing

Thanks for sending the traffic-editor files. The lane edge from waiting1 to lift_middle is not connected to the rest of the lane graph -- you can see this by using the Move tool (pressing M) and dragging the waiting1 vertex around.
To attach it to the rest of the graph, the easiest way is to go into traffic-editing mode (Ctrl+T), select tool (Esc) and clicking on the edge between eight and seven and deleting that lane edge. Then you can add new lane edges from seven to waiting1 and then from waiting1 to eight
I'm guessing that you'd probably want lift_lobby to also be in the traffic lane edge network, so you might want the edges to stop at lift_lobby along the way
hopefully then after rebuilding the map package, it will generate a nav_graph.yaml that can reach all the endpoints you need.

Have done as you said, it works now! Thanks for your help!

Thanks for sending the traffic-editor files. The lane edge from waiting1 to lift_middle is not connected to the rest of the lane graph -- you can see this by using the Move tool (pressing M) and dragging the waiting1 vertex around.
To attach it to the rest of the graph, the easiest way is to go into traffic-editing mode (Ctrl+T), select tool (Esc) and clicking on the edge between eight and seven and deleting that lane edge. Then you can add new lane edges from seven to waiting1 and then from waiting1 to eight
I'm guessing that you'd probably want lift_lobby to also be in the traffic lane edge network, so you might want the edges to stop at lift_lobby along the way
hopefully then after rebuilding the map package, it will generate a nav_graph.yaml that can reach all the endpoints you need.

Have done as you said, it works now! Thanks for your help!


pipeline: {} init

Closes #1677
This PR lays the groundwork for the pipeline work to be built upon.

A new dev-vm has been added, to use in dev + testing without polluting the existing user resources
Early docs have been added that will be updated at each stage as features are added

/test

/test-force-approve
have run tests already


Added All Builders Profile in Homepage

Description
I added a section in the index.tsx that directs users to all the builder's profile;

When the "click here" is tapped, it directs the operator to:

The operator then clicks any address to view the profile.
Also added my profile.
Additional Information

[ ] I have read the contributing docs (if this is your first contribution)
[ ] This is not a duplicate of any existing pull request

Related Issues
Closes #{issue number}
Note: If your changes are small and straightforward, you may skip the creation of an issue beforehand and remove this section. However, for medium-to-large changes, it is recommended to have an open issue for discussion and approval prior to submitting a pull request.
Your ENS/address: akwuba.eth

Hey @Chinwuba22 , thanks for the PR!
@solo1g is also working on this (see assigned on #3)
Also:

You committed your profile page in this PR. Let's do that on a different PR as part of #8
The list of builders should be dynamic: we can read the "CheckedIn" event from the contract.
You are getting some linting errors:

2:10  Error: 'Address' is defined but never used.  @typescript-eslint/no-unused-vars
21:21  Error: 'address' is defined but never used.  @typescript-eslint/no-unused-vars


Ascertain proper number of RF estimators and add to code

Like http://scikit-learn.org/stable/auto_examples/ensemble/plot_ensemble_oob.html.


pulse.Instruction.draw refers to non-existent visualization.pulse_drawer

Environment

Qiskit Terra version: 0.22.2
Python version: 3.10
Operating system: macOS

What is happening?
The method pulse.Instruction.draw refers to visualization.pulse_drawer: https://github.com/Qiskit/qiskit-terra/blob/dfbc738be57619b503b931a2d0741433a1bbe21b/qiskit/pulse/instructions/instruction.py#L223-L275
This function does not exist - it was removed in #8306, as it had been deprecated in Terra 0.17.  This wasn't caught by lint because visualization had a getattr function defined (to handle other unrelated deprecations).
How can we reproduce the issue?
...
What should happen?
Possibly this needs transitioning to the v2 drawer, but it's not clear to me if there's an entry point for single instructions any more.
Any suggestions?
As a stop-gap measure, we can probably directly call visualization.pulse.ScheduleDrawer, which while logically deprecated (because its entry point was), wasn't actually removed as part of #8306.

Fixed by #9144.


Update Kubernetes and local cluster examples.

Fix clients for _vt port rename.
Other minor fixes.

LGTM


New PR to test SignCLA at 2020-12-08 11:09:17.792073847 -0800 PST m=+4.954089191

Body of PR

indiv-pass-staging: test succeeded


Remove TVO_CanBindToHole flag in ConstraintSystem::openGenericParameter when generic parameter is parameter pack

Removed TVO_CanBindToHole flag in ConstraintSystem::openGenericParameter when generic parameter is parameter pack, so that passing nil to generic parameter pack produces fix correctly.
Resolves #69432 .

Before
func foo<each T>(_ bar: repeat each T) { }

foo(nil) // Failed to produce diagnostic for expression; please submit a bug report

After
func foo<each T>(_ bar: repeat each T) { }

foo(1, "hello", 2) // OK

foo(nil) // error: 'nil' requires a contextual type

foo(1, nil) // error: 'nil' requires a contextual type

Diagnostic messages:
/Projects/SwiftPlayground/Sources/main.swift:5:5: error: 'nil' requires a contextual type
foo(nil)
    ^
/Projects/SwiftPlayground/Sources/main.swift:7:8: error: 'nil' requires a contextual type
foo(1, nil)

One more thing - please add test-cases from description to test/Constraints/variadic_generic_functions.swift and we’d be good to go!

@li3zhen1 So I've been looking at the the constraint system state with these problematic cases and I'm thinking that we should just produce a tailored fixed for each pack element that cannot be inferred with diagnostic like 'cannot infer pack element # from contextand a tailored version of that when it's converted tonil(that shouldn't be hard to detect because the locator forpack element` is going to have ApplyArgToParam in the path for these cases).
It seems reasonable to let pack element to be bound to a hole and propagate that to the nil type variable instead of other way around because we don't want to attempt binding nil directly until absolutely necessary.

@li3zhen1 So I've been looking at the the constraint system state with these problematic cases and I'm thinking that we should just produce a tailored fixed for each pack element that cannot be inferred with diagnostic like 'cannot infer pack element # from contextand a tailored version of that when it's converted tonil(that shouldn't be hard to detect because the locator forpack element` is going to have ApplyArgToParam in the path for these cases).
It seems reasonable to let pack element to be bound to a hole and propagate that to the nil type variable instead of other way around because we don't want to attempt binding nil directly until absolutely necessary.

How about this?

Sorry, I'm trying to decide how I keep about all of the locators manipulation that is going on here.

@li3zhen1 We are getting pretty close. I think all we need is to address a few remaining comments and add a few more test-cases that test newly add diagnostic.

@xedin Sorry for the delay. I'm still trying to learn how repairFailure works here. Could you give more hint about this? like do I try avoid the unexpected fixForHole call or just silence the hole's diagnostics?

No rush! Regarding repairFailures, there is a simpler problem you can look at which doesn't involve variadic generics:

The failure to convert Array to Set here is "fixed"  via IgnoreContextualType fix - (attempting fix [fix: ignore specified contextual type] @ locator@0x14b13f320 [Call@<stdin>:5:19 → contextual type])
This block of code is repairFailures that deals with ContextualType locator could detect that both types are BoundGenericTypes with the same number of arguments and use matchTypes (like we do in other places there) to match them, that would mean that Int from Set is propagated to type variable that opens T of Array and would avoid generic parameter 'T' could not be inferred failure.

But as I mentioned before, you don't have to fix it in this PR - you can just adjust the changed diagnostics, we'll take a look what changed and whether it makes sense.

We shouldn't produce a fix in this case because this would just result in a fallback diagnostic this looks like more of an assert to me

So does this mean that I should move the condition applyArguments -> applyArgToParam -> PackElements to fixForHole?

In general if there is a case that cannot be diagnosed, it shouldn't produce a fix just to fail a diagnostic.  If we detect that a type variable represents a pack element it shouldn't matter what path it uses because it can always fallback to newly added "cannot infer pack element ..." diagnostic right? In the diagnostic we could check for presence of ApplyArguments or ArgToParam etc. and produce tailored diagnostic messages but the fallback is always there.

If I understand correctly, I need to either:

fix the unexpected <placeholder> first
or add the fallback diagnostic, and allow that case to emit this cannot infer pack element ...
Right?

Not really, I think check for PackElement can be turned into an assert and check for apply arguments could be removed completely for the diagnostic.

I see. Below is the result accordingly. Guess I can't pass the test without removing that placeholder?
/Users/lizhen/Projects/swift-projects/build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64/bin/swift-frontend /Users/lizhen/Projects/swift-debug/generic_mismatch.swift -typecheck
/Users/lizhen/Projects/swift-debug/generic_mismatch.swift:9:21: error: cannot convert value of type '(repeat Array<_>)' to specified type 'Set<Int>'
 7 │ func patternInstantiationConcreteInvalid() {
 8 │
 9 │   let _: Set<Int> = patternInstantiationTupleTest1()
   │                     ╰─ error: cannot convert value of type '(repeat Array<_>)' to specified type 'Set<Int>'
10 │
11 │ }

/Users/lizhen/Projects/swift-debug/generic_mismatch.swift:9:21: error: could not infer pack element #0 from context
 7 │ func patternInstantiationConcreteInvalid() {
 8 │
 9 │   let _: Set<Int> = patternInstantiationTupleTest1()
   │                     ╰─ error: could not infer pack element #0 from context
                              ❌  this is not expected in test case.
10 │
11 │ }

bool UnableToInferGenericPackElementType::diagnoseAsError() {
  auto *locator = getLocator();
  auto path = locator->getPath();

  const auto packElementElt = (path.end() - 1)->getAs<LocatorPathElt::PackElement>();

  if (isExpr<NilLiteralExpr>(getAnchor())) {
    emitDiagnostic(diag::unresolved_nil_literal);
  } else {
    // ❌ generic mismatch hits here:
    emitDiagnostic(diag::could_not_infer_pack_element,
                   packElementElt->getIndex());
  }

  if (isExpr<ApplyExpr>(locator->getAnchor())) {
    // emit callee side diagnostics
    if (auto *calleeLocator = getSolution().getCalleeLocator(locator)) {
      if (const auto choice = getOverloadChoiceIfAvailable(calleeLocator)) {
        if (auto *decl = choice->choice.getDeclOrNull()) {
          const auto applyArgToParamElt =
              (path.end() - 2)->getAs<LocatorPathElt::ApplyArgToParam>();
          if (auto paramDecl =
                  getParameterAt(decl, applyArgToParamElt->getParamIdx())) {
            emitDiagnosticAt(
                paramDecl->getLoc(), diag::note_in_opening_pack_element,
                packElementElt->getIndex(), paramDecl->getNameStr());
          }
        }
      }
    }
  }

  return true;
}

@swift-ci please test

@swift-ci please test Windows platform

@li3zhen1 Thank you for seeing this through. The next time you wish to take an issue with relatively recent prior activity, please ask the previous volunteers if they are still working on it and mind you taking it. We don’t want good first issues to be a competitive game.

@li3zhen1 Thank you for seeing this through. The next time you wish to take an issue with relatively recent prior activity, please ask the previous volunteers if they are still working on it and mind you taking it. We don’t want good first issues to be a competitive game.

Sorry. I’ll keep this in mind.

The failure to convert Array to Set here is "fixed" via IgnoreContextualType fix - (attempting fix [fix: ignore specified contextual type] @ locator@0x14b13f320 [Call@<stdin>:5:19 → contextual type])
This block of code is repairFailures that deals with ContextualType locator could detect that both types are BoundGenericTypes with the same number of arguments and use matchTypes (like we do in other places there) to match them, that would mean that Int from Set is propagated to type variable that opens T of Array and would avoid generic parameter 'T' could not be inferred failure.

@xedin Did you mean that for this case there should be no generic parameter 'T' could not be inferred? I tried swift 5.9 and the main branch and they are all emitting two errors including this. Is this a desired behavior? If so, I feel like it's reasonable to emit 2 errors in the generic pack element case as well.


Use plugin on ant based project

hi,
is there any jar file of your plugin that i can use in my ant based project?

You can find any artifact for any version on bintray page (for example aar for the version 2.6.4)


Server crash, exiting wrong phase

I'm fairly sure that this is only happening when I have this plugin installed.  I'm using a version built from source ec97ade.  I was previously using the version published on Ore and was getting similar state issues.
Gist: https://gist.github.com/treyzania/6227f2c74e627d912be33a0c644400ef
And another crash log too big for gist (scroll to the end): https://tr3y.io/media/2018-05-15-7.log
I'm also getting a lot of "can't keep up" warnings that only happen with this plugin.  I'm running this on a reasonably powerful DigitalOcean droplet so I don't think that I'm actually underprovisioning the server.  I'm trying to run the timings to check to be sure if this is actually the case.
This is in the log but I'll copy it here for ease of reference:
Minecraft: 1.12.2
SpongeAPI: 7.1.0-SNAPSHOT-7d8a9382
SpongeVanilla: 1.12.2-7.1.0-BETA-42

Here's a zip that should reproduce the issue, using the 1.2.7. release: https://tr3y.io/media/antixray-example.zip
You can start the server with harness.sh.  I also included the crash-reports folder from Sponge, which has some crash logs from when I was initially testing the server, so the ones you would be looking for would be the more recent ones.

About your crash, I did not found anything related to my plugin for now, these kind of crashes are very generics and can be caused by almost anything. Keep me informed if this crash reappear.
About your performances issues, try to run some timings (10 minutes are usually enough but longer is better) and then post the timings link here or in private.
Can you also give me a copy of your worlds configurations ?

This issue was probably fixed during the numerous modifications of the last version. I close this issue as long as it does not reappear.


Azure Functions VSCode extension causes unresponsiveness

The three extensions that have potential to cause this are below, with Azure Functions being the leader:

I have no idea what to check to help troubleshooting this issue and get away of this annoying notification (which is later followed by another one):

Extension Name: vscode-azurefunctions
Extension Version: 0.11.0
OS Version: Windows_NT x64 10.0.17763
VSCode version: 1.28.0

We have written the needed data into your clipboard. Please paste:
{
	"activationTimes": {
		"startup": true,
		"codeLoadingTime": 17897,
		"activateCallTime": 50,
		"activateResolvedTime": 0,
		"activationEvent": "workspaceContains:local.settings.json"
	}
}

@chojrak11 you're on insiders, correct? That warning is very new and was just removed. Per https://github.com/Microsoft/vscode/issues/60318:

The notification was a great way to become aware that some extensions keep the extension host busy. However, as is, the notification is not helpful for the general user and it motivates the creation of unhelpful issues. We need to remove the notification until we have a better flow.

I'll leave this issue open to double check our actiavtion times - but hopefully you won't see that warning again anyways

@EricJizbaMSFT sorry I was AFK for a few days. It was the interim 1.28 release of VSCode that caused these warnings, which were then removed in 1.28.1 if I remember correctly. The warning is one thing, but I'm having hard time understanding what requires 16-17 seconds during initialization of this extension. Hope you'll get it sorted.


Bug: {csp-style-nonce} Doesn't Work

PHP Version
8.1
CodeIgniter4 Version
4.2.10
CodeIgniter4 Installation Method
Composer (using codeigniter4/appstarter)
Which operating systems have you tested for this bug?
Linux
Which server did you use?
apache
Database
none
What happened?
{csp-style-nonce} is not getting resolved when my view is processed. I have to use <?= csp_style_nonce() ?> to get it to work. ($autoNonce = true in app/Config/ContentSecurityPolicy.php).
Steps to Reproduce
Turn CSP on. Create a view with a style tag, i.e., <style {css-style-nonce}>...</style>
Expected Output

Anything else?
No response

Not {css-style-nonce}, but {csp-style-nonce}.

Typo. {csp-style-nonce} doesn't work either. {csp-script-nonce} also does not work.
On Thursday, December 15, 2022 at 04:36:59 AM EST, kenjis<EMAIL_ADDRESS>wrote:

Not {css-style-nonce}, but {csp-style-nonce}.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Cannot reproduce it.
diff --git a/app/Config/App.php b/app/Config/App.php
index 79e5741..7b56cf6 100644
--- a/app/Config/App.php
+++ b/app/Config/App.php
@@ -462,5 +462,5 @@ class App extends BaseConfig
      *
      * @var bool
      */
-    public $CSPEnabled = false;
+    public $CSPEnabled = true;
 }
diff --git a/app/Controllers/Home.php b/app/Controllers/Home.php
index 7f867e9..4cc76d0 100644
--- a/app/Controllers/Home.php
+++ b/app/Controllers/Home.php
@@ -6,6 +6,6 @@ class Home extends BaseController
 {
     public function index()
     {
-        return view('welcome_message');
+        return '<style {csp-style-nonce}>...</style>';
     }
 }

I can see <style nonce="9ada8f523ef9afdc5593b739">...</style> in the response.

Must be something gummed up with my config settings. I'm using app.CSPEnabled = true in .env if that matters. I'll mess around with it some more. Also the views are in app/Views/errors/html too. I don't know why that would matter.

Well, it's working now. I had another problem with ini_set() and "headers already have been set.." errors at the same time as this problem because of a typo. Maybe that caused everything to shut down. I've since fixed that and the nonce stuff now works.

We use GitHub issues to track BUGS and to track approved DEVELOPMENT work packages. We use our forum to provide SUPPORT and to discuss FEATURE REQUESTS.


Remove certain users from People page

Change the way that people are show on the People page.
The People index view should only use the current template for accounts who satisfy the following requirements:
Have completed their profiles, a complete profile would include:

First and last name
contact email
Company name
avatar with Gravatar or uploading a file
at least 60 characters in their bio OR three News or Event posts

All other People should have a simplified template, showing just their first name, last name

Hi, can I help out with this issue? If yes, would like to know what's the criteria of completing the profile i.e. first name and last name as in validation or completed all fields in edit profile page? What's the details about "company if those exist"? And do we still include the "inactive person" in filtering?

@yoonwaiyan I'd love your help.
I'm editing the issue with more detail to answer your questions and clarify further.

Alright, will work on this.

@kentonh is this issue resolved ?

@sa77 it looks like @yoonwaiyan is working on this issue.


Migrate to hosting on GitLab

GitLabForm is fundamentally a tool for managing GitLab.

If we migrated to hosting on GitLab we would benefit from the OpenSource Initiative: https://about.gitlab.com/solutions/open-source/join/ - qualifying us for among other things, an Ultimate license and free runner minutes.

We could then control our own repository config using GitLabForm, providing a neat demonstration to other Organisations wanting to adopt the solution.

We could take advantage of GitLab's inbuilt scanners such as Vulnerability scanning, SAST et, to provide a greater degree of confidence in the security of our project.

We can enforce approvals on MRs whilst still retaining our current Manual Release process, with a view to moving to an automated flow

Likely hurdles

We will need to re-write our GitHub workflows into gitlab-ci.yml files
We will need to deploy to GitLab pages and redirect from the existing GitHub pages
We will need to setup something like Renovate to replace the native Dependabot support in GitHub

Process

SPIKE: Migrate GitHub into GitLab - using a test repo first as a destination
Develop GitLab ci pipelines and release process
Ensure all secrets are captured
Final migration to end repo and enabling of the developed GitLab CI and auto-dependancy updates
Apply for OpenSource Programme

https://gitlab.com/gitlabform is already claimed, I think we might need to archive and re-import in order to ensure we get the current state of issues/PRs etc, but 🤞 I can use that to apply for the OpenSource programme.

First I'll do some testing of the migration and ci/cd pipeline creation over on https://gitlab.com/gitlabform1 -> which I've just grabbed a free Ultimate trial for

https://gitlab.com/gitlabform1/gitlabform/-/merge_requests/782 -> prelim CI work

https://gitlab.com/gitlabform is already claimed

It's mine so we can use it if we want to. ☺

So far the main blocker on the CI front is getting the acceptance tests to run with how GitLab handles Docker commands on it's runners

We should check how it's done in https://gitlab.com/gitlab-org/terraform-provider-gitlab, we probably need a similar solution to them.

We have now working an OpenSource UItimate License for the GitLab hosted GitLabForm.
@amimas @gdubicki -> here is an MR to build out the majority of the CI: https://gitlab.com/gitlabform/gitlabform/-/merge_requests/1
It is missing

Arm64 build
Codecov integration
Automatic dependency updates

publishing to GitLab not GitHub Pages

This is great, we will switch to those with a redirect from the old address.

…and Container Reg

I would switch back to DockerHub here, like @jimisola suggested at some point.

We have now working an OpenSource UItimate License for the GitLab hosted GitLabForm. @amimas @gdubicki -> here is an MR to build out the majority of the CI: https://gitlab.com/gitlabform/gitlabform/-/merge_requests/1

Amazing work, thank you @TimKnight-DWP! 👏 🥳
I noticed that the tests are running a bit slower, the acceptance ones take ~13 minutes in GitLab CI while they took ~10 minutes in GitHub Actions.
I have enabled timestamps to try to track what exactly is slower, see https://gitlab.com/gitlabform/gitlabform/-/jobs/7207889811

So comparing https://github.com/gitlabform/gitlabform/actions/runs/9698230309/job/26764299132 to https://gitlab.com/gitlabform/gitlabform/-/jobs/7207889811 my first observations are:

the tests themselves are ~2 minutes slower in GitLab CI. That's strange because we are using 4 CPU, 16GB RAM runners in GitHub Actions (https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories) while 8 CPU, 32GB RAM in GitLab CI...
setting up the environment takes ~1 minute longer in GitLab CI, including downloading the Docker images - perhaps we should try switching to Docker (dind) and Python images hosted by GitLab? https://docs.gitlab.com/ee/user/packages/dependency_proxy/, perhaps?

Yeah if we can setup the proxy that would be good.

I've just got an email with a Mend.io Renovate CE license key we can use:
Getting started:
Docker - Installation guides: GitHub, GitLab, Bitbucket
Kubernetes - Use Helm charts
Other docs and guides:
Mend Renovate Community Edition configuration
Renovate CLI configuration


npm ERR! code ELIFECYCLE if 1 test fails

If 1 test fails npm return an error
Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 passed, 2 total
Snapshots:   0 total
Time:        1.197s
Ran all test suites matching /.\/assets/i.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>unit: `jest --verbose ./assets`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>unit script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:

This is not a bug. Jest exits with code 1 because a test failed.

Any setting to prevent that ? i read mostly of the documentations and didn't find information about that.

I don't think so (and I cannot imagine why you would want that).
But jest --verbose ./assets || true would always exit with 0, no guarantees on Windows compatibility though 😅

I disagree with this as a general statement - and I think there are other programs where the sensible thing is to forward an error caused by something that is not under the control of the program. For example, you wouldn't want yarn run somescript to swallow the exit code of somescript and just return 0 because it successfully executed somescript.
I think this choice needs to be made on a case-by-case basis, and for Jest clearly the implications of exiting with 0 despite failing tests would be immense - think of all the CI pipelines that would stop detecting failing tests.

@jeysal So what do you suggest in order to get rid of those 10 lines and the continuous creation of useless debug log files?
It's immensely annoying to get that every single time your test fails, merely because you were not done programming yet.

@pedzed As suggested in this Stack Overflow post, you can use the --silent or -s flag to avoid npm error messages if tests fail.

@pedzed As suggested in this Stack Overflow post, you can use the --silent or -s flag to avoid npm error messages if tests fail.

Ah, thanks!
npm -s run test

npm -s run test
it work for me, thank you so much


disable management features

I need a docker container to provide only the runtime environment to run the NorthStar server. The only other container available doesn't work at all, so I need an option for this one (or alternate version) which only runs the server and does not attempt to manage the files or configuration, as I already have that set up outside the container and the options documented in the readme are insufficient.

After discussing this on discord, I have decided to add an undocumented (and unsupported) option to use assembled game files from a mounted directory instead of the overlay and config logic.

Another idea that occurred to me after our conversation was to add a mount point for config files ( specifically ns_startup_args_dedi.txt and mods\Northstar.CustomServers\mod\cfg\autoexec_ns_server.cfg since they contain all know vanilla config options that I'm aware of ) to allow all options on top of your container's features

It also just now occurs to me, could I already include autoexec with your existing mods mount point? if so I guess the only real issue is the options used in ns_startup_args_dedi.txt.

Testing my theory reveals that config path is specifically disallowed for overwriting. I would like to figure out a solution that allows NorthStar to be updated with the container if possible.

@sugoidogo, if you have a prepared Northstar dir and you don't need Docker for anything, you can try my new nswrap tool (compile it along with the gfsdk and d3d11 stubs, then create the wineprefix manually and launch it with nswrap).

I don't know anything about C, do you have compiled binaries or build instructions?

Which distro are you on?

Proxmox. I'm most familiar with debian but I can spin up whatever distro is easiest.

You can now do this with something like the following command: docker run -v /path/to/northstar/installation:/northstar -e DISPLAY=xvfb -p 8081:8081/tcp -p 37015:37015/udp -it --entrypoint nswrap ghcr.io/pg9182/northstar-dedicated:1-tf<IP_ADDRESS>-ns1.6.3 /northstar [args]. You will need to ensure that the install directory is writable by UID 1000.
Note that you don't need to pin the container image to a specific northstar version, but I recommend doing so (to the corresponding version) just in case a future nswrap version breaks compatibility with your installation.


run tools/demo.ipynb error

[libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 6305:21: Message type "caffe.LayerParameter" has no field named "roi_pooling_param".
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0216 21:21:02.787189 22074 upgrade_proto.cpp:90] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: /bottom-up-attention/models/vg/ResNet-101/faster_rcnn_end2end/test.prototxt
*** Check failure stack trace: ***
Aborted
could you please give me a solution to fact this error? Thank you!!

I have the same problem. Could you tell me how to solve it? thank you!

I have the same problem. Could you tell me how to solve it? thank you!