Intuition about hyper parameters for Atari games

Hi!
I decided to put Sokoban off for now as it doesn't seem to work the same as in World Models. So I started to experiment with Atari once again. I easily made World Models work with e.g. Boxing environment, but in PlaNet I get very blurry open loop predictions:

Please, could you share your intuition about hyper-parameters I might try to tune? I tried to disable overshooting (to make the case more similar to World Models) and now I'm looking into divergence scales, but without luck for now. As I have no cluster to run hyper-params optimisation, maybe you have some ideas what is worth focusing at? :D
Thanks!

At least for me, 50k train, 1.5k test and collect_every 8k works for Assault-V0 with 1.2k max length bound. The agent learns somewhat decent behaviour (~300 score per episode). Note that Atari games already do action repeat, thus I disabled this in the PlaNet's source code.

@piojanu The predictions don't seem that blurry (I do see agents fighting). Take a look again into World Models paper - their predictions look blurry as well.

Thanks guys, I still work on that. Nevertheless, PlaNet continuous control tasks had really sharp reconstructions even after 15 steps. Furthermore, I've implemented World Models and trained it for Boxing. Here are the open loop predictions I was able to obtain there:

As you can see those are MUCH clearer. Do you have other ideas?

@piojanu Interesting, they do seem much clearer. That's a dimension to explore though, maybe the VAE of World Model's is bit better :)

Also:
@astronautas the agent performance is weak too. Please keep your responses more succinct. What do you mean "maybe the VAE of World Model's is bit better"?
@Kwander thank you for noting that OpenAI Gym already does action skipping. I'll disable it too!

Also:
@astronautas the agent performance is weak too. Please keep your responses more
@Kwander thank you for noting that OpenAI Gym already does action skipping. I'll disable it too!

Closed by mistake, sorry.
Lowering global prior divergence scale seems to help. I'll disable it and run the test again ;)

@piojanu What do you mean the performance is weak too? I've edited my answer.

He gets on average below 0 points. In World Models it was 18 points on average, but planning algorithm is different so it's not fair to compare. Please lets focus on prediction metrics (reconstruction in that case) as it's the main focus of my question.

What's the batch you're using?

@astronautas that's funny a little, expanding VAE to Variational Auto Encoder doesn't add value I was thinking about 😄
I use default batch size (50, 50).

Sorry, the author did not use the term vaes in the paper, Variational encoder was the definition used :D

@Kwander do you observe this too:

Please note the scale of y-axis. Those values are barely changing e.g. graph/summaries/closedloop/posterior/log_prob/image stays pretty much the same all the time. Is it expected? For me it's sign that it's not learning much beyond first steps before first summary gets written.

@piojanu I have noticed this as well, for dmcontrol cartpole swingup problem.

Okay, thanks for clarification ;) Do you use default config in "Assault-V0"? Can you post your open loop reconstructions and agent's score to see what to expect?

@piojanu By default we use a decoder variance of 1, which means the model explains a lot of variation in the image as random noise. While this leads to more robust representations, it also leads to more blurry images. If the predicted images are all the same, the posterior collapsed because the model explains everything as observations noise. Try to reduce the decoder variance in conv_ha.py or equivalently set a lower divergence_scale parameter.
@astronautas That's a cool result. I assume you're still using CEM with a Gaussian belief over action sequences for planning on even though the Atari actions are discrete, and later take the argmax along the action dimension?

@danijar Thanks, yes, I am using those. My action space consists of N actions (-1 to 1), which I later argmax in the step function of the environment wrapper.

Thanks guys, status update:
For smaller state size (16) and smaller global divergence scale (0.01) I got to this:

Today I'll run hyper-param tuning and leave it for ~3-4 days. I'll get back to you with results (and even more questions, haha 😊) in here so let's leave this issue open.

Quick update: I'm working on the hyper-param tuning. I wasn't able to run it during the last weekend. I run it yesterday though.

I did several more runs on Assault and Breakout Atari. I'll update my comment in the evening with the results. The algorithm seems to perform worse on Breakout game, although it's probably the simplest one for model-free algorithms (no early conclusions, will share the results first).

@danijar check me if I understand correctly what is happening.

Consecutive frames in Boxing are quite similar so the decoder explains it as a random noise. It means that the posterior starts to infer similar codes for consecutive frames (this is what you mean by "the posterior collapse", right?).
The transition model (prior) is trained to minimise KL-divergence to the posterior. The posterior tells it that consecutive frames have (almost) the same codes, so the transition model doesn't learn anything and open loop predictions are more or less the same blurry blob.

Now, lowering the variance of the decoder should result in sharper reconstructions, because a blurry blob is less probable under the lower variance i.e. MSE loss is higher for blurry recostructions and the posterior needs to encode concrete information about what is in the frame so the decoder can use it (i.e. it needs to stop encoding bull**** 😄).
The other way is to lower the divergence scale, so one-, two-, three-... step predictions (that will have the same or very similar codes as I explained above) will have lower impact than long-term predictions (which should differentiate the frames better, because frames further apart have more differences). Am I right?
One thing that I don't understand is why you called the two approaches "equivalent":

Try to reduce the decoder variance in conv_ha.py or equivalently set a lower divergence_scale parameter.
Those seems to me to address the problem in fundamentally different ways and maybe both changes will need to be applied. Could you expand on why those are equivalent?

I suggest to reduce the divergence_scale (I'd try values around 1e-2 or 1e-3) and to increase the action repeat. The action repeat will result in a bigger difference between consecutive frames and thus more signal for the model to learn from, that cannot easily modeled as noise as you explained.
Divergence scale and the (constant and scalar) decoder variance are the same. You can see this by writing the ELBO for a Gaussian decoder in the standard form E_q(z)[lnp(x|z)]-KL[q(z) || p(z)]. The log-likelihood terms is lnp(x|z) = -0.5(x-f(z))/std^2-lnZ. Multiplying the ELBO by std^2 removes it from the log-prob term and puts it in front of the KL term as in beta-VAE. The objectives have different values because of the Gaussian normalizer Z but they have the share the same gradient since the normalizer is a constant.

So here's some result when training to play Breakout-v0:

Episodes: 170
Average test performance: 4
Action repeat (probability): 0.25 <- Atari default

Sadly, I accidentally deleted the log directory, thus I cannot share images. Yet, what I noticed was that the ball was not visible in graph/summaries/closedloop/posterior/image/animation/gif. I'll try running some experiments with @danijar suggested divergence rate.
Note: I am mapping [-1, 1] interval to [0, action_space.n] and doing math.floor to get a discrete action. E.g. if the network predicts -0.1, it gets mapped ~2.95, which gets transformed into action 2.
Things yet to try:

Sampling actions from a discrete Gaussian in planning to avoid the mapping above.
Trying out different divergence rates, as suggested.

EDIT: I did another run with this configuration:
params.batch_shape = [24, 50]
params.train_steps = 40000
params.test_steps = 1000
params.collect_every = 20000

default

Max_length: 1200, maximum_episode_duration: 600.

@astronautas Here you have the graphs from training simple VAE on MNIST dataset (I've implemented it in TensorFlow Probability using Danijar coding patterns from his blog to better understand PlaNet summaries and codebase ~and take over his researcher identity~. I start to feel like some creep/crazy fanboy 😮 @danijar do you or PlaNet have fanpage? haha 😄):

So, let's get serious, my intuition is that KL-divergence from a global prior (which is a normal distribution) should get higher as the posterior is trained to encode useful information. At the same time, it makes an entropy of the posterior lower (it's more and more certain about the latent state (code) as it learns). On the other hand, KL-divergence between the posterior and a prior from a transition model should get lower as the transition model learns to predict future latent states (codes) more and more accurately. loss (negative ELBO) needs to get lower, but all the log prob. needs to get higher.
@danijar please confirm that it holds for PlaNet. Also, I still don't understand if it's correct that summaries has so small ranges in y-axis (see my comment above).
@danijar also thank you for your explanation about beta-VAE, I didn't catch that.
Also, @astronautas please post openloop and closedloop/prior predictions as those are interesting too (or even more interesting). closedloop/posterior in 0-step prediction (encode and decode the frame), closedloop/prior is 1-step future prediction (+ decode) and openloop is n-step future prediction (+ decode). I start to gather interesting results from my hyper-param tuning for Boxing. I'll post the results have tomorrow or next week after Easter holidays.

Here's some results from my latest run (divergence scale 1e-4, learning rate 0.5e-3)
Openloop:

Closedloop posterior:

Closedloop prior:

Entropy:

Correct if I am wrong guys but the posterior entropy should be decreasing, and log_probs should all be increasing, am I right?
Thanks!

The entropy and losses don't necessarily have to decrease during training, because the agent is collecting new data that could be surprising. I would mostly look at the openloop images summaries and the predicted reward trajectories (another image summary).
@piojanu I don't have a fan page, unfortunately :D I'm not sure if your intuition is correct regarding the KL. the KL to the global prior may be roughly constant throughout training.

@danijar Good tip, here are the predicted reward screenshots:

It makes sense why the performance is so poor - the planner is not capable of successfully predicting reward spikes. Concerning the openloop, how should one interpret them? Does it follow the same structure as it was in the paper:

Yes, the open loop images have 5 context frames and all other frames are predicted open loop. For the paper, we additionally only showed every 5th frame to have it fit on a page. The TensorBoard summaries don't skip any frames.
It seems like the reward predictions are not very accurate (yet). How long did this train for? I would expect it to take at least a couple of million steps. One million steps means 200 episodes since it collects an episode every 5000 steps.

Cool, thanks for clarifying this!
I trained it for about 6 hours and 800k steps (370 episodes).
The problem with these Atari games is that more than one episode is collected in one collect session, as the "done" signal is issued when the agent loses all lives. I was thinking maybe of writing a wrapper to reset the game when the agent runs out of lives to solve this or collect data till first done is fired.

@danijar could you share your insight about KL-divergence to global prior then?
@astronautas I think episodes are reset after "done", it's here:
https://github.com/google-research/planet/blob/9cd9abed5b9a8831388f4d9da16e5604cfbd7c20/planet/control/simulate.py#L225-L230
@danijar could you confirm that?

@danijar @piojanu I have some new Breakout results after fiddling with the divergence_scale.
It seems it's going be tough yet interesting. With high divergence scale, ball 0 step reconstructions fail. With the reduced rate to 1e-4, it mostly succeeds at 0 step reconstruction, though hallucinates incorrect future ball positions or incorrectly infers ball movement direction.
Sometimes it does succeed. Last row of the image below - in the 6th frame, it confidently chooses a corrective action. Still, it is kind of "shy" about predicting the ball movement.

The fact that the ball is only a few pixels large might be a problem, with a loss that treats all pixels equally there is little motivation for the model to learn to predict it -- it's much easier to get the loss down by getting really good at predicting the background. Lowering the divergence scale should help a bit as you said. You can also try to reduce the color depth further, e.g. to image_bits: 3. At the same time, it might be easier to start with an environment that doesn't have important but tiny objects.

Thanks @danijar for the tips. I am still eager to maximize PlaNet's accuracy on Breakout - lots of games contain small yet important details which should not be ignored.
I am intrigued by the World Model's @piojanu shared. This probably suggests that it's worth fiddling with the loss functions as well. Thanks for keeping this discussion alive. Hopefully, this will provide some insights to you as well about what PlaNet still lacks.

@astronautas The global_divergence_scale is for the fixed global prior and the divergence_scale is for the temporal prior predicted by the model. The prior and posterior are the same as described in the paper, prior means p(s_t | s_t-1, a_t-1) and posterior means q(s_t | s_t-1, a_t-1, o_t). The state s_t is a dict that contains both the deterministic GRU state and the stochastic state.
I've found that I can train on the MuJoCo tasks without overshooting, global prior, and reward scale, by setting future_rnn: True and switching the activation functions from tf.nn.relu to tf.nn.elu. We'll post an update to the paper at some point but if you like you could already try and see if this works better on Atari as well. Another thing to try to help with small objects is to set free_bits to a larger value.

@danijar Thanks! If we disable overshooting though, does the loss still include training of priors against posteriors (i.e. KL (p(s_t | s_t-1, a_t-1); q(s_t | s_t-1, a_t-1, o_t)) ? It looks like this is disabled as we disable latent overshooting loss, though correct me if I am wrong.

There will still be a KL term from each posterior to the temporal prior predicted from one step before. This is the term from the standard ELBO that is the only term left when setting overshooting: 0 and its scale is controlled by divergence_scale.

Thanks for clarifying this @danijar. I got confused, as in the paper it seemed to be included in the 3rd equation, yet it wasn't there anymore in the final 7th equation. But maybe it's just me ¯_(ツ)_/¯.

Exactly, with D=1 latent overshooting reduces to the standard ELBO. It's just a bit confusing that the overshooting parameter in the code is not D and D-1, such that overshooting: 0 means standard ELBO.

What is future_rnn: True and free_bits?

@danijar I haven't found free_bits parameter in the source code as well.

Sorry, it's called free_nats and means the model is allowed to use this amount of nats without KL penalty, a trick that's often used for static VAEs. It helps the model focus on smaller details which don't contribute much to improving the reconstruction loss. The future_rnn flag fixes a somewhat somewhat subtle bug in the RSSM code, where RNN and stochastic state were both used but didn't interact with each other at future steps.

@danijar Thanks for the info, I'll try these!
I am also curious, what's the use of the observation model? I've read it's only used for training as a "signal", which intuitively sounds valid, as we want latent states to contain visual information. Could we though train only the reward model, so that the latent states would be more "focused" on encoding the reward relevant visual information?
TLDR; What was the motivation behind including the observation model for training although it's not used for planning?

I've experimented with learning only reward models a while ago. This works in surprisingly many cases, however, it doesn't work well when there are sparse rewards, as in cup catch. Moreover, it's a bit less data efficient, but still much more data efficient than most model-free approaches. I think the effect of the observation model is comparable to the effect of auxiliary losses in UNREAL, etc. But I also want to move towards agents that learn without reward, which makes predicting sensory inputs more important.

@danijar thanks for all your hard work open-sourcing this. I'm working on a PyTorch port and it was fundamental in working out the architecture (even though the paper was very clear about most other aspects). Just to confirm, your reported results were without fixing the future_rnn bug? I've managed to replicate your results (i.e., overshooting is important) having fixed that in my code on the walker-walk environment, so any comments of the importance of that vs. replacing all ReLUs with ELUs?
FYI don't think I spotted any other oddities in your code. Looking to do some research on top of PlaNet, hope to open source my code once done. Unfortunately I'm looking at almost 2 weeks per run atm and a not-insignificant amount of memory usage, so will have to look into possible speedups, but the algorithm in general is quite sequential so not sure much can be done.

@Kaixhin That's great! The results where without the fixed bug and I'll update them in the paper. There are a few other small updates to the paper that I want to make (add some citations, etc), so it might take me a couple of days. I've confirmed that elu is not the deciding factor.
I'd be great to see some plots comparing with and without overshooting using your implementation, if you have them. Maybe you could open a new thread for everything related to your PyTorch rewrite?

Hi @astronautas, a few updates from my side:

It turns out when I added the future_rnn parameter in the open source repo, I passed it to SSM instead of RSSM in configs.py. This is fixed now.
The updated agent uses tfd.Normal distributions instead of the custom MSEDistribution I used before in networks/basic.py and networks/conv_ha.py.
I've also confirmed that ELU is not important so I've switched back to ReLU. It only made a different for a follow-up project I've worked on recently.

I think this could be worth another try for Breakout. I'll also sync all my changes to Github at some point for the camera-ready version of the ICML paper, but it could take a few weeks until I get to it.

@danijar thanks for your update! I need to catch up with your posts. However, here are my results from random search of overshooting (from 0 to 50), divergance_scale (from 1E-5 to 1E-2) and global_divergence_scale (from 1E-5 to 1E-2) parameters. Every data point is from Freeway training for 1M steps. Those three scores are last three logged test scores in TensorBoard:
https://docs.google.com/spreadsheets/d/1VCQRyFVZzzbSE_LPpNIvu_QnzFSx9tV8g99mfVaK7gw/edit?usp=sharing
Well... for me it's some random noise :/ It seems like really low overshooting makes things worse, but I have other experiment where with no overshooting and global divergence set to 0.01 it does much much better (score ~16) after ~2M steps of training so it seems like random chance.
My conclusion is it doesn't matter that much if those scales are 1E-3 or 1E-5. What I can see clearly is that lowering those scales from default in deed helps. Also overshooting doesn't seem to have obvious advantage as well. @astronautas and @danijar could you look at this and tell what do you thing?

@danijar I'll give it a shot with the new configuration on VizDoom, as I've wanted to do this from the very beginning. Thanks for the fixes!
@piojanu Thanks for sharing the results! Looks like big overshooting coupled with small global prior is needed, coupled with somewhat low, but not too low divergence scale. Enforcing consistency yet weak one for predictions of multiple steps seems to have helped, as well as reducing the dependency on forcing models to be Gaussian-like.

I've also run training with future_rnn: true, divergence_scale: 0.0001, global_divergence_scale: 0.0001 parameters for Boxing and Freeway. I intent to leave it to train for longer time. We will see what will turn out from that ;)
@astronautas what do you mean by "as well as reducing the dependency on forcing models to be Gaussian-like"?

I don't think lowering the divergence scale (normal or global) makes anything more or less Gaussian. The posteriors are conditionally Gaussian already. Note that the trajectory distribution is highly non-Gaussian because of the non-linear transformation at each time step. So starting from a Gaussian, the next time step is already a mixture of infinitely many Gaussians. One reason that lowering the divergence scale can help is that it allows the model to absorb more information from its observations by loosening the information bottleneck.

@danijar Yup you're right, it makes sense :).
One question - in Atari games, it crucial that the agent does not learn it's good to die to get a reward after environment is resetted. Are the episodes concatenated during training, or is the training done for each episode separately? Could be there training chunks containing <states,action,reward> tuples from different episodes?

All episodes are trained separately in the code, so as long as the episode terminates when the agent dies, this is not a problem.

I'm back with good news :)
TL;DR
Atari Freeway and Crazy Climber seems to start working. I get sharp open loop reconstructions, but still with some random "glimmer" of the chicken in case of Freeway and the building in case of Crazy Climber. Boxing also starts to work (posterior doesn't collapse any more)!
Details
Hyper-params (if not stated otherwise):

global_divergence_scale: 0.0001,
divergence_scale: 0.001,
future_rnn: true - this helped a lot!,

Boxing
Experiments:
I've run those for ~4.3M steps which is around ~870 episodes of Boxing.

action_repeat: 8 - this didn't help with collapsing posterior, only made agent worse. Final score is below 0 which is worse than random agent which scores around 0. Before this higher action repeat experiment it is around 0 too, so it acts like random agent).
free_nats: 4 and action repeat from above - higher free_nats helped a lot with collapsing posterior! Final score like above.

Openloop and rewards predictions:

It still doesn't look great, but at least boxers aren't turning into blurry blob!

Rewards prediction doesn't seem to work. Any ideas how to improve it @danijar and @astronautas?
Next actions: I've run it again with standard action_repeat: 4 and free_nats: 4 for longer.
Freeway
Experiments:
Hyper-params listed at the top. Here I've verified that future_rnn: true helps a lot. I've run it for ~4.2M steps or ~850 Freeway episodes.
Openloop and rewards predictions:

In gif animation it can be seen even better that cars are precisely modelled and sharply reconstructed. However, chicken is reconstructed somehow randomly up and down. This needs to be improved.

Well, here rewards are really sparse, so there is not much signal to learn from to predict rewards.
Next actions: I've run it with free_nats: 3 with hope that chicken will be modelled better. Also there is problem with rewards prediction to solve, any ideas guys?
Crazy Climber
Experiments:
Again, hyper-params from the top. I've run it for ~3M steps which is ~620 Crazy Climber episodes.
Openloop and rewards predictions:

Reconstructions are sharp, but building structure isn't modelled well (e.g. this black hole in the middle usually is much higher/lower then reconstructed). It can be seen better in gif.

Again, it didn't train to predict rewards right.
Also agent have problems with moving up (some combination of actions is needed to raise a hand and then grab). Random agent does better. It seems like problem with exploration, what do you think?
Next actions: I've run it with disabled global prior to check if it changes anything. @danijar found out that it isn't important with fixed RSSM, see #28.
Conclusions
I suspect that rewards aren't predicted too good and that's why it does so purely (in case od Crazy Climber it does worse them random, in this paper they have nice benchmarks and comparisons e.g. table 1. Their algo seems great too).
Other possibility, which I'll check, is poor exploration. We also need to remember that CEM isn't designed for discrete actio-space and maybe hacky way of argmaxing sampled action scores isn't the best idea (maybe we should use something softer, e.g. sampling from proportional/softmax dist. of those action scores?).
Im interested in what do you thing @astronautas and @danijar? What next steps can you see?

Quick updates:

I've implemented this CEM modification to discrete action-space (one-hot actions + e-greedy exploratory policy in mpc_agent.py, not additive noise as in continuous action-space case).
free_nats=3 helped with Freeway! Although there are still some errors in prediction, chicken moves are smooth and stable (don't break env dynamics with e.g. teleportation). I'm truly amazed right now and I start to believe it might work 😄

I'll post more probably tomorrow.

@piojanu Did you try running same experiment with and without the discrete planner? Does it really improve the results?

Sorry, I've edited my post. I'm now testing it ;)

I have new results and I hope for your insights @astronautas and @danijar.
Freeway - discrete planner experiment
Well, the agent have hard time getting to the other side of the road (where it gets a reward of 1). I've issued multiple runs with epsilon in e-greedy policy 0.3 and 0.7 and in three from four runs it didn't collected any reward (in data collection phase and in test phase):

It's even worse than without this discrete planner, I'll try to find out why. It it comes to modeling, then results aren't clear too. In 2 runs it models the actions really well in other 2 runs the chicken is jumping unpredictably. So I'll now try with disabling global prior.
Do you have some thoughts?
Boxing - higher free nats and lower divergence scale
Still fighting, currently I tested divergence scales 1E-4 and 1E-5 with free nats 4 in multiple runs. Reconstructions are still looking like some randomly jumping blobs, EVEN for closedloop prior reconstructions (one-step prediction) in case of divergence scale 1E-5. So lowering the divergence scale further doesn't seem to make much sense. I'll try with higher free nats (5).
It's worth noting that closedloop posteriors looks fine, so it's still some problem with the transition model I bet. @danijar do you have some ideas?
Here are closedloop predictions for 1E-4 and then 1E-5:

And here are openloop predictions for 1E-4 and then 1E-5:

Crazy Climber - higher free nats
Nothing new, reconstructions are still not precise. I wan't post anything more as it looks much the same as in Boxing case. There are little differences between the frames and the transition model doesn't seem to get that. @astronautas any ideas how we can work that out?

I made Boxing work really well! :D Score is high and open loop predictions are "sharp".

TL;DR ~7M steps; disabled global prior; enabled future rnn; used "discrete" CEM described above; divergence scale 3E-02; free nats 12;
Hyper-params
The most important parameters seems to be free nats and divergence scale (and turning on future rnn). I did some tuning with many parameters and no other seem to have that big impact (I didn't try to change reward scales though). Random search results are: the lower divergence scale the nosier predictions are. The higher free nats the better actions movement predictions are (more stable one could call). Although, there were quite good reconstructions for low free nats too (when divergence scale was a bit higher or equal 1E-02). I'll try it next.
Questions
@danijar what might cause so dramatic leap in image loss/log_prob statistics around step 2M? See the diagrams:

Reward open loop log_prob also started to decrease and then increase in the same range:

Other reward statistics doesn't have this clear pattern:

Also in this interval it was scoring less and then it started to score more and more:

So can it be that reward model started to learn at the expense of observation model and then both found balance again? Because I don't see much difference in collected dataset around 2M step (although, I only looked at those images of dataset in TensorBoard and it's not much).
Further work
Now I try with lower free nats in Boxing. Also, I'm working on Freeway and MsPacman. Stay tuned!

That's great news! It could be a numerical instability or just that the agent has discovered a new part of the environment that is more surprising. I would suggest to look at the video predictions right before and after this happens (to the degree TensorBoard allows). The reward scales are not very important as I've found. Have you tries disabling overshooting yet?
Besides this, I would recommend a divergence scale that is as high as possible while still allowing for good performance. For example, when you set the divergence scale to zero it could learn to become a deterministic autoencoder which and reconstruct well but is less likely to generalize to state in latent space that the decoder hasn't seen during training.

@danijar thanks for your insights!

I didn't try disabling overshooting yet (well, I did once at the very beginning of playing with PlaNet and it seemed to help, but this was before many other tweaks, so this experiment needs to be redone). I can't understand how overshooting might make things worse? It seems like great idea to calculate loss over future predictions (this is what we really care about at the end, to predict future well, not only one step ahead).
I've looked into image reconstructions before, during and after this "pit". Before it reconstructions were quite good, then in the pit there is noting but green deck with barriers (?!) and then after the pit reconstructions are amazing. Strange 😮 Second run (I run it more times to confirm results) doesn't have this pit, but it just got to the point when score should start to rise so I don't know just yet if this will work.
Yea, I even saw it in my experiments that low divergence scale yields really noisy future predictions. Now I'm running Boxing with 3E-02 and Freeway with 8E-03 so the best parameters from Random Search, but I'll try with higher scale for Freeway too (same as in Boxing).


Power Run hooking Alt+Space is an accessibility issue

Microsoft PowerToys version
0.74.0
Installation method
Microsoft Store
Running as admin
None
Area(s) with issue?
PowerToys Run
Steps to reproduce

Accidentally put MS Terminal into "Focus mode". I can't tell you how that happened, just that it did. The only mention of this mode I find online tells me it's in the startup preferences for Terminal, or by invoking the command palette and typing "toggle focus mode". My cat must have done that.
Observe that the Terminal window has no frame or title bar, so there's no way to move the window with a mouse, or close the application, or to access the Settings dialog that's normally accessible through the tabs (that are hidden) or the system menu (that's hidden). So now my cat's got a Terminal window covering up his cat videos and can't move it out of the way.
Press Alt+Space, which normally would open the system menu for the window with focus, and offer Move among other options. That form of Move would allow you to move the window with arrow keys, or put your mouse on the title bar if it were there, which it isn't.

I researched this in your bug base, and I know the argument for not fixing it before has been that anyone can remap Alt+Space if they want to. While that's true, it's not a reason to take the affirmative action of breaking an accessibility feature with your default shortcut. PowerToys has way too many shortcut keys, and I sympathize that there are no easy answers for what the right key should be, it's clear to me that Alt+Space is always going to be the wrong key.
There are two reasons this is a bad idea:

That Alt+Space has worked as the default in every version of Windows since at least Windows 2.0. You're breaking something that's been in place forever, for the sake of a mere convenience.
Accessibility: users who use the keyboard to navigate are frozen out of using the only means for moving windows. There are plenty of people who don't have the visual acuity or motor coordination to pick things out with a mouse (or maybe they're using MS Terminal, whose Focus Mode is a little bit too good...) and those folks are just stuck.

The fact that I can't tell what the menu belongs to shouldn't be glossed over, here, BTW. PowerToys has enough things in it now that users can't be expected to know what every one looks like or how to invoke it--ESPECIALLY when it's a toy that intercepts an ancient and venerable system shortcut. Some non-interactive "PowerToys Run" text in the background bitmap would have saved me several minutes of trying to identify the window. There's room to fit the name in all the localized languages, too.
If you won't do it for me, do it for my cat.
✔ Expected Behavior
Expected: Alt+space invokes a system menu for the currently-focused window. The menu includes Move as an option that enables moving a window using arrow keys.
❌ Actual Behavior
Actual: A window pops up that doesn't even identify itself as PowerToys Run, which does not offer the opportunity to move the window (or if it does, does so in a non-standard and non-obvious way.) I eventually narrowed it down to PowerToys Run by turning off PowerToys until I narrowed down the culprit.
Other Software
No response

@EXPErdelfavero you can change the activation shortcut for any PowerToy which is interrupting your workflow. Does this solve this issue? /needinfo

related to (maybe a duplicate) #3838

No, me changing the shortcut isn't a solution.
The defaults should not conflict with longstanding, well-known system shortcuts. That's implicit in the amount of backward compatibility Windows has always taken extra measures to maintain.
While me changing the PowerToy shortcut would solve the problem for me on whichever machine I make the change on, that doesn't change my other machines, or other users' setups. As someone who recognized that the errant window called up by alt+space was a PowerToy almost immediately, I still wasted ten minutes finding the right one. Someone on #3838 said it took them an hour to figure out. The typical user isn't going to recognize the cause of the problem as a PowerToy quickly, or see the solution you propose very quickly. Especially if, as I allude to in the title, the user is relying on default behaviors because they have accessibility needs that aren't going to be met in the meantime, and their machine is only partly usable as they try to investigate. What's the experience of a screen reader user when they get the PowerToy, versus when they use default Windows behavior? Is there enough information there for the screen reader user to find and make the appropriate change? I don't think so, especially if behaviors they rely on to get information are disabled. It's not like there's any other way to move windows without the mouse once Run has co-opted the default shortcut.
More to the point, overriding a default Windows shortcut with a utility app's default, then requiring the user to find and change the utility's shortcut is doing things backwards. Utilities, especially those from Microsoft (or at least sanctioned by Microsoft and carrying Microsoft branding in the name) shouldn't conflict with Windows shortcuts out of the box. If someone who doesn't use Alt+Space wants to assign it to Run, that's up to them. But it shouldn't be the default that's activated when I install PowerToys. If all I wanted was FancyZones to arrange things on my wide screen monitor; I shouldn't have to find and fix a conflict with a different PowerToy I didn't even know about or want to use. That makes no sense to me as a user or as a Windows developer, and I'm frankly surprised that the discussion even needs to be had. Not Breaking Windows seems like Acceptance Criterion One for PowerToys.
It looks like #3838 is an earlier report of the same problem, and the discussion there has already made many of the same points I'm trying to make here. It's pretty disappointing to see that although @crutkas said it would be fixed, either nothing was changed then or the fix was reverted in the meantime. Perhaps the consideration of what this does for accessibility changes the determination of how important it is to fix this and keep it fixed?

We will be clearly show conflicts like this with the upcoming dashboard system over the next few iterations but there currently is no plans to change the default.


Pre-installed Python Option

Currently, dask-yarn only works with envs shipped to or cached in HDFS.  Occasionally, all the requisite packages and dependencies are in place on the cluster.  If I am fortunate to work on a pre-configured cluster for dask-yarn I should be allowed to refer to the pre-configured Python binary and use any HDFS based environments.
My proposal would is to add another option: cluster_python_prefix or pythonprefix or another name, to the DaskYARNCluster constructor and pass that string to the knit command:
pref = self.pythonprefix
if self.env:
    bn = os.path.basename(self.env)
    pathsep = '/'  # assume execution is always on posix
    pref = pathsep.join([bn, os.path.splitext(bn)[0]])  # like myenv.zip/myenv
command = ('{pref}/bin/python {pref}/bin/dask-worker....')

Certainly I think this is useful.
We need to tell the scala code NOT to upload anything (there is already a path for this) and to pass the python executable path through rather than constructing one.

Skein (https://jcrist.github.io/skein/) allows for a more flexible way of describing applications to run on YARN. Dask-Yarn has been rewritten to use skein (http://dask-yarn.readthedocs.io/), and can accept a generic skein specification to start dask. The above can be achieved by:
# spec.yaml
name: dask
queue: default
max_attempts: 1

services:
    dask.scheduler:
        resources:
            memory: 512
            vcores: 1
        commands:
            - /path/to/dask-yarn-scheduler

    dask.worker:
        instances: 0
        max_restarts: 2
        resources:
            memory: 512
            vcores: 1
        depends:
            - dask.scheduler
        commands:
            - /path/to/dask-yarn-worker 1 --memory_limit 512MB

from dask_yarn import YarnCluster

cluster = YarnCluster.from_specification('spec.yaml')
cluster.scale(10)

Closing, development has moved to https://github.com/dask/dask-yarn.


What means 'In some corner cases optional parameters may not work as you expect...'

Can you please clarify this with some example?

I am working on some improvements that will address this issues, and then I will add detailed description about corner cases

I've been working with router for while and I added regex support, for parameters like this:
$Router->addRoute("get", '/user/{id}/{name}', 'UserController@show',[
   "id" => "([0-9]++)",
   "name" => "alnum"
]);

Can you somewhere add your improvements? To better collaborate on this router.


Use the AddOriginatingEntity/RemoveOriginatingEntity handlers to handle PRs in the webhook provider

Summary

Remove DeleteEntityByName - With multiple providers, we might have entities with the same name.  Let's just use DeleteEntity which deletes by ID, forcing the API user to do a search before.
Add tests for DeleteOriginatingEntity - We didn't have any tests for this codepath before. Also switches to making an explicit search through getEntityInner for the PR and then deleting by ID
Use the originating webhook handlers for handling PRs in the github webhook handler - Instead of bespoke github-specific code, let's use the new AddOriginatingEntity and RemoveOriginatingEntity handlers.

Fixes: #4619
Change Type

[ ] Bug fix (resolves an issue without affecting existing features)
[ ] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[x] Refactoring or test improvements (no bug fixes or new functionality)

Testing
manually + unit tests
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[x] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[x] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

coverage: 53.638% (+0.3%) from 53.356%
when pulling c6f6c1704aeb315971af6d70624b4f7fb5f73984 on jhrozek:webhook_pr_new_handler
into 65b5ddf8471658aca4c0c26f9e490389ba7cf193 on stacklok:main.

I did run the following 2 tests locally before I merge:

pr_trusty_check_go
pr_vulnerability_check_go

(and they passed)


Create gsf-action-legacy module

Move html sub-module into legacy and out of gsf-action.

Need to add gsf-action-legacy into gsf-foundation-legacy dependency
gsf-action-legacy module should appear in parent pom.xml before gsf-foundation-legacy


Package - Remove rocDecode

rocDecode disabled

@nunnikri this is for rocm 6.2.2

Closing this for 6.2.X, this will be available for 6.3


When I ran bert_classifier_main.py, I got a value error

ValueError: Method not found in ['tensorflow', 'tensorflow.nn', 'texar.custom', 'texar.core.layers']: gelu

gelu is defined in texar.core.layers.gelu. can you access it? you may need to install the latest version of Texar


layer.conf: Add uart2 to uart5 overlays

We add the uart2 to uart5 overlays to the rootfs alongside the
existing uart0 and uart1 overlays.
Changelog-entry: Add the uart2 to uart5 overlays to rootfs
Signed-off-by: Florin Sarbu<EMAIL_ADDRESS>

@floion maybe the i2c3 to i2c6 overlays could be added as well for orthogonality?
reference: https://github.com/raspberrypi/linux/commit/bd7aebbcdce2d5c5cdba3601764799382271606a#diff-916605b47e18e502c20876d1513ae33f


fix: Immediately redraw on Vim

The denops.redraw() is required to be called to change Vim's UI immediately. It does nothing on Neovim.

Ty!


runtime error CreateCapabilities must be invoked after the StartEdge or StartChromeMethods, and before OpenBrowser Methods

I use the DLL and get the abovementioned error:

The error occurs whenever I start a VBA routine second time. All objects are properly ended (CloseBrowser / Shutdown) and object references are set to nothing (what should happen when a program ends anyway). Also the chromedriver.exe process is not active anymore.
It seems the DLL lacks some internal clean up ?

Check for a difference between DLL and addin - error occurs for addin version as well what I would not have expected.

I can't replicate on my end (Windows 11 64-bit, Excel 365 64-bit). Can you provide more info about your system and some reproducible code?
On your system, I thought you said you were using Office 2003 - is that true?

Yes, using Office2003.
I did some additional testing - I use a own initialisation routine to allow to set some stuff lieke stealthening (i e something I could achieve with application specific ini file). I think I execute the commands in the appropriate order.

I reconsidered - I think it was a strange error in my code. I used a static var to track the status and I thought this would be deleted/reset after the run which is obvoiously not the case. but with the status-checker s(see other thread) I could make it work the way I want it to

Own error


Remove reliance on targetObject from tests

DEPRECATION: <dummy@component:sortable-item-mixin::ember446> Usage of targetObject is deprecated. Please use target instead. [deprecation id: ember-runtime.using-targetObject]

Fixed by https://github.com/jgwhite/ember-sortable/pull/172


Platform: add UserEnv module to WinSDK

This allows access to the GetUserProfileDirectoryW API which is required for the swift-foundation implementation on Windows.
(cherry picked from commit 5919346b32c286b132a3224c6e8f49e053170131)
Explanation: Add a new submodule for API access on Windows
Scope: This impacts only the WinSDK module, which is Windows specific.
Issue:
Original PR: #73008
Risk: If the WinSDK module is not tested, it could result in the module breaking which would make Swift nearly unusable on Windows.
Testing: CI testing which will build modules with WinSDK
Reviewer: @jmschonfeld

@swift-ci please test


Can't run tests with bundle exec rspec

New Issue Checklist

[x ] Updated fastlane to the latest version
[x ] I have read the Contribution Guidelines

Issue Description
Having trouble getting the tests to run. Using bundle exec rspec in root directory.
Main error I am seeing:
Could not load native IDN implementation. LoadError: cannot load such file -- rack/mount
FULL OUTPUT
$> [Coveralls] Set up the SimpleCov formatter. [Coveralls] Using SimpleCov's default settings. Changing stdout to /tmp/fastlane_tests, set DEBUGenvironment variable to print to stdout (e.g. when usingpry) /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.1/spec/addressable/idna_spec.rb:145: warning: already initialized constant LONG /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.1/spec/addressable/idna_spec.rb:29: warning: previous definition of LONG was here Could not load native IDN implementation. LoadError: cannot load such file -- rack/mount   /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.1/spec/addressable/rack_mount_compat_spec.rb:21:in require'
/Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.1/spec/addressable/rack_mount_compat_spec.rb:21:in <top (required)>'   /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in load'
/Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in block in load_spec_files'   /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in each'
/Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in load_spec_files'   /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:100:in setup'
/Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:86:in run'   /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in run'
/Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in invoke'   /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/exe/rspec:4:in <top (required)>'
/Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/bin/rspec:22:in load'   /Users/joelklabo/src/fastlane/vendor/bundle/ruby/2.4.0/bin/rspec:22:in <top (required)>'
Environment
✅ fastlane environment ✅
Stack

Key
Value

OS
10.12.5

Ruby
2.4.1

Bundler?
true

Git
git version 2.11.0 (Apple Git-81)

Installation Source
~/src/fastlane/vendor/bundle/ruby/2.4.0/bin/fastlane

Host
Mac OS X 10.12.5 (16F73)

Ruby Lib Dir
~/.rbenv/versions/2.4.1/lib

OpenSSL Version
OpenSSL 1.0.2k  26 Jan 2017

Is contained
false

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcode.app/Contents/Developer/

Xcode Version
8.3

System Locale

Variable
Value

LANG
en_US.UTF-8
✅

LC_ALL

LANGUAGE

fastlane files:
`./fastlane/Fastfile`
skip_docs

desc "Runs the unit tests and other verifications for the fastlane repo"
lane :test do |options|
  validate_repo
end

desc "Increment the version number of this gem"
lane :bump do |options|
  ensure_git_branch(branch: "master")
  ensure_git_status_clean

  github_api_token = ENV["FL_GITHUB_RELEASE_API_TOKEN"]
  UI.user_error!("Please provide a GitHub API token using `FL_GITHUB_RELEASE_API_TOKEN`") if github_api_token.to_s.length == 0

  path = "./fastlane/lib/fastlane/version.rb"

  # Verify everything is in a consistent state
  latest_version = current_version
  local_version = version_get_podspec(path: path)
  UI.user_error!("Version on RubyGems doesn't match local repo: #{latest_version} != #{local_version}") if latest_version != local_version

  changelog_text = show_changelog

  bump_type ||= 'minor' if prompt(text: "New feature, method or API?", boolean: true)
  bump_type ||= 'patch'

  slug = "fastlane/fastlane"
  new_version = version_bump_podspec(path: path, bump_type: bump_type)
  sh "git checkout -b 'version-bump-#{new_version}'"
  commit_message = "Version bump to #{new_version}"
  git_commit(path: path,
             message: commit_message)
  push_to_git_remote

  pr_body = ["Auto-generated by fastlane 🤖"]
  pr_body << "**Changes since release '#{latest_version}':**"
  pr_body << changelog_text
  pr_url = create_pull_request(
    api_token: github_api_token,
    repo: slug,
    title: commit_message,
    body: pr_body.join("\n\n")
  )

  if ENV['SLACK_URL']
    slack(
      channel: "action",
      default_payloads: [],
      message: "Version bump: #{pr_url} ✨",
      payload: {}
    )
  end

  # Revert to master branch
  sh "git checkout master"
end

desc "Does everything that's needed for a release"
desc "This includes running tests and verifying the GitHub release"
lane :release do
  slack_train_start(distance: 2,
                       train: "🚀",
           reverse_direction: true,
                        rail: "✨")

  update_fastlane
  slack_train

  # Git verification
  #
  ensure_git_status_clean
  ensure_git_branch(branch: 'master')
  git_pull

  validate_repo

  # Verifying RubyGems version
  #
  require "../fastlane/lib/fastlane/version.rb"
  version = Fastlane::VERSION
  old_version = current_version
  puts "Deploying #{version}"
  if Gem::Version.new(version) <= Gem::Version.new(old_version)
    UI.user_error!("Version number #{version} was already deployed")
  end

  # Then push to git remote
  #
  push_to_git_remote

  # Preparing GitHub Release
  #
  github_release = get_github_release(url: "fastlane/fastlane", version: version)
  if (github_release || {}).fetch('body', '').length == 0
    show_changelog(old_version: old_version)

    title = prompt(text: 'Title: ')
    description = prompt(text: "Please enter a changelog (make sure to rephrase changes to make it easy to understand, and remove non-important changes that don't affect the end-user): ",
                         multi_line_end_keyword: "END")

    github_release = set_github_release(
      repository_name: "fastlane/fastlane",
      name: [version, title].join(" "),
      tag_name: version,
      description: description,
      is_draft: false
    )

    # Actual release of the gem
    #
    sh "gem push ../pkg/fastlane-#{version}.gem"

    release_url = github_release['html_url']

    message = [title, description, release_url].join("\n\n")
    add_fastlane_git_tag(tag: "fastlane/#{version}", message: message)
  end

  # After publishing
  #
  if ENV['SLACK_URL']
    release_notes = github_release['body']
    # markup - turn github references into links
    release_notes.gsub!(/\(#([0-9]+)\)/, '(<https://github.com/fastlane/fastlane/issues/\1|#\1>)')
    slack_message = "Successfully released [fastlane #{version}](#{release_url}) :rocket:"
    slack(
      channel: "releases",
      default_payloads: [],
      message: slack_message,
      payload: {
        "New" => release_notes
      }
    )
    slack(channel: "action", default_payloads: [], message: slack_message)
  end

  clubmate
  donate_food

  puts "You can now tweet:".green
  releases_url = "https://github.com/fastlane/fastlane/releases/tag/#{version}"
  puts "[fastlane] #{github_release['name']} #{releases_url}"

  send_mac_app_ci_reminder

  update_docs
  slack_train
end

desc "Makes sure the tests on https://docs.fastlane.tools still work with the latest version"
lane :verify_docs do
  clone_docs do
    Bundler.with_clean_env do
      puts `sed -i -e "s/activate_bin_path/bin_path/g" $(which bundle)` # workaround for bundler https://github.com/bundler/bundler/issues/4602#issuecomment-233619696
      sh "bundle install"
      sh "bundle exec fastlane test skip_building:true" # skip_building since we don't have a proper python environment set up
    end
  end
end

desc "Generates the actions.md that will get used on https://docs.fastlane.tools"
desc "This will NOT also automatically submit a pull request to fastlane/docs"
desc "This SHOULD only be used for testing changes to the docs"
lane :test_generate_docs do
  ENV['USE_ENHANCE_TEST_DATA'] = "true"
  generate_markdown_docs
  ENV.delete("USE_ENHANCE_TEST_DATA")
end

desc "Update the actions.md on https://docs.fastlane.tools"
desc "This will also automatically submit a pull request to fastlane/docs"
lane :update_docs do
  if ENV["ENHANCER_USER"].to_s.length == 0 || ENV["ENHANCER_PASSWORD"].to_s.length == 0
    UI.error("No ENHANCER_USER or ENHANCER_PASSWORD environment variables found, which are required to generate a new Actions.md")
    next
  end

  clone_docs do
    actions_md_path = generate_markdown_docs

    Bundler.with_clean_env do
      sh "bundle update"
    end

    if `git status --porcelain`.length == 0
      UI.success("No changes in the actions.md ✅")
    else
      # Create a new branch
      sh "git checkout -b 'update-actions-md-#{Time.now.to_i}'"

      Dir.chdir("fastlane") do # this is an assumption of fastlane, that we have to .. when shelling out
        # Commit the changes
        git_commit(path: [actions_md_path, "Gemfile.lock"], message: "Update actions.md for latest fastlane release 🚀")
        # Push them to the git remote
        push_to_git_remote

        # Submit the pull request
        pr_url = create_pull_request(
          api_token: ENV["FL_GITHUB_RELEASE_API_TOKEN"],
          repo: "fastlane/docs",
          title: "[Bot] Update actions.md for latest fastlane release 🚀",
          body: "Auto-generated by _fastlane_ bot 🤖"
        )
        UI.success("Successfully submitted a pull request to fastlane/docs: #{pr_url} 🚀")
      end
    end
  end
end

def generate_markdown_docs
  require 'fastlane/documentation/markdown_docs_generator'
  actions_md_path = File.expand_path("docs/actions.md")
  Fastlane::MarkdownDocsGenerator.new.generate!(target_path: actions_md_path)

  return actions_md_path
end

def clone_docs
  require 'tmpdir'
  git_url = ENV['FASTLANE_DOCS_GIT_URL'] || "https://github.com/fastlane/docs"

  Dir.mktmpdir("fl_clone") do |tmp_dir|
    Dir.chdir(tmp_dir) do
      sh "git clone #{git_url} --depth=1"
      Dir.chdir("docs") do
        yield
      end
    end
  end
end

error do |lane, exception|
  if ENV['SLACK_URL']
    slack(channel: "testing", message: exception.to_s, success: false)
  end
  slack_train_crash
end

private_lane :send_mac_app_ci_reminder do
  if ENV['FABRIC_SLACK_URL']
    slack(
      slack_url: ENV['FABRIC_SLACK_URL'],
      channel: 'deployment-tools',
      default_payloads: [],
      message: "Please run the Fastlane Mac App Package CI job in TeamCity\n#{ENV['FABRIC_MAC_APP_CI_JOB_URL']}"
    )
  end
end

desc "Verifies all tests pass and the current state of the repo is valid"
private_lane :validate_repo do
  # Verifying that no debug code is in the code base
  #
  ensure_no_debug_code(text: "binding.pry", extension: ".rb", exclude: "playground.rb", exclude_dirs: ["\.bundle"]) # debugging code
  ensure_no_debug_code(text: "# TODO", extension: ".rb", exclude_dirs: ["\.bundle"]) # TODOs
  ensure_no_debug_code(text: "now: ", extension: ".rb", exclude_dirs: ["\.bundle"]) # rspec focus
  ensure_no_debug_code(text: "<<<<<<<", extension: ".rb", exclude_dirs: ["\.bundle"]) # Merge conflict

  # spaceship and credentials_manager don't have access to fastlane_core
  ensure_no_debug_code(text: " UI\\.", path: "spaceship/lib", extension: ".rb", exclude_dirs: ["\.bundle"])
  ensure_no_debug_code(text: " UI\\.", path: "credentials_manager/lib", extension: ".rb", exclude_dirs: ["\.bundle"])

  rubocop

  # Verifying the --help command
  #
  # tools_to_test = Fastlane::TOOLS
  tools_to_test = [:fastlane] # until we have the binaries for all tools included here
  tools_to_test.each do |tool_name|
    next if no_binary.include?(tool_name)
    Dir.chdir("..") do
      binary_path = File.join("bin", tool_name.to_s)
      content = sh("PAGER=cat #{binary_path} --help")
      ["--version", "https://fastlane.tools", tool_name.to_s].each do |current|
        UI.user_error!("--help missing information: '#{current}'") unless content.include?(current)
      end
    end
  end

  Dir.chdir("..") do
    # Install the bundle and the actual gem
    sh "bundle check || bundle install"
    sh "rake install"

    # Run the tests
    #
    sh "bundle exec rake test_all"

    # Verify shell code style
    sh 'find -E . -regex ".*\.(sh|bash)" -not -name "Pods-*" -name ".bundle" -exec shellcheck {} +'
  end

  test_generate_docs
  ensure_code_samples

  # Verify docs are still working
  verify_docs
end

desc "Get the version number of the last release"
private_lane :current_version do
  puts "Checking the latest version on RubyGems"
  download(url: "https://rubygems.org/api/v1/gems/fastlane.json")["version"]
end

desc "All repos that don't have a binary to test"
private_lane :no_binary do
  ['fastlane_core', 'spaceship', 'credentials_manager']
end

desc "Print out the changelog since the last tagged release and open the GitHub page with the changes"
lane :show_changelog do |options|
  old_version = options[:old_version] || current_version

  changes = sh("git log --pretty='* %s via %aN' #{old_version}...HEAD --no-merges ..", log: $verbose).gsub("\n\n", "\n")
  changes.gsub!("[WIP]", "") # sometimes a [WIP] is merged

  github_diff_url = "https://github.com/fastlane/fastlane/compare/#{old_version}...master"
  sh "open #{github_diff_url}"

  puts "Changes since release #{old_version}:\n\n#{changes.cyan}"
  changes # return the text without the modified colors
end

desc "Add a git tag in the fastlane repo for this release"
private_lane :add_fastlane_git_tag do |options|
  `git tag -am #{options[:message].shellescape} #{options[:tag].shellescape}`
  push_git_tags
end

# Run the code samples, yo
private_lane :ensure_code_samples do
  UI.message "<U+1F557>  Verifying all code samples work with the latest fastlane release"
  test_sample_code(path: File.expand_path("docs/Actions.md"))
  UI.success "<U+2705>  All fastlane code samples work as expected"
end

lane :donate_food do
  if ENV["SHARETHEMEAL_USERHASH"].to_s.length > 0
    sharethemeal(
      amount: "0.5",
      currency: "USD",
      team_id: "fastlane"
    )
  end
end

No Appfile found
fastlane gems

Gem
Version
Update-Status

fastlane
2.30.2
✅ Up-To-Date

Loaded fastlane plugins:

Plugin
Version
Update-Status

fastlane-plugin-ruby
0.1.3
✅ Up-To-Date

fastlane-plugin-clubmate
0.1.0
✅ Up-To-Date

fastlane-plugin-sharethemeal
0.1.9
✅ Up-To-Date

fastlane-plugin-slack_train
0.2.0
✅ Up-To-Date

Loaded gems

Gem
Version

did_you_mean
1.1.0

bundler
1.14.6

io-console
0.4.6

rake
11.3.0

CFPropertyList
2.3.5

public_suffix
2.0.5

addressable
2.5.1

ast
2.3.0

babosa
1.0.2

debug_inspector
0.0.3

binding_of_caller
0.7.2

builder
3.2.3

byebug
9.0.6

claide
1.0.1

colored2
3.1.2

cork
0.3.0

nap
1.1.0

open4
1.3.4

claide-plugins
0.9.2

coderay
1.1.1

colored
1.2

highline
1.7.8

commander-fastlane
4.4.4

json
2.1.0

docile
1.1.5

simplecov-html
0.10.0

simplecov
0.14.1

tins
1.13.3

term-ansicolor
1.6.0

thor
0.19.4

coveralls
0.8.21

safe_yaml
1.0.4

crack
0.4.3

multipart-post
2.0.0

faraday
0.12.1

faraday-http-cache
1.3.1

git
1.3.0

kramdown
1.13.2

sawyer
0.8.1

octokit
4.7.0

unicode-display_width
1.1.3

terminal-table
1.7.3

danger
4.3.5

ox
2.5.0

danger-junit
0.7.3

diff-lcs
1.3

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20170404

dotenv
2.2.1

excon
0.55.0

fakefs
0.8.1

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

faraday_middleware
<IP_ADDRESS>

fastimage
2.1.0

gh_inspector
1.0.3

jwt
1.5.6

little-plugger
1.1.4

multi_json
1.12.1

logging
2.2.2

memoist
0.15.0

os
0.9.6

signet
0.7.3

googleauth
0.5.1

httpclient
2.8.3

hurley
0.2

mime-types
1.25.1

uber
0.0.15

representable
2.3.0

retriable
2.1.0

google-api-client
0.9.28

mini_magick
4.5.1

multi_xml
0.6.0

plist
3.3.0

rubyzip
1.2.1

security
0.1.3

slack-notifier
1.5.1

terminal-notifier
1.8.0

tty-screen
0.5.0

word_wrap
1.0.0

nanaimo
0.2.3

xcodeproj
1.4.4

rouge
2.0.7

xcpretty
0.2.8

xcpretty-travis-formatter
0.0.4

fastlane-plugin-clubmate
0.1.0

fastlane-plugin-ruby
0.1.3

rest-client
1.6.9

stm_api
0.1.7

fastlane-plugin-sharethemeal
0.1.9

fastlane-plugin-slack_train
0.2.0

hashdiff
0.3.4

interception
0.5

method_source
0.8.2

parser
<IP_ADDRESS>

powerpack
0.1.1

slop
3.6.0

pry
0.10.4

pry-byebug
3.4.2

pry-rescue
1.4.5

pry-stack_explorer
<IP_ADDRESS>

rack
1.6.7

rack-protection
1.5.3

rainbow
2.2.2

rb-readline
0.5.4

rspec-support
3.5.0

rspec-core
3.5.4

rspec-expectations
3.5.0

rspec-mocks
3.5.0

rspec
3.5.0

rspec_junit_formatter
0.2.3

ruby-progressbar
1.8.1

rubocop
0.48.1

tilt
2.0.7

sinatra
1.4.8

webmock
2.3.2

xcode-install
2.1.1

yard
<IP_ADDRESS>

generated on: 2017-05-15

@joelklabo I'm having the same issue trying to run the tests. Did you resolve this issue? If so, how?

 Let us know if you are still having issues after doing that and we will be happy to help! 😄  🐙  🚀

I'm having the same problem, and bundle update did not update anything.

Unfortunately I'm having the identical problem to the original poster, running bundle update did not solve the issue.
Output of bundle exec rspec from the fastlane root is below:
$ bundle exec rspec
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's default settings.
Changing stdout to /tmp/fastlane_tests, set `DEBUG` environment variable to print to stdout (e.g. when using `pry`)
/Users/workingDir/fastlane/vendor/bundle/gems/addressable-2.5.2/spec/addressable/idna_spec.rb:151: warning: already initialized constant LONG
/Users/workingDir/fastlane/vendor/bundle/gems/addressable-2.5.2/spec/addressable/idna_spec.rb:29: warning: previous definition of LONG was here
Could not load native IDN implementation.
LoadError: cannot load such file -- rack/mount
  /Users/workingDir/fastlane/vendor/bundle/gems/addressable-2.5.2/spec/addressable/rack_mount_compat_spec.rb:21:in `require'
  /Users/workingDir/fastlane/vendor/bundle/gems/addressable-2.5.2/spec/addressable/rack_mount_compat_spec.rb:21:in `<top (required)>'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `load'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `block in load_spec_files'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `each'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:100:in `setup'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:86:in `run'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
  /Users/workingDir/fastlane/vendor/bundle/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'
  /Users/workingDir/fastlane/vendor/bundle/bin/rspec:23:in `load'
  /Users/workingDir/fastlane/vendor/bundle/bin/rspec:23:in `<top (required)>'

Any thoughts @ohayon ?

Are you trying to run the unit tests of fastlane?

I am getting the same error when running the tests on Ubuntu:
foo@foo-VirtualBox:~/fastlane$ DEBUG=1 bundle exec rake
rspec --pattern ./**/*_spec.rb --format progress
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's default settings.
/home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.2/spec/addressable/idna_spec.rb:151: warning: already initialized constant LONG
/home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.2/spec/addressable/idna_spec.rb:29: warning: previous definition of LONG was here
Could not load native IDN implementation.
[Coveralls] Outside the CI environment, not sending data.
/home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.2/spec/addressable/rack_mount_compat_spec.rb:21:in `require': cannot load such file -- rack/mount (LoadError)
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/addressable-2.5.2/spec/addressable/rack_mount_compat_spec.rb:21:in `<top (required)>'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `load'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `block in load_spec_files'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `each'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:100:in `setup'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:86:in `run'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `load'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `<main>'
rake aborted!
Command failed with status (1): [rspec --pattern ./**/*_spec.rb --format pr...]
/home/foo/fastlane/Rakefile:19:in `block in <top (required)>'
/home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'
/home/foo/.rbenv/versions/2.4.2/bin/bundle:23:in `load'
/home/foo/.rbenv/versions/2.4.2/bin/bundle:23:in `<main>'
Tasks: TOP => default => test_all
(See full trace by running task with --trace)
foo@foo-VirtualBox:~/fastlane$

Could anyone of you fix it somehow? I have no idea where/how to start.

Looking at the stacktraces we all posted, it seems bundle exec rspec, bundle exec rake or even bundle exec rake test_all is actually executing tests of the adresseable gem - which doesn't make any sense!?

Indeed, deleting the spec folder of the addressable gem gives this error message:
foo@foo-VirtualBox:~/fastlane$ bundle exec rake test_all
rspec --pattern ./**/*_spec.rb --format progress
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's default settings.
Changing stdout to /tmp/fastlane_tests, set `DEBUG` environment variable to print to stdout (e.g. when using `pry`)
/home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/babosa-1.0.2/lib/babosa/utf8/active_support_proxy.rb:1:in `require': cannot load such file -- active_support (LoadError)
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/babosa-1.0.2/lib/babosa/utf8/active_support_proxy.rb:1:in `<top (required)>'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/babosa-1.0.2/spec/utf8_proxy_spec.rb:4:in `require'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/babosa-1.0.2/spec/utf8_proxy_spec.rb:4:in `<top (required)>'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `load'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `block in load_spec_files'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `each'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:100:in `setup'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:86:in `run'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `load'
	from /home/foo/fastlane/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `<main>'
rake aborted!
Command failed with status (1): [rspec --pattern ./**/*_spec.rb --format pr...]
/home/foo/fastlane/Rakefile:19:in `block in <top (required)>'
/home/foo/fastlane/vendor/bundle/ruby/2.4.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'
/home/foo/.rbenv/versions/2.4.2/bin/bundle:23:in `load'
/home/foo/.rbenv/versions/2.4.2/bin/bundle:23:in `<main>'
Tasks: TOP => test_all
(See full trace by running task with --trace)
foo@foo-VirtualBox:~/fastlane$

This is the next gem with a spec folder.
Why is it running tests of gems?

Because in Ubuntu (or at least the way we set it up, even if accidentally), unlike in other installations, the gems are installed in the vendor folder of our project. The rspec pattern rspec --pattern ./**/*_spec.rb --format progress includes that, so dutifully executes these files.

We all probably have this:
foo@foo-VirtualBox:~/fastlane$ bundle config
Settings are listed in order of priority. The top value will be used.
path
Set for your local app (/home/foo/fastlane/.bundle/config): "vendor/bundle"

I think you get instructions to do this when you try to run bundle install with the system ruby. We probably all did this when we first checked out fastlane and messed up our project that way.
Now to find out how to change that...

Solution:

Delete .bundle
Delete vendor
Run bundle install again
Tests work with bundle exec rake

(You can check where individual gems are installed with e.g. bundle info addressable)


[SPARK-24626] Add statistics prefix to parallelFileListingInStatsComputation

What changes were proposed in this pull request?
To be more consistent with other statistics based configs.
How was this patch tested?
N/A - straightforward rename of config option. Used git grep to make sure there are no mention of it.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3206/
Test PASSed.

Test build #96207 has started for PR 22457 at commit 8d46d72.

Test build #96207 has finished for PR 22457 at commit 8d46d72.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96207/
Test FAILed.

Test build #96209 has started for PR 22457 at commit dde781f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3207/
Test PASSed.

Test build #96209 has finished for PR 22457 at commit dde781f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96209/
Test PASSed.

Merged build finished. Test PASSed.


I want to make a forks.

Hello!
I like a Raylib but Claylib not everything suits me:

Raylib and Raygui are merged into Claylib as a single wrapper library.
The Raygui executable is bundled with the wrapper instead of being installed separately. This is slightly inconvenient for me as I would prefer to install the executable myself.
Probably due to the second point, Claylib is not available in Quicklisp. I have to clone the repository into local-projects.
I am not satisfied with the implementation of vectors and matrices. In the Lisp community, there are very successful libraries for 3d-vectors, 3d-matrices, and 3d-quaternions (let's call them the "linear trio"). The function names in these libraries are much more convenient (although the names are shorter, so it will take some getting used to). It would be great to use them instead of reinventing the wheel.

I would like to suggest splitting the library into two separate libraries. Additionally, I would like to replace your implementation of linear algebra with the "linear trio" libraries that are widely used in the Lisp community. Do you think it would be difficult to make these changes? I'm ready to write new libraries on my own.
Thanks!

Hi there, thanks for your enthusiasm and willingness to push the project forward! I'll say that overall the changes you're proposing are more work than I (personally) am willing or able to do right now -- but that doesn't mean I wouldn't merge any PRs you submit, as long as they make sense. You could also of course make a fork; no offense taken.
It may be helpful though to understand some of the design choices before you go on that path:

Raylib and Raygui are merged into Claylib as a single wrapper library.

Yes, the vision of Claylib was as a beginner-friendly "batteries included" tool, which meant including things like Raymath and Raygui and wrapping them in a strong Lispy layer. cl-raylib exists for people who might want just a thinner wrapper over Raylib only. I suspect it wouldn't be too difficult to split into something like Claylib and Claygui. Where you'll run into trouble is with Raygui itself, because you actually have to build Raygui differently depending on whether you're using it with Raylib or standalone. That would in turn cause issues using Claylib and Claygui at the same time.

The Raygui executable is bundled with the wrapper instead of being installed separately. This is slightly inconvenient for me as I would prefer to install the executable myself.

Claylib does support this, via the CLAYLIB_USE_SYSTEM_RAYLIB_LIBRARIES environment variable. See #71. Though, do also see #77 -- it expects the tagged Raylib 4.5 release.

Probably due to the second point, Claylib is not available in Quicklisp. I have to clone the repository into local-projects.

That's not due to the second point, but the fact we've only tested comprehensively on Linux and SBCL. This is relatively low-hanging fruit and I'd love to be able to build for Mac and Windows, and ideally other CL distributions too.

I am not satisfied with the implementation of vectors and matrices. In the Lisp community, there are very successful libraries for 3d-vectors, 3d-matrices, and 3d-quaternions (let's call them the "linear trio"). The function names in these libraries are much more convenient (although the names are shorter, so it will take some getting used to). It would be great to use them instead of reinventing the wheel.

We didn't reinvent the wheel -- not exactly. What we did was create "pass-through" CLOS objects. If you look under the hood you'll see that they're really just convenient ways of interfacing with Raylib's own C structs. I'm not entirely familar with how those Lisp libraries work so am willing to be proven wrong, but I suspect it won't be as easy as you're hoping to just plug those in and have them work seamlessly with the passthrough Raylib functions. For Claylib's own native shape classes, maybe, but then that seems like more trouble than it's worth.

Thanks, I'll consider if it's worth creating a fork for professional development. I don't think you should split the library. Look, you've got a huge amount of code generated there that connects C and Lisp. Could you talk about how you did it? Can I post your answer on my blog?

Look, you've got a huge amount of code generated there that connects C and Lisp. Could you talk about how you did it?

Are you talking about the claw wrapper layer, or all the Lispification on top of it? Or both?
For the wrapper layer, to rebuild that you can check the docs in wrap/lib/BUILD.md and wrap/README.md. Not sure I'd have much more to say on that part.
For Lispification, the short answer to that is: Macros. :grin: The critical parts are in src/helpers.lisp. I am certain some of those macros could be better written, and they may steepen the learning curve if you're creating a fork. But once you get used to them, extending the toolkit is a breeze.

Can I post your answer on my blog?

Of course! Did you have any specific questions?

I don't know the answer to that one. You'd have to check the Raygui issues/docs. If Raygui does but Claylib does not -- maybe I can help with that.


python311Packages.pytype: init at 2023.11.29;

Description of changes
Resolves https://github.com/NixOS/nixpkgs/issues/272786
This includes python311Packages.pycnite: init at 2023.10.11 since it's a dependency not in nixpkgs yet.
This builds fine, however, there's a runtime error:
No module named ninja when running pytype on a python file.
I believe it's because the ninja in propagatedBuildInputs is the ninja package, not the application
Any hints on how to resolve this?

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

I believe it's because the ninja in propagatedBuildInputs is the ninja package, not the application

You can reference both the application (pkgs.ninja) and the Python package (pkgs.python3.pkgs.ninja) independently.

I believe it's because the ninja in propagatedBuildInputs is the ninja package, not the application

You can reference both the application (pkgs.ninja) and the Python package (pkgs.python3.pkgs.ninja) independently.

I tried that right now using
{
# other imports
, pkgs
}:

propagatedBuildInputs = [
  # other build inputs
  ninja # i assume this comes from pythonPackages
  pkgs.ninja
]

still the same issue, unfortunately. i must have missed something here


How to pass computer resource information

Hello,
I am looking to see where/how cpu and/or gpu information is passed during server start but I am unable to find it.
Thank you

Are you trying to configure CatAI only to use some CPU/GPU cores?
In general llama.cpp is the actual engine behind the scenes, you can configure with CatAI how many CPU/GPU cores and memory the model uses.
You can configure CatAI like this:

open the settings tab:

Add custom settings, which you can find here: https://withcatai.github.io/node-llama-cpp/types/LlamaModelOptions.html

Restart CatAI

Hello,
I am trying to get the model to utilize all of the cpu/gpu that I have available...
Looking at the utilization, node is using 6 cores (of 16) and ~13% of a v100 gpu.

I have tried altering some of the settings but I wasn't able to see a difference.
By chance is it because of the model that I am using?
vicuna-7b-v1.5-q4_1
Thank you.

Yes, this may be related. The binding of node-llama-cpp has some bugs with the configuration.
This binding allows you to use ggmlV3 (legacy format).
The recommended format is gguf (node-llama-cpp-v2),
If you want to utilize more VRAM you should use a bigger model, such as

llama-2-13b-chat-limarp-v2-merged-q3_k_s
phind-codellama-34b-q3_k_s

You can check this page for more models: https://huggingface.co/TheBloke
Most GGUF models are supported, you can install a custom model by coping the model link to catai install

The more parameter the model the more resources it utilizes, so this model (70B) https://huggingface.co/TheBloke/ARIA-70B-V2-GGUF
Can be pretty heavy.
If a model split into files like https://huggingface.co/TheBloke/Falcon-180B-GGUF/tree/main
You can install it like this:
canai install model-part-a.gguf,model-part-b.gguf
I have not tried this method yet, so if there is a problem feel free to report it :)

Ok. I'll continue to look into it.
For the gif that is in the documentation, would you be able to tell me what model was used and about how long the answer took to generate? I feel that what I am seeing is slower than what is to be expected.
Thank you again.

This git was generated from screen recording in mac pro m1. llama.cpp has spesifict optimization for mac silicon.
If I remember right this was from vicuna-13b-v1.5-q4_1 (but older version of the model).
I will soon update the models index to support vicuna with GGUF (gguf have more optimizations).
Are your cuda working when the model generating tokens?

I uploaded new models, in my Mac token/s is faster than in the gif, I use vicuna-7b-16k-q4_k_s
https://github.com/withcatai/catai/assets/50456099/7a971dd3-8fd5-45d4-840a-2325d03d2de4


Search Editor - Peek Definition displays confusing results

Issue Type: Bug

Open a folder (I used the vscode-mock-debug example extension)
Search for a word (I used 'provide')
Click the "Open in editor" link
Double-click the underlined name of a file in which matches were found (I used CHANGELOG.md). Or alternatively use its context menu to choose Peek > Peek Definition.
Try to understand what gets shown in the peek area.

When no context lines are being shown in the Search Editor, see how the Peek reports 6 definitions, whereas the search panel (and the editor text behind the peek) shows 3 matches for the search string.
See how the three lines containing the search term are each shown twice in the right-hand panel of the peek, and the matching word is not what is highlighted.
Repeat the peek after enabling context lines:

VS Code version: Code 1.63.2 (899d46d82c4c95423fb7e10e68eba52050e30ba3, 2021-12-15T09:40:02.816Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No

Good catch, thanks. The peek's count will now match the number of result lines shown. There is still a corner case of having multiple results on the same line which is not accurately handled, unless the results are far enough apart to get a ⟪ ([0-9]+) characters skipped ⟫ added.


Add support for skip-ssl-validation

Support added for skip-ssl-validation. Also looks like gofmt changed a few formatting items.

Thanks for this!
Sorry it took so long, I don't currently have time for open source projects during working hours.


V2 : enhanced polyhedron to accept an array of colors

These changes add support for colors (arrays) to polyhedron. The colors are applied to each face (polygon) of the polyhedron (geom3).
The reason behind these changes is that the several deserializers use polyhedron to produce geometries.
All Submissions:

[X] Have you followed the guidelines in our Contributing document?
[X] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[X] Does your submission pass tests?

I am generally ok with this but unsure about two points:

the api itself: so you can now pass multiple arrays for different data, it is good because you can omit some of the data, but would an interlaced data structure not also be practical ? (I would say no to be honest, but it has perks): ie : [[posX, posY, posZ, colR, colG, colB], ...]
would we not want color information per point and not just per face ? (gradients & co ?)

would we not want color information per point and not just per face ? (gradients & co ?)

I think you are correct but there would be major changes require throughout the packages. We are already pushing the limits of JavaScript and WeGL, so I’m not sure this will be practical for a few years.

@kaosat-dev requested changes complete. :)


Assembly load error when running C# cmdlet

Describe the bug
After importing custom PS module in Jupyter PowerShell notebook, an error occurs when running a cmdlet from the module. When running in "plain" powershell 7 (GA) it works as expected. I.e. the failing assembly is located in the same directory as the PS module assembly.
Import-Module "c:\Novelda\<snip_path>\Novelda.DataTwister.PowerShell.dll"

try
{
    Get-TwisterInventory
}
catch
{
    Out-Host $_.Exception.ToString()

}

Out-Host:
Line |
7 |      Out-Host $_.Exception.ToString()
|      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| A positional parameter cannot be found that accepts argument 'System.TypeInitializationException: The type initializer for 'Novelda.ServiceLocator' threw an exception.
---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Could not find or load a specific file. (0x80131621)
File name: 'Microsoft.Extensions.DependencyInjection, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60'
---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly)
at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
at System.Runtime.Loader.AssemblyLoadContext.OnAssemblyResolve(RuntimeAssembly assembly, String assemblyFullName)
at Novelda.ServiceLocator..cctor()
--- End of inner exception stack trace ---
at Novelda.ServiceLocator.get_Provider() in C:\Novelda\XeThru\Novelda.DataTwister\Novelda.DataTwister.Settings\Settings.cs:line 86
at Novelda.DataTwister.PowerShell.GetTwisterInventoryCmd.BeginProcessing() in C:\Novelda\XeThru\Novelda.DataTwister\Novelda.DataTwister.PowerShell\GetTwisterInventoryCmd.cs:line 23
at System.Management.Automation.Cmdlet.DoBeginProcessing()
at System.Management.Automation.CommandProcessorBase.DoBegin()'.
Please provide as much information as you can.
Loading dependent assemblies behaves differently in dotnetinteractive vs powershell 7
Did this error occur while using dotnet interactive?

[x] .NET Jupyter Notebook

Screenshots
If applicable, add screenshots to help explain your problem.
Please complete the following:

OS

[x] Windows 10
[ ] macOS
[ ] Linux (Please specify distro)
[ ] iOS
[ ] Android

Browser

[ ] Chrome
[x] Edge
[ ] Safari

Frontend

[x] Jupyter notebook
[ ] Jupyter lab
[ ] nteract

@thomas-t1 Your module depends on Microsoft.Extensions.DependencyInjection, is that assembly also in the module folder? PowerShell doesn't ship that assembly.

Yes, the module folder contains all dependencies including the DI assembly, as I mentioned the module works as expected in "plain" powershell 7.0.0

Not sure if it is relevant, but the error occurs inside a static constructor in the assembly that is dependent on Microsoft.Extensions.DependencyInjection

It turns out this is because the <IP_ADDRESS> version of Microsoft.Extensions.DependencyInjection is already loaded by the dotnet-interactive at its startup.

I can update the version of the reference package in dotnet-interactive project, but the same project is bound to show up again until we finish the module isolation work in PowerShell.


Failure to launch on OSX 10.11.13

Fresh download of 1.0 version from github page, permissions are set to launch any application, however on launch this error is displayed:

Hello I'm aware of this issue, and going to fix, upload correct portable as soon as possible.

cool, much appreciated

Could you please retry from the same link ? App name will be my-electrified-app

yep that works!

Great, closing issue for now, I need to package for OsX with a better icon and name soon :)


Implement Initial Collections Artwork Filter Navigation Stack

#skip_new_tests

Warnings

:warning:
Missing Test Files:

src/lib/Components/ArtworkFilterOptions/__tests__/SortOptions-tests.tsx

If these files are supposed to not exist, please update your PR body to include "#skip_new_tests".

  Generated by :no_entry_sign: dangerJS

Fails

:no_entry_sign:

🃏 FAIL in SortOptions-tests.tsx

renders properly

Snapshot name: `Sort Options Screen renders properly 1`- Snapshot+ Received@@ -35,10 +35,11 @@ }, ] } >
Full message

Error: expect(received).toMatchSnapshot()
Snapshot name: Sort Options Screen renders properly 1

Snapshot

Received

@@ -35,10 +35,11 @@
},
]
}
>
<View

    data-test-sort-back-nav="item"
    ml={2}
    mt={2}
    onTouchStart={[Function]}
    style={
      Array [

at Object.toMatchSnapshot (/home/travis/build/artsy/emission/src/lib/Components/ArtworkFilterOptions/tests/SortOptions-tests.tsx:20:31)
at Object.asyncJestTest (/home/travis/build/artsy/emission/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:102:37)
at /home/travis/build/artsy/emission/node_modules/jest-jasmine2/build/queueRunner.js:43:12
at new Promise ()
at mapper (/home/travis/build/artsy/emission/node_modules/jest-jasmine2/build/queueRunner.js:26:19)
at /home/travis/build/artsy/emission/node_modules/jest-jasmine2/build/queueRunner.js:73:41
at runNextTicks (internal/process/task_queues.js:58:5)
at processImmediate (internal/timers.js:412:9)

  Generated by :no_entry_sign: dangerJS

Cool! @ashleyjelks I'm going to merge this in 😄 but it's worth revisiting the few remaining items from @ds300's review in one of your subsequent PRs!

:rocket: PR was released in v1.21.52 :rocket:


Graphics not displayed until refreshed

I am reporting an issue with

[ ] TypeScript definitions
[x] another resource in this repository

Hello, I am facing an issue with Graphics when using ES modules with angular. I am referring to the jsapi-angular-cli sample application in this repository. I've updated the code to display a Point using Graphics. See gist here.
When  I execute the application, or refresh page, the API does not display the point. But when I zoom in, it appears.
See the screen capture here.
As you can see, when I refresh the page, the Point does not appear. After a few seconds, I zoom in on the map and it shows up.
I've ended up with workaround, where I programmatically zoom in after a short delay - but that seems to have led to a subsequent issue in which part of graphics are rendered but not all.
If this is not the right repository to log this issue, I would appreciate if someone points me to the right one.
Thank you.

@gauravsamudra thanks for reporting. This is a known issue and we hope to have it fixed in 4.19 (April).  I'll go ahead and close this out.

Thanks @andygup. Appreciate about the April timeline information. Helps plan things better.
I'm not sure if this helps, but I'd like to mention that, as a workaround, I switched over to esri-loader - and its version 3.0.0 has the same issue. I downgraded to 2.16.0 and it works well.

@gauravsamudra we believe we fixed this bug. Can you retest using npm install @arcgis/core@next? /next is a preview build for the 4.19 production release in April.


cargo-wasi behind proxy

Support cargo-wasi to fetch resources behind a proxy.
Find the proxy in order:

cargo's 'http.proxy'
http_proxy env var
HTTP_PROXY env var
https_proxy env var
HTTPS_PROXY env var

It seems that there is some problem with the wasm-opt and wasm-bindgen that this project used, and cause check fail.

Thanks for the PR, but I think pulling in the entire cargo library just to read the proxy settings is a bit overkill. I think it's fine to check the env vars for now and that can be extended in the future if necessary.

It sounds great to me, and have moved the code relating to the cargo library.
BTW, it seems that reading cargo's global config is a common behavior for subcommands.


chore: update aegir

Codecov Report

Merging #72 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #72   +/-   ##
=======================================
  Coverage   84.41%   84.41%
=======================================
  Files           8        8
  Lines         494      494
=======================================
  Hits          417      417
  Misses         77       77

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aafe59a...a7edf4f. Read the comment docs.


Interface changed to improve multiplatform support

Hi Bob,
I extended hscdtd_device_t and use it as handle for the open,close, write, read functions.
In addition, I added another begin method with an additional parameter device in the signature.
And to allow complication I had to move hscdtd_device_t into a separate header file common.h.
For the rpi-target, it compiles and runs.
platform_arduino.cpp would need to be extended/changed to support the changed signatures as in platform_rpi.cpp.
Please review and check, if this is a suitable approach for the support of different platforms.
Thanks Tilman

I changed the Makefile, so that the library is now compiled as static library that can be  linked to a main progam (in this case, the example program)

@tilman1 Apologies for not responding earlier, it got lost in my email spam.
It looks like you used an existing branch without fetching changes from master. Could you make a new merge request / branch with just the changes related to this interface change?

I will also add a linter, that should also help with the formatting of some whitespace issues.

Hi Bob

Could you make a new merge request / branch with just the changes
related to this interface change?

How do I open a new branch ? Can I do this via command line? Or do I
need to interacte with the webgui of github?
Thanks
Tilman
On 29.09.22 00:40, Bob wrote:

@tilman1 https://github.com/tilman1 Apologies for not responding
earlier, it got lost in my email spam.
It looks like you used an existing branch without fetching changes
from master. Could you make a new merge request / branch with just the
changes related to this interface change?
—
Reply to this email directly, view it on GitHub
https://github.com/bobveringa/HSCDTD008A-Library/pull/12#issuecomment-1261536002,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFA7EQNYEH6VCNSWZMNUTLDWATCPNANCNFSM6AAAAAAQVI6KFE.
You are receiving this because you were mentioned.Message ID:
@.***>

You can do this both from the command line and from webgui of GitHub. I use the command line for most of these things.
But first you have to make sure that there is a branch that is up-to-date with the main of this repository. Then you can branch from there, and make any changes you would like to make.
You can find more information on syncing a fork here. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
I generally recommend creating a new branch for the feature you want to submit as a PR, it makes updating your own repository much easier and avoids merge conflicts. Here are is some more information on creating branches, https://devconnected.com/create-git-branch/


NPE in REST

If I call createServiceTicket (/v1/tickets/{tgtId:.+}) with invalid TGT it causes NPE.
ERROR [org.apereo.cas.support.rest.TicketsResource] -
java.lang.NullPointerException: null
at org.apereo.cas.authentication.DefaultAuthenticationResultBuilder.lambda$buildAuthenticationHistory$1(DefaultAuthenticationResultBuilder.java:118) ~[cas-server-core-authentication-5.1.4-SNAPSHOT.jar:5.1.4-SNAPSHOT]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_66]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_66]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[?:1.8.0_66]
at org.apereo.cas.authentication.DefaultAuthenticationResultBuilder.buildAuthenticationHistory(DefaultAuthenticationResultBuilder.java:117) ~[cas-server-core-authentication-5.1.4-SNAPSHOT.jar:5.1.4-SNAPSHOT]

But you closed my Pull Request without explanation!

That's not quite true. Let me explain: I am not saying this issue is invalid or does not exist. I am simply saying your pull request is not the appropriate fix, and you need to dig deeper to find the root cause and prevent it there. That's the explanation.


Init base

Core changes
Backend: NodeJS
The backend is built using Node.js with the following features:

Express.js: A web framework for handling HTTP requests and routing.
Database: SQLite3 is used for data storage and retrieval. Its lightweight nature makes it ideal for local or small-scale applications.
Middleware: Various middleware functions are used for logging, error handling, and validating requests.

Frontend: React
The frontend is built with React and communicates with the Node.js backend via the REST APIs. Key features include:

State Management: React's built-in state management, or libraries like Redux or Context API, are used to manage app state.
Component-Based Structure: The UI is built with reusable components that represent different parts of the user interface.
Routing: React Router is used for navigating between pages and handling different views.
HTTP Requests: Axios or Fetch API is used to make HTTP requests to the backend for data operations.
Responsive Design: The application is designed to be fully responsive, providing a smooth user experience across various devices.

LGTM