[Submission]: New galaxy generators for X3 Terran Conflict

URL Link
https://www.nexusmods.com/x3terranconflict/mods/121
Game Series
X
Game(s)
X3 Terran Conflict
Summary
Galaxy generator.
Open source?

[ ] Open source

URL Link
https://www.moddb.com/mods/mayhem
Game Series
X
Game(s)
X3 Terran Conflict
Summary
Galaxy generator (external tool bundled with the modification).
Open source?

[ ] Open source

I'm not sure what is the relationship between these two or what relationship they have with the one already listed for X3... They seem to be completely different projects, from the couple of screen-shots I've seen.
Thought I'd share this here since I had no idea this game had a randomizer - let alone multiple! Hopefully it'll help others find them too.
Thanks for open-sourcing this list and keeping it alive! Cheers!

added them in this commit: https://github.com/video-game-randomizers/rando-list/commit/2fbec3b38009f5c1bac88c45ef332e3078496432#diff-016f5b1ce73c428f22267c28c8996da5f66016d320178e7401c61608de9d05d6
thank you!
https://randomizers.debigare.com/#recent-randomizers


Gunicorn/multiprocess with restarting workers, .db files are not cleaned up

We are using Gunicorn with multiple workers with the Gunicorn max_requests option. We found that memory usage in our containers increases with every worker restart. By reducing the max_requests value so workers restart very frequently, the problem became apparent.
Using the method described in the readme:
from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)

While running a performance test, I see the following in the container:
[root@mock-adapter-8568fdcf6d-lz8tx app]# cd /app/prometheus_tmp
[root@mock-adapter-8568fdcf6d-lz8tx prometheus_tmp]# ls
counter_10.db   counter_214.db  counter_319.db  counter_41.db   counter_91.db     histogram_196.db  histogram_296.db  histogram_401.db  histogram_69.db
counter_111.db  counter_222.db  counter_31.db   counter_425.db  counter_97.db     histogram_1.db    histogram_305.db  histogram_409.db  histogram_77.db
counter_117.db  counter_232.db  counter_328.db  counter_433.db  counter_9.db      histogram_202.db  histogram_311.db  histogram_419.db  histogram_84.db
counter_130.db  counter_23.db   counter_338.db  counter_444.db  histogram_10.db   histogram_214.db  histogram_319.db  histogram_41.db   histogram_91.db
counter_138.db  counter_244.db  counter_345.db  counter_450.db  histogram_111.db  histogram_222.db  histogram_31.db   histogram_425.db  histogram_97.db
counter_148.db  counter_251.db  counter_353.db  counter_458.db  histogram_117.db  histogram_232.db  histogram_328.db  histogram_433.db  histogram_9.db
counter_154.db  counter_258.db  counter_361.db  counter_470.db  histogram_130.db  histogram_23.db   histogram_338.db  histogram_444.db
counter_175.db  counter_269.db  counter_36.db   counter_48.db   histogram_138.db  histogram_244.db  histogram_345.db  histogram_450.db
counter_187.db  counter_277.db  counter_374.db  counter_56.db   histogram_148.db  histogram_251.db  histogram_353.db  histogram_458.db
counter_18.db   counter_286.db  counter_384.db  counter_61.db   histogram_154.db  histogram_258.db  histogram_361.db  histogram_470.db
counter_196.db  counter_296.db  counter_401.db  counter_69.db   histogram_175.db  histogram_269.db  histogram_36.db   histogram_48.db
counter_1.db    counter_305.db  counter_409.db  counter_77.db   histogram_187.db  histogram_277.db  histogram_374.db  histogram_56.db
counter_202.db  counter_311.db  counter_419.db  counter_84.db   histogram_18.db   histogram_286.db  histogram_384.db  histogram_61.db

Watching that directory, new files are created with every new worker and old ones are not being cleaned up. Deleting the files in that directory reduce down the memory usage on that container, which then starts building again.
What is the proper way of dealing with these database files? Is this mark_process_dead's responsibility?
And a general question about Prometheus: do we need to keep metrics around after they are collected? Could we wipe our metrics after the collector hits our metrics endpoint? If so, can this be done via the prometheus_client?

This is the expected behaviour, you can only delete these safely when the whole application restarts. I'd suggest increasing max_requests.

I'd suggest increasing max_requests.

We are using max_requests to deal with some memory leaks introduced with Gunicorn with gevent workers. Not ideal at all.
But even if we didn't have this problem, workers can restart for other reasons. Is it also expected that memory usage would increase like this? Because if so, it seems like eventually over the lifetime of the service, at some point all services using this pattern would run out of memory.

Memory usage for workers should be constant.

We have the same issue in our uWSGI deployment. We also restart workers every N requests, and they could restart by themselves because of harakiri (if the request takes too long). I don't think we could just keep the same workers forever, some sneaky memory leak might persist and max-requests is widely used to mitigate it.
Is it at all possible that the prometheus client would support this use case? Or should we just look for another solution?

There's not much we can do here if you have high churn I'm afraid.

We disabled worker restarts and we found that using Gunicorn and Flask, once in a while, we do have unplanned worker restarts infrequently. So for us, we were able to reduce the large memory leak and now have a very slow one that grows over the course of months.

Couldn't mark_process_dead() summarize the data into a single "dead" file so that there wasn't a need to leave a bunch of files hanging around?  Are the values in each individual file needed after the process dies?

That'd require locking across processes, which we want to avoid, and cause issues if that pid was reused.

Hey folks.
We have a workaround for this currently which uses gunicorn's child_exit hook to merge the dead worker's counter/histogram metrics into archive files and deletes the pid file. This has kept our file count and thus scrape response times stable. The attached image shows our scrape times (yellow line) when we deployed our workaround.
Our thinking was that using child_exit avoids the need to lock, as it's run only in the master. We still call the current mark_process_dead() function too, so gauge files are removed (although we are not heavy users of gauges).
Regards pid recycling, I think it is only an issue for gauges, which are still deleted anyway in our experiment. Also, given filenames with the pid in should be deleted very soon after the worker death, the chance of a recycling pid clash is very remote? I may have missed something though...
It seems to be working for use, but I would appreciate any feedback on anything we may have missed.
Our experiment involves a slightly altered version of the MultiProcessCollector.collect() function, which you can see here: https://github.com/canonical-ols/talisker/blob/master/talisker/metrics.py#L193
The changes are documented there, but for convenience are copied here:
This almost verbatim from MultiProcessCollector.collect().

The original collects all results in a format designed to be scraped. We
instead need to collect limited results, in a format that can be written
back to disk. To facilitate this, this version of collect() preserves label
ordering, and does not aggregate the histograms.

Specifically, it differs from the original:

1. it takes its files as an argument, rather than hardcoding '*.db'
2. it does not accumulate histograms
3. it preserves label order, to facilitate being inserted back into an mmap file.

It needs to be kept up to date with changes to prometheus_client as much as
possible, or until changes are landed upstream to allow reuse of collect().

We'd like to create a PR that upstreams some of this work, if you think the approach is sensible? Specifically, we'd like to slightly refactor the collect() method to support preserving label order, allow for not accumulating histograms and not hardcoding '*.db'. This would allow us to reuse it, rather than reimplement it.
Also, if this approach is deemed worthwhile, then we'd be happy to work on a PR to expand to include this aggregation as appropriate.
Thanks for reading!

Our thinking was that using child_exit avoids the need to lock, as it's run only in the master.

That's not a good assumption. For example if another worker is started up and gets the old pid, that'll break things. We also need this to work on things that aren't Gunicorn.
For this to work, you'd need a lock of some form.

refactor the collect() method to support preserving label order

Does it not already? That's usually something we have.

allow for not accumulating histograms

That violates the Prometheus text format, histograms must be cumulative.

Isn't the current approach already broken if the new worker gets the same PID?  How does it currently keep the values for the old and new processes if the PID is the same? Won't the new process start overwriting the old process's values since it uses the same file.  Just trying to understand this better to figure out potential workarounds.

Isn't the current approach already broken if the new worker gets the same PID?

No, it's designed to handle it. It reads in the previous values at startup.

Strawman naive solution: add process creation timestamp to the filename? I think that would ensure that they are never reused, regardless of pid recycling?

That could make the number of files worse. Not everyone can mark things dead.

Using SIGCHLD to the master process is one way - and that is what child_exit boils down to. Would it perhaps be beneficial to offer an opt-in documented solution for gunicorn, than no solution at all?

We need to work on Windows, and we shouldn't interfere with the signal handling of the process we're instrumenting.

Sadly not - the wrapping them in dict() is what loses the order:

Ah yes. That can't really be changed, as that's part of our API. Why do you want to change this?

Re windows, I suspect we there is no single solution that would work for both, anyway?

I don't know offhand.

Regards interfering, it would be up to the application to ensure the cleanup is called at the right time, just like it is currently?

Yes. But there's no concurrency restrictions around that, so for example two different processes could be performing it at once.

The json key used for the mmap file format has the in label order. To write the collect()ed data back to the aggregated archive files, I need the same order, which sadly the dict() loses.

The file format can be changed.

I don't know offhand.

Looking around, multiprocessing.Lock() might provide a cross-platform solution. I've added it to solve an issue in our solution, seems to work, but it's not yet in production, so unproven.

Yes. But there's no concurrency restrictions around that, so for example two different processes could be performing it at once.

Right, I wonder if documenting what the synchronization requirements are might be a good start? Then applications can use the right approach given their concurrency model?

The file format can be changed.

Right, fair enough. But I don't think it needs to, we just need to avoid losing the order of labels, which seems fairly simple?

Looking around, multiprocessing.Lock() might provide a cross-platform solution.

Hmm, how does that work behind the scenes?

Right, I wonder if documenting what the synchronization requirements are might be a good start? Then applications can use the right approach given their concurrency model?

I think this is something that should Just Work in any environment, I rather not have a system that requires extensive documentation and reasoning to work - as users will get it wrong.

Hmm, how does that work behind the scenes?

From the source, it looks like it's using WaitForSingleObjectEx syscall in windows, and a POSIX semaphore on Unix.

I think this is something that should Just Work in any environment, I rather not have a system that requires extensive documentation and reasoning to work - as users will get it wrong.

Fair enough, it's just a lot more work to get there :).
If multiprocessing.Lock is suitable, then I think mark_process_dead could maybe be expanded to aggregate into a single file on worker death by default, using that lock?
Rough stab at general solution: register an atexit function that calls mark_process_dead. That should work for any multiprocess scenario, with the caveat that processes forcibly killed will not run the atexit handle, and thus leak mmapped files. But that's a common enough issue with killing processes that way, and the leak would be limited to rare error scenarios, rather than every process recycle.

How is the handle/semaphone id shared between the processes?

I think mark_process_dead could maybe be expanded to aggregate into a single file on worker death by default, using that lock?

That only works if you know that no new processes will be created with that pid while you're doing that.

How is the handle/semaphone id shared between the processes?

The lock has to be inherited, so it has to be created in the parent process, and thus only works for a fork()ing model.

That only works if you know that no new processes will be created with that pid while you're doing that.

Right, that could probably be addressed by locking the mmap file opening with the same lock, or as discussed earlier, adding a timestamp or similar to the mmap filename (if we have full worker death cleanup, then extra files are no longer a problem, anyway).

The lock has to be inherited, so it has to be created in the parent process, and thus only works for a fork()ing model.

More particularly it only works if the parent process loads the Prometheus client library, which certain configurations of Gunicorn will not.

More particularly it only works if the parent process loads the Prometheus client library, which certain configurations of Gunicorn will not.

Right. You would probably need to document adding a gunicorn on_starting hook or something that did that.

Gunicorn isn't the only thing in use though, what if it's say mod_python in Apache (which I admit to having never used, and haven't even read the docs) in which a C process is what spawns the children.

Made a PR with the small changes to MultiProcessCollector.collect() I mentioned above.
This doesn't solve the problem, but it makes it easier for others to roll their own solution to the problem, as we have done (we had to duplicate the whole of collect() in our code - this would allow us to just call the new merge() function directly)

Did anybody manage to solve this? I'm using uwsgi and it's not obvious how I can call mark_process_dead

We managed to solve this for Gunicorn, the bulk of our (rather complex) solution can be seen here:
https://github.com/canonical-ols/talisker/blob/master/talisker/prometheus.py
It's not pretty :(
The core of the solution is this: The master process runs a cleanup function for each worker that dies. This function aggregates the dead worker's files into some central files, and then deletes the old files. This write and delete is a critical section that needs protecting with a multiprocess lock of some kind, or else workers reading the files could double read.
The Gunicorn specifics are using a SIGCHLD handler on the master process, and other messy details, but it's been working well for us, after a few initial bumps.
However, that solution is very specific to gunicorn's multiprocess model. uWSGI is a whole other kettle of fish. AFAICS, it doesn't support hooking in to SIGCHLD handling on the master, which is in C, so not so easily extensible. One option might be to define a custom worker as documented here:
https://uwsgi-docs.readthedocs.io/en/latest/WorkerOverride.html
You can maybe do the work in the destroy() function, which I assume happens in the worker process, but you'd really need multiprocess locking here. We used the stdlib's multiprocess.Lock(), but as discussed in this thread, it requires a python master process to initialise it. But under the hood, it's just a semaphore based on a shared memory file path, some form of that might be usable, if you can figured out how to ensure each worker is using the same path.
Are you aware of https://github.com/Lispython/pyprometheus ? That is an alternative prometheus client library that has specific support for using uwsgi's shared memory support, as opposed to using mmap files in a shared directory. I have not used it, and it's not the official prometheus client, but it might help here - you may be able to use it's approach with prometheus_client.
HTH

Have you found any solution for this issue ?

hello @bloodearnest , I have attempted to use your solution to tackle db file number problem. I used the solution on three metrics types: Counter (counter_{pid}.db), Histogram (histogram_{pid}.db), and Summary (summary_{pid}.db). I tried decreasing the max_requests config in Gunicorn and ran some stress tests and the result is promising, the number of files are kept at minimum.
However, I have some questions about the procedure:

I occasionally got the error RuntimeError: Read beyond file size detected, /opt/prometheus/summary_archive.db is corrupted. and that made the db files not merged correctly, leaving some of dead pid's summary db files behind. The error has not happened to counter_archive.db and histogram_archive.db yet from my observation. Is that the reason you did not archive the summary db files in your code?
Assuming the db files are all merged well, will there be a day when {metrics_type}_archive.db grows too large and causes problem?
Will this procedure slow down and/or block the gunicorn master at some points? I am concerned about the impact of this when it comes to production.
Thank you for your time.

hello @bloodearnest , I have attempted to use your solution to tackle db file number problem. I used the solution on three metrics types: Counter (counter_{pid}.db), Histogram (histogram_{pid}.db), and Summary (summary_{pid}.db). I tried decreasing the max_requests config in Gunicorn and ran some stress tests and the result is promising, the number of files are kept at minimum.
However, I have some questions about the procedure:

I occasionally got the error RuntimeError: Read beyond file size detected, /opt/prometheus/summary_archive.db is corrupted. and that made the db files not merged correctly, leaving some of dead pid's summary db files behind. The error has not happened to counter_archive.db and histogram_archive.db yet from my observation. Is that the reason you did not archive the summary db files in your code?
Assuming the db files are all merged well, will there be a day when {metrics_type}_archive.db grows too large and causes problem?
Will this procedure slow down and/or block the gunicorn master at some points? I am concerned about the impact of this when it comes to production.

Thank you for your time.

I'm using
if "prometheus_multiproc_dir" in os.environ:
    import uwsgi

    prometheus_client.values.ValueClass = prometheus_client.values.MultiProcessValue(
        uwsgi.worker_id
    )

which makes it so that it doesn't use the pid but the worker id, which is reused on worker restart.
This looks like it works in dev. @brian-brazil would there be problems with this setup?
The only thing that looks suspicious to me is not having a _created gauge for counters. Is that to be expected with multiprocess setup?

This issue has gotten a bit long covering many related issues. The initial report was not a bug, so let's open new issues for any new discussion around improvements here.

Hi, sorry, I know this issue is now closed, but I wanted to reply to @xuansontrinh, I'd missed the question addressed to me.

hello @bloodearnest , I have attempted to use your solution to tackle db file number problem. I used the solution on three metrics types: Counter (counter_{pid}.db), Histogram (histogram_{pid}.db), and Summary (summary_{pid}.db). I tried decreasing the max_requests config in Gunicorn and ran some stress tests and the result is promising, the number of files are kept at minimum.
However, I have some questions about the procedure:

I occasionally got the error RuntimeError: Read beyond file size detected, /opt/prometheus/summary_archive.db is corrupted. and that made the db files not merged correctly, leaving some of dead pid's summary db files behind. The error has not happened to counter_archive.db and histogram_archive.db yet from my observation. Is that the reason you did not archive the summary db files in your code?

No - the reason is that we don't use summaries! I expect you'll need to extend the code to do so. If the summaries are a different format from counters/histograms, then you may need to write code to merge them.

Assuming the db files are all merged well, will there be a day when {metrics_type}_archive.db grows too large and causes problem?

I don't think this should be an issue. I think the file size should depend on the number of metrics/labels your app has, not the length of time it's been around, as the merge should collate all of them into one entry per metric/label combination.
Also, we avoid this by using a new tmpdir for prometheus_multiproc_dir when the gunicorn service restarts, to meet the requirement of clearing it between service starts as in the documentation. And as each deploy is a new service, it will only grow for the lifetime of a deployment, which is typically only a few weeks at most.

Will this procedure slow down and/or block the gunicorn master at some points? I am concerned about the impact of this when it comes to production.

The gunicorn master does do more work. But it spends most of its time idle anyway, as it handles no traffic. I guess if you have many workers, there might be an issue, especially if you SIGHUP gunicorn, as all the current workers will die. FTR, we use 64 workers, and regularly SIGHUP, with out any issue that we've detected.

Thank you for your time.
Edit: Below is the image describing the 1st question. My gunicorn is configured to run with 3 workers:

Hi everyone! Sorry for updating it after two years but I will don't understand what should I do with flooding prometheus multiproc dir with .db files? It slows the /metrics collection. Now we just clean up the dir for each deploy but if where are not deploys for some days it leads to timeouts for /metrics request.
What is the best practice here? Cron cleaning job?

Hi All. I am also facing the same issue. Memory will reach the limit and application will restart each time. I tried to use all types of workers but still no success. I tried to run a cron job to delete files but after each deletion I will get a pod error for prometheus. Also if a file was being used by some worker metrics endpoint would give an exception and again error will be registered via prometheus.
For the suggestion above to restart the server is not suitable for me as I need it to run without any restart and would not want application depend of metrics. Any help is appreciated. Thanks!


⚠ Bofrost.lu has degraded performance

In d7f71f0, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1184 ms

Resolved: Bofrost.lu performance has improved in 5a58ea1 after 9 minutes.


Skip failing test on parallel executer

Because assertTasksExecutedInOrder may fail with --parallel

OK, I've already triggered a build for you.


Topological alignment between GPUs and NICs in DRA (exposing pci device topology as device attribute?)

I understand DRA will finally promote to Beta in v1.32🎉  Thank you very much contributors for your hard work standardizing flexible device scheduling and implementing NVIDIA's dra-driver.
Do you have a plan exposing intra-node topology as device attribute?? Especially distances between GPU<->GPU and GPU<->NIC or HCA (I imagine nvidia-smi topo -m equivalent information)? Or, would you have a plan to provide some extension point to add user-defined device attribute in this dar-driver??
I imagine below usecases for optimizing training performance:

~Single Node Multi GPUs:~

~a user wants to have 1 pod with 2 gpus which are connected via NVLink each other (NV# in nvidia-smi topo -m)~
→ discussed in #214

Multi Node Multi GPUs:

a user wants like to have N pods per 4 gpus each of which have adjacent NIC or HCA (PIX in nvidia-smi topo -m) in specific zone(achieved by node selector)

probably, it needs integration with cni and network device plugins (e.g. https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin)

Thanks, in advance.

I'm open to suggestions on what these attributes would look like and how they would be used, but as I mentioned in my comment here https://github.com/NVIDIA/k8s-dra-driver/issues/214#issuecomment-2517240003, I've struggled to come up with something that would actually be useful.

Thanks,

Or, would you have a plan to provide some extension point to add user-defined device attribute in this dar-driver??

How about this? If this driver provides such knob, user will be able to publish their own extra attributes for thier needs.

Multi Node Multi GPUs:

a user wants like to have N pods per 4 gpus each of which have adjacent NIC or HCA

For this use case, I found the presentation excactly matched this case.
Better Together! GPU, TPU and NIC Topological Alignment with DRA - John Belamaric, Google & Patrick Ohly, Intel
So, If both NVIDIA/k8s-dra-driver and kubernetes-sigs/cni-dra-driver exposed k8s.io/pcieRoot attribute, user can define the ResourceClaim like below, as described in the session:
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaim
metadata:
  name: big-gpu-with-aligned-nic
spec:
  devices:
  requests:
  - name: gpu
    deviceClassName: gpu.nvidia.com
    selectors:
    - cel:
        expression: "device capacity['memory'].compareTo(quantity ('80Gi')) >= 0"
  - name: nic
    deviceClassName: rdma.nvidia.com
    selectors:
    - cel:
        expression: "device.attribute[ 'sriovType'] == 'vf'"
  constraints:
  - requestNames: ["gpu", "nic"]
    matchAttribute: k8s.io/pcieRoot

I would like to know if NVIDIA/k8s-dra-driver plans to expose k8s.io/pcieRoot attribute.

Unfortunately, we can't include this until we start to standardize the set of attributes we put under the k8s.io/* prefix. I could include this under the nvidia.com/* prefix, but this is less useful since you can't then use it in a matchAtrribute to match with an attribute with a different prefix from a different driver.

Thanks for the quick reply. OK, then, let me keep this open for now.


Gulp not copying images from src to dist on Windows

Vanilla Zurb Master Template watch on images not copying from src to dist.  Images are copied if you restart or run a production build, but not being copied over in normal workflow.

What version of Windows, what command line, and what version of Node do you use?

Windows 7 SP1
Standard Command prompt, or Node.JS Command prompt
Upgraded npm yesterday to v3.5.2, was using 3.3.x previously.
I’ve got it working now (of sorts) by installing and using Foundation CLI.  After npm upgrade it still didn’t work without the CLI.
From: Geoff Kimball<EMAIL_ADDRESS>Sent: 16 December 2015 18:18
To: zurb/foundation-sites
Cc: tallchap67
Subject: Re: [foundation-sites] Gulp not copying images from src to dist on Windows (#7552)
What version of Windows, what command line, and what version of Node do you use?
—
Reply to this email directly or view it on GitHub.

Strange, how did you set it up initially? By cloning the template from GitHub?

Downloaded the latest zip from Github and unpacked into a new folder.
npm install
Bower install
npm start
From: Geoff Kimball<EMAIL_ADDRESS>Sent: 17 December 2015 19:49
To: zurb/foundation-sites
Cc: tallchap67
Subject: Re: [foundation-sites] Gulp not copying images from src to dist on Windows (#7552)
Strange, how did you set it up initially? By cloning the template from GitHub?
—
Reply to this email directly or view it on GitHub.

We had a few other reports about this issue, and I believe zurb/foundation-zurb-template#25 is the solution. You can look at the diff of the PR to see what change is necessary.


accounts directory created in certificate directory

I'm not sure what caused this, honestly... but here's what happened.
This occurred using dehydrated from master, revision b116e6b, during a normal cron run. One of my certificates has a certificate-specific config file there, which specifies a different hook file. There are no other variables set in that cert-specific config.
During the cron run, I got the following output:
Processing <domain> with alternative names: <altnames>
 + Using certificate specific config file!
   + HOOK = /srv/dehydrated/hooks/dns-01-manual/cron-hook.sh
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Apr  8 23:19:24 2018 GMT (Less than 30 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 6 authorizations URLs from the CA
 + Handling authorization for <domain>
 + Handling authorization for <domain>
 + Handling authorization for <domain>
 + Handling authorization for <domain>
 + Handling authorization for <domain>
 + Handling authorization for <domain>
 + 6 pending challenge(s)
 + Deploying challenge tokens...

The output for that certificate stopped there. As I had keep-going set, the run continued on to the next certificate in the list and wrapped up normally. I caught it on review of the log and didn't think much of it--I'd recently been fiddling with the hook script for that certificate, and figured I had created an error of some sort that crashed the hook.
But I also noticed today that there was a new accounts directory in the certificate output directory for that certificate, with a CA hash corresponding to the ACME v2 endpoint, with no contents. That is:
BASEDIR
|-accounts
||-ACME v1 CA hash (with account keys)
||-ACME v2 CA hash (symlink to v1 folder)
|-certs
||-domain
|||-accounts
||||-ACME v2 CA hash (no contents)
|||-cert.pem
|||-fullchain.pem
|||-etc.

I don't know how this new accounts directory got there. The time of creation matches the time when the cron job was executing. And there's zero chance my hook script was creating an accounts directory with the CA hash underneath it.
The other certificates processed at the same time do not have a local accounts folder created. It only shows up in the directory for the certificate with the per-certificate config file--which, incidentally, never returned from the deploy hook, but died quietly enough to move on to the next certificate without leaving an error in the cron log.
I'm not sure what to make of it, honestly. Any thoughts?

Mhm, do you have a config file in that certificate directory and is there the slightest possibility that dehydrated was somehow executed inside of that directory?
accounts gets appended to BASEDIR, which normally comes from the config location, which (if there is nothing in /etc/dehydrated and /usr/local/etc/dehydrated) will be the current working directory
But if that happened it shouldn't have continued as it would basically have used a broken config and/or an empty accounts directory... but there is no other way, the word accounts doesn't occur anywhere else in the script and ACCOUNTDIR is only really used in the beginning of the script...

The command as written in cron is:
/srv/dehydrated/dehydrated —cron —keep-going
Main config file is located at /srv/dehydrated/config. The certificate-specific config file is located within the certificate directory, but does not override the value of BASEDIR (which is set within the main config file).
I know, it’s weird.

I'm closing this for now as I can't reproduce this other than by running dehydrated directly in a certificate directory (any possibility that your hook-script somehow ran dehydrated?).
I might change the (preferred) name for that config file in the future so dehydrated doesn't ever get confused by this...


python3.pkgs.opensearch-py: disable broken tests

Fix broken opensearch-py tests.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 352871

x86_64-linux

:fast_forward: 4 packages marked as broken and skipped:

python311Packages.parsedmarc
python311Packages.parsedmarc.dist
python312Packages.parsedmarc
python312Packages.parsedmarc.dist

:x: 2 packages failed to build:

parsedmarc
parsedmarc.dist

:white_check_mark: 8 packages built:

python311Packages.awswrangler
python311Packages.awswrangler.dist
python311Packages.opensearch-py
python311Packages.opensearch-py.dist
python312Packages.awswrangler
python312Packages.awswrangler.dist
python312Packages.opensearch-py
python312Packages.opensearch-py.dist

I believe these tests can likely also be fixed by applying https://github.com/opensearch-project/opensearch-py/commit/0ccd4173cc49b8e15d2c2f1a5fa7459f6e390a86 from upstream.

I believe these tests can likely also be fixed by applying opensearch-project/opensearch-py@0ccd417 from upstream.

Indeed that works. The patch doesn't cleanly apply, so I'll drop this here so you won't have to repeat my work:
--- a/pkgs/development/python-modules/opensearch-py/default.nix
+++ b/pkgs/development/python-modules/opensearch-py/default.nix
@@ -2,6 +2,7 @@
   lib,
   buildPythonPackage,
   fetchFromGitHub,
+  fetchpatch2,

   # build-system
   setuptools,
@@ -39,6 +40,17 @@ buildPythonPackage rec {
     hash = "sha256-GC0waXxHRiXVXjhTGbet3HvDKmUBKzoufu/J4fmrM+k=";
   };

+  patches = [
+    (fetchpatch2 {
+      name = "opensearch-py-aiohttp-3.10.6-compat.patch";
+      url = "https://github.com/opensearch-project/opensearch-py/commit/0ccd4173cc49b8e15d2c2f1a5fa7459f6e390a86.patch";
+      excludes = [
+        "dev-requirements.txt"
+      ];
+      hash = "sha256-qqS3Nw03q6U3UW1sgQZV3l9SB/Y6Q/RllarTAANgj2A=";
+    })
+  ];
+
   nativeBuildInputs = [ setuptools ];

   propagatedBuildInputs = [

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 352871

x86_64-linux

:fast_forward: 4 packages marked as broken and skipped:

python311Packages.parsedmarc
python311Packages.parsedmarc.dist
python312Packages.parsedmarc
python312Packages.parsedmarc.dist

:x: 2 packages failed to build:

parsedmarc
parsedmarc.dist

:white_check_mark: 8 packages built:

python311Packages.awswrangler
python311Packages.awswrangler.dist
python311Packages.opensearch-py
python311Packages.opensearch-py.dist
python312Packages.awswrangler
python312Packages.awswrangler.dist
python312Packages.opensearch-py
python312Packages.opensearch-py.dist

Why does nixpkgs-review say that parsedmarc both fails and is skipped because it's marked as broken?

Oh, one set is the python package, and the other is top-level package.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 352871

x86_64-linux

:fast_forward: 4 packages marked as broken and skipped:

python311Packages.parsedmarc
python311Packages.parsedmarc.dist
python312Packages.parsedmarc
python312Packages.parsedmarc.dist

:x: 2 packages failed to build:

parsedmarc
parsedmarc.dist

:white_check_mark: 8 packages built:

python311Packages.awswrangler
python311Packages.awswrangler.dist
python311Packages.opensearch-py
python311Packages.opensearch-py.dist
python312Packages.awswrangler
python312Packages.awswrangler.dist
python312Packages.opensearch-py
python312Packages.opensearch-py.dist

Why was this closed? Is this resolved elsewhere?


[#151370733] Block CF pipeline on api-availability-tests failures

What
We will block the CF pipeline (we make the staging pipeline fail so we don't run the production deployment automatically) if the api-availability-tests job fails. We already do this with the app-availability-tests.
We still have some 503 errors when we release gorouter, but we want to ignore these errors if there are only a few, so I modified the tests monitor to handle specific errors as warnings. The tests will still fail if we see more than 5 warnings.
Additional changes:

added a helper script to be able to run the api-availability-tests locally
cd platform-tests/src/platform/availability/api && PIPELINE_TRIGGER_VERSION=0.0.123 ./run_tests_dev.sh
added basic units tests for the monitor which resulted in minor changes/refactors in the monitor code
moved the monitor to a separate package so we can run the unit tests on Travis

Note for the api-availability-tests monitor:

it was not the scope of this task to cover the monitor fully
we can't check for the exact error/warning/etc counts as it's nondeterministic how the monitor behaves if it receives the stop while a result is available - also not the scope of this to try to change this.

How to review
The new monitor functionality is covered in the unit tests, so Travis will cover that.
To test the pipeline:

update your pipeline from this branch
BRANCH=block_pipeline_on_failures_151370733 make dev pipelines
deploy CF and the pipeline should succeed
run a second CF deploy, but shortly after the api-availability-tests started to run remove the instances from the uaa elb and add them back - this will cause several errors and the api-availability-tests should fail

I tested the pipeline with warnings by changing the stemcell for uaa and the gorouter - I don't think there is value in repeating this method as it can take some tries to finally see some errors.
Who can review
Not @bandesz

All comments addressed.


conceptual documentation: readability of the code

We should make a clear statement in the documentation on whether the code kiota generates is meant to be readable by humans or not. And the general philosophy behind it.
@maisarissi to make a decision on the topic before we make changes either way.

I provided my thoughts on the original issue https://github.com/microsoft/kiota/issues/4796#issuecomment-2180743708 but I'll quote it also.

Regarding the topic of readability. While we will always try to make it as readable as possible, there will always be cases where machine-generated code will need to take a path that impacts the readability of the code in favor of better generation. There are areas where the code is pretty much very readable, where it's a bit less because of the constraints. I don't think we should be peeking into the generated code and we should leave it as is. I see this code to be as opaque as can be while offering really easy debugging and learning experience.
So my stance: We should not tax any of our development efforts to make the code "more readable" at the expense of complex solutions.

This is the docs page... go put a PR together now :D

Thanks for pushing @baywet... Thanks for pushing 😂


Retry after HTTP2 GOAWAY errors

Describe the bug
[2023-02-13T11:10:36Z WARN  aws_smithy_client::hyper_ext] unrecognized error from Hyper. If this error should be retried, please file an issue. err=http2 error: connection error received: not a result of an error: connection error received: not a result of an error (hyper::Error(Http2, Error { kind: GoAway(b"", NO_ERROR, Remote) }))
Expected Behavior
Should retry
Current Behavior
Error out
Reproduction Steps
none
Possible Solution
No response
Additional Information/Context
No response
Version
aws-smithy-client = "0.54.2"

Environment details (OS name and version, etc.)
docker ubuntu:latest
Logs
[2023-02-13T11:10:36Z WARN  aws_smithy_client::hyper_ext] unrecognized error from Hyper. If this error should be retried, please file an issue. err=http2 error: connection error received: not a result of an error: connection error received: not a result of an error (hyper::Error(Http2, Error { kind: GoAway(b"", NO_ERROR, Remote) }))

https://github.com/golang/go/issues/18639

@ozgrakkurt Thanks for submitting this. Are you actually encountering this error in production right now?

Hey, yes. We get this regularly

I can reproduce it easily with the following code.
Note that 60s sleep duration between calls is important (there was no issue with 10s sleep).
Version
aws-config = "0.55.1"
aws-sdk-cognitoidentityprovider = "0.26.0"
use aws_sdk_cognitoidentityprovider as cognitoidentityprovider;
use tokio;
use std::thread;
use std::time::Duration;

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let config = aws_config::load_from_env().await;
    let client = cognitoidentityprovider::Client::new(&config);
    let username = "fake_user";
    let password ="fake_password";
    let client_id = "fake_client";
    for i in 1..4 {
        let cognito_result = client.initiate_auth().set_client_id(Some(client_id.to_string())).auth_flow(cognitoidentityprovider::types::AuthFlowType::UserPasswordAuth).auth_parameters("USERNAME", username).auth_parameters("PASSWORD", password).send().await;
        println!("Result {:}: {:?}", i, cognito_result);
        thread::sleep(Duration::from_secs(60));
    }
}

Results:

✅ Err(ServiceError(ServiceError { ... { message: Some("User pool client fake_client does not exist.")
❌ Err(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: hyper::Error(Http2, Error { kind: GoAway(b"", NO_ERROR, Remote) }), connection: Unknown } }))
✅ Err(ServiceError(ServiceError { ... { message: Some("User pool client fake_client does not exist.")

ah perfect! thanks for the reproducer. it's a quick fix but I wasn't able to reproduce it in tests so I wasn't able to figure out if my fix actually worked. We'll get this prioritized and fixed—or we're happy to accept a PR, the relevant code is here:
https://github.com/awslabs/smithy-rs/blob/main/rust-runtime/aws-smithy-client/src/hyper_ext.rs#LL207C1-L224C2
If we correctly classify that error is kind: Io it should be fixed

I get a similar error when trying to invoke dynamodb in Rust AWS SDK:
DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: NoMatchingAuthScheme, connection: Unknown } })
Not sure what this error means...

This is a different error—no matching auth scheme. Can you open a separate
issue and include some logs?
On Tue, Sep 12, 2023, 11:04 AM Harrison Kaiser @.***>
wrote:

I get a similar error when trying to invoke dynamodb in Rust AWS SDK:
DispatchFailure(DispatchFailure { source: ConnectorError { kind:
Other(None), source: NoMatchingAuthScheme, connection: Unknown } })
Not sure what this error means...
—
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-sdk-rust/issues/738#issuecomment-1715903047,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADYKZ4SEETLOGKTYJ5O76DX2B2XRANCNFSM6AAAAAAU2EGWYA
.
You are receiving this because you commented.Message ID:
@.***>

It looks like the fix for this was merged but then accidentally removed as part of a merge. We'll get a fix out soon

The fix has been merged and will go out in a future release: https://github.com/smithy-lang/smithy-rs/pull/3250

This fix for this went out in the December 5, 2023 release.


story(sbb-pearl-chain-vertical): initial implementation

Feature Description
Feature Description
We want to provide the sbb-pearl-chain-vertical component for the second and third level
Figma Spec
Design Spec

same as the horizontal sbb-pearl-chain. Except -> not-part-of-the-journey leg

Technical Spec

departure and arrival time is not optional

interface:
type legConfig: {
 leg: LegDetail[]
}

markup:
 <sbb-pearl-chain leg-config={} />

Definition of Done

[ ] Component is implemented
[ ] Documentation is provided
[ ] Tests are implemented
[ ] E2E Tests are implemented
[ ] Storybook stories are implemented
[ ] Navigation via keyboard is tested
[ ] Screen reader output is tested
[ ] High-contrast is tested
[ ] Remaining accessibility is tested
[ ] UX approved

Will be tracked/decided in #1901


CanExecute in non-reactive binding does not work.

Describe the bug
CanExecute method does not work when binding a Button's command directly with ViewModel's method. I am following this tutorial:
https://docs.avaloniaui.net/docs/guides/data-binding/how-to-bind-to-a-command-without-reactiveui
View:
<UserControl xmlns="https://github.com/avaloniaui"
             xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
             xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
             xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
             xmlns:vm="clr-namespace:TestApp.ViewModels.Settings"
             mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
             x:Class="TestApp.Views.TestView"
             x:DataType="vm:TestViewModel">
    <StackPanel>
        <ComboBox ItemsSource="{Binding Values}" SelectedItem="{Binding Value}" />
        <Button Command="{Binding Save}">Save</Button>
    </StackPanel>
</UserControl>

ViewModel:
public sealed class TestViewModel : ViewModelBase
{
    public void Save(object _)
    {
        //
    }

    public bool CanSave(object _)
    {
        return Value > 0;
    }

    public int[] Values { get; } = [0, 10, 20, 40];

    private int _value;
    public int Value
    {
        get => _value;
        set => this.RaiseAndSetIfChanged(ref _value, value);
    }
}

Expected behavior
The Button must be enabled when selected dropdown value is more than 0. CanSave method remains uncalled.
Screenshots

Environment

OS: Windows 2019
Avalonia-Version: 11.0.7

You missing to read this.
For trigger it, you should add code like this in your vm
public sealed class TestViewModel : ViewModelBase
{
    public void Save(object _)
    {
        //
    }

    [DependsOn(nameof(Value))]
    public bool CanSave(object _)
    {
        return Value > 0;
    }

    public int[] Values { get; } = [0, 10, 20, 40];

    private int _value;
    public int Value
    {
        get => _value;
        set => this.RaiseAndSetIfChanged(ref _value, value);
    }
}

Yes, DependsOn attribute is mandatory, when you bind to methods. Otherwise, there is no way for Avalonia to know if CanSave should be executed again.

@workgroupengineering @maxkatz6
Thanks for the help, guys. Sorry for the trouble.
Just tested, and found that for use of multiple values in Can method, we can add more DependsOn attribute.


add parents and lineage methods

I have a subclass of MimeMagic that I use that has a handful of useful methods:

.binary? will sample an input to determine if it's binary (or text)
.default_type will return application/octet-stream if the input is binary, text/plain otherwise
#parents will return the list of a type's immediate parents
#lineage will return a type's full list of ancestors

I also make a cursory affordance for type parameters in the constructor and inspect methods.
I'm filing this issue rather than coming in hot with a PR because I also change the behaviour of .by_magic and .all_by_magic, such that they will always return a value (the value of .default_type); I figured that merited some discussion. I'm happy to do the patch that addresses this and another issue.

Update: I did the patch.


Improve situation when new spacy version gets published

It seems recently a new spacy version got published (3.5), without there yet being spacy models for this version.
What happens in a feedstock where I have a test-dependency on space-models-en_core_web_sm is that now it pulls in a really old build:
    spacy:                      3.5.0-py38h507a481_0        conda-forge
    spacy-legacy:               3.0.12-pyhd8ed1ab_0         conda-forge
    spacy-loggers:              1.0.4-pyhd8ed1ab_0          conda-forge
    spacy-model-en_core_web_sm: 2.3.1-pyh9f0ad1d_0          conda-forge

that then fails pip check:
+ pip check
en-core-web-sm 2.3.1 has requirement spacy<2.4.0,>=2.3.0, but you have spacy 3.5.0.

It seems this is a question of the metadata of the old spacy-model-* builds having bad metadata (not depending hard enough on a given spacy version), so the resolver believes it found a non-conflicting solution, while keeping the newest spacy...
It seem to me that we should patch the old metadata; additionally/ideally, the faster spacy-models-* follows a new spacy version, the better.

Yep patching the old metadata would work. I'll work on getting the new one out, in the meantime, but there's only so much we can do...

I'll work on getting the new one out, in the meantime, but there's only so much we can do...

Yeah, this is not to stress you here in getting a new model version out, but to ensure that during that time-period, we don't produce broken environments.

It might be easiest to just patch in a global spacy<3 cap to all spacy-model-* 2.x packages.

Ok, #6 is merged.
Also ran into:

https://github.com/conda-forge/transformers-feedstock/issues/106#issuecomment-1413844866
expanded https://github.com/conda-forge/spacy-models-feedstock/issues/2

@h-vetinari Are you going to make the repodata patches PR? I haven't done that before.

@h-vetinari Are you going to make the repodata patches PR? I haven't done that before.

Opened https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/396 for you. Now you know how to do it! :)


Can you tell how the datasets are preprocessed?

Hi, is it possible to share the preprocessing settings steps for each dataset? E.g. What was the threshold for removing frequent/infrequent words?

Hi,
each dataset folder contains a "metadata.json" file, with the preprocessing details (see the field preprocessing-info). For example BBC news dataset.
As far as I remember, we selected those values by iteratively trying different values and inspecting the resulting topics.

Thanks, Silviatti!


carthage build

$carthage update --platform macOS
results in:
*** Skipped building Then due to the error:
Dependency "Then" has no shared framework schemes for any of the platforms: Mac
If you believe this to be an error, please file an issue with the maintainers at https://github.com/devxoul/Then/issues/new
Using carthage 0.34, Swift 5, xcode 10.2

i get the same error

same error

same error


Distinguish Official OSM forum and Discourse in general

for example https://github.com/matkoniecz/osm-community-index/blob/f45f979f26dbbe679b9094e2029d3e2a86b4cca9/resources/europe/poland/OSM-PL-discourse.json has   "type": "discourse",
maybe   "type": "official_osm_forum", would be better? The primary info is that it is OSMF run space, where you can log in with your OSM account and so on, right?
Not the implementation detail which software powers it.
And of community (for example in Iran, I think) is using Discourse forum setup somewhere separately then it should be shown differently...

I really don't want to do any additional development to highlight the "official OSM forum run by the OSMF".  If this is information that is important to the OSMF, can you just update the description / extendedDescription properties to include this detail?


Describe the bug
test
URL of the page where the issue is observed.
test
Steps To Reproduce
test
Expected Behavior
test
Screenshots/Videos
No response
What device are you using?
Desktop
Operating System
Windows
What browsers are you seeing the problem on?
No response
Browser version
No response
Additional context
No response
Tips for developers
Before addressing the bug, please identify which PR caused the issue (you can follow the steps here). If you identify the PR, comment on the issue with a link to it. If not, mention the commit hash of the oldest commit you saw the bug on (and the month and year it was made in).
Also, if this is your first issue, please make sure to follow https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia#choosing-a-good-first-issue and https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia#setting-things-up before claiming it. Thanks!

test

test

test

test

oppiabot rqri

test

test

oppiabot rqri

oppiabot rqri

oppiabot rqri


Improve logs when resolving a secret value

Minor improvement in the logs, the previous output is:
Decrypted Env [VAULT_ADDR:https://theknowledge.nitro.us:8200, path: EXACT_TARGET_CLIENT_SECRET]
The new output:
Decrypted DB_USER_PASS  [VAULT_ADDR: https://theknowledge.nitro.us:8200, path: vault://secret/db_pass]

:shipit:


Stack install failing

Bindings-GLFW won't compile on my macOS 10.11.6: fatal error: OpenGL/gl.h: No such file or directory
Any ideas how I can resolve this issue?
Context:
$ stack install bindings-GLFW
Warning: File listed in .stack-work/downloaded/0pCfpnxANlB-/freetype2.cabal file does not exist: objs/ftmodule.h
Warning: File listed in .stack-work/downloaded/F_GRW2V-q8oW/nodejs-exec.cabal file does not exist: node/node
WARNING: Ignoring out of range dependency (allow-newer enabled): template-haskell-<IP_ADDRESS>. bindings-GLFW requires: >=2.10 && <2.12
bindings-GLFW-<IP_ADDRESS>: configure
bindings-GLFW-<IP_ADDRESS>: build

--  While building package bindings-GLFW-<IP_ADDRESS> using:
      /Users/vilem/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_<IP_ADDRESS>_ghc-8.2.1 --builddir=.stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS> build --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
    Logs have been written to: /Users/vilem/code/github/lamdu/.stack-work/logs/bindings-GLFW-<IP_ADDRESS>.log

    Configuring bindings-GLFW-<IP_ADDRESS>...
    Preprocessing library for bindings-GLFW-<IP_ADDRESS>..
    In file included from GLFW.hsc:9:0:
    /private/var/folders/12/1kppc1y16nzfg7ntgz9pyvfc0000gn/T/stack15987/bindings-GLFW-<IP_ADDRESS>/glfw/include/GLFW/glfw3.h:156:25: fatal error: OpenGL/gl.h: No such file or directory
    compilation terminated.
    compiling .stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/Bindings/GLFW_hsc_make.c failed (exit code 1)

BTW, also tried cloning and installing from master, but I get the same issue:
$ stack install
[1 of 2] Compiling Main             ( /Users/vilem/.stack/setup-exe-src/setup-mPHDZzAJ.hs, /Users/vilem/.stack/setup-exe-src/setup-mPHDZzAJ.o )
[2 of 2] Compiling StackSetupShim   ( /Users/vilem/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /Users/vilem/.stack/setup-exe-src/setup-shim-mPHDZzAJ.o )
Linking /Users/vilem/.stack/setup-exe-cache/x86_64-osx/tmp-Cabal-simple_mPHDZzAJ_<IP_ADDRESS>_ghc-7.10.2 ...
bindings-DSL-1.0.22: download
bindings-DSL-1.0.22: configure
bindings-DSL-1.0.22: build
bindings-DSL-1.0.22: copy/register
bindings-GLFW-<IP_ADDRESS>: configure (lib)
Configuring bindings-GLFW-<IP_ADDRESS>...
bindings-GLFW-<IP_ADDRESS>: build (lib)
Preprocessing library bindings-GLFW-<IP_ADDRESS>...
In file included from GLFW.hsc:9:0:
/Users/vilem/code/github/bindings-GLFW/glfw/include/GLFW/glfw3.h:156:25: fatal error: OpenGL/gl.h: No such file or directory
compilation terminated.
compiling .stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/Bindings/GLFW_hsc_make.c failed (exit code 1)

Do you have the "Xcode command line tools" installed? (xcode-select --install)

Yup.

I even tried updating Xcode.

Hmm.. What about glfw? Do its examples successfully build on your machine?

Yes.

Hmm, what's version of stack do you have installed?

I had Stack 1.4.0, but upgraded to 1.5.1 now but it still won't install:
$ stack install
--  While building package digest-<IP_ADDRESS> using:
      /Users/vilem/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_<IP_ADDRESS>_ghc-8.2.1 --builddir=.stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS> configure --with-ghc=/Users/vilem/.stack/programs/x86_64-osx/ghc-8.2.1/bin/ghc --with-ghc-pkg=/Users/vilem/.stack/programs/x86_64-osx/ghc-8.2.1/bin/ghc-pkg --user --package-db=clear --package-db=global --package-db=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/pkgdb --libdir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/lib --bindir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/bin --datadir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/share --libexecdir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/libexec --sysconfdir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/etc --docdir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/doc/digest-<IP_ADDRESS> --htmldir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/doc/digest-<IP_ADDRESS> --haddockdir=/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/doc/digest-<IP_ADDRESS> --dependency=base=base-<IP_ADDRESS> --dependency=bytestring=bytestring-<IP_ADDRESS> -f-bytestring-in-base --exact-configuration
    Process exited with code: ExitFailure 1
    Logs have been written to: /Users/vilem/code/github/lamdu/.stack-work/logs/digest-<IP_ADDRESS>.log

    Configuring digest-<IP_ADDRESS>...
    Cabal-simple_mPHDZzAJ_<IP_ADDRESS>_ghc-8.2.1: Missing dependency on a foreign
    library:
    * Missing (or bad) header file: zlib.h
    This problem can usually be solved by installing the system package that
    provides this library (you may need the "-dev" version). If the library is
    already installed but in a non-standard location then you can use the flags
    --extra-include-dirs= and --extra-lib-dirs= to specify where it is.
    If the header file does exist, it may contain errors that are caught by the C
    compiler at the preprocessing stage. In this case you can re-run configure
    with the verbosity flag -v3 to see the error messages.

--  While building package bindings-GLFW-<IP_ADDRESS> using:
      /Users/vilem/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_<IP_ADDRESS>_ghc-8.2.1 --builddir=.stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS> build --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
    Logs have been written to: /Users/vilem/code/github/lamdu/.stack-work/logs/bindings-GLFW-<IP_ADDRESS>.log

    Configuring bindings-GLFW-<IP_ADDRESS>...
    Preprocessing library for bindings-GLFW-<IP_ADDRESS>..
    /private/var/folders/12/1kppc1y16nzfg7ntgz9pyvfc0000gn/T/stack5873/bindings-GLFW-<IP_ADDRESS>/In file included from GLFW.hsc:9:0:
    /private/var/folders/12/1kppc1y16nzfg7ntgz9pyvfc0000gn/T/stack5873/bindings-GLFW-<IP_ADDRESS>/glfw/include/GLFW/glfw3.h:156:25: fatal error: OpenGL/gl.h: No such file or directory
    compilation terminated.
    compiling .stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/Bindings/GLFW_hsc_make.c failed (exit code 1)
    command was: /Users/vilem/.nix-profile/bin/gcc -c .stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/Bindings/GLFW_hsc_make.c -o .stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/Bindings/GLFW_hsc_make.o -m64 -fno-stack-protector -m64 -fno-stack-protector -m64 -D__GLASGOW_HASKELL__=802 -Ddarwin_BUILD_OS=1 -Dx86_64_BUILD_ARCH=1 -Ddarwin_HOST_OS=1 -Dx86_64_HOST_ARCH=1 -Iglfw/include/GLFW -Iglfw/include/ -Iglfw/src -Iglfw/include/os/darwin -D_GLFW_USE_CONFIG_H -I.stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/autogen -I.stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/global-autogen -include .stack-work/dist/x86_64-osx/Cabal-<IP_ADDRESS>/build/autogen/cabal_macros.h -I/Users/vilem/.stack/snapshots/x86_64-osx/nightly-2017-08-10/8.2.1/lib/x86_64-osx-ghc-8.2.1/bindings-DSL-1.0.23-GnirsnGaJOzLEFMHVhMFIe/include -I/Users/vilem/.stack/programs/x86_64-osx/ghc-8.2.1/lib/ghc-8.2.1/base-<IP_ADDRESS>/include -I/Users/vilem/.stack/programs/x86_64-osx/ghc-8.2.1/lib/ghc-8.2.1/integer-gmp-<IP_ADDRESS>/include -I/Users/vilem/.stack/programs/x86_64-osx/ghc-8.2.1/lib/ghc-8.2.1/include -I/Users/vilem/.stack/programs/x86_64-osx/ghc-8.2.1/lib/ghc-8.2.1/include/

BTW, I got it running fine on a friend's mac, but I did install everything from scratch (it didn't even have brew). Some state is messed up somewhere. Maybe I should consider a clean install of my OS? 😭

Maybe brew doctor could notice something wrong? I'd further investigate why the digest package doesn't build because that seems like a smaller scope of problems which affect more people..

I've been working on OS X for a while without problems. I'm going to close this. If the problem persists feel free to reopen with additional details.

Apologies I had forgotten about this. Thanks for closing.


Add OpenAI component

Contributes to #4567
Microsoft Reviewers: Open in CodeFlow

Is there also work to enable Aspire.Azure.AI.OpenAI to work with just an OpenAIClient when configured to talk to OpenAI or a local emulator like Ollama? Or do you see that as a separate PR?

Failing on unrelated cosmos tests

/azp run


Hopes  for  your search  code , I  think it  will  be  very  useful  to this world

Hopes too! I think it will be very useful, and I really want to try if this algorithm work to my project.
Is the author currently open-source search code now? I skimmed through the code and not found.

It will be released within one or two week! I'm cleaning up some of codes now...

@monkeyDemon @heartInsert Released. I tested all training codes and released trained model checkpoints. But note that I didn't tried to rerun the search phase due to our gpu resources.
Closing this issue but if you have any question, please let me know with new issue.


Stable Baselines 3's DummyVecEnv Reset ->  too many values to unpack (expected 2)

Hi guys. I try to use examples notebooks in the repo for training. Both on enviroment reset functions and agents' step I get this error about Stable Baselines 3's DummyVecEnv Reset:
.venv/lib/python3.10/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py:77, in DummyVecEnv.reset(self)
     75 print("Env #",self.num_envs)
     76 for env_idx in range(self.num_envs):
---> 77     obs, self.reset_infos[env_idx] = self.envs[env_idx].reset(seed=self._seeds[env_idx])
     78     self._save_obs(env_idx, obs)
     79 # Seeds are only used once

ValueError: too many values to unpack (expected 2)

My python is 3.10, stable-baselines3's Version is 2.0.0 and finrl's version is lastest (0.3.6)
How can I fix it?

Hello! I'm here to assist you with the issue you're facing. The error you're encountering with Stable Baselines 3's DummyVecEnv Reset function is a common issue that is often related to the Python version being used.
Stable Baselines 3 version 2.0.0 may not be fully compatible with Python 3.10. Therefore, to resolve the error, it would be recommended to downgrade your Python version to a more compatible version such as 3.8 or 3.9.
Here are the steps you can follow:

First, delete or deactivate your current Python 3.10 virtual environment. Then, create a new virtual environment with a lower Python version.

For example, install Python 3.8 or 3.9:

python3.8 -m venv myenv

or
python3.9 -m venv myenv

Activate the new virtual environment:

source myenv/bin/activate

Install finrl and stable-baselines3 in your new virtual environment:

pip install finrl stable-baselines3

By creating a new virtual environment with a more compatible Python version, you should be able to resolve the issue. You can then try running the example notebooks in FinRL.
I hope this solution helps! If you have any further questions, please feel free to ask.

Facing the same issue.

Facing the same issue

Facing this issue as well. my reset method returns a 3D NdArray if that is useful.

How can we solve this?


Allow attribute deletion without a value

Currently it seems that once an attribute is created, it can only be set to null, or - if it is a set/array - a value can be deleted from it.
With this change, the attribute appears to be deleteable entirely.

Looks good. Thank you for this improvement.


Make gRPC service structs forward-compatible

Description
gRPC code generation generates Unimplemented... structs that return an unimplemented status code for all methods. This struct should be embedded in all gRPC service implementations. While this is not currently enforced, future gRPC code generator versions will make this mandatory (this behavior can be disabled, but it is not recommended; see here for a lengthy discussion on this matter with the clear verdict that gRPC authors see embedding of Unimplemented.. as something that should always have been required).
Alas, this makes service type assertions ineffective, but that seems to be the pill to swallow.
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[x] ~Evaluated and added CHANGELOG entry if required~
[x] ~Determined and documented upgrade steps~
[x] ~Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)~

If any of these don't apply, please comment below.
Testing Performed

CI

How do make sure none of the services are missed? Is there a way to unit test it?

We could write a unit test or linter. But because a soon-to-be-upgraded-to version of the proto library will enforce this via the type system, this seems unnecessary.

Test failure is not related.

/retest


hotfix: fix reference cycle caused by print

PR Checklist
PR Structure

[x] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[x] This PR's title starts with name of package that is most changed in the PR, ex.
paymentservice.stellar, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[x] This PR adds tests for the most critical parts of the new functionality or fixes.

What
Fix reference cycle caused by print.
Why
For some reason, when we printed the request, it caused a reference cycle that went on until the computer memory is over, to finally throw a 500.

Reference Server Preview is available here:https://anchor-ref-pr330.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr330.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr330.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr330.previews.kube001.services.stellar-ops.com/


Serving plain text file using a default label is not working in Finchley M9

Using a config server at version Finchley.M9 with the following properties
spring:
  application:
    name: testapp
  profiles:
    active: native
  cloud:
    config:
      server:
        default-label: default-label-test

and config directory layout src/main/resources/config/default-label-test/default-label-nginx.conf
When I request the contents of  default-label-nginx.conf with a request
curl http://localhost:9999/testapp/native/default-label-nginx.conf\?useDefaultLabel, the mapped controller is https://github.com/spring-cloud/spring-cloud-config/blob/master/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/EnvironmentController.java#L99
but should be
https://github.com/spring-cloud/spring-cloud-config/blob/master/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/resource/ResourceController.java#L81
Using Edgware.SR2 the correct controller method is mapped
@snicoll mentioned that request mapping has changed in Boot 2, see https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.0-Migration-Guide#spring-mvc-path-matching-default-behavior-change

hey @ojhughes - can we close it?

Let's document this as a note.