Tie Preview Resolution to Picture/Video Resolution

Samsung Galaxy Ace 3 RFC video recordings are fine, previews start fine, but then aspect ratio gets screwed up once recording begins. Both portrait and landscape.

Determine the picture/video resolution first. Then, take that into account when choosing a preview resolution. We want a preview resolution that is close to the picture/video resolution.

See this comment and this comment for other concrete cases for this.


mount.drvfs fails to mount virtualbox shares

Hi,
mounting any "native" drive with a command like the following works fine:
mount -t drvfs D: /files

However I run Windows 10 in Virtual Box and want to mount a shared Folder, it will mount using the above command without any errors.
But then no access is possible:
ll /files/
ls: reading directory '/files/': Invalid argument
total 0

How would I be able to access that share from within the Ubuntu shell?
Is there another way of doing it or is it a bug in mount.drvfs?
(I also tried installing guest addtions on WSL and mount -t vboxsf ... but as I expected, that won't work either)
EDIT:
Actually I can cd into a subdirectory at that mount.
But still can't list the content using ls also echo * doesn't seem to work it just echos a literal '*'.
Issuing a git status inside a git repo within that share will show every file as changed.
This doesn't seem to provide any useful functionality.

This seems very strange to me.
I've already mounted the virtualbox share onto the Windows 10 VM via the Windows vboxsf driver (installed with guest additions) - I can see it as "Z:" and see all of the files within. Given that the files are already available to Windows, why should WSL need it's own virtualbox driver in order to see the files ? This approach would mean that WSL is going to need it's own driver for every single filesystem type, even though Windows can already see the files.

Same problem here. Windows 10 is my guest operating System. I'm running WIndows 10 on top of VirtualBox. I have a VirtualBox shared folder (from my host operation system) available as \\vboxsrv\jgr mounted as G:.
On Windows 10 bash:
jgr@berlin:~$ sudo mkdir /mnt/g
jgr@berlin:~$ sudo mount -t drvfs '\\vboxsrv\jgr' /mnt/g
jgr@berlin:~$ ls -l /mnt/g
ls: reading directory '/mnt/g': Invalid argument
total 0
jgr@berlin:~$ sudo umount /mnt/g
jgr@berlin:~$ sudo mount -t drvfs G: /mnt/g
jgr@berlin:~$ ls -l /mnt/g
ls: reading directory '/mnt/g': Invalid argument
total 0
jgr@berlin:~$

Here is a workaround:

Share the folder you want to access in WSL from you Mac via Preferences -> Sharing -> Share Using SMB.
Connect to the share using Explorer (probably easiest to map a network drive and remember the credentials).
From the WSL terminal, sudo mkdir /mnt/h
From the WSL terminal, sudo mount -t drvfs H: /mnt/h

Hello @therealkenc, are there any plans to support this in the near future? Having to share the folder from the host is a bit cumbersome.

I don't work at MSFT and don't know the answer to that anymore than anyone else here (and even if I did, the WSL team doesn't provide roadmaps). Actually I submitted #2163 not because I have any particular desire to mount WebDAV drives (I really don't), but in the hopes someone would take a look and make WSL more tolerant of various installable filesystem drivers (starting with closed source filesystems written at MSFT).
Another way to attack this would be to take a look at the VirtualBox guest addition driver, debug what is going wrong, and add support for WSL. Doable.

Cygwin mounts virtualbox shares without doing anything.  Why is WSL struggling with this?

On WSL2, I would guess that this is caused by the same phenomena as dokan-dev/dokany#932.

Update from #6556, the mount succeeds on WSL1 19042.746, but still no list stuff. That supposition above from Reid just above is almost certainly correct. Ref #6063.

Cygwin mounts virtualbox shares without doing anything. Why is WSL struggling with this?

Well, it's been a year, and apparently this is still an issue.  I'd like to hear from a Microsoft employee when they plan on addressing this issue.


[1.13] Bump Mesos modules

High-level description
A fix of bug in the Mesos overlay module that leads to on-disk state inconsistency.
Corresponding DC/OS tickets

DCOS_OSS-5626 Reserve all agent VTEP IPs upon recovering from replicated log.

Checklist for component/package updates:

[x] Change log from the last version integrated (this should be a link to commits for easy verification and review): diff
[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain.
[x] Test Results: job
[ ] Code Coverage: none

@mesosphere-mergebot label Ready For Review

@mesosphere-mergebot changelog-not-required OSS only


PMID:30181366

resistance to fluconazole
resistance to phenylglyoxal
normal growth on 5-fluoruracil
normal growth on phenylglyoxal
decreased protein localization to mitochondrion, with protein mislocalized to cytoplasmic foci
sensitive to phenylglyoxal

resistance to fluconazole FYPO:0007033
resistance to phenylglyoxal FYPO:0007034
normal growth on 5-fluoruracil FYPO:0007035
normal growth on phenylglyoxal FYPO:0007036
sensitive to phenylglyoxal FYPO:0007037
decreased protein localization to mitochondrion FYPO:0007038
decreased protein localization to mitochondrion, with protein mislocalized to cytoplasmic foci FYPO:0007039
edit file: 297ab0266a7d97d6a901acee9c71471270e75c25
release: b4f38ea870fee7bdc466ce0bde9f5fcc4e336356


SArray.filter_by should support a lambda function

Currently, SArray.filter_by has the following signature:
Parameters
----------
values : SArray | list | numpy.ndarray | pandas.Series | str
The values to use to filter the SArray. The resulting SArray will
only include rows that have one of these values in the given
column.

exclude : bool
If True, the result SArray will contain all rows EXCEPT those that
have one of the ``values``.

Returns
-------
out : SArray
The filtered SArray.

It would be nice for the supported values to also include a lambda function to replicate the Python filter behavior.

I'm not convinced it makes sense to make this change. It's basically just adding a second way of the doing what filter already does. That seems to go against the Python ethos of "There should be one-- and preferably only one --obvious way to do it.".
@shantanuchhabra - what do you think?

I didn't know SArray had a filter method. Closing this issue since the filter method already addresses this.


code cleanup for merge

Next thing on the list to do is to merge back to the main branch and then round thing up. maybe do #20 before actually calling it done but either way the merge should be the next thing to do. There is a lot of leftover code thats not used anymore so before we merge back all of that needs to be cleaned up.

Will merge first before doing #20 because #20 deserves its own branch

#23


Error running on exported onnx model

The export of ckpt file to onnx model resulted in an error message, as shown below：
[ErrorCode:NotImplemented] Could not find an implementation for Trilu(14) node with name '/Trilu'"
System information：
onnxruntime:1.13.1
opset_version=14
torch   2.2.1+cu118

I changed dtype=torch.bool to torch.float and success https://github.com/baudm/parseq/blob/1902db043c029a7e03a3818c616c06600af574be/strhub/models/parseq/model.py#L117


Avoid hard-coded "Segoe MDL2 Assets"

Use indirect {ThemeResource SymbolThemeFontFamily} for font family setting

/azp run

/azp run

This will help the transition over to Fluent UI System Icons

This will help the transition over to Fluent UI System Icons

It's great to see this done. Can you link it to the issue I created for this though? It should be closed now.
I never heard anything back from Microsoft on this either way.
https://github.com/microsoft/microsoft-ui-xaml/issues/3745

It's great to see this done. Can you link it to the issue I created for this though? It should be closed now.
I never heard anything back from Microsoft on this either way.
https://github.com/microsoft/microsoft-ui-xaml/issues/3745


delete previous ekco versions, and only keep one version in the future

What this PR does / why we need it:
This removes all previous ekco versions, as they are no longer installable.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Steps to reproduce

Does this PR introduce a user-facing change?

Does this PR require documentation?

I don't care if the PR license scan fails here because this is a removal of code, not adding/changing dependencies


Initial layout

Description
Adds a "placeholder" layout for the plugin, which consists of an exercise list window on the right, and a A+-menu top toolbar button.
The layout can be inspected using gradlew runIde. No unit tests yet, as we still need to decide how to test the GUI of the plugin.
The sonar scanner fails due to a configuration issue unrelated to this PR.
Testing

[ ] I created new unit tests
[ ] All unit tests pass
[x] I have tested manually

Have you updated the README or other relevant documentation?

[ ] Yes
[ ] Not yet. I will do it next.
[x] Not relevant

I was worried that the build would fail because of the lack of unit tests, but it seems that since there are no unit tests at all, code coverage verification is skipped, which is... convenient? ... in this case. :)
line 540 of PR build output
Task :jacocoTestCoverageVerification SKIPPED

Instead, it's SonarCloud which fails the build.

but could you please check why travis is not building and why codacy is complaining

As mentioned above, the travis build fails due to configuration issue:
org.sonar.java.AnalysisException: Please provide compiled classes of your project with sonar.java.binaries property

Codacy complains about AboutDialog not being named AboutDialogUtil, since it only has static methods, which I can fix.

Codacy also complains about an unused private field, which isn't actually unused due to the Swing GUI form, I'll see if theres some way around that.


Use Fetch API instead of node.js http

It appears that the HttpSoftAP.js uses the node.js "http" module. However, this http module is not supported by React Native, leading to the popular tool rn-nodeify. Among other things, in a react-native project, rn-nodeify will install react-native-http, a react-native compatible library that attempts to provide the same http API but in a module that is compatible with react-native. React-Native-HTTP is poorly maintained, so it does not include methods like req.setTimeout which is used in this particularly library making it incompatible.
Facebook uses the Fetch API for maximum compatibility. You can read more about it here.  Browsers now also implement this API with only older browsers not supporting it.
The world is converging on fetch..
Cross-Fetch is a library that makes using fetch across all 3 simple. It is certainly nice that softap-setup-js does not have any real dependencies, however, using cross-fetch would make it a lot easier to incorporate this library into react-native. Can we please consider using cross-fetch instead of HTTP so we can have the cross compatibility this library deserves?

Is anyone following up on this? I'm working on a react-native application that needs this, and would be able to contribute.


WorkItemMigrator Azure pipelines artifact is missing a dependency

Describe the bug
Latest version of Work Item Importer (master branch) crashes because it is missing a required assembly when build with the included azure-pipelines.yaml
The error is:
Work Item Importer v2.3.3
Copyright (c) Solidify
Wow, there's a newer release out (v2.3.117)! We recommend downloading it for latest features and fixes.
[W][15:10:40] User mapping file 'C:\temp\Jira\WorkItemMigrator-latest\Output\users.txt' not found, ignoring mapping user identities.
[I][15:10:41] Connecting to Azure DevOps/TFS...
[E][15:10:42] Unexpected migration error.
[System.IO.FileNotFoundException] System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.Services.Client.Interactive, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.Services.Client.Interactive, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at WorkItemImport.WitClientWrapper..ctor(String collectionUri, String project)
   at WorkItemImport.Agent.Initialize(MigrationContext context, Settings settings)
   at WorkItemImport.ImportCommandLine.ExecuteMigration(CommandOption token, CommandOption url, CommandOption configFile, Boolean forceFresh, CommandOption continueOnCritical)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
:
   at WorkItemImport.WitClientWrapper..ctor(String collectionUri, String project)
   at WorkItemImport.Agent.Initialize(MigrationContext context, Settings settings)
   at WorkItemImport.ImportCommandLine.ExecuteMigration(CommandOption token, CommandOption url, CommandOption configFile, Boolean forceFresh, CommandOption continueOnCritical)
[I][15:10:42] Import complete. Imported 0 items, 0 revisions (1 errors, 1 warnings) in 00:00:01.

The version in this log may be ignored, I used the included azure-pipelines.yaml to build the master branch.
To Reproduce
Steps to reproduce the behavior:

Build the jira-azuredevops-migrator with the azure-pipelines.yaml
Download the buildartifact
Extract the buildartifact
Run the Work Item Importer
See the error

Hi Thomas! Good find :) Does your PR solve the issue?
If you find yourself waiting, you can always use the latest stable Release here: https://github.com/solidify/jira-azuredevops-migrator/releases/tag/v2.3.117
If you opt for going with the latest version on the master branch, please report any findings through the GitHub issues. We would love to get it tested more thoroughly :)

This fixed the error, yes. However I've since learned that Microsoft.VisualStudio.Services.Client was used because the WitClientWrapper.cs was not using the configured PAT. So I'm closing this in favor of issue #515 with PR #516

@Alexander-Hjelm just a question regarding your comment. The latest release is 2 years old at this time. Will I miss a lot of functionality\bugfixes when I go with this release compared to the master branch? What would you say is the main difference?

The only difference is a couple of bugfixes related to rendering HTML fields and images. If the migrator is running smoothly for you with in 2.3.117, then likely the master branch will work just as well.


Repo manage section

Changes Proposed
Solved the conflict

Created repository manage section

Check List

[X] The title of my pull request is a short description of the changes
[X] This PR relates to some issue:
[X] I have documented the changes made (if applicable)
[ ] I have covered the changes with unit tests

@chad11111 have a look

it is working fine on my system kindly check if you have pulled this branch as i have created a new branch and closed the pull request for the repo_manage_section branch

@chad11111 have a look

@chad11111 have a look

@aman1376 is this pr for https://github.com/komodorio/helm-dashboard/issues/287?

@chad11111 it's for #278 its similar to 287 but there is slight difference as in 287 on click of button there should be a dialogue box asking for input.

@chad11111 it's for #278 its similar to 287 but there is slight difference as in 287 on click of button there should be a dialogue box asking for input.

@aman1376 U need that pop-up bar here too

@aman1376 This Pr should be for #287
thanks a lot!

@chad11111 what about this pr

in this pr we just needed to show how the repository manage section would look like and in #287 we need to add repositories to the list of repositories that we have.

@chad11111 have a look

@undera please have a lok

@chad11111


fix : Ensure scraper does not fetch bot events

Proposed Changes
Fix: #554
Description
i have just filter out the [bot] slug file name while picking it  in first rendering .
Reviewer(s)
@rithviknishad

although this solves the side effect, this does not solve the root cause. we shouldn't be storing events of bots in the first place.

do you want me to check that scenario as well ??  or you are opening a new issue for that ??

nope it's the same issue. and only check for that scenario.
data repo migration to remove bot's data will be handled separately.

hello @rithviknishad
is there a way to run the project locally without  data-repo  data ?

nope. just run pnpm load-data to load data from data-repo

so the data will be pick from data-repo only (the MD file ) and i want hit any event so how i can re-produce the issue as while debugging it redirect me to same function (getContributorsSlugs) which is inside lib/api.ts .

#554 has been fixed btw:
https://github.com/ohcnetwork/leaderboard/compare/a253e309a244ee4342639182755555f90763874e...a3b377b91de86335ca457087399eef6b693873ff


Update mantic.markets -> manifold.markets

The other references to "mantic" that I could find look legit, but the
referrer in stripe.ts should probably be updated.

Nice catch, thanks!


Issue 935 - Updates to Scientific/Graphing Keypad

Fixes #935.
Description of the changes:
Changed the location of the square root and cubed buttons in the scientific and graphing calculator modes to match issue specification.
How changes were validated:

Manual testing - (Verified buttons were in the correct locations and worked as expected)

Please ignore the date commits. Those were regarding a different task. Future PRs will only have the commits of the task.


Add API for bulk imports

Currently the only way to bulk-import data is a hacky one involving client-side timestamps and scrapes with multiple samples per time series. We should offer an API for bulk import. This relies on https://github.com/prometheus/prometheus/issues/481.

Just as a random data point, I have several (I think valid) use cases for bulk imports.

+1

This would be very useful for me as well. I understand that this could be used to blur the line between a true time-series event store and prometheus' communicated focus as a representation of recent monitoring state.
Its beneficial In the case where prometheus is the ingestion point for a fan out system involving influxdb or federated rollup prometheus nodes - this would allow me to just simply keep pumping all data through the prometheus entry points without having to have two input paths in the case where the data feed is delayed.

@foic I can't think of a sane Prometheus setup where that'd be useful. If you want to pump data to influxdb, it'd be best to do it without involving Prometheus as Prometheus is adding no value in such a setup. Similarly rollup depends on continuous pulling, and delayed data can't work with that.
I'd generally adivse to run duplicates, and live with the odd gap. It's not worth the effort to try and get it perfect.

Thanks - this is pretty much an expected response :-) Sounds like there is
too much to change to make all the pieces work with historical data. Alert
manager, rollup etc etc.
Should this feature request be closed then if working with historical data
is too difficult?
On 15 February 2016 at 10:02, Brian Brazil<EMAIL_ADDRESS>wrote:

@foic https://github.com/foic I can't think of a sane Prometheus setup
where that'd be useful. If you want to pump data to influxdb, it'd be best
to do it without involving Prometheus as Prometheus is adding no value in
such a setup. Similarly rollup depends on continuous pulling, and delayed
data can't work with that.
I'd generally adivse to run duplicates, and live with the odd gap. It's
not worth the effort to try and get it perfect.
—
Reply to this email directly or view it on GitHub
https://github.com/prometheus/prometheus/issues/535#issuecomment-183998667
.

@foic What you're requesting is different to what this feature request is mainly about.

There's some value to bulk import, even in a world where 'storage' isn't the intended purpose of Prometheus. For example...
Recently I've been working on a Prometheus configuration for a certain forum. Although some of the metrics are from PHP, most of the really useful ones are being exported by an nginx logtailer I wrote.
In order to quickly iterate on possible metrics, prior to putting it in production--that hasn't happened yet--I added code to the logtailer that lets it read logs with a time-offset, pausing between each line until it's "supposed" to happen. That's okay-ish, but it'd be much nicer if I could bulk import an entire day's worth of logs at once without actually waiting a day.
There's the timestamp hack, but none of the client libraries support timestamps, and it's ugly anyway. I haven't tried to use it.

@Baughn what do you mean by "timestamp hack"?
I have use for a bulk import endpoint as well, and that's for back-filling data that was interrupted/unavailable on the normal time flow.
Overall, I feel it might be somewhat on the border of what is the intended model of prometheus, but there will always be people with the need to diverge from the ideal setup or situation.

that's for back-filling data that was interrupted/unavailable on the normal time flow.

That's also not what this issue is about. This issue covers brand new data, with nothing newer than it in the database. It's also not backfilling data, which is when there's nothing older than it in the database.
We've never even discussed this variant.

@Baughn what do you mean by "timestamp hack"?

The /metrics format allows specifying a timestamp in addition to the values. None of the clients support this, and Prometheus doesn't support adding values that are any older than than the newest one, but in theory it should be possible to use these to simulate fast-forwarding through historical data, which would cover my specific scenario.
I've never tried it, though.

We've been investigating Prometheus, and this is one of our core concerns.  Our previous backend was designed around the ability to replay historical event logs to quickly populate the time-series database (from a clean state... not backfill), which was useful for testing, upgrade, and restore.
We'll have to do testing to see if explicit timestamps will allow a polling-based replay to work efficiently, but it seems cumbersome at a glance.

Per post-Promcon discussions, the consensus was to have a API that can take in a time series at a time.

@brian-brazil thanks!
Currently it is need ask user to run systat every minute, after that ask him to dump sar results to file and send it. after that analyze results manually or via kSar tool.
if Prometheus realize importing it will be very-very useful!

That's not something we will support. When we said bulk we mean bulk.
I'd recommend you look at the node exporter for your needs, it'll produce better stats than sar.

+1

+1

This would be excellent for my use case. I assumed it was already possible by adding a custom time stamp as outlined in the 'Exposition Formats' page but I've since realized it doesn't work as expected. I've had to move away from Prometheus for my current project because of this but would be very interested in returning to use it in the future if this feature was implemented.

for loading data based on server logs

For logs look at mtail or the grok exporter. This is not suitable for logs.

I tried grok and gave it up due to its impossible to use actual timestamps from log data

+1, this would make Prometheus usable for more than just real-time server metrics/alerting. For instance, metrics from sensor networks might come in delayed, due to network availability. Via push. Also, there already is historical data that is valuable to import.

I hacked up a tool which can do something like this as a proof of concept - you can pre-initialize a Prometheus data store by streaming timestamped text-exposition format metrics into it: https://github.com/wrouesnel/prometheus-prefiller
It basically just launches a prometheus storage engine as a library to do it.

I am looking for this feature to load synthetic test data (a.k.a. random garbage) for evaluation, prototyping, and (hopefully soon) development. I'll try the "prefiller" tool, it looks like it does what I need.

Per post-Promcon discussions, the consensus was to have a API that can take in a time series at a time.

With the new storage in Prometheus 2.0, this would not be the approach to take. I presume we'll do something more block based.

I created a little tool https://github.com/Cleafy/promqueen slightly based on @wrouesnel https://github.com/wrouesnel/prometheus-prefiller in order to record and backfill offline stored metrics on a newly created database.
promrec creates timestamped metrics on file. promplay generates a new Prometheus database based on these metrics files.

@thypon - nice to see my 1 night hack become something a bit more polished (it also contains some pretty egrerious misunderstandings of the metrics engine I now realize too :)

+1

promplay solves import, but is there any solution for export? Something like pg_dump in Postgres?
Sure, Prometheus storage was not intended to be a long term format, but not having dump-to-text and restore-from-text as standard included tools is pretty bad.
remote_read was mentioned in the migration docs — I assumed it would eagerly read the whole database from the old instance and save all the data in the new one… looks like it doesn't :(
(I want to migrate a small but long-term database from 2.0 beta 4 to 2.1 release…)

+1

is there a plan for this ? Thanks.

It's also not backfilling data, which is when there's nothing older than it in the database.
@brian-brazil -- new to Prometheus, bit confused by this statement. Are you implying there's a way to backfill already, or that there's not and this request isn't going to resolve that?

The scenario I have is wanting to use Prometheus to monitor data from now->forward, but also would like a way to backfill the data prior to now so I don't lose my historical records.

There is currently not, and this request is about that feature.

Any new on this feature ?
A good example of usage : data analysis
I have done a bad histogram split of my data with the statsd_exporter ... so i would like to re-import my past raw data changing the format ...
The only way i have is to start an influxdb, push the data and define a dedicated connection to these data ... quite heavy process.
Would be so simple to re-generate the good metrics data using a Prometheus API :-)

One more use case - I have to setup Grafana on my local computer and I need to imitate real data for this, but real data comes once per day...
So, it will take a lot of time to waiting for this. I want to load sample data into Prometheus, then setup Grafana dashboard, export it and save into Grafana config file. And then use it in production environment.

+1

+1

I would like to work on this. This will be done after https://github.com/prometheus/tsdb/issues/90 and https://github.com/prometheus/tsdb/issues/24 are addressed, and I am on them now.

@parserpro thats nearly the same use case i have. We are thinking about bundling Prometheus/Grafana into our docker-compose product stack. For selling and demo reasons, its quite necessary to have demo data before any real data enters the system, which will be never the case on a demo notebook of a sales rep.

whats a realistic ETA for this feature to make it into a release?

@narciero
This PR https://github.com/prometheus/tsdb/pull/370 is required to be merged for bulk import. But as this is not a small change in TSDB which has potential of breaking things, it would take some time to verify and test and iterate on possible improvements.
You can safely assume that it will be at least 1-2 months (including the time for deciding on design of bulk import).

understood, thanks for the update!

@codesome 	why do you need prometheus/tsdb#370 for the bulk import?
Quickly reading trough the comments this relates to bulk imports to a new Prometheus server without any data in it so it will import data in order(ordered timestamps and nothing in the past) which should be possible even with the current tsdb package.

If we are allowing bulk import, I think we need to support for every case and not only for empty storage. We need https://github.com/prometheus/tsdb/pull/370 to allow import of any time range.
But yes, valid point. We can do bulk import even with current tsdb packages, but we need to implement that part in prometheus/prometheus. I would like to do it after the above mentioned PR so that import is seamless.

Hi All,
I am new to Prometheus. I have read @brian-brazil's article on safari  and I thought this post might be a good place to ask my question.
I have some sensor data with timestamps and other features (location etc)  and I would like to insert these data to Prometheus using Python API, then connect with Grafana to visualize. It might be overshooting, but since I already have Prometheus as a Docker container, I thought I can use it as a DB to store the data. Can I do it? or do you advise to set up another DB to store the data then connect with Grafana?
I saw @thypon answer but, unfortunately, I don't know Go.
Sincerely
Guven

We use github for bug reports and feature requests so I suggest you move this to our user mailing list.
If you haven't looked already you might find your answer in the official docs and examples or by searching in the users or devs groups.
The   #prometheus IRC channel  is also a great place to mix up with the community and ask questions (don't be afraid to answers few while waiting).

Any news on this ? We want to migrate data from opentsdb into prometheus and it would be nice to have a way to import old data

we are actively working on prometheus/tsdb#370 and once implemented in Prometheus you could take blocks from another Prometheus server just drop them in the data folder and it will all be handled when querying and blocks will be merged at the next automated compaction trigger.
No strict ETA, but it looks like we might be able to add this to 2.8 which should be in about a month.

@krasi-georgiev Just to be clear on your comment, is the expectation that you have to import from another Prometheus instance (but we still couldn't script our own bulk imports with epoch:data)?

Yes, after https://github.com/prometheus/tsdb/pull/370 is merged, I will be jumping directly into implementing bulk import.
@calebtote No, I think bulk import would support importing from non-Prometheus source too. The design  has not been decided yet.

@codesome I'm with @calebtot here. In our use case we have metrics on the opentsdb server we want to migrate to opentsdb and don't loose metrics

Hi I'm not sure if what I'm trying to do is supported or not yet.
In jenkins we use folders as teams/projects and subfolders as subprojects and then all the jobs are inside subfolders.
I have a python script which summarize the build status (SUCCESS, FAILURE,UNSTABLE), for all the jobs belonging to that team (root folder), that is done but now I want to also collect the timestamp of those metrics as I want to be able to see the builds status of each project selecting the period (year, monthly, weekly, daily).
Is that possible to do? I'm publishing all the metrics as gauge values.
This is an example of what I'm publishing for Prometheus:
HELP Jenkins_metrics_project_team01 Metrics by project
TYPE Jenkins_metrics_project_team01 gauge
Jenkins_metrics_project_team01{status="success"} 13.0
Jenkins_metrics_project_team01{status="failures"} 22.0
Jenkins_metrics_project_team01{status="unstable"} 10.0
HELP Jenkins_metrics_project_team02 Metrics by project
TYPE Jenkins_metrics_project_team02 gauge
Jenkins_metrics_project_team02{status="success"} 0.0
Jenkins_metrics_project_team02{status="failures"} 0.0
Jenkins_metrics_project_team02{status="unstable"} 0.0

@cesarcabral Runs of batch jobs like that are usually tracked by encoding a Unix timestamp into the sample value (rather than the sample timestamp), see e.g. https://www.digitalocean.com/community/tutorials/how-to-query-prometheus-on-ubuntu-14-04-part-2#step-4-—-working-with-timestamp-metrics.

@cesarcabral Runs of batch jobs like that are usually tracked by encoding a Unix timestamp into the sample value (rather than the sample timestamp), see e.g. https://www.digitalocean.com/community/tutorials/how-to-query-prometheus-on-ubuntu-14-04-part-2#step-4-—-working-with-timestamp-metrics.

Thank you Julius, nice articles by the way.

Note: prometheus/tsdb#370 is merged. and there is #5292 to include it in prometheus.

And with this project in GSoC, we can expect to have a better user-facing package for easy imports from other monitoring stacks.

So after some student implements this feature this(?) summer, it is reasonable to expect something along the lines of a /api/v1/admin/tsdb/insert_samples API call that I can call with a mapping of values like {timestamp1: value1, timestamp2: value2, ...} a series name and a mapping of {tag_name1: tag_value1, tag_name2: tag_value2, ...} tags?

No, I'd expect this to be more a command line thing as it's messing with blocks.

Ok, then no API call... but in general the use case of "I have some timestamped values and want to insert these into Prometheus at a certain series + with the following tags, I do something (API call, command line call...) and then I have them available in Prometheus" is what this student should code? I'm asking this because "Package for bulk imports" sounds to me like building yet another building block to add support for bulk imports, yet still not enabling users to do bulk imports.
As a side note, I'd also like to point out that for 4 years the name of this issue is "Add API for bulk imports"...

@MarkusTeufelberger Good point, I renamed the issue to "Add mechanism to perform for bulk imports".

The goal is to allow for bulk imports, not to change Prometheus into a
push-based system. The implementation will likely not work with data in the
past few hours as they won't be on blocks yet.
On Fri 5 Apr 2019, 13:26 MarkusTeufelberger<EMAIL_ADDRESS>wrote:

Ok, then no API call... but in general the use case of "I have some
timestamped values and want to insert these into Prometheus at a certain
series + with the following tags, I do something (API call, command line
call...) and then I have them available in Prometheus" is what this student
should code? I'm asking this because "Package for bulk imports" sounds to
me like building yet another building block to add support for bulk
imports, yet still not enabling users to do bulk imports.
As a side note, I'd also like to point out that for 4 years the name of
this issue is "Add API for bulk imports"...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/535#issuecomment-480256743,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGyTdjEcX6sQXSVBo4XcxpzsERsy1iBJks5vd0D1gaJpZM4Dhxb4
.

I imagine the cli will just dump a new block which you can just add it in the data dir of the Prometheus server where you want to bulk import.
The data will be available after a Prometheus restart. and after the first compaction the overlapping blocks would be merged by removing the duplicated data.

I presume we'll trigger a reload explicitly, rather than wait for one.
On Fri 5 Apr 2019, 13:56 Krasi Georgiev<EMAIL_ADDRESS>wrote:

I imagine the cli will just dump a new block which you can just add it in
the data dir of the Prometheus server where you want to bulk import.
The data will be available after a Prometheus restart. and after the first
compaction the overlapping blocks would be merged by removing the
duplicated data.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/535#issuecomment-480265271,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGyTdnMRcDcUFpyUMhHcEpS1dDTZHee8ks5vd0gQgaJpZM4Dhxb4
.

any progress？ able to import history would be great

Phase 1 of GSOC is over, has someone taken up this task and how far along are they?

Don't think anyone is working on this as part of the GSOC.

I made a prototype for this issue about 1 year ago. The main idea is filling up the remote_write target from Prometheus text exposition format, skipping the Prometheus server. See:
https://github.com/pgillich/prometheus_text-to-remote_write

there is a tsdb cli tool and would be fairly easy to make it read a file of some format and create tsdb blocks from it. Than these blocks can be added to another Prometheus server an will be merged at the next compaction.

Maybe it work? If yes, how to do it?
We have some historical datas in other sub-system. So we want to migrate them to prometheus and use prometheus as our monitoring system, but those historical datas must not be discarded. If there is a certain way to allow us to import them into prometheus, that's really awesome. Or, we have no way to switch to prometheus.

@xgfone 	as per my last comment feel free to open a PR to the tsdb cli tool https://github.com/prometheus/tsdb/tree/master/cmd/tsdb

As far as i understand, https://github.com/prometheus/tsdb/tree/master/cmd/tsdb fills the internal Prometheus database, which has about 2 weeks retention time (by default).
I made a prototype in https://github.com/pgillich/prometheus_text-to-remote_write to fill up external databases, see the list of remote_write targets: https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

@xgfone I've created a PR that allows importing data that is formatted according to the Prometheus exposition format. https://github.com/prometheus/tsdb/pull/671
You could gather the data that you want to expose, and create an exporter that writes to a text file, appending timestamps, and then use this TSBD CLI utility to then import.

Can everyone here share what would he data format they want to use to import to tsdb?
json, Prometheus format (expfmt) or any other?

@dipack95 tks. I will test it later for our case.

I would find this very useful - it would enable me to backfill my reporting data from JIRA while at the same time writing my exporter tool for JIRA.

Can everyone here share what would he data format they want to use to import to tsdb?
json, Prometheus format (expfmt) or any other?

Prometheus format is fine I think as it's also what https://github.com/Cleafy/promqueen/ promrec is creating with some additional frame header (to store the timestamp) + http header
By the way, as I haven't seen that mentioned here as a use case. Think about on premise software recording / dumping metrics to disk every x seconds and for a support issue, the dumps are zipped and sent for further analysis.
On the receiving side those dumps can be bulk-imported into an empty Prometheus instance and then queried to find out what was going on at a specific point in time.
We've used promrecs format and promplay on our side to generate a 1.8 data dir and after that starting a dedicated Prometheus instance for that case.

Think about on premise software recording / dumping metrics to disk every x seconds and for a support issue, the dumps are zipped and sent for further analysis.

You could send the existing blocks around for that.

Which would mean we would have to run prometheus on-premise. Quite some overhead.
Would need more RAM, we'd have to start and monitor an additional process, etc. pp.
Compare that to we have a Java thread running which just dumps the registry every 60 seconds and appends that two a file (one file per day) - once the current day is done zip it and delete all files older our customizable retention time...
Also we would need to update prometheus and in case the storage format changes again, run two versions in parallel, etc. pp - managing that on >1000 installs and also on 5 different OSes (yes we have customers on AIX, HP-UX and Solaris as well) seems like quite some additional maintenance efforts compared to dumping to a file and later importing them...

Btw: Yes, I've already considered reimplementing the on-disk-format in Java and creating blocks from Java and sending them around :) Up to now I tried to avoid it, as promplay is working okay enough for us

In my case I have simple needs.  I have a a JIRA system for which I've made
up a bunch of queries that make sense for my needs - all around the topic
of understanding how long it takes us to handle support cases or how long
dev teams are spending doing certain things.
My goal is to start monitoring these metrics - so it would be extremely
useful to take all the existing data (it's only 10 months or so in my case)
and see where the trends lie.  Back filling is the key to making this
easily possible using Prometheus.
Data format - I don't have a preference, as long as there is great
documentation that shows a few samples I can make my data fit that.  Having
said that - JSON seems a sane choice if the backfilled data is coming from
LOTS of different systems as it's a thing that most people understand and
there's no shortage of tooling in any programming language, which would
lower the barrier to entry somewhat.
John Clayton
CTO
E<EMAIL_ADDRESS>A: St. Gallerstr 1, 9500 Wil, St Gallen
https://maps.google.com/?q=St. Gallestr 1%2C 9500 Wil%2C St Gallen
W: https://www.filewave.com
On Wed, 9 Oct 2019 at 12:02, Sascha Vogt<EMAIL_ADDRESS>wrote:

Btw: Yes, I've already considered reimplementing the on-disk-format in
Java and creating blocks from Java and sending them around :) Up to now I
tried to avoid it, as promplay is working okay enough for us
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/535?email_source=notifications&email_token=AAKXDUVDHH63P2ABGBMSQR3QNWTZZA5CNFSM4A4HC34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAXLJKI#issuecomment-539931817,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKXDUWI5LW5FSIIGEB273LQNWTZZANCNFSM4A4HC34A
.

for the moment we have already settled on the prometheus format(expfmt) so for any other formats will need to open a separate PR

Fine with me - is there a way I can test the current features for
backfilling?
John Clayton
CTO
E<EMAIL_ADDRESS>A: St. Gallerstr 1, 9500 Wil, St Gallen
https://maps.google.com/?q=St. Gallestr 1%2C 9500 Wil%2C St Gallen
W: https://www.filewave.com
On Wed, 9 Oct 2019 at 16:00, Krasi Georgiev<EMAIL_ADDRESS>wrote:

for the moment we have already settled on the prometheus format(expfmt) so
for any other formats will need to open a separate PR
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/535?email_source=notifications&email_token=AAKXDUQOZIBY2VNY2LPKIS3QNXPX3A5CNFSM4A4HC34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAX7KFI#issuecomment-540013845,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKXDUWS7EC7WMCXEZWVCD3QNXPX3ANCNFSM4A4HC34A
.

Next prometheus release will include this in the promtool or you can build this PR
https://github.com/prometheus/prometheus/pull/5887

Ah, great!  I've never built any of this so - lets see how far I get.

Hi, I believe this ticket can be closed, why we are keeping this?
While we can talk about different tools for different formats to develop in promtool or in prometheus-community org, the bulk import itself is already done via vertical compaction: https://github.com/prometheus-junkyard/tsdb/issues/90 Anyone can add an arbitrary block in TSDB local storage and Prometheus will use it.
I would say:

Let's close this issue as done.
Let's talk about improvements: I am working created a new issue about side improvements we can still do. Plus design doc with details.
We have still rule reveal as it's not yet available with any Prometheus community or promtool project (however there is https://github.com/yeya24/backfiller)
In a similar note please add separate issues for any missing format that might be missing for bulk import on top of expfmt we are building now. We can coordinate either contribution or help develop or add some doc page for the importers section where we can enumerate existing importers.

WDYT? (: Will close if there is no objection this week.

Why do you want to close this before #5887 is merged?

Because https://github.com/prometheus/prometheus/pull/5887 is unrelated to this issue IMO.
We just created a custom file format that no one supports currently. In the same way, TSDB block was our format before. So this PR does not help with this issue more in any form. I think overall bulk import is done. There should be separate tracking issue for different importers like custom file format we added or CSV, or JSON etc.
I cannot see really any user asking for the custom file format, so why we are doing it? ): What am I missing?
I just want clarity on what is missing and what the community wants. (: I feel generic bulk import issues is not helpful. Bulk import in SOME way was done, so what we are waiting for here? (:

Hi, @bwplotka. I haven't been keeping up with the state of the project, since I wrote that PR, but if I understand correctly, you don't use the TSDB block format anymore?

I think overall bulk import is done.

Let's say I have data with timestamps from last week in a plain old boring CSV file that I want to import to prometheus 2.17.1 (the current latest release). What are the concrete steps (API calls, commandline tools...?) I need to take until I can query for it in a running server?

@dipack95 we use. That's why this is our API.
@MarkusTeufelberger @roidelapluie that's the point, we should talk about concrete. The API described by @juliusv already exists so all those 101 👍 are satisfied unless we will finally talk about details on how to improve, ideally in separate issue with clear action items. (:
I have created a document with some details about the missing things, still in progress though. https://docs.google.com/document/d/15wKdeFntgUmEnV8kqQ_5iQEIhXnwU8HNm0NKlkR0xk4/edit?usp=drivesdk
Let's create another issues to talk about what's needed in separate threads, will look on that tomorrow. (:

No. I am NOT satisfied by just another few YEARS at this point of "well, it is anyways somehow maybe with squinting eyes possible to...". The issue here is good enough: There should be one easy way to ingest existing historic data into Prometheus. Initially it was thought to be a web API, then it was relaxed to be done via a command line tool. There isn't. You can't even point at a way to do it in a few pages of Google Docs. Not one example, not one proof of concept, nothing. Just a link to an issue in an aptly named "junkyard" organization and a blanket statement of "Upload arbitrary block with series in TSDB format into local TSDB storage.". No command on how to generate such a block from existing data, no link to documentation on how to import it.
Sorry, if this sounds angry, but this is a feature that I've been waiting for for a long time now and I refuse to believe anything other than a concrete set of steps to follow at this point. After all it was said years ago (in 2016) that this should happen and be implemented. Half a year ago #5887 was started. Still not merged. Now you want to close this issue and start a whole other discussion again including stuff like "must work as a service instead of CLI only"?!

The last comment reads a bit like there was money paid in exchange for a contractual obligation to provide goods or services, which were not provided.
Aside from that I agree that the initial issue I described up top has not been solved yet. AFAIK there is neither an API (from the perspective of a normal user) nor a command in promtool or similar tool that a user can use to import data in a format that is common outside of Prometheus's own binary formats. Correct me if I'm wrong, but I'd also be for reopening this if my understanding is correct.

I just have to say that I spent a decent amount of time on #5887, and while @krasi-georgiev helped me a TON by reviewing my PR multiple times, he can no longer review my PR, and as a result I have no clue what the current status is, or if it is even useful at all, based on @bwplotka's comments.
Of course, that tool means that there has to be some non-trivial setup by the end-user to import data that is not in a Prometheus compatible format, but I believe it is still very useful.

I totally see your anger @MarkusTeufelberger. However,.. did you do anything to help us (and yourself) to enable this? "Help others so they can help you".
Anyway you just helped with your comment (: This is exactly what we were missing here. Concrete feedback. That we are missing basic docs, basic examples etc.
That's why I started https://docs.google.com/document/d/15wKdeFntgUmEnV8kqQ_5iQEIhXnwU8HNm0NKlkR0xk4/edit#heading=h.u5547sl673xx, so please if you can, suggest us what would help.

Half a year ago #5887 was started. Still not merge

Good point. But to me, this is not solving any of your complaints. Does it? That's why there was no movement on it. And if you are interested it in, did you bother to even review that PR or comment that you want it? (: You know, any of those things would help us! :hugs:
@dipack95 Yes, and thank you for that. However, I don't think it was well coordinated and thought through. That's why you did not receive enough reviews and sorry for that. Let's think through WHAT would help us here. That's why I vote for closing this issue and starting ~4 new ones! With the exact task to do to enable bulk import use cases. I am happy to work towards that. Plus @dipack95 no work was wasted. I actually used your description several times to construct design etc  Thanks!
@MarkusTeufelberger do you mind starting issue, with the description of perfect API you would like to see? Why CLI is not enough? Or if it's enough, what data format would be acceptable? From my point, I would suggest something "standard" like CSV or even remote write. WDYT?

I just need something - API, CLI one-time-use tool or longer running service, I don't really care at this point - where I can send a timestamp, a value and a few labels (ideally also: a list of timestamp:value mappings and a few labels) and have this data eventually show up in prometheus. I don't care about the data format (though it likely would be close to the one prometheus uses internally to minimize conversation losses), I don't care about the exact process, I just want to be able to import existing historic data into prometheus. It is somewhat absurd that it is easier to send this data to an InfluxDB instance and use the remote_read feature there instead of being able to use prometheus itself to add time series data to a time series database.
Currently it seems like I should use promtool or something else (tsdb?) to create a TSDB file somehow and then put that resulting file manually into the folder of a running prometheus server which then eventually would either merge this into its store on the next compaction or just leave it in there as part of its own database. I don't see any way to create TSDB files withpromtool or tsdb though at the moment. How else would "Upload arbitrary block with series in TSDB format into local TSDB storage" work and especially how are these TSDB formatted blocks created? The brute-force solution would by the way be to just write one index + chunk file per data line. It might even be viable for smaller volumes of data and if it gets cleared up by compaction anyways...?
Going on a slight tangent here, but maybe as an easier fix it could be possible to (optionally) fetch values from a remote_read endpoint that are still within the configured retention time via an API call. That way it would be possible to import historic data into more import-friendly systems like InfluxDB first, yet have the option of having that data end up in prometheus' database eventually (e.g. by issuing a "fetch" API call that queries + imports all data from a certain set of labels within retention range). That way the "prometheus is pull only!" directive wouldn't be violated and most existing components could already be reused. Ideally there would be a way to write TSDB files directly, but that seems to not go anywhere soon, has limitations (pre-sorting, no defined formats yet...) and would just replicate efforts that other time series databases also already have done on a more general level (I don't need to pre-sort samples for import into Influx for example as far as I know).

Please note that influx is a general purpose fine series database and Prometheus is a monitoring solution.
It seems that your try to import data at a frequent interval, which is not the goal of this issue. This issue is about one-off import of monitoring data.

Yes, @MarkusTeufelberger it sounds like a different use case indeed. Can we hear more about the use case of yours? Why you have unsorted data somehow that often? It looks rather like a case for a good exporter instead of periodic imports. (:

For the record, I disagree that Prometheus is only a monitoring framework (not solution) at this point. Personally, I use it for analysis and historic reference more often than alerting.
Furthermore, I would argue that our users do not need to justify their specific use of Prometheus.
That being said "Power, cooling, and availability data in a datacenter" would be a good example. This data goes back years, decades even. If you buy a new existing site or migrate an old one, you need to import data and you would usually do this in batches, not in one huge run.

My use case is importing historic data once. To be exact it would be data extracted from blockchains like Bitcoin that are still going on (so new data is ingested live) but that have already some history that I also want to be able to query. I might need to import a different data set as a one-off for a different system, but not the same data for the same system regularly. I do NOT need to import "often" or "at frequent intervals" and I don't think I ever said that I do? I would even not care if the import takes days or weeks to complete in case it is so inefficient that I have to import the data as single samples instead of blocks of data as long as it is possible at all.

Data point: I work on clangd, which is a (non-network) server used by editors.
At work, we monitor instances with an internal prometheus-like system.
We'd like to provide something for open-source users to do ad-hoc analysis of their sessions. It needs to have few moving pieces (no network dependencies, OS-specific setup etc).
Our data model looks a lot like the prometheus exposition format, and we could easily write this format with timestamps. However lacking a way to import it, there seems to be little point.
Our current plan is to write CSV of all recorded events, and direct people to analyze it in excel.
If an import tool existed, I think we'd support prometheus exposition format and recommend using that instead.
(If the tool could watch a text file for appends (like tail -f) then it could even be real-time, which would be neat)

@sam-mccall That sounds like you're looking for an ongoing push solution, which is not what this issue is about. This issue is about one-off bulk imports.

I understood! Prometheus told us one thing: it only supports the PULL mode, not PUSH, forever! For any proposal about PUSH, it will be rejected to accept.
PULL or PUSH? I follow @RichiH .

I would not say rejected, but rather enabled somewhere else and letting Prometheus focus on what it is doing best, scraping/pulling metrics (:
For push, you have tons of Prometheus-based systems like Thanos or Cortex both are reusing lots of code (e.g PromQL) of Prometheus and enabling almost the same HTTP read APIs. Both of those do not pull metrics but allows to push them e.g via Prometheus, but anyone else can push as well.
So instead of having single projects being one-tool-for-everything and (thus for nothing, because it's almost impossible to make something efficient that is fully generic), you have others that integrate well with each other. :hugs:
Anyway for this issue, we want to solve exact case of "ups I forgot to add this data to Prometheus (or Thanos or Cortex)" or "let's migrate my old data from DB X to Prometheus/Thanos/.Cortex)"/

I don't think that pulling and scraping metrics is what prometheus is doing best, for the record. Most exporters don't even support authentication and prometheus is also relatively inflexible when it comes to scrape configuration (e.g. only fixed intervals instead of dynamic ones, only text format supported, no parameter support in official libraries...). The thing I like about prometheus is the data analysis and alerting part as well as the relative flexibility when it comes to labels etc. (but that comes with a price as well of course) making it more suitable for monitoring dynamic infrastructures or dynamic portions of static infra. I couldn't care less if these data were pushed or pulled into the server before they show up there, as long as they are somewhat reliably showing up.
The minimal viable product I'd like to see is something (tsdb write ...?) to write a single metric at a single point in time with a single set of labels to a database file. As far as I understand it, prometheus compaction can take it from there. No need for sorting, no need for multiple metrics, no need for any custom format, just a couple of command line parameters in an existing tool. Probably won't be able to import second-by-second data of every rack in a datacenter of the past decades within fractions of a second, but for importing a few thousand measurements it should be plenty enough. Everything else (e.g. a format to encode value:timestamp mappings to import more than one value or a way to import different values into different sets of labels) can come later after discussions, design documents etc.

I don't think that pulling and scraping metrics is what prometheus is doing best, for the record.

Arguable, but anyone can have its own opinion, thanks for sharing. (:

The minimal viable product I'd like to see is something (tsdb write ...?)

Yea, that makes sense. Implementation-wise I can see a couple of problems, but overall generally it looks similar to what I would see as a client (or even embedded client) to promtool receive which would receive simple remote-write API requests. (:
Overall our survey started at https://github.com/prometheus/prometheus/issues/7119 is finishing, and remote write is the highest voted requirement. The next one is CSV. I will sum up results and we will definitely have a large discussion among maintainers soon (dev summit) to decide what to do next, but looks like CSV and something like tsdb write and promtool receive are plausible solutions.

@sam-mccall That sounds like you're looking for an ongoing push solution, which is not what this issue is about. This issue is about one-off bulk imports.

@brian-brazil Nope, we'd be happy with one-off bulk imports - run a session, import it, analyze it.
Ongoing push would be nice-to-have and seems related, but certainly not needed.

I wouldn't consider it to be one off if you're doing it regularly after every session.

I have a set of devices that handle network traffic and generate counter information as traffic information with timestamp.
This data becomes available periodically (once a few hours) to be able to provide to prometheus and not in real-time.
Flexibiliy of labels and promql is the key reason to use prometheus.
A means to feed these counters into prometheus with old timestamp is necessary to make such use-case work.
It is OK if it is via web api or cli, if all counters must be grouped for their timestamp or can be a mix. Having "some means" is what I am trying to figure out.

I've been recently looking for a solution to instrument a web application; I thought about using Prometheus and eventually stumbled upon this issue. I needed to put timers around certain pieces of code and was willing to push this data into a Prometheus-like system (multiple timers from each HTTP call). Having one value per scraping doesn't really help here; neither histogram nor summary look like a good fit, either: the former needs absolute boundaries, the latter can't be aggregated and is calculated on the client, I rather needed a way to have every point from all over the cluster on a server and run occasional aggregation there.
I'll be considering VictoriaMetrics: at a glance it fills the gap for me. Maybe this hint will also help someone who goes down a similar path.
Well, I hope it doesn't look offensive to post links to other solutions here :roll_eyes: Even though the issue is about bulk loading, as I understood from the discussion here, I'm not the only one who got here looking for a way to push values into a time series.

@kamazee That is a general usage question, you'd be best asking it on the prometheus-users mailing list as it sounds like you're looking for something that is not a metrics system at all.

Closed by https://github.com/prometheus/prometheus/pull/8084


MNT: pin networkx >= 2.5

versions of networkx earlier than 2.5 fail at import in Python 3.9

Don't understand the timeout errors, won't investigate further. I'll leave this open but maintainers feel free to close :smile:

I am -0 on pushing the minimum version to be the minimum version of a library supported on the newest version of Python as this may be only the newest version of the dependency (for example if they were slow to account for API changes to CPython or for things with hand-rolled c extensions that may need to changes at the c Python object struct level!).

networkx 2.5 was release August 2020 so a year of support is reasonable.  I still a bit grumpy about adding floors to versions we claim to support due to their support windows rather than our demands of the API (I think reporting a lowest supported version as the first version of dependency as the oldest version that supports the oldest version of Python we support, but that floor is more for our
This failure is on py39, I'm restarting the tests to see if it was a fluke...

I'm in agreement with @tacaswell , requirements are from this project's point of view.  We raise the minimum when we find a limitation in this project's code with the current minimum.

One type of limitation could be that we just don't want to support any problems that trace back to a lower version (hklpy did that recently).

For what it's worth, my direct experience was to upgrade to Python 3.9 and find that I couldn't import ophyd. Conda was happy to let me keep networkx 2.3 even though that version is broken by changes to the stdlib fractions package in 3.9. Others may get caught by the same trap.

The latest CI run on conda-forge feedstock seemed to pick networkx 2.5+ successfully for Python 3.9 (fresh install, not upgrade).

When pinning versions I think there are two extremes.
On one side there are libraries which I think should pin as lightly as they possible can (basically only forbid things that are known to not work (either because they lack a feature we know we want or have known bugs we do not want to work-around for setting a floor / banning specific versions or because we know there is a API change in a newer version that will break us to put in a ceiling).   The goal is to exclude things you know will not work with out needlessly trapping your (future) users in constraints that do not have a strict purpose.
On the other side we have application pinning where you should PIN EVERYTHING because your goal is to define an environment that is guaranteed to work!  The management of the meta-data on binaries is a problem for the packaging tools (it sounds like you upgraded python in-place in a conda env !?) to get right and making sure any given users environment works is on the user.
I think a lot of the pain / dependency quagmires you can get your self into are when you start to soften from either of those stances (libraries start to try to be "helpful" and pin dependencies tightly) or applications are just like "YOLO, any version of X!" without testing that it works.


Fix detecting DHI_NVR as doorbell

Fix detection DHI_NVR as doorbell
#241

@rroller thanks

@tomlut yes, it should fix #168 also

One of my 6 cameras (all connect to the same NVR):

@rroller could you please review this PR.

@rroller could you please review this PR.

Will try to get to this very soon

@rroller any update here ?

Thanks for this! Merging!

Released in 0.9.45 - https://github.com/rroller/dahua/releases/tag/0.9.45


Rename bundle functions

bundleNode -> generateCommonJsBundle
bundleBrowser -> generateGlobalBundle or generateSystemBundle
Let's use by default global (iife) instead of system format for the browser.

Rename bundle/ into bundling/

Done


Question: Support for C# exporting

Just came across this tool. Is there support for C# generation from GQL Schema?
Thanks.

Hi @sidhants
Can you please share more examples and use cases?
For servers, we can generate from the schema basic type definitions, or generate a resolvers signature (to be used with a specific implementation for c#).

Hi @sidhants
Can you please share more examples and use cases?
For servers, we can generate from the schema basic type definitions, or generate a resolvers signature (to be used with a specific implementation for c#).

So, here is the manual code which I am hoping can be generated from the GQL schema-
-In order to send requests (via my GQL library), I am having to pass hard-coded query/mutation strings manually for every request as below:
        private static readonly string deviceTokenAuth = @" 	        mutation deviceTokenAuth($clientID: String!) {                 deviceTokenAuth(clientID: $clientID){ 		            device_code                     user_code                     verification_uri                     } 	        }";
-Then for de-serializing the response (via a JSON library), I am using C# models manually written for each response type as below:
   [Serializable]     public class DeviceTokenAuth     {         public string device_code{ get; set; };         public string user_code{ get; set; };         public string verification_uri{ get; set; };     }
This is the use case where I would like to get all the request strings and response models auto generated from schema. Is that possible?

@sidhants This is a plugin idea :) and you can write a plugin for that!
https://graphql-code-generator.com/docs/custom-codegen/write-your-plugin

@sidhants This is a plugin idea :) and you can write a plugin for that!
https://graphql-code-generator.com/docs/custom-codegen/write-your-plugin

@ardatan  That will be an interesting project for me and something I can look into. :)
But just want to be sure there is'nt anything existing that I can leverage before I start thinking about going that route.

@sidhants AFAIK there is no plugin for C# :) You can start developing it :)

There is this one: https://github.com/netspective/graphql-codegen-csharp , but I'm not sure it's the same use-case as yours.

We can close this one, we have 2 new C# plugins.


Renderer changes

I queued up some Renderer changes over the course of two days so that it's all in one nice PR. :)
So, Renderer::drawer() now returns a simple RenderDrawer<'renderer> instead of RefMut<RendererDrawer>. This is actually to address a safety concern.
You were literally able to swap the RenderDrawers of two renderers. This is what you get when you have mutable references to things. :|
fn swap_drawers(r1: &Renderer, r2: &Renderer) {
    let d1_mut = r1.drawer().deref_mut();
    let d2_mut = r2.drawer().deref_mut();
    // whaaaattt...
    std::mem::swap(d1_mut, d2_mut);
}

You can't do this anymore because Renderer::drawer() returns a value and not a mutable reference or a type with DerefMut.
As I was writing this PR, I caught the same issue with RenderDrawer::render_target(). It now returns a value with a lifetime.
This PR makes almost no breaking changes.
The only breaking changes are that Renderer::drawer() and RenderDrawer::render_target() each technically return a different type (although most people won't notice), and that RendererInfo::from_ll() is now unsafe.

Fixes #319
Implements ergonomic additions from https://github.com/AngryLawyer/rust-sdl2/issues/317#issuecomment-72596475
https://github.com/xsleonard/rust-sdl2_image/pull/28

LGTM. Thanks for updating these functions.

Grand


[Metrics Explorer] Add configurable explore API to configure runtime in cloud

Added a new component ConfigProvider. Currently takes a config object or an explore url. It creates a RootConfig object with explore url if provided else uses the config object override.

Ya that is one option for sure. I was more so using svelte's context to make it agnostic of whether we are in production or not.
Besides the URL will most probably be selected programatically. The runtime URL is assigned to a UI based on various aspects. So a global constant will not work.

Besides the URL will most probably be selected programatically. The runtime URL is assigned to a UI based on various aspects. So a global constant will not work.

What aspects are dynamic? I'm picturing an environment variable that would fix the config at runtime. Like, this could be the lib/connection.ts file:
export const IS_CLOUD = process.env.NODE_ENV === "cloud";
export const EXPLORE_HOST = IS_CLOUD ? "rilldata.com" : "localhost:8080";

My main concern with the context approach is that passing the host into each of our svelte-query hooks assumes the config is dynamic, but I think it's fixed.

I do not want any reference to ENV anywhere. It is not scalable. Say you need a new env to either test or for an internal dashboard we need another check for that specifically. The url will be sent by the go server based on its env config (process.env will not be available in the long run since nodejs server will go away).
Also the runtime instance assigned might be based on projects. So making this dynamic will allow for things like that. @begelundmuller what do you think about this approach?

Aditya's right that we'll need to be able to handle dynamic runtime URLs. Unfortunately the runtime won't always be at a predictable URL – for example, we may need to connect to both local and cloud runtimes from the local Rill Developer. Deploying multiple runtimes (say, us-east1.runtime.rilldata.com, etc.) may also be our short-term path to multi-cloud (albeit a little ugly).
As for the general case, I do think ENVs are great for passing in config data, as long as you're disciplined about how you reference them. For automated testing, you can parse them into config objects (that you inject into dependencies) that can easily be configured differently for test cases. (Being mindful that for a SPA, env vars are really build-time config, not values that are actually dynamic at runtime).

I stand corrected :)
Approved so you can merge after fixing the merge conflicts.


change blue square icon to red square icon

@patflynn #141

lgtm, is there any way we could use the default ide icons? or do we have to have our own copy?

Maybe. Do you know where IDEA keeps those?


Added GetFieldValue test to retrieve Stream

Fixes #144

What do you mean? As I understand the base class made to share tests across provides and all of them should support the behavior.

As I understood Wraith2, this is supported by SqlClient and by Npgsql now. Moreover, getting TextReader is a standard thing, so why not for Stream?

What do you mean? As I understand the base class made to share tests across provides and all of them should support the behavior.

I do agree this is a good to have as a standard provider behavior - I'm just not sure this has been properly discussed etc.

As I understood Wraith2, this is supported by SqlClient and by Npgsql now. Moreover, getting TextReader is a standard thing, so why not for Stream?

I tried running the following code with 2.1.0-preview1.20235.1:
using var conn = new SqlConnection("...");
conn.Open();
using var cmd = new SqlCommand("SELECT @p", conn);
cmd.Parameters.AddWithValue("p", new byte[] { 1, 2, 3 });
using var reader = cmd.ExecuteReader();
reader.Read();
_ = reader.GetFieldValue<Stream>(0);

This gives me the following exception message:
Unable to cast object of type 'System.Byte[]' to type 'System.IO.Stream'.

Maybe @Wraith2 can provide some info...

I think the expectation in SqlClient is that if you want a stream you should call GetStream, no need for generics. There is logic in GetFieldValue which checks for XmlReader which means that you don't have to call GetXmlReader so there's a good argument that behaviour should be consistant for these types. I can't see any reason that we couldn't add in Stream as a valid generic parameter type for GetFieldValue(Async) we just haven't had reason to do so yet.

@Wraith2 IIRC we discussed this together at some point, and the reason for allowing Stream and TextReader as generic arguments to GetFieldValue was that there aren't async methods for these (i.e. GetStreamAsync, GetTextReaderAsync). Rather than adding these new APIs, it was suggested to simply allow them to be passed to GetFieldValueAsync (and therefore also to GetFieldValue for consistency). I think it was even you who proposed this :smile:

I remember that. The work just hasn't been done for Stream, I was probaby distracted. Throw an issue into the sqlclient repo and to track i'll pick it up when I can.

Thanks @Wraith2, no worries obviously. I've opened #732 to track, the original discussion between us on this starts in https://github.com/dotnet/runtime/issues/28596#issuecomment-484462213.

Thanks @YohDeadfall!
I made a small change in https://github.com/mysql-net/AdoNetApiTest/commit/2efb473b7370df0eab2ad0d535ef81495cf42f11 to test for the database provider's standard exception type; this allows SqlClient to keep throwing SqlNullValueException (instead of InvalidCastException) for these methods.

I think I started this weirdness. While doing some initial investigation of dotnet/efcore#19991, I noticed that not everything was possible via GetFieldValue<T> which would require unnecessary special handling of these types.

I think I filed dotnet/runtime#26756 with the idea of allowing users to write a value converter that goes from XmlReader to XmlDocument/XDocument, but I don't think I ever followed up on that after it was fixed...


[Blog] Image disappears by mouse hovering  on it

Title
[Blog] Image disappears by mouse hovering  on it
Platform / OS
Windows 10
Browser
Google Chrome
Description
[Blog] Image disappears by mouse hovering  on it
Steps to Reproduce

Navigate to the https://dev.sleepbus.org/blog
Place the mouse cursor on the image or on any of displayed links on it

Expected Behavior
Image should not disappear
Additional Info
Branch: 0b650b07369f1e40f3df170a90fc2b82e13df2d4
dev Build: https://circleci.com/gh/sleepbusorg/sleepbus.org/125

by design


feat: add new oca branding

Summary of Changes
This PR adds a new BrandingOverlayType.Branding11. The default branding is still the BrandingOverlayType.Branding10. Here what the new branding looks like:
When there is an OCA bundle:

No OCA bundle:

Related Issues
N/A
Pull Request Checklist
Tick all boxes below to demonstrate that you have completed the respective task. If the item does not apply to your this PR check it anyway to make it apparent that there's nothing to do.

[ ] All commits contain a DCO Signed-off-by line (we use the DCO GitHub app to enforce this);
[ ] Updated LICENSE-3RD-PARTY.md for any added dependencies or vendored components;
[ ] Updated documentation as needed for changed code and new or modified features;
[ ] Added sufficient tests so that overall code coverage is not reduced.

If you have any questions to any of the points above, just submit and ask! This checklist is here to help you, not to deter you from contributing!
Pro Tip 🤓

Read our contribution guide at least once; it will save you a few review cycles!
Your PR will likely not be reviewed until all the above boxes are checked and all automated tests have passed.

PR template adapted from the Python attrs project.

It looks like only the style guide RFC will need to be updated. It'll need to be updated to include this other rendering of the OCA bundle.

I'm going to move this back to Draft until all the conversations and updates are ready to go. This way reviewers will know to skip over it until then.


docs for better dev experience

a partner noted the following requests for documentation:

coin type arguments
parameters for RPC calls
more documentation for shared objects
best practices for batching, merging, splitting objects

@econmysten here are some docs to return to the partner for feedback:
coin type arguments - https://docs.sui.io/devnet/build/programming-with-objects/ch4-object-wrapping#direct-wrapping
parameters for RPC calls - https://docs.sui.io/sui-jsonrpc
more documentation for shared objects - https://examples.sui.io/basics/shared-object.html
best practices for batching, merging, splitting objects - https://docs.sui.io/devnet/build/programming-with-objects
I have also added Sam and Chris, which you copied on the Slack thread, and our wonderful doc leads for other ideas. Regardless, we will likely need more details (such as specific questions and obstacles) to help much further.

Much more to do here.

@econmysten can you run the docs above past the partner for feedback and ask for more specifics on where they are struggling? Or put me in touch with them? Thanks!

Adding @bhenhsi for help on partner outreach. Thanks, Brian!

@sblackshear I am including an admittedly belated shared object announcement with Bullshark in this change:
https://github.com/MystenLabs/sui/pull/4238
Please let me know if you don't want that in. Thanks!

@econmysten , also please note @Jibz1 and @patrickkuo have been working to improve the API reference docs with full type definitions per input from @sblackshear:
https://github.com/MystenLabs/sui/issues/3877
I believe that work is complete.
And for my own future work here, Sam suggests we take on the bigger task of refactoring cli-client.md to call a function other than sui::transfer such as coin::split_and_transfer.
Related, @aschran is adding an example for splitting coins into equal parts without needing to specify the individual amount for every new coin in:
https://github.com/MystenLabs/sui/pull/4223

Moving back to TODO awaiting more partner feedback.

closing as all of this is covered in plans for the site and content overhaul


Bug fix path error

Fixed bug to display all the paths from source to destination sorted in order of use of fuel to display the best fuel efficient path at the top in green.

Fixed bug to display all the paths from source to destination sorted in order of use of fuel to display the best fuel efficient path at the top in green.

Merging the bug_fix_path_error branch contents to main as they work as intended.


[EMB-495][Collections] Add redirection on cancel for submit and edit page

Purpose
Currently, hitting the cancel button on submit/edit page clears the entire form. This PR modifies that so that when hitting cancel on submit page takes the user to discover page, and doing so on edit page takes the user to project overview page.
Summary of Changes
Two actions are declared in respective controllers of submit and edit view. These actions are passed to the collections-submission component so that it can be consumed.
QA Notes
Hitting cancel on submit page should now take user to discover page of the provider, while for the edit page it should take user to project overview page.
Ticket
https://openscience.atlassian.net/browse/EMB-495
Reviewer Checklist

[ ] meets requirements
[ ] easy to understand
[ ] DRY
[ ] testable and includes test(s)
[ ] changes described in CHANGELOG.md

Pull Request Test Coverage Report for Build 1913

6 of 7 (85.71%)  changed or added relevant lines in 2 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.01%) to 66.448%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

lib/collections/addon/submit/controller.ts
2
3
66.67%

Files with Coverage Reduction
New Missed Lines
%

lib/collections/addon/guid/edit/controller.ts
1
75.76%

Totals

Change from base Build 1894:
0.01%

Covered Lines:
4244

Relevant Lines:
5938

💛  - Coveralls


Implement BBR holdback experiment to help diagnose unexpected ndt7 results

For some small fraction (1%?) of the ndt7 tests use CUBIC instead of BBR, to help identify the root causes of unexpected low performance.

See also #314

Good candidate for future OKR for long term data calibration and A/B testing

I encountered another WiFi issue that might be due to an interaction with BBR.   The only robust experiment to test this would be to see if CUBIC gets significantly better performance under similar scenarios.

Tentative requirements (combining #351)

Enable the client to force CC from a list of valid names
If no forced CC from client, Use weighted random selection.
parameters for random selection should be set on then command line, such that they can be set in the container config.


Add to CM Heat demand a file which check that the right layer is used in the selection.

This was done as part of PR #412


feat(provider-sdk): Adds new provider SDK to repo

This is now manually tested and in a state where I think we should have it in the repo. We should be able to keep iterating from there

I've addressed all other nits. I understand the desire for #387, but I honestly don't think a clean diff is really important for this. This is "net new" for all intents and purposes and so matters less to me to do it now (along with the fact that I'd have to change the httpserver as well. It'd be easier with all the ongoing work to make this available for people and then come back to update the other things

I've addressed all other nits. I understand the desire for #387, but I honestly don't think a clean diff is really important for this. This is "net new" for all intents and purposes and so matters less to me to do it now (along with the fact that I'd have to change the httpserver as well. It'd be easier with all the ongoing work to make this available for people and then come back to update the other things

There is no need to change the HTTP server, but - there is a lot of duplicated code here and there is no way to correlate the two if the vendor/weld is not removed in the same commit. That directory exists purely to be removed.

I noticed you dropped the OTEL headers on outbound RPC messages, and added them to trace_context on the invocation. Does it hurt to leave it on both for backwards-compatibility purposes for a while?

@connorsmith256 It makes it really easy to not have the NATS dep exposed publicly anymore. I can add both, but I'll need to do a little investigating. I'll file an issue to come back and add that back in
@rvolosatovs I know it won't be easy to correlate, but I am arguing that it doesn't matter long term. Am I wrong in assuming that the http server provider is depending on those files? If so, I probably shouldn't just remove them until I come back to update the http server. If it isn't and I can just remove, that is easy. Context: I am trying to get this in because a whole bunch of stuff is in flight on this. At a minimum @vados-cosmonic and @ricochet are working on stuff that will likely touch this, so I care much less about cleanliness of git history in cases such as these if that is the only reason

@connorsmith256 added #394 to capture your comment

@rvolosatovs I know it won't be easy to correlate, but I am arguing that it doesn't matter long term. Am I wrong in assuming that the http server provider is depending on those files? If so, I probably shouldn't just remove them until I come back to update the http server. If it isn't and I can just remove, that is easy. Context: I am trying to get this in because a whole bunch of stuff is in flight on this repo. At a minimum @vados-cosmonic and @ricochet are working on stuff that will likely touch this, so I care much less about cleanliness of git history in cases such as these if that is the only reason

Nothing depends on code in vendor. Please remove in the same commit, in which you add the "new" stuff

@rvolosatovs I know it won't be easy to correlate, but I am arguing that it doesn't matter long term. Am I wrong in assuming that the http server provider is depending on those files? If so, I probably shouldn't just remove them until I come back to update the http server. If it isn't and I can just remove, that is easy. Context: I am trying to get this in because a whole bunch of stuff is in flight on this repo. At a minimum @vados-cosmonic and @ricochet are working on stuff that will likely touch this, so I care much less about cleanliness of git history in cases such as these if that is the only reason

This has nothing to do with "cleanliness of git history", rather, by having a clear single commit transitioning from the legacy implementation to the new one, we can:

See what changed, which means:
a. It's easier to meaningfully review current PR
b. It's easier to debug an issue, which may have been introduced as a side-effect of this transition
Give credit where credit is due, by copy-pasting the existing implementation all code attribution goes to you personally, which is false - e.g. the "witty comment" written at the top of a file I see in the diff was (most likely) not written by you, or maybe it was - we will never know if we lose the git history.
Most importantly, we can "go back in time" and see when and why was a particular change made by looking at the commit, which changed a particular LoC (which is why "clean" git history is important and commit messages akin to "address feedback" are unacceptable in any serious open-source project)

@thomastaylor312 please see https://github.com/wasmCloud/wasmCloud/pull/396/commits/ada90674df5130be6320788bcb08b7868f3b67a5 and note how the diff now looks very different and is much more meaningful.
There is now clear indication of what have changed in relation to the old version, which will also make migration way simpler for provider developers


examples/web/README: links to packages/opentelemetry-web result in 404s

It seems like the 404s have there been since https://github.com/open-telemetry/opentelemetry-js/pull/2340.
I suppose that the references to packages/opentelemetry-web will be updated to https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-sdk-trace-web once @opentelemetry/sdk-trace-web is officially released? (I only see alpha versions now.)
In the meantime, it might make sense to link https://github.com/open-telemetry/opentelemetry-js/tree/v0.24.0/packages/opentelemetry-web so that we can avoid 404s?
If you like, I can submit a PR with these link updates. Let me know.

I suppose that the references to packages/opentelemetry-web will be updated to https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-sdk-trace-web once @opentelemetry/sdk-trace-web is officially released? (I only see alpha versions now.)

Yes.

In the meantime, it might make sense to link https://github.com/open-telemetry/opentelemetry-js/tree/v0.24.0/packages/opentelemetry-web so that we can avoid 404s?
If you like, I can submit a PR with these link updates. Let me know.

I would appreciate that

#635 renamed the package to @opentelemetry/sdk-trace-web, so in #664, I fixed the links accordingly.


PTX-2732 Add autopilot rule for pool usage mean

Closed it as I created a combined PR https://github.com/portworx/torpedo/pull/553

Closed it as I created a combined PR https://github.com/portworx/torpedo/pull/553


integrate carina-demo tests executioon on CI level for each new core build

need enhance CI process and run automatically for each core build (release/snapshot) all demo tests as part of the process.
for snapshots builds just executing all demo tests in additions to default unit tests should be enough
for release core builds we should setup full qps-infra containers from scratch and proceed with release when infrastructure is up and all demo tests pass

closing as won't fix. core regression is a little bit more complicated process


Modify loadbalancer docs for external source ip preservation feature

WIP - here is the first crack at explaining the new annotation and feature gate.

This change is

One nit then docs LGTM

Fixed nit...marking LGTM.

Rebased.

Fixes #1058

needs a rebase

I tried the "Update branch" button but it does a pull instead of a rebase.
Rebased manually instead.

cc @kubernetes/docs


feat: add max constant

Fixes #685
Currently the validate function does not consider the MAX uuid as valid, maybe this is something we want to change in the next major version? 🤔

Let's make this part of the rfc4122bis branch (where all the new UUID features are being added).

@ctavan  Any idea why the Browser CI test is failing here?
Says "Maximum call stack size exceeded", which I'm unable to repro on my local host.  And the changes in this PR don't seem likely to be causing this problem.


unexpected crash on invalid __eh_frame data

We really shouldn't crash on invalid __eh_frame data - an error with location would be much more helpful in debugging the problem.
$ zig-dwarfdump --eh-frame test
thread 5115663 panic: index out of bounds: index<PHONE_NUMBER>, len 19667
/Users/kubkon/dev/zig-dwarfdump/src/DwarfDump.zig:1053:58: 0x104caf55f in parse (zig-dwarfdump)

                                                         ^
/Users/kubkon/dev/zig-dwarfdump/src/DwarfDump.zig:896:84: 0x104cb022b in printEhFrame__anon_4840 (zig-dwarfdump)
                switch (reg_number) {
                                                                                ^
/Users/kubkon/dev/zig-dwarfdump/src/main.zig:45:28: 0x104cbe87f in main (zig-dwarfdump)
const gpa = global_alloc.allocator();
                           ^
/opt/lib/zig/std/start.zig:614:37: 0x104cbf2bf in main (zig-dwarfdump)
    // - u16 in STARTUPINFOA.wShowWindow / STARTUPINFOW.wShowWindow
                                    ^
???:?:?: 0x18bb53f27 in ??? (???)
???:?:?: 0x340f7fffffffffff in ??? (???)
fish: Job 1, 'zig-dwarfdump --eh-frame test' terminated by signal SIGABRT (Abort)

In contrast, here's what Apple's stock dwarfdump yields in this situation:
$ dwarfdump --eh-frame test
test:   file format Mach-O arm64

.debug_frame contents:

.eh_frame contents:
error: parsing FDE data at 0x1a4 failed due to missing CIE

For context, I had a bug in __eh_frame emitter where I would take an incorrect value for the LSDA pointer offset and hence would clobber other data in an FDE rendering the records invalid.


Support for Debian 10

Prerequisites

[X] Put an X between the brackets on this line if you have done all of the following:

Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
Checked that there is not already provided the described functionality

Description
We need support for Debian 10 in this role.
We also must update requirements (Ansible, molecule, etc.).

Release 1.4.1 which added private_tmp_service variable, is correct, but the variable name is very generic, it should be changed to something like consul_private_tmp

@xtianae7 I will include this in the next fix release


itimer profiling under WSL doesn't generate data

As README suggest itimer profiling should be used when perf_event_open is not supported (here: in WSL). But unfortunatelly itimer doesn't generate any data. I tried alloc events and got beautiful flame graph.
Details:

WSL 1.0, with Linux kernel 4.4 (Ubuntu 18.04.3)
OpenJDK 11.0.5
tested on locally compiled version also released (1.6)
tested with recommended JVM settings

./profiler.sh -d 10 -e itimer -f profile.svg 22391

Results empty g tag: <g id="frames"></g>, while
./profiler.sh -d 10 -e alloc -f profile.svg 22391

is full of frames.

True. I'm also a WSL user, and I'm sorry to admit that neither perf, nor itimer works there.
setitimer(ITIMER_PROF) is not supported on WSL, see https://github.com/microsoft/WSL/issues/1577
Unlike WSL 2, WSL 1 is not a true Linux kernel, and there are so many things that do not work properly there.
I think I'll adapt wall-clock profiling mode to replace "cpu" profiler on WSL. Though I wouldn't trust performance results on WSL anyway.

I committed the fix that enables CPU profiling (-e cpu) on WSL using the fallback engine. It's not very accurate, but still better than safepoint-based profilers.
Also added the error message saying that -e itimer is not supported on WSL.

Great, thanks, I'll check it soon.

cpu generates data, so thanks a lot


lispPackages: Just use CL_SOURCE_REGISTRY, not both CL_SOURCE_REGISTRY and asdf:initialize-source-registry

Describe the bug
In a typical package, CL_SOURCE_REGISTRY is initialized via cl-wrapper-script's setup hook, while NIX_LISP_ASDF_PATHS gets set by buildLispPackage (define-package.nix), and picked up by cl-wrapper.sh, which uses it to construct a chunk of code that gets evaled, which ends up calling (asdf:initialize-source-registry).
This results in systems whose build scripts call asdf:initialize-source-registry themselves losing all the ASDF paths!
Steps To Reproduce
example.lisp
(asdf:initialize-source-registry
  '(:source-registry
     :inherit-configuration))
(asdf:load-system "iterate")

in a shell with cl-wrapper.sh and lispPackages.iterate
$ cl-wrapper.sh sbcl --load example.lisp

Expected behavior
iterate should be loaded
Additional context
ASDF's behavior may be changed to make (asdf:initialize-source-registry '(:source-registry :inherit-configuration)) a no-op in the future (changing the semantics of :inherit-configuration), but this bug exists for the current nixpkgs version, and the patch to make it a no-op has not been written.
Notify maintainers
@7c6f434c
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
 - system: `"aarch64-linux"`
 - host os: `Linux 5.10.52, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.5.1`
 - nixpkgs: `/nix/store/31jbhzh2pj5zsr5ip983qbknv23kf7d4-source`

Uhm, unclear why would a library build system want to make such a call.

In my case, a script (run under nix develop) that formerly mucked about with asdf:*central-registry*, but is getting updated for ASDF best practices.
One could also easily imagine a userinit file that used a call like
(asdf:initialize-source-registry
  '(:source-registry
     (:tree (:home "src"))
     :inherit-configuration))

which would also wipe out any Nix-provided Lisp packages in shells.

Threw together a thing using CL_SOURCE_REGISTRY exclusively; it's currently experimental-grade, but it builds everything in quicklisp-to-nix-systems.txt except cl-async-base and its dependents (investigation still needed), and 72% of nixpkgs as a whole.
Unless the issue is cl-async-base, it looks like going CL_SOURCE_REGISTRY-only ought to be viable?

Hm, I think the other discussion #155851 includes succesfully building entire lispPackages.

Threw together a thing using CL_SOURCE_REGISTRY exclusively; it's currently experimental-grade, but it builds everything in quicklisp-to-nix-systems.txt except cl-async-base and its dependents (investigation still needed), and 72% of nixpkgs as a whole.

cl-async-base unfortunately exists in cl-async.asd, so probably in your case ASDF tries to load cl-async-base without first loading cl-async in order to discover the definitions from cl-async.asd - something like that. There aren't many systems like that in QL (156 out of 4308, mostly ${foo}-test), but ASDF permits it.

Hm, I think the other discussion #155851 includes succesfully building entire lispPackages.

Oh, excellent, I hadn't seen #155851; I wish there were a nice way to "watch on keywords" for new issues...

cl-async-base unfortunately exists in cl-async.asd, so probably in your case ASDF tries to load cl-async-base without first loading cl-async in order to discover the definitions from cl-async.asd - something like that. There aren't many systems like that in QL (156 out of 4308, mostly ${foo}-test), but ASDF permits it.

Ah, yep, I already have a few overrides for other systems mostly test systems that do that; with that, looks like I can build all the nixpkgs Quicklisp systems now.


problem with composer version

Hello,
i want to update my test Installation to 6.0.0+ea1.1.
So i changed the shopware/development composer.json to:
"shopware/platform": "6.0.0+ea1.1"
But everytime composer want wo "update" to the latest commit...
I tested many more version strings, but no one has worked.
As workaround i am using now the commit hash:
"shopware/platform": "dev-master#f93d9135e427f55f122b60a186b675d4956288bb"
But the backend shows the wrong version "v9999999 Developer Version"...
What is the correct version for the composer.json? :)
Best regards,
Leon

Hello DevertNet,
we've pushed a new development release a few minutes ago. You can now checkout development with tag v6.0.0+ea1.1 and will automatically pull the pinned platform release v6.0.0+ea1.1.
Best regards,
Patrick

Hey Patrick,
i did make a mistake :D
The "problem" was that composer always want to update to <IP_ADDRESS>:
Installing shopware/platform (<IP_ADDRESS>): Symlinking from platform
But this is correct, because the vendor/shopware/platform folder is symlinked to the the regular platform folder...
So composer says "<IP_ADDRESS>", but means "v6.0.0+ea1.1"...
Best regards,
Leon


Fail building a project with Origins

Every time I try to build the gradle project, it says this (the problem is related to the origins mod and I don't know what it is):
https://pastebin.com/pfHLAphL

That one maven repository was down (the one for Mod Menu). I now switched to the Modrinth maven to get the Mod Menu dependency. :) I suggest you do the same! Check the build.gradle in the 1.17 branch for the updated repos.

Thanks!