Multi-org/peer example/test

To make sure all our design really work and also to illustrate to users how a multi-org (and hence, multi-peer) would work we should have a test-case with at least 2 orgs and peers ...

This is a good idea! For solid multi-peer testing the use of the client SDK will make our life much easier.
In such a test we would execute (endorse) a transaction on peer A and check if peer B successfully commits this tx.
We could use the peer CLI for that (https://hyperledger-fabric.readthedocs.io/en/release-1.4/commands/peerchannel.html#peer-channel-fetch) but this requires polling blocks and parsing them in order to check. Not nice.

Using the fabric client SDK (https://github.com/hyperledger/fabric-sdk-go) one can easily register for events. Eg.
client.RegisterChaincodeEvent(ccID, eventID)

seems the peer CLI indeed provides an event listener, which we can use here (and also remove the sleep command)
--waitForEvent                   Whether to wait for the event from each peer's deliver filtered service signifying that the 'invoke' transaction has been committed successfully
--waitForEventTimeout duration   Time to wait for the event from each peer's deliver filtered service signifying that the 'invoke' transaction has been committed successfully (default 30s)

See here https://hyperledger-fabric.readthedocs.io/en/release-1.4/commands/peerchaincode.html#peer-chaincode-invoke

relevant pointers of setting up multi-peer/multi-org:
https://hyperledger-fabric.readthedocs.io/en/latest/channel_update_tutorial.html

Moving to multi-peer we also will have to adapt the handling of __setup in the peer wrapper by persisting info on whether setup has happened and check before invoke and query whether it has happened and if not, call it there.  See issue #141 and PR #144

Note: while in old lifecycle, our peer.sh wrapper logic should have worked with multiple peers and the ercc install&instantiate and the ecc onDemand setup, with the new lifecyle this is not true anymore, e.g., the ercc configuration in channel join would require tighter coordination between multiple peers.  So this might require now more explicit additional management commands but also has to be looked at from the perspective of the additional steps necessary for th  new key-managment protocols

depends on resolution of issue #12

Done in conjunction with #394 ..

Another useful feature and test that can be added in this setting is the following:

3 peers/orgs, only one of which is the SGX-capable FPC endorser, and only that has the code; the remaining two, however, must still be FPC peers (because of the validation plugins)
ensure that the LifecycleEndorsement policy is set to majority of orgs (i.e., only the majority can modify the chaincode definition; #313  handles upgrades)
ensure that the fabric network can go though the approveformyorg step (note: this requires a majority, and require an org to install a package; so it should check whether an empty package or an empty package id is necessary at some org; the Fabric code here suggests that using an empty package id should be viable.)

fabric-sample test-network does already do now in a minimalistical cut multi-peer setup (though no automated test yet)


Expensive recording rules cause high memory/malloc use in 2.0.0

What did you do?
Have some recording rules that require a large amount of data to evaluate.
What did you expect to see?
Moderate heap use/growth.
What did you see instead? Under which circumstances?
Large heap use/growth.
Environment
This recording rule requires about 12k metrics, which is admittedly expensive, but the heap grows to 10-15GB.
groups:
- name: recordings/recording.rules
  rules:
  - record: gitaly:grpc_server_handled_total:error_avg_rate12h
    expr: avg(rate(grpc_server_handled_total{grpc_code!="OK"}[12h])) BY (job, grpc_method,
      environment)

System information:
insert output of uname -srm here

Prometheus version:

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

See attached pprof heap.svg.gz.

Was the memory usage any different in Prometheus 1.x?
I'd assume the allocations happen in the query layer, so 1.x and 2.x should be affected in the same way.
And BTW, we have found predict_linear and deriv being especially prone to allocating loads of memory when run over long time frames.

Attaching the heap output for 1.8.1.
heap-1.8.2.svg.gz.
I don't think it's much different, but it was being masked by the target-heap-size use.

So is this 2.0 specific or not? If not we should remove the 2.0 tag as it's not a regression but a general issue then.

I also have issues with memory usage on 2.0, so not sure it's unrelated.
Here is my prometheus 2.0 memory usage (was runnig beta2 until today but still runnig OOM):

mem usage: https://i.imgur.com/myQY6IH.png
head series: https://i.imgur.com/AFeY2fD.png

It's possible that something in my infra caused this increase but I can't explain it. I assume the only thing that could cause such growth would be an growing number of timeseries but not sure if that's what open head series shows?
I have a few recording rules which are more or less expensive:
- name: default
  rules:
  - record: probe_uptime:avg6h
    expr: avg_over_time(probe_success[6h]) * 100
  - record: probe_uptime:avg24h
    expr: avg_over_time(probe_success[1d]) * 100
  - record: probe_uptime:avg7d
    expr: avg_over_time(probe_success[1w]) * 100
  - record: probe_uptime:avg14d
    expr: avg_over_time(probe_success[2w]) * 100
  - record: probe_duration_seconds:success
    expr: probe_duration_seconds * (probe_success == 1)
  - record: probe_duration_seconds:failure
    expr: probe_duration_seconds + (probe_success == 0)
  - record: probe_duration_seconds:15m:99th
    expr: quantile_over_time(0.99, probe_duration_seconds:success[15m])
  - record: probe_duration_seconds:15m:95th
    expr: quantile_over_time(0.95, probe_duration_seconds:success[15m])
  - record: probe_duration_seconds:15m:90th
    expr: quantile_over_time(0.9, probe_duration_seconds:success[15m])
  - record: probe_duration_seconds:15m:75th
    expr: quantile_over_time(0.75, probe_duration_seconds:success[15m])
  - record: probe_duration_seconds:15m:median
    expr: quantile_over_time(0.5, probe_duration_seconds:success[15m])

Either way, this was running just fine until yesterday. Now it runs out of memory within minutes.

We've made quite a few performance improvements to PromQL since this was filed, so this should be a lot better now.


unix: enforce uv_tcp_keepalive delay parameter

Current UNIX systems define defaults for TCP_KEEPINTVL
and TCP_KEEPCNT, which extend the time between
TCP_KEEPIDLE delay is reached and timeout effectively
occurs (Linux: /proc/sys/net/ipv4/tcp_keepalive_intvl
, tcp_keepalive_probes). As a result, there is little
relationship between the "uv_tcp_keepalive()" delay
parameter in seconds, and actual timeout in seconds.
Do the following: take the delay parameter, substract
1 second, then pass this to TCP_KEEPIDLE and define
TCP_KEEPINTVL/TCP_KEEPCNT to only occur once with
a 1-second-between interval.

Hi there! Thanks for submitting a patch to libuv!
Question: why the 1 second substraction?
Counter-proposal: remove what we do in keepidle, since we now have uv_fileno so the user is welcome to set the socket options they deem appropriate without libuv having to provide a wrapper.

Hi @saghul, and thanks for reviewing !

Response-to-Question: having personnally measured time with several utilities, here is what happens when you do "uv_tcp_keepalive(..., true, 10)" with this patch:

you disconnect the server/close the server socket brutally. The KEEPALIVE feature starts sending ACK packets;

after 9 seconds (= 10 - 1), THE KEEPIDLE delay expires. TCP_KEEPINTVL/TCP_KEEPCNT values become relevant ;

the client waits for TCP_KEEPINTVL (= static 1 second) and resumes the probe TCP_KEEPCNT times (= static 1 time).
If it works, the connection isn't closed, and everything is reset. If it doesn't work, then the connection is  closed.
At the end, ~10 seconds have elapsed.

Counter-proposal: this would effectively deport the platform-specific work onto the user/consumer. As the whole point, on my side, is to make use of the libuv facilities, I cannot support it.

This would effectively deport the platform-specific work onto the user/consumer. As the whole point, on my side, is to make use of the libuv facilities (hence the patch), and more specifcally through high-level languages which do not expose system sockets ; I of course cannot support it.

Isn't that already the case, since Windows doesn't support this?

Isn't that already the case, since Windows doesn't support this?
Hmmm, yet it seems to me that the code is there.
More specifically, reading the Microsoft documentation, I notice the following:
For TCP, the default keep-alive timeout is 2 hours and the keep-alive interval is 1 second. The default number of keep-alive probes varies based on the version of Windows.
On Windows Vista and later, the number of keep-alive probes (data retransmissions) is set to 10 and cannot be changed.
This seems to indicate that on Windows, the effective minimal timeout is 10 seconds (1 second * 10 probes). I could change my PR so that it mirrors this behavior (hard-code TCP_KEEPCNT to 10), which is still acceptable to me, and document it.

Here it is; setting a delay parameter of 10 will effectively make a timeout occur after ~20 seconds.
This should be the same than on Win32.
A UNIX user still has the opportunity to fiddle with the Interval by playing with the socket -as a Windows user would do by editing the registry.

Oops, I just realized this PR targets master. Since it doessn't break the ABI, can you please re-target it to v1.x ?

Oops, I just realized this PR targets master. Since it doessn't break the ABI, can you please re-target it to v1.x ?

My pleasure. Looks live I cannot simply re-target it without pulling ~100 commits from master. I plan to push another PR -should I let this one live or abandon it ?

Yeah, maybe make a new PR then.

@saghul Done! Here it is:
https://github.com/libuv/libuv/pull/2669


Improve performance of dashboard

Dashboard is a bit slow on first load

Closing, different charts make the whole thing faster


Implemented Story Upload

Now it is possible to upload stories automatically with the simple command bot.upload_story_photo('image.jpg') as shown in the examples folder.
Only pictures are implemented for now, I will look into adding videos support when I have time.

Codecov Report

Merging #885 into master will decrease coverage by 0.24%.
The diff coverage is 26.92%.

@@            Coverage Diff             @@
##           master     #885      +/-   ##
==========================================
- Coverage   46.69%   46.44%   -0.25%
==========================================
  Files          28       28
  Lines        2831     2872      +41
==========================================
+ Hits         1322     1334      +12
- Misses       1509     1538      +29

Impacted Files
Coverage Δ

instabot/bot/bot_unfollow.py
10.41% <0%> (ø)
:arrow_up:

instabot/api/api_photo.py
7.4% <0%> (ø)
:arrow_up:

instabot/bot/bot_follow.py
48.59% <0%> (+0.89%)
:arrow_up:

instabot/bot/bot_story.py
10% <14.28%> (+2.3%)
:arrow_up:

instabot/api/api_story.py
25.53% <26.66%> (+2%)
:arrow_up:

instabot/api/api.py
54.47% <60%> (-0.03%)
:arrow_down:

instabot/bot/bot.py
75.59% <66.66%> (-0.14%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d734f89...d456b78. Read the comment docs.

Many thanks for your commit! Thanks!!

@marco2012 did you know how to send data of stories stickers?


Trackers aren't detected

On Poly's PC literally nothing happen when you plug in the tracker, even though the serial exists.

On my laptop I see a popup but it can't provision the tracker.

Even though I can actually open serial console and send GET INFO.
No idea how to even start debugging it.

After random figieting it provisioned on my laptop. I had to reboot the tracker for it to send commands, but I can't find where in the code it looks for boot sequence or anything.

Also if the "connect trackers" window is opened for a some period of time, even just 5 minutes, it starts a lot of connect loops, resulting in many connections per second.
15:05:16 [INFO] Trying to open:Auto  auto: true
15:05:16 [INFO] Trying to open:Auto  auto: true
15:05:16 [INFO] Trying to open:Auto  auto: true
... 50+ lines


Add Spanish

Description
Add Spanish translations / Añade traducciones al Español
Type of change

[ ] Bug fix (added a non-breaking change which fixes an issue)
[x] New feature (added a non-breaking change which adds functionality)
[ ] Updated documentation (updated the readme, templates, or other repo files)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Misc change (updated other files non-breaking change)

How Has This Been Tested?

[ ] Tested locally with a valid username
[ ] Tested locally with an invalid username
[ ] Ran tests with npm test
[ ] Added or updated test cases to test new features

Checklist:

[x] I have checked to make sure no other pull requests are open for this issue
[x] The code is properly formatted and is consistent with the existing code style
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings

Screenshots

Looks great! Thanks for the contribution 🎉


Esc button for escaping full screen and removing markers

The Esc button is used to escape full screen, but also to remove markers. This is confusing, especially since I usually work in full screen since the film strip is partially out of view without full screen. Either use a different key to remove markers, or update the ReadtheDoc.

https://github.com/Neural-Systems-at-UIO/WebAlign/commit/cbef4f51ba58b6fab7e186ae5393f4aa1850a7da: Delete and Backspace also hide the marker now. Documentation update is still needed.


PUP-6164: Tech-debt: links.puppetlabs.com -> links.puppet.com

Resolving old tech-debts: Renaming links.puppetlabs.com to links.puppet.com

Hi @angrox thanks for filing this! We have a commit message format checker - and it wants your commit message title to have the ticket number in parens, ie (PUP-6164) Tech debt: rename.. Any chance you wouldn't mind amending the title and re-pushing?

@MosesMendoza Done :-) Thanks for the info!

@angrox thanks! looks like the .pot file has a conflict w/the change, so i'm going to merge this manually.


security_monkey should track lambda resource policies

Lambda resource policies aren't displayed in the console but are very important.
Track lambda functions and their resource policies.
Alert on cross account access.
Currently, it seems that cross account principals in a lambda resource policy can only be granted to the entire account, and not to a specific IAM role within that account.  Until that is cleared up, it's probably a good idea to avoid cross account lambda invocation all together.
Wide open cross account lambda resource policy:
{
  "Statement": [
    {
      "Sid": "1",
      "Resource": "arn:aws:lambda:<region>:<accountnumber>:function:<name>",
      "Principal": {
        "AWS": "arn:aws:iam::<other_account_number>:root"
      },
      "Effect": "Allow",
      "Action": "lambda:InvokeFunction"
    }
  ],
  "Version": "2012-10-17",
  "Id": "default"
}

What steps would you take if you were going to add this to security monkey? Maybe I can

PR to the cloudaux library adding lambda:
https://github.com/Netflix-Skunkworks/cloudaux

PR to security_monkey to add a watcher that uses cloudaux.  (See IAM Role/IAM User/S3 for examples)

[Optional] PR to security_monkey to add an auditor for lambda.

Fixed in SM #771
Fixed in https://github.com/Netflix-Skunkworks/cloudaux/pull/41
Turns out Lambda resource policies are pretty complex.  The PR tracks alias, version, and default policies.
A lambda auditor still needs to be created.


Add InstanceName trace option

This PR adds TraceOption to define database instance name so that traces and metrics can be separated when multiple instances are being used.
Resolves https://github.com/opencensus-integrations/ocsql/issues/33.

Does it make sense to add this as a tag in the span as well?

@jcchavezs thanks for pointing that, I think it totally makes sense.

could we have this thing merged? I have the same usecase with different databases.

@basvanbeek thanks for reviewing! Would it be possible to tag a version too?

Thanks @vearutop for the change and bug report, and thanks @jcchavezs and @basvanbeek for the code reviews! I have cut a new release at https://github.com/opencensus-integrations/ocsql/releases/tag/v0.1.6


Use continue when skip a value in a while-loop.

Coverage remained the same at 81.379% when pulling 8a95302ce93403bdbf8e3317d2e6256a77144f37 on bogdandrutu:bugmeasure into c008cb07ee83bab189f826a9b6fd98dc02bab00a on census-instrumentation:master.

Codecov Report

Merging #879 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master     #879      +/-   ##
============================================
- Coverage     79.36%   79.34%   -0.03%
  Complexity      911      911
============================================
  Files           156      156
  Lines          4076     4076
  Branches        383      383
============================================
- Hits           3235     3234       -1
  Misses          760      760
- Partials         81       82       +1

Impacted Files
Coverage Δ
Complexity Δ

...io/opencensus/implcore/stats/MeasureToViewMap.java
96.38% <100%> (ø)
26 <0> (ø)
:arrow_down:

...census/implcore/trace/export/SpanExporterImpl.java
96.66% <0%> (-1.67%)
7% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c008cb0...8a95302. Read the comment docs.

Coverage remained the same at 81.379% when pulling 317cd6387c668af38733d6a0605d15843f349b9a on bogdandrutu:bugmeasure into c008cb07ee83bab189f826a9b6fd98dc02bab00a on census-instrumentation:master.


No project files in Xcode

When I open up ios folder in Xcode I can't see module.m or module.h file. Just a folder named Products. However I can see the files in Finder.
Using Xcode 11.6

Sorry for my noobiness. I found that I needed to add files to the project. Closing


expose execSync

@niftylettuce could you elaborate more on what you're asking for?

Don't need this anymore, I'm using https://www.npmjs.com/package/child-process-promise instead to do await childProcess.exec(...) instead.


Support nested options.

For example (using the JS API):
{
  discardComments: {removeAll: true}
}

Blocked by https://github.com/ben-eb/cssnano/issues/91 (implement a config file).

Note that when writing up this issue initially, I didn't realise that it is possible to do this already using minimist's defaults;
cssnano --discardComments.removeAll
However in the longer term I think a config file is a much saner solution.


Unsure how to deal with the removal of r2d2::Config

Sorry to ask here, I'm not sure where a better place to ask is (there's no IRC or gitter link in the README)... crates.io has this function which I am completely stumped on how to migrate to 0.8.0. Any help would be appreciated.
pub fn diesel_pool(
    url: &str,
    config: r2d2::Config<PgConnection, r2d2_diesel::Error>,
) -> DieselPool {
    let mut url = Url::parse(url).expect("Invalid database URL");
    if env::var("HEROKU").is_ok() && !url.query_pairs().any(|(k, _)| k == "sslmode") {
        url.query_pairs_mut().append_pair("sslmode", "require");
    }
    let manager = ConnectionManager::new(url.into_string());
    r2d2::Pool::new(config, manager).unwrap()
}

This is a reasonable place to ask! The analogous function would just I think be
pub fn diesel_pool(
    url: &str,
    builder: r2d2::Builder<ConnectionManager>,
) -> DieselPool {
    let mut url = Url::parse(url).expect("Invalid database URL");
    if env::var("HEROKU").is_ok() && !url.query_pairs().any(|(k, _)| k == "sslmode") {
        url.query_pairs_mut().append_pair("sslmode", "require");
    }
    let manager = ConnectionManager::new(url.into_string());
    builder.build(manager).unwrap()
}

You can also use build_unchecked to avoid the unwrap if you'd like.

I'll give that a try, thanks!


Added new card

Fixes Issue🛠
Closes #931
Description👨💻
I have added a new card that is code snippet card. The code-snippet is in form of card and can be attached in many websites.
Type of change📄

[x] New feature (non-breaking change which adds functionality)

How this has been tested✅
Cross-compatible
Checklist✅

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have added demonstration in the form of GIF/video file
[x] I am an Open Source Contributor

Screenshots/GIF📷

Please do not create a PR until you have been assigned to an issue @hars-21

Sure, I'll be careful next time but please can you assign me now @Rakesh9100


Feature request: Select format based on resolution (mainly CBS)

[x] I've verified and I assure that I'm running youtube-dl 2016.05.10

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[ ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[x] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue
Example formats from a CBS clip:
[info] Available formats for 6681E9BA-2685-F7FE-7103-EA222276F6A4:
format code  extension  resolution note
hls-47-0     mp4        audio only   47k , mp4a.40.2
hls-47-1     mp4        audio only   47k , mp4a.40.2
hls-47-2     mp4        audio only   47k , mp4a.40.2
hls-47-3     mp4        audio only   47k , mp4a.40.2
hls-meta-0   mp4        multiple   Quality selection URL
hls-meta-1   mp4        multiple   Quality selection URL
hls-meta-2   mp4        multiple   Quality selection URL
hls-meta-3   mp4        multiple   Quality selection URL
hls-1        mp4        400x224       1k
hls-110      mp4        400x224     110k
http-150-0   mp4        400x224     150k
http-150-1   mp4        400x224     150k
hls-156-0    mp4        400x224     156k , avc1.66.30,  mp4a.40.2
hls-156-1    mp4        400x224     156k , avc1.66.30,  mp4a.40.2
hls-156-2    mp4        400x224     156k , avc1.66.30,  mp4a.40.2
hls-156-3    mp4        400x224     156k , avc1.66.30,  mp4a.40.2
http-240-0   mp4        400x224     240k
http-240-1   mp4        400x224     240k
hls-250      mp4        400x224     250k
hls-309-0    mp4        400x224     309k , avc1.66.30,  mp4a.40.2
hls-309-1    mp4        400x224     309k , avc1.66.30,  mp4a.40.2
hls-309-2    mp4        400x224     309k , avc1.66.30,  mp4a.40.2
hls-309-3    mp4        400x224     309k , avc1.66.30,  mp4a.40.2
rtmp-364-0   flv        384x216     364k
rtmp-364-1   flv        384x216     364k
http-440     mp4        400x224     440k
rtmp-496-0   flv        640x360     496k
rtmp-496-1   flv        640x360     496k
hls-500      mp4        512x288     500k
hls-568-0    mp4        512x288     568k , avc1.66.30,  mp4a.40.2
hls-568-1    mp4        512x288     568k , avc1.66.30,  mp4a.40.2
hls-568-2    mp4        512x288     568k , avc1.66.30,  mp4a.40.2
hls-568-3    mp4        512x288     568k , avc1.66.30,  mp4a.40.2
http-740     mp4        640x360     740k
rtmp-796-0   flv        640x360     796k
rtmp-796-1   flv        640x360     796k
hls-800      mp4        640x360     800k
hls-845-0    mp4        640x360     845k , avc1.66.30,  mp4a.40.2
hls-845-1    mp4        640x360     845k , avc1.66.30,  mp4a.40.2
hls-845-2    mp4        640x360     845k , avc1.66.30,  mp4a.40.2
hls-845-3    mp4        640x360     845k , avc1.66.30,  mp4a.40.2
hls-1200     mp4        768x432    1200k
rtmp-1296-0  flv        640x360    1296k
rtmp-1296-1  flv        640x360    1296k
hls-1307-0   mp4        768x432    1307k , avc1.77.30,  mp4a.40.2
hls-1307-1   mp4        768x432    1307k , avc1.77.30,  mp4a.40.2
hls-1307-2   mp4        768x432    1307k , avc1.77.30,  mp4a.40.2
hls-1307-3   mp4        768x432    1307k , avc1.77.30,  mp4a.40.2
hls-1800     mp4        960x540    1800k
hls-1931-0   mp4        960x540    1931k , avc1.77.30,  mp4a.40.2
hls-1931-1   mp4        960x540    1931k , avc1.77.30,  mp4a.40.2
hls-1931-2   mp4        960x540    1931k , avc1.77.30,  mp4a.40.2
hls-1931-3   mp4        960x540    1931k , avc1.77.30,  mp4a.40.2
hls-2683-0   mp4        1024x576   2683k , avc1.77.30,  mp4a.40.2
hls-2683-1   mp4        1024x576   2683k , avc1.77.30,  mp4a.40.2
hls-2683-2   mp4        1024x576   2683k , avc1.77.30,  mp4a.40.2
hls-2683-3   mp4        1024x576   2683k , avc1.77.30,  mp4a.40.2
hls-3719-0   mp4        1280x720   3719k , avc1.77.30,  mp4a.40.2
hls-3719-1   mp4        1280x720   3719k , avc1.77.30,  mp4a.40.2
hls-3719-2   mp4        1280x720   3719k , avc1.77.30,  mp4a.40.2
hls-3719-3   mp4        1280x720   3719k , avc1.77.30,  mp4a.40.2 (best)
Description of your issue, suggested solution and other information
A lot of videos come in different resolution with often individual format codes for each clip.
I thought it would be neat, to be able to tell Youtube-DL:
"I want the download to go via HLS/RTMP/HTTP protocol."
"I want the best resolution possible within that protocol (often, but not always 1280x720)"
And: "Pick the first/second/third etc. available format within the HLS-best you have in that section."
In the example output it would be "hls-3719-0".
That would save a lot of time, since I wouldn't have to to youtube-dl -F on every clip you want to download and then wait for the format processing twice.
Additional information: There are also some clips that have another twist. Instead of one group of formats (group: "hls-3719-0" to "hls-3719-3") there might be two groups on one resolution.
Maybe something like "--prefer-protocol" for the protocol selection "--prefer-resolution" for the resolution" and "--prefer-edition" for the group number (use 0 as default). Maybe replace "prefer" by "use", I don't know. Just my ideas.

You can already do that.

From
hls-3719-0 mp4 1280x720 3719k , avc1.77.30, mp4a.40.2
hls-3719-1 mp4 1280x720 3719k , avc1.77.30, mp4a.40.2
hls-3719-2 mp4 1280x720 3719k , avc1.77.30, mp4a.40.2
hls-3719-3 mp4 1280x720 3719k , avc1.77.30, mp4a.40.2 (best)
is hls-3719-0 the worst?

They all the same.

Well, it appears to be, yes. But 4 different hls streams all with the same properties? Why?
To be sure, maybe download all four and then compare checksums ;-)


Network connectivity

[Enter feedback here]
Network connectivity for configuration server only documents Ports 443 and 9443. It would be very helpful if direction is also documented. This is mandatory information when configuring on-prem firewalls to allow ASR traffic internally (between  servers replicated and configuration server,  and externally to ASR).
Example:
Between Source machines being replicated and Configuration Server
Between Configuration server and Azure ASR (define ASR IPs)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bc12f304-5826-fb32-1728-5e43f1b05d8a
Version Independent ID: 7cc3d5a7-9b0e-dd6b-3b34-2036219f5bd6
Content: Support matrix for VMware/physical disaster recovery in Azure Site Recovery - Azure Site Recovery
Content Source: articles/site-recovery/vmware-physical-azure-support-matrix.md
Service: site-recovery
GitHub Login: @JYOTHIRMAISURI
Microsoft Alias: v-jysur

@saadatahmed
Thank you for feedback. At this time we are reviewing the ask and will provide an update as appropriate

For both the ports, 443 and 9443 (or whatever equivalent user has set for 9443), inbound traffic should be allowed on the configuration server.
We will make sure that this info is added to the doc.

Thanks for the review and confirmation, @Sharmistha-Rai.
@saadatahmed - thank you for the inputs.  We will update the relevant docs and keep you posted.

As advised by my manager @jyothisuri  closing the issues that are older than a year.

#please-close


Hale test

This PR fixes #
Nothing. This is a test. Added an extra space to two different files.

@halerankin Please don't merge anything in without signoff from at least one other core contributor. Thanks!


Support SSL using internal openshift CA

@mwringe Could you please review?

Removed SSL Version.
Use Service Serving Certificate Secrets
Removed parameter SSL enable/disable

Can one of the admins verify this patch?

Do we know how this will affect current clients? The agent is now going to have to know about the ca certificate used to sign Hawkular Services

@mwringe
I'm not sure how the customers are affected by this. if they are not using SSL this doesn't have any effect.
But I don't know how they used SSL with this template now,  the current version just generated a self-signed certificate, and store the certificate and private key in the container, the certificate wasn't exposed using some external volume or secret, so as far as I know there wasn't a way to point  the agent to the certificate or a CA to trust on it. ( I don't know if it has some configuration to skip that validation)
But anyway I'll do some tests on agents. Any other observation or issue on this?
// cc @theute @hhovsepy

We don't have to keep backward compatibility here

I did compared old and new templates, and found differences [1] and [2] which are in comments.

@hhovsepy I removed SSL support from ephemeral as requested in the review. as  @mwringe I don't see why we need to parametrize the secret, but if we need that I think I can do in another PR. WDYT?

have tried the new template persistence.yaml, it starts in non ssl mode

Verified the hawkular-services-persistent.yaml


Update es.json (English | Spanish Version)

I've updated the translations until the 66 line :)

Hey, actually you need to translate right thing, left are keys to match translation :)

Hey thank you for your translation, don't forget to post KSM address!

Here is my KSM Address i created the address on Polka.Js idk if that's compatible with NFT's :)
EoefzRoSeepV3Kzg2aABQu6F2kkdUzCMWiJcn7aDsBg91rx


system_tests_topology failure

38: self = <system_tests_topology.TopologyTests testMethod=test_01_topology_failover>
38:
38:     def test_01_topology_failover(self):
38:         name = 'test_01'
38:         if self.skip[name]:
38:             self.skipTest("Test skipped during development.")
38:         test = TopologyFailover(name,
38:                                 self.client_addrs,
38:                                 "closest/01"
38:                                 )
38:         test.run()
38: >       self.assertIsNone(test.error)
38: E       AssertionError: "expected trace ['0/A', '0/D', '0/C', '0/B'] but got ['0/A', '0/D', '0/B']" is not None

https://github.com/skupperproject/skupper-router/runs/5481493366?check_suite_focus=true

Known issue from Dispatch, https://issues.apache.org/jira/browse/DISPATCH-2021

still present, https://github.com/skupperproject/skupper-router/runs/5692504231?check_suite_focus=true#step:26:114

This test has a long history of being flakey.  I propose we disable it until we have a chance to fix it.


"More Tips" link in the header of the "Tip of the Day" block!

Changes:

Added "more tips" (can be renamed to a better name) link in the header of the "tip of the day" block
Changed layout of "latest release" and "what's happening" blocks, so layout become more flexible
For example now you can put "latest release" block in any place without <hr> element in bottom of this block

These are small changes, but they improve the UX in some way :)

All looks good. Thanks! :)


Fix a bug in One-To-N-broadcast connector

This PR fixes a bug in OneToNBroadcastConnector.

LGTM.


Cherry pick csv writer doc example  mistype to active_release

Automatic cherry-pick of d5a4bd7

Originally appeared in https://github.com/apache/arrow-rs/pull/904: csv writer doc example  mistype

Codecov Report

Merging #918 (a327c19) into active_release (03d95e6) will decrease coverage by 0.14%.
The diff coverage is n/a.

@@                Coverage Diff                 @@
##           active_release     #918      +/-   ##
==================================================
- Coverage           82.45%   82.31%   -0.15%
==================================================
  Files                 168      168
  Lines               48175    48091      -84
==================================================
- Hits                39725    39586     -139
- Misses               8450     8505      +55

Impacted Files
Coverage Δ

arrow/src/csv/writer.rs
82.05% <ø> (-0.37%)
:arrow_down:

arrow/src/array/transform/fixed_binary.rs
75.00% <0.00%> (-3.95%)
:arrow_down:

parquet/src/schema/visitor.rs
67.10% <0.00%> (-2.64%)
:arrow_down:

arrow/src/compute/kernels/cast_utils.rs
87.50% <0.00%> (-2.50%)
:arrow_down:

parquet/src/encodings/decoding.rs
90.45% <0.00%> (-2.02%)
:arrow_down:

arrow/src/datatypes/ffi.rs
70.85% <0.00%> (-2.02%)
:arrow_down:

parquet/src/arrow/arrow_reader.rs
89.32% <0.00%> (-1.95%)
:arrow_down:

arrow/src/ffi.rs
84.69% <0.00%> (-1.89%)
:arrow_down:

arrow/src/csv/reader.rs
90.00% <0.00%> (-1.78%)
:arrow_down:

arrow/src/util/data_gen.rs
75.44% <0.00%> (-1.66%)
:arrow_down:

... and 70 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03d95e6...a327c19. Read the comment docs.


feat: add jump instructions tests

Description
Tests were developed and integrated to specifically validate the jump instructions.
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

lgtm


Multi remote for electron apps

Could we consider extending multi-remote capabilities to this service?
Use cases are to drive more than one app in parallel as well as a browser, etc.

Thanks for reporting!
That makes totally sense to me!
Any contributions that resolves the request are highly appreciated. Please take a look into our contribution guidelines and let us know if you have any questions. Cheers!

I think this should be the next thing to go in, need to finish off the API PR but there shouldn't be too much collision if someone wants to start looking at this.

Just a few more unit tests to go on #19 and I'll move my attention to this; it will tie in with an investigative piece around E2Es I'm doing as part of my new role but the work is not high priority so will likely still be some weeks.

Started a PR - just took code from wdio-chromedriver-service and wdio-geckodriver-service.
https://github.com/webdriverio-community/wdio-electron-service/pull/26

Merged


Error defining POST body schema for class-based controller

Hello. I'm having an issue defining a controller for a POST endpoint containing a body with a ninja schema. It seems like ninja_extra is importing the ProductCreateSchema after it creates the api_controller causing it to not be able to import ProductCreateSchema. I also attempted importing my schema before importing ninja_extra, but no luck.
# schemas.py
"""Schemas for the shop app."""

from __future__ import annotations

from ninja import Schema
from pydantic import BaseModel

class ProductCreateSchema(Schema):
    """Schema for creating products."""

    name: str
    description: str = ""
    price: float = 0.0

# api.py
from shop.schemas import ProductCreateSchema
from ninja import Router
from django.http import HttpRequest  # noqa: TCH002
from ninja_extra import ControllerBase, http_post, permissions, api_controller
from ninja_jwt.authentication import JWTAuth

router = Router(tags=["Shop"])

class IsCmsAdmin(permissions.BasePermission):
    """Custom permission for cms_admin role."""

    def has_permission(self, request: HttpRequest, controller: ControllerBase) -> bool:  # noqa: ARG002
        """Check if user has cms_admin role."""
        user = request.auth
        return user and user.groups.filter(name="cms_admin").exists()

@api_controller
class ProductController:
    """Product API controller."""

    @http_post("/product", auth=JWTAuth(), permissions=[IsCmsAdmin])
    def create_product(
        self,
        request: HttpRequest,  # noqa: ARG002
        payload: ProductCreateSchema,
    ) -> dict:
        """Create product API endpoint."""
        product = Product.objects.create(
            name=payload.name,
            description=payload.description,
            price=payload.price,
        )
        return {
            "id": product.id,
            "name": product.name,
            "description": product.description,
            "price": product.price,
        }

File "/code/backend/urls.py", line 8, in <module>
  from backend.api import api
File "/code/backend/api.py", line 12, in <module>
  from shop.api import router as shop_router
File "/code/shop/api.py", line 33, in <module>
  @api_controller
   ^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja_extra/controllers/base.py", line 558, in api_controller
  return APIController(
         ^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja_extra/controllers/base.py", line 405, in __call__
  self._add_operation_from_route_function(v)
File "/usr/local/lib/python3.12/site-packages/ninja_extra/controllers/base.py", line 470, in _add_operation_from_route_function
  route_function.operation = self.add_api_operation(
                             ^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja_extra/controllers/base.py", line 505, in add_api_operation
  operation = path_view.add_operation(
              ^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja_extra/operation.py", line 361, in add_operation
  operation = operation_class(
              ^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja_extra/operation.py", line 66, in __init__
  super().__init__(path, methods, view_func, **kwargs)
File "/usr/local/lib/python3.12/site-packages/ninja/operation.py", line 82, in __init__
  self.signature = ViewSignature(self.path, self.view_func)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja/signature/details.py", line 48, in __init__
  self.signature = get_typed_signature(self.view_func)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja/signature/utils.py", line 43, in get_typed_signature
  annotation=get_typed_annotation(param, globalns),
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja/signature/utils.py", line 54, in get_typed_annotation
  annotation = make_forwardref(annotation, globalns)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja/signature/utils.py", line 60, in make_forwardref
  return evaluate_forwardref(forward_ref, globalns, globalns)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ninja/signature/utils.py", line 20, in evaluate_forwardref
  return cast(Any, type_)._evaluate(globalns, localns, recursive_guard=set())
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/typing.py", line 924, in _evaluate
  eval(self.__forward_code__, globalns, localns),
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 1, in <module>
backend  | NameError: name 'ProductCreateSchema' is not defined

@tmorgan497 I will look into this. But at the moment can you share your python version, ninja-extra version and your pydantic version

Also, the error message does not show where you registered the controller

I copied your code to a new Django project and it worked perfectly

@tmorgan497 I will look into this. But at the moment can you share your python version, ninja-extra version and your pydantic version

My versions are:

Python==3.12.3
django-ninja-extra==0.21.4
pydantic==2.9.2

I purged my docker volumes and reinstalled and now I'm not getting the import error. I also switched to using the router that I already have in this module. However, I now have an issue where I have to pass self as a parameter to the endpoint. Is there a way to suppress self as a query parameter for the endpoint? If I remove self from the args, then I get a linting error (which I can suppress, but I didn't know if this is the proper way to go about this).

"""API endpoints for the shop app."""

# from __future__ import annotations

import logging

from shop.schemas import BaseQueryParams, ProductResponse, ProductReadSchema, ProductCreateSchema  # isort: skip

from ninja import Query, Router
from django.http import HttpRequest
from ninja_extra import ControllerBase, permissions, api_controller
from django.db.models import Q
from django.core.paginator import Paginator
from ninja_jwt.authentication import JWTAuth
from django.contrib.postgres.search import TrigramSimilarity

from api.api import GenericResponse
from shop.models import Vendor, Product, ProductImage

router = Router(tags=["Shop"])

logger = logging.getLogger(__name__)

class IsCmsAdmin(permissions.BasePermission):
    """Custom permission for cms_admin role."""

    def has_permission(self, request: HttpRequest) -> bool:
        """Check if user has cms_admin role."""
        user = request.auth
        return user and user.groups.filter(name="cms_admin").exists()

@api_controller("shop_class/", auth=[JWTAuth()], permissions=[IsCmsAdmin])
class ProductController(ControllerBase):
    """Product API controller."""

    @router.post("/product")
    def create_product(
        request: HttpRequest,
        payload: ProductCreateSchema,
    ) -> dict:
        """Create product API endpoint."""
        product = Product.objects.create(
            name=payload.name,
            description=payload.description,
            price=payload.price,
        )
        return {
            "id": product.id,
            "name": product.name,
            "description": product.description,
            "price": product.price,
        }

UPDATE: I think I figured it out. Just leaving this comment here in case future me runs into this issue again 😅 (or if anyone else has an issue). The issue seems to be caused by from __future__ import annotations, so I removed it and followed the docs closely to come up with this fix:
"""API endpoints for the shop app."""

import logging

from ninja import Query, Router
from django.http import HttpRequest
from ninja_extra import ControllerBase, route, permissions, api_controller
from django.db.models import Q
from django.core.paginator import Paginator
from ninja_jwt.authentication import JWTAuth
from django.contrib.postgres.search import TrigramSimilarity

from api.api import GenericResponse
from shop.models import Vendor, Product, ProductImage
from shop.schemas import BaseQueryParams, ProductResponse, ProductReadSchema, ProductCreateSchema

router = Router(tags=["Shop"])

logger = logging.getLogger(__name__)

class IsCmsAdmin(permissions.BasePermission):
    """Custom permission for cms_admin role."""

    def has_permission(self, request: HttpRequest, controller: ControllerBase) -> bool:  # noqa: ARG002
        """Check if user has cms_admin role."""
        user = request.auth
        return user and user.groups.filter(name="cms_admin").exists()

@api_controller("shop_class/", auth=[JWTAuth()], permissions=[IsCmsAdmin()])
class ProductController:
    """Product API controller."""

    @route.post("/product")
    def create_product(
        self,
        payload: ProductCreateSchema,
    ) -> dict:
        """Create product API endpoint."""
        product = Product.objects.create(
            name=payload.name,
            description=payload.description,
            price=payload.price,
        )
        return {
            "id": product.id,
            "name": product.name,
            "description": product.description,
            "price": product.price,
        }

Additionally, I ensured that the class controller was being registered.
"""Main API entrypoint.

backend/backend/api.py
"""

from __future__ import annotations

from ninja_extra import NinjaExtraAPI
from ninja_jwt.controller import NinjaJWTDefaultController

from api.api import router as api_router
from shop.api import ProductController
from shop.api import router as shop_router
from authutils.api import router as auth_router

api = NinjaExtraAPI()
api.register_controllers(
    NinjaJWTDefaultController,
    ProductController,
)

api.add_router("/", api_router)
api.add_router("/auth", auth_router)
api.add_router("/shop", shop_router)

I will close the issue. Thanks @eadwinCode!


EME 2021 test 22 is loading an incorrect file

As you can see in this , test 22 fails because it creates a source buffer for video/webm and feeds an mp4 file.

Same problem with test 38, source buffer for video is opened with video/webm; codecs="vp<IP_ADDRESS>" and loaded file is the same, mp4.

Same problem with test 38, source buffer for video is opened with video/webm; codecs="vp<IP_ADDRESS>" and loaded file is the same, mp4.

Hi and sorry for the late reply. That's actually intended, the test steam is in mp4 container, the test is failing because of a typo, please expect a new release with the fix in the coming weeks. We will also update this repo.

Hi and sorry for the late reply. That's actually intended, the test steam is in mp4 container, the test is failing because of a typo, please expect a new release with the fix in the coming weeks. We will also update this repo.


Downloading one snapshot per day

Hi,
Is there a way to do it, e.g. download only the first or the last snapshot of the day?

You should be able to achieve one-snapshot-per-day via the --collapse parameter documented in waybackpack -h, which uses the Wayback Machine CDX Server API's "collapse" parameter. Does that work for you?


Migrate transport-native-unix-common tests to JUnit 5

https://github.com/netty/netty/issues/10757#issuecomment-848445551

@kashike thanks!


Can't connect to PostgreSQL via SSL

Hi
I can not connect via SSL to a PostgreSQL instance. I get the following error message :
"Could not read SSL key file....".
I have tried with pgadmin 4 and it's OK.
Is it a bug ?
Thanks in advance
Didier

Did you configure SSL certificates?
If no then you can set SSL factory to NonValidating:

Hi Serge
I have set up SSL certificates
Root certificate : C:\Users\A74708-A\AppData\Roaming\postgresql\root.crt
SSL Certificate : C:\Users\A74708-A\AppData\Roaming\postgresql\postgresql.crt
SSL Certificate key : C:\Users\A74708-A\AppData\Roaming\postgresql\postgresql.key
but the connexion is impossible.

Do you use the most recent version of DBeaver?
Some PG SSL issues were fixed recently in #1972

I use DBeaver version 4.2.0 September 4, 2017
NB : For a security matter , it is very important that we can connect via SSL.
PG_ADMIN 4 can do it. And I try to impose DBeaver as the main tool for querying our PostgreSQL database.
Thanks in advance
Didier

Agreed. This can be a DBeaver bug or some SSL misconfiguration.
Please send error stacktrace (you can get it in Error Log view or in log files: https://github.com/serge-rider/dbeaver/wiki/Log-files)
(note: strip any secure information from logs before posting).

Hi
Sorry for the delay. I was too busy.
Here are the log-files+
.dbeaver4.zip

oops . I make a mistake. I close the case and it is not fixed !

Looks like your SSL key certificate is in unsupported (by Java) format.
Check this: https://stackoverflow.com/questions/11621414/reading-an-x-509-certificate-with-java
Converting key cert format could be done with command
openssl x509 -outform der -in certificate.pem -out certificate.der

This command can be used to convert client.key file, in case you've followed postgres document to generate the pair client.crt and client.key.
openssl pkcs8 -topk8 -inform PEM -outform DER -in client.key -out client.pk8 -nocrypt

Btw, I really hope DBeaver can support PEM format by default.

is there a way to disable ssl check ?

Just select disable option in SSL mode

 Just select disable option in SSL mode

If you want to connect via SSL, disable is not what you need! It disables SSL
For me, setting SSL Mode to require fixed the issue and ensures SSL is used


process _read is dropping methylation events on reads mapping to the reverse strand (16)

I have two bam files :
watson_read.bam which contains one read with SAM flag 0
crick_read.bam which contains one read with SAM flag 16.
Both BAM files have  MM:Z:A+a. and  ML:B:C,  m6a methylation positions and quality scores.
!modbedtools bam2mod -c .6 -b A watson_read.bam -o watson_read  --> generates a correct modbed file
!modbedtools bam2mod -c .6 -b A crick_read.bam -o crick_read --> generates an empty modbed file
        if base == 'A':
            modbase_key = ('T', 1, 'a') if read.is_reverse else ('A', 0, 'a')
        if modbase_key not in read.modified_bases:
            return []

walking through the code, modbase_key = ('T',1,'a')  is set correctly when read.is_reverse (16)
However, when modbase_key=('T',1,'a')  this does not match read.modified_bases which returns ('A', 1, 'a')  and empty data is returned.

Hi @mitomac thanks for reporting this issue, possible you can share a small fraction of both the bam files to me? we probably need apply some patch to the code but would like to test on your files as well.

Thanks for the prompt reply. Here are two toy bam files each with a watson
(0) or crick read (16)
And the command I used.
modbedtools bam2mod -c .6 -b A watson_read.bam -o watson_read
modbedtools bam2mod -c .6 -b A crick_read.bam -o crick_read
I also noticed that when using the addbg command that for reads near the
telomere and which might extend past the telomere that there can be an
indexing error.   So I added a simple sanity check to the addbg function.
This might only be an issue in the smaller s. cerevisiae genome and might
be a function of how the repetitive telomeres are present and accounted for
in the genome file.
for x in range(start, end + 1):
if x >= len(s):  # Check if x exceeds the chromosome length
continue  # Skip if it does
Cheers!
Dave
On Wed, Apr 3, 2024 at 12:03 PM Daofeng Li @.***> wrote:

Hi @mitomac https://github.com/mitomac thanks for reporting this issue,
possible you can share a small fraction of both the bam files to me? we
probably need apply some patch to the code but would like to test on your
files as well.
—
Reply to this email directly, view it on GitHub
https://github.com/lidaof/modbedtools/issues/1#issuecomment-2035008971,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD5VBMHXXMPWVJVRES57UZDY3QR5JAVCNFSM6AAAAABFUMCV72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZVGAYDQOJXGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi @mitomac thanks for the reply, i assume the here should be a link? :)
Bye the way, possible you can try change that line:
modbase_key = ('T', 1, 'a') if read.is_reverse else ('A', 0, 'a')

to
modbase_key = ('T', 1, 'a') if read.is_reverse else ('A', 1, 'a')

see if it works for both strand bams? (from what you described?)
also thanks for the sanity check for the length issue.
If above change works I would like to invite you to send a PR, then we can merge your contributions to this tool.
Thank you!

Hi @mitomac, would like to follow up on this issue see if the fix works? I think github might remove your attachment :)

My apologies for the delay -- day job got in the way.  :)
I  forked your repository, added the test bam files, and the sanity check
for length.  Unfortunately, your suggested change didn't work and
returned empty files for both 0 and 16 reads.

Bye the way, possible you can try change that line:
modbase_key = ('T', 1, 'a') if read.is_reverse else ('A', 0, 'a')
to
modbase_key = ('T', 1, 'a') if read.is_reverse else ('A', 1, 'a')

Here is the repository with the test bam files.
https://github.com/mitomac/modbedtools
Best,
Dave
On Fri, Apr 5, 2024 at 9:55 AM Daofeng Li @.***> wrote:

Hi @mitomac https://github.com/mitomac, would like to follow up on this
issue see if the fix works? I think github might remove your attachment :)
—
Reply to this email directly, view it on GitHub
https://github.com/lidaof/modbedtools/issues/1#issuecomment-2039871058,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD5VBMFZZGA6SNUVRVW4D6DY32UNVAVCNFSM6AAAAABFUMCV72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZZHA3TCMBVHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi @mitomac David, sorry for the delay, apparently the key should be ('A', 1, 0), my bad...
I have pushed a new version, also included your length check, which should fix this:
pip install modbedtools==0.1.6
Thank you!


Canabalt [x86]: Missing button text

Desktop running 1491
Run the app (debug/release)
Expected: App runs, shows buttons after splash screen appears.
Actual: App runs, but missing button text

This is a duplicate of #924.


It's easy to get files outside of the static dir

We don't stop things like http://host:8080/../../../etc/passwd

If static root dir is not set, it is even simplier: http://host:8080/etc/passwd


Serve legacy bundles to Safari 10.1 and 11

Supersedes https://github.com/fusionjs/fusion-cli/pull/640
This solution is better because 10.1 and 11 are uncommon versions and this fix doesn't degrade the fast path for compliant browsers.

!merge


Add perms for remote snapshot cache eviction on scripted query

Description
Scripted queries require elevated permissions to write to a remote snapshot local file cache. Since accessing the file cache can trigger an eviction the AccessController doPrivileged call needs to move up the call stack so we have permissions to perform file deletions for these evictions.
Reproduced and verified fix with OSB StackOverflow workload:

Run OSB StackOverflow workload.
Create snapshot from so index.
Restore from remote snapshot.
Query with a scripted field.

curl -X POST "http://localhost:9200/so/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "body": "java"
    }
  },
  "script_fields": {
    "combined_field": {
      "script": {
        "source": "params[\"_source\"][\"user\"] + \" \" + params[\"_source\"][\"body\"]"
      }
    }
  },
  "size": 100
}
'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "failed to parse / load source"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "so",
        "node" : "Q87dFmoFQ4ivpVr0ng82NQ",
        "reason" : {
          "type" : "script_exception",
          "reason" : "runtime error",
          "script_stack" : [
            "org.opensearch.search.lookup.SourceLookup.loadSourceIfNeeded(SourceLookup.java:115)",
            "org.opensearch.script.FieldScript.lambda$static$2(FieldScript.java:72)",
            "org.opensearch.script.DynamicMap.get(DynamicMap.java:84)",
            "params[\"_source\"][\"user\"] + \" \" + params[\"_source\"][\"body\"]",
            "       ^---- HERE"
          ],
          "script" : "params[\"_source\"][\"user\"] + \" \" + params[\"_source\"][\"body\"]",
          "lang" : "painless",
          "position" : {
            "offset" : 7,
            "start" : 0,
            "end" : 59
          },
          "caused_by" : {
            "type" : "parse_exception",
            "reason" : "failed to parse / load source",
            "caused_by" : {
              "type" : "access_control_exception",
              "reason" : "access denied (\"java.io.FilePermission\" \"/local/home/carrofin/DEBUG/STORES/DATA/nodes/0/cache/nI-w4Uy5QzeP73xjaqu4SQ/0/RemoteLocalStore/_nd.fdt.87\" \"delete\")"
            }
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "parse_exception",
      "reason" : "failed to parse / load source",
      "caused_by" : {
        "type" : "access_control_exception",
        "reason" : "access denied (\"java.io.FilePermission\" \"/local/home/carrofin/DEBUG/STORES/DATA/nodes/0/cache/nI-w4Uy5QzeP73xjaqu4SQ/0/RemoteLocalStore/_nd.fdt.87\" \"delete\")"
      }
    }
  },
  "status" : 400
}

Related Issues
Resolves #14268
Check List

[ ] Functionality includes testing.
[ ] API changes companion pull request created, if applicable.
[ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Thanks @finnegancarroll! Can you see if you can add a test to SearchableSnapshotIT that fails without this fix?

Let's add a changelog entry under "Fixed" as well. Something like "Fix searchable snapshot failure with scripted fields".

Sorry for the delay @finnegancarroll. Can you rebase this PR? Also, did you look at adding a test case for this scenario?

Sorry for the delay @finnegancarroll. Can you rebase this PR? Also, did you look at adding a test case for this scenario?
I'm having trouble reproducing in an integration test. After hacking the cache capacity and block size such that I get evictions I do not see permission issues when running a scripted query. I think the way scripted queries are mocked in the integration tests by overriding protected Map<String, Function<Map<String, Object>, Object>> pluginScripts() is not adequately reproducing the permissions.

Flaky tests
https://github.com/opensearch-project/OpenSearch/issues/5329
https://github.com/opensearch-project/OpenSearch/issues/13846

@andrross when you have a moment can you take a look?
Thanks!


riverpod migration

Riverpod migration
Initilaize the repository and adjust the sharedpreferences

i am ready to work on this issue


scipy.spatial.transform.rotation documentation

I would like to see the documentation that better explains the "angles" argument to functions like from_euler() and elsewhere in the Rotation class. It does not seem to be explicitly stated anywhere, for example, that the angles are [rotation_about_z, rotation_about_y, rotation_about_x]. And while one can specify the order of rotations with the seq argument, without that information one has to reverse engineer which angles apply to which rotation.
Thanks!
-Val

Hi @valschmidt Care to submit a PR to make the documentation clearer? It is sometimes difficult for us to come up with better explanation than what we have but reviewing is epsilon-more easier for us to get the context.

Done!


Clicking on Tab with cd command in the cmd should end with a slash

Description of the new feature/enhancement

Proposed technical implementation details (optional)

I'm going to presume you meant "pressing tab with a cd command...".
@venuBondugula Are you using cmd.exe or powershell.exe/pwsh.exe?

I'm going to presume you meant "pressing tab with a cd command...".
@venuBondugula Are you using cmd.exe or powershell.exe/pwsh.exe?

Yes pressing tab in cmd.exe

Sorry -- we're not going to change CMD. You need to "confirm" the tab completion anyway, so you could just do it by pressing \ instead of .. whatever other thing you were doing that \ automatically appearing would have made easier.


TERM and NO_COLOR envirnoment vars still not respected

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce
As per "Enable use of TERM and NO_COLOR env var plaintext #14969" the TERM and NO_COLOR variables should disable coloring of errors in pwsh. But executing the showbug.bat in command.exe in the root directory of pwsh version 7 4 1 with this content:
pwsh.exe -ExecutionPolicy Unrestricted ^
   -Command "$host.version ; " ^
   "Get-Item  Env:TERM ; " ^
   "Get-Item  Env:NO_COLOR ; " ^
   "Get-It1"

Shows:
 Major  Minor  Build  Revision
 -----  -----  -----  --------
 7      4      1      -1

 PSPath        : Microsoft.PowerShell.Core\Environment::TERM
 PSDrive       : Env
 PSProvider    : Microsoft.PowerShell.Core\Environment
 PSIsContainer : False
 Key           : TERM
 Value         : xterm-mono
 Name          : TERM

 PSPath        : Microsoft.PowerShell.Core\Environment::NO_COLOR
 PSDrive       : Env
 PSProvider    : Microsoft.PowerShell.Core\Environment
 PSIsContainer : False
 Key           : NO_COLOR
 Value         : True
 Name          : NO_COLOR

 Get-It1: The term 'Get-It1' is not recognized as a name of a cmdlet, function, script file, or executable program.
 Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

Here, the error message, triggered intentionally to demonstrate the bug, is RED colored.
Expected behavior
The error message should be printed without the colors.

Actual behavior
The error is RED colored, in spite of the environment variables requesting no colors and being clearly readable by the pwsh.exe.

Error details
The error in the above example is triggered intentionally to show that the error message is colored RED.
The error here is that the error message itself is displayed using colors, which is not what is supposed to happen according to the #14969

Environment data
PS C:\tmp\pwsh741> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.4.1
PSEdition                      Core
GitCommitId                    7.4.1
OS                             Microsoft Windows 6.3.9600
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Visuals

NO_COLOR does get recognized and affects $PSStyle.OutputRending
$ NO_COLOR=true pwsh -c '$PSStyle.OutputRendering'
PlainText
$ pwsh -c '$PSStyle.OutputRendering'
Host

According to about ANSI terminals

OutputRendering only applies to rendering in the Host, Out-File, and Out-String. Output from native executables isn't affected.

So I question if the error channel is included in that handling.

The rationale is simple: if the colors are enforced by default, they will make the output for some users completely unreadable. There should be a way for users set a variable to prevent colors from being used:
https://no-color.org/
At the moment not only the error output is always colored, the same happens for the PS prompt input: on a gray background the input is yellow or light green both of which are unreadable.
NO_COLOR is not recognized and not respected for errors and for prompts.

the input is yellow or light green both of which are unreadable.

Agreed!
PowerShell on osx how to force to be completely monochrome

You can set monochrome pwsh:

It means not using any colour escape codes in any output.

NO_COLOR also set colors with escape sequences, just reduce base 16 console colors to 2

escape 37 m sets the foreground colour to white, it does not mean 'no colour'. So if the background is white or yellow it becomes hard to read.

escape 37 m sets the foreground colour to white

No! This is 8th color of terminal color palette.

It means not using any colour escape codes in any output.

NO_COLOR also set colors with escape sequences, just reduce base 16 console colors to 2

From no-color.org

The goal is not have any ANSI colour code escape sequences in the output.

prevents the addition of ANSI color.

How when terminal does understand what color it has to output? Any symbol, any color and any hotkey are esc-sequences

How when terminal does understand what color it has to output? Any symbol, any color and any hotkey are esc-sequences

A true monochrome terminal does not have that problem, there are no colours to choose from, It may be white on black, green on black or amber on black, some monochrome terminals allow you to have reversed characters.  The application does not deal with any colours at all, it just prints text and it is printed how the terminal sees fit.

A true monochrome terminal does not have that problem

What does it mean "true"? Kernel console 80x25?

A true monochrome terminal does not have that problem

What does it mean "true"? Kernel console 80x25?

VT102, VT52 were all monochrome terminals

VT102, VT52 were all monochrome terminals

These are hardware terminals, let's get closer to life.

The original xterm was monochrome as not everyone could afford colour.

I do not remember "original" xterm. But I do remember that xterm configured with ~/.Xresources
https://invisible-island.net/xterm/ctlseqs/ctlseqs.html

These are hardware terminals, let's get closer to life.

Modern Macintosh Terminal Console offers many themes, including black on yellow. But PowerShell uses Yellow on Yellow, It also offers black and white. PowerShell gives you yellow on white. The terminals also work on according to the light/dark mode and will change colours during the day. So more that 50% of the time the
PowerShell colour configuration is wrong, if you reconfigure it, then it will be wrong again when the dark/light mode changes. It will be wrong if you have multiple terminals open using different themes. So the solution is we want, when asked, is for PowerShell to not do any colours at all.

But PowerShell uses Yellow on Yellow

Just test two commands, there is difference in order of colors:
0..8 | % { Write-Host $([consoleColor] $_) -f $_ }
30..37 | % { "`e[$($_)mTEST"

The .Net...

How when terminal does understand what color it has to output? Any symbol, any color and any hotkey are esc-sequences

For a programmer, the correct answer is to see how it's implemented elsewhere.
And example of using https://github.com/termcolor/termcolor

demo-temcolor-nocolor.zip

For a programmer, the correct answer is to see how it's implemented elsewhere.

Or to ask what the user requirement is.

demo-termcolor.py

Python 3.11.6:
ModuleNotFoundError: No module named 'termcolor'

ModuleNotFoundError: No module named 'termcolor'

On windows:
py -m pip install termcolor

Adding  -NoProfile doesn't change the bug reported in this entry. The error output is still colored and it shouldn't be. Everything else is a tangential discussion.

The error output is still colored and it shouldn't be.

Not here:

I haven't seen in your outputs that you ever see red errors? What's the step when the red errors stop being red in your case, and how does it look before that step?

and how does it look before that step? I think PSReadLine is completely irrelevant for that

I'm simply extracting

I do the same on Linux and there is not red error. What terminal emulator are you using? And I set in registry:
$ (Get-ItemProperty HKCU:\Console).VirtualTerminalLevel
1

I'm opening a %windir%\system32\cmd.exe  and I haven't created any profile.ps1 myself and the only ones I see are something like in C:\Windows\System32\WindowsPowerShell\v1.0\Examples\profile.ps1 and these have only comments inside and should not be seen by the pwsh at all. And as you see adding -nop changes nothing.
I also don't have VirtualTerminalLevel anywhere in registry. Can you try to remove your entry and see what happens?

This gives black on white in PuTTY

You can switch between black-on-white or white-on-black using the [X] System Colours checkbox

I'm reporting here that pwsh.exe fails to behave according to NO_COLOR.
I'm not looking for workarounds using some other programs which can change behavior of all programs started through them.

I was looking for a way of easily testing the black text on white background scenarios, and agree that pwsh should simply be using the existing terminals colours.

Why not using Windows features directly?

Why not using Windows features directly?

I am sure there are ways, I found it the easiest way to have different profiles save under different names.

I found it the easiest way to have different profiles

I have this also only using default features and no external programs: the shortcuts remember the settings: starting from the shortcut to cmd.exe in the taskbar, you see black text on grey. That black background was cmd.exe started from the start menu.

But that really doesn't matter for this issue. The issue is still:
The pwsh.exe fails to behave according to NO_COLOR.

You can switch between white and black background/foreground:
only background and foreground:
"`e]4;0;#ffffff;7;#000000`a"

all 16 colors:
$esc = (1..15 | % { '{0};{1}' -f $_,'#000' }) -join ';'
"`e]4;0;#fff;$esc`a"
$PSStyle.OutputRendering = "PlainText"

This does not affect PSReadline. It has to be configured separately.

Just seeing the initial
OS                             Microsoft Windows 6.3.9600

Is this a facet of the way PowerShell behaves on older console hosts. ?

As you can see on Windows 11 with Windows Terminal,  It works as expected. BUT PSReadline (IIRC) does not use ansi sequences

In the no_color instance writing the ansi blue sequence has no effect, but unsurprisingly Write-host goes around it, and I believe that's the method used by PSReadline.

BUT PSReadline (IIRC) does not use ansi sequences

If you type Get-PSReadlineOption you can see that it uses them. This is by default:
CommandColor                           : "`e[93m"
CommentColor                           : "`e[32m"
ContinuationPromptColor                : "`e[37m"
DefaultTokenColor                      : "`e[37m"
EmphasisColor                          : "`e[96m"
ErrorColor                             : "`e[91m"
InlinePredictionColor                  : "`e[97;2;3m"
KeywordColor                           : "`e[92m"
ListPredictionColor                    : "`e[33m"
ListPredictionSelectedColor            : "`e[48;5;238m"
ListPredictionTooltipColor             : "`e[97;2;3m"
MemberColor                            : "`e[37m"
NumberColor                            : "`e[97m"
OperatorColor                          : "`e[90m"
ParameterColor                         : "`e[90m"
SelectionColor                         : "`e[30;47m"
StringColor                            : "`e[36m"
TypeColor                              : "`e[37m"
VariableColor                          : "`e[92m"

Unfortunately the colours can't be defined as $Null to say do not set the colour. However the goal is not to have the colours configured to achieve the effect, the goal is to have all colours ignored when NO_COLOR is set.

the goal is to have all colours ignored when NO_COLOR is set

Yes. That's why #3918

Yes, but what we missed was NO_COLOR affects the output channel, but not the error channel in pwsh.

Everything works for me, stderr is not red, look at screenshots above, why it doesn’t work for you, I don’t know. Maybe it depends on the version of Windows.

Yes, just tested on Linux
$ NO_COLOR=true pwsh -c "this-does-not-exist"
this-does-not-exist: The term 'this-does-not-exist' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

and that did not print any colours. But if removed the NO_COLOR=true then it was all red.

@janko-js:

Microsoft Windows 6.3.9600 suggests you're running Windows 8.1, which is no longer supported (since v7.2, the minimum version has been Windows 10) - see https://learn.microsoft.com/en-us/powershell/scripting/install/powershell-support-lifecycle?view=powershell-7.4#supported-platforms

While the MSI installer in 7.4 includes a minimum version check, unfortunately it apparently thinks Windows 8.1 is still supported:
https://github.com/PowerShell/PowerShell/blob/4f02a89906a92f684ff4511ae74958f602edf800/assets/wix/Product.wxs#L136-L141

Like others, I don't see the problem (on my W11 22H2 machine). If you can reproduce the problem on a still-supported OS version, please let us know.

A few general points:

The interactive (input) shell experience is not covered by the NO_COLOR and TERM behavioral contract, if you will - only output.

The command-line editing experience and its coloring is provided by a separate module, PSReadLine, and possibly adding support for a monochrome mode for it too is the subject of the following feature request in the latter's repo: https://github.com/PowerShell/PSReadLine/issues/3918

In repro steps involving the PowerShell CLI, it is advisable to use -NoProfile, so as to rule out potential side effects from profiles.

The VirtualTerminalLevel registry setting is unrelated to the issue at hand: it tells conhost.exe whether regular console windows should render VT (ANSI) sequences as such when emitted by programs. However, PowerShell manages this setting dynamically: for output from PowerShell commands, it unconditionally enables this support, and for output from external programs, it respects the registry setting.

The dynamic switching between support and non-support is currently broken in pipelines involving an external program followed by a PowerShell command that produces colored output (Windows Terminal is _not affected), e.g. cmd /c echo hi | Select-String h - see #20898

BUT PSReadline (IIRC) does not use ansi sequences

If you type Get-PSReadlineOption you can see that it uses them. This is by default:
CommandColor                           : "`e[93m"
CommentColor                           : "`e[32m"

Yes but I have something in my memory which says that's not what actually goes to the console. PSR knows how to translate "blue" to an ansi sequence, and I wish I could find where I got the idea that it does the equivalent of a write-host -foreground and not write-output "ANSI_seq"
But that is what this the screen shot below suggests

Note the colour goes from the battery-life bars in my prompt as well, it is only PSR which seems able to emit colour.

what actually goes to the console

The "blue" is 94m ))

You can test color palette:
0..15 | % { Write-Host $([consoleColor] $_) -f $_ }

what actually goes to the console

The "blue" is 94m ))
You can test color palette:

Yes. But  Write-host -foreground Blue  and   write-host "e[94mblue"` work differently. `No_color` renders ANSI escape sequences ineffective but doesn't stop  `-Foreground` working.

I do not know for 100% certain what happens inside PSReadline. But the fact its colours are not disabled when colours of everything else using ANSI is would suggest it is translating the ANSI it is given to console commands, but I can't find where I got the idea that is what is happening.

This behavior is strange. $env:NO_COLOR = 1 sets $PSStyle.OutputRendering = "PlainText", but this has no effect on Write-Host with System.ConsoleColor enum in parameters.
But if I entirely set mono for terminal then Write-Host is also monochrome:
# mono.ps1

param ( [Parameter()] [string] $color = "black" )

if ($color -eq "white") { $bg, $fg = "#ccc", "#000" }
else { $bg, $fg = "#111", "#ddd" }

[console]::Write("`e]12;$fg`a")

$PSStyle.OutputRendering = "PlainText"
Set-PSReadlineOption -Colors @{
    Command = "`e[37m"
    Comment = "`e[37m"
    ContinuationPrompt  = "`e[37m"
    Default = "`e[37m"
    Emphasis = "`e[37m"
    Error = "`e[37m"
    InlinePrediction  = "`e[37m"
    Keyword = "`e[37m"
    ListPrediction = "`e[37m"
    ListPredictionSelected  = "`e[37m"
    ListPredictionTooltip  = "`e[37m"
    Member = "`e[37m"
    Number  = "`e[37m"
    Operator  = "`e[37m"
    Parameter  = "`e[37m"
    Selection =  "`e[7m"
    String  = "`e[37m"
    Type =  "`e[37m"
    Variable  = "`e[37m"
}

$col = (1..15 | Foreach-Object { '{0};{1}' -f $_, $fg }) -join ';'
[console]::Write("`e]4;0;$bg;$col`a")

Works in conhost and it can work in terminals without living config reloading:
PS > mono white  # or black

@237dmitry, the Write-Host behavior seems to be an (actual) bug - or at least a baffling cross-platform inconsistency; see:

#21168

This behavior is strange. $env:NO_COLOR = 1 sets $PSStyle.OutputRendering = "PlainText", but this has no effect on Write-Host with System.ConsoleColor enum in parameters.
But if I entirely set mono for terminal then Write-Host is also monochrome:
# mono.ps1

param ( [Parameter()] [string] $color = "black" )

if ($color -eq "white") { $bg, $fg = "#ccc", "#000" }
else { $bg, $fg = "#111", "#ddd" }

[console]::Write("`e]12;$fg`a")

$PSStyle.OutputRendering = "PlainText"
Set-PSReadlineOption -Colors @{
    Command = "`e[37m"
    Comment = "`e[37m"
    ContinuationPrompt  = "`e[37m"
    Default = "`e[37m"
    Emphasis = "`e[37m"
    Error = "`e[37m"
    InlinePrediction  = "`e[37m"
    Keyword = "`e[37m"
    ListPrediction = "`e[37m"
    ListPredictionSelected  = "`e[37m"
    ListPredictionTooltip  = "`e[37m"
    Member = "`e[37m"
    Number  = "`e[37m"
    Operator  = "`e[37m"
    Parameter  = "`e[37m"
    Selection =  "`e[7m"
    String  = "`e[37m"
    Type =  "`e[37m"
    Variable  = "`e[37m"
}

$col = (1..15 | Foreach-Object { '{0};{1}' -f $_, $fg }) -join ';'
[console]::Write("`e]4;0;$bg;$col`a")

Works in conhost and it can work in terminals without living config reloading:
PS > mono white  # or black

Write-host is not selecting (e.g) RGB of 0,0,255 for blue it's selecting terminal colour 9
What your code does is set all the terminal colours to the same thing

Note also (1)

Write-host strips ansi when No_color is true, but -Foreground  is not affected.
(2) What in one place is called blue is either "BrightBlue" or "DarkBlue" in the the other
(3) These changes affect the console even after the process which applied them exits.

Write-host strips ansi when No_color is true, but -Foreground is not affected.

Rather than "stripping ansi", I suggest code should not be writing ANSI colour escape sequences to the output in the first place.

What your code does is set all the terminal colours to the same thing

Yes. What does it do to reset color palette to two colors (background and foreground)? Monochrome is two colors 37m and 40m and their values are taken from the terminal settings, where the 1st is background and the 8tn is foreground.

Monochrome is two colors 37m and 40m and their values are taken from the terminal settings, where the 1st is background and the 8tn is foreground.

If the terminal is monochrome terminal and does not handle ANSI colour escape sequences and it will just print out the text the best it can, so your output would be littered with the literals "37m" and "40m" in the content.

@rhubarb-geek-nz, regarding Write-Host  stripping vs. not writing VT escape sequences:

I agree that it should not write them to begin with when -ForegroundColor / -BackgroundColor are used - that's the subject of #21168; the fact that it always writes them on Windows (whereas  on Unix only with NO_COLOR it does not)  is probably due to the fact that the legacy WriteConsoleOutput WinAPI function is used there, whose color support is unrelated to and predates VT support.

Separately, active escape-sequence stripping is performed when $PSStyle.OutputRendering = 'PlainText' is in effect, as follows:

When PowerShell performs to-terminal output , as opposed to an external program call's output being passed through to the terminal.
Intra-session, this only affects the display of data, but an outside caller actually receives the stripped text.

Here's a Unix example; run from bash:
# "green" prints in green, because the external printf utility prints directly to the terminal
# (and itself doesn't honor NO_COLOR)
NO_COLOR=1 pwsh -nop -c "printf 'It ain''t easy being \033[32mgreen\033[m.\n'"

# "green" does NOT print in green, because the use of (...) means that PowerShell captures and relays
# the output, and performs active VT-sequence stripping due to NO_COLOR -> $PSStyle.OutputRendering = 'PlainText'
NO_COLOR=1 pwsh -nop -c "printf 'It ain''t easy being \033[32mgreen\033[m.\n'"

Here's a Unix example; run from bash:

This has to be colored or not?

If you explicitly print any escape sequence then that is what should be output because that is what you said to do.
With NO_COLOR there should be no need for any filtering, it is up to the code to abide by the request to not output any colour.

My bad, @237dmitry - I forgot to add the (...) enclosure, which triggers PowerShell's VT-sequence stripping  (I've updated my comment):
# From Bash
NO_COLOR=1 pwsh -nop -c "(printf 'It ain''t easy being \033[32mgreen\033[m.\n')"

Now the word "green" should NOT be colored.

@rhubarb-geek-nz

If you explicitly print any escape sequence then that is what should be output

I don't disagree, but - unlike the buggy, inconsistent Write-Host -Fore/BackgroundColor behavior - this stripping is by intentional design.
That said, you could argue that outside callers, specifically, then see actively modified data is a buggy (possibly accidental implementation) aspect of this design.
(Intra-session, the problem doesn't arise: the data itself isn't modified during further processing, only the display is impacted).
In other words: it may worth opening a separate issue to question the CLI behavior when stripping is in effect.

If the terminal is monochrome terminal and does not handle ANSI colour escape sequences

Try to see rgb value of background (for example):
"`e]4;0;?`a"

All colors are escape sequences. Why they do not handle? I think they are write directly into the host.

All colors are escape sequences. Why do they not handle? I think they write directly into the host.

All colours are escape sequences, and monochrome terminals do not deal with colours.

and monochrome terminals do not deal with colours.

Where does your yellow-black terminal get the colors?

Where does your yellow-black terminal get the colors?

I don't know and I don't care because that is internal to the terminal implementation. What I want is my terminals to work with their existing themes.
Years ago I wrote a nice simple terminal, WishStream, this was Windows Mobile/Windows 8 Store app. The foreground and background colours were defined by the user through the XAML Theme, if they wanted it dark it would be light on dark, if they wanted light, they got dark on light.
As it was a monochrome terminal it implemented the VT100 escape sequences, which do not include colour. So this terminal ignored all colour escape sequences because they were meaningless, and as it set TERM=vt100, no application should have been sending any colour escape sequences. So vi etc all worked happily because they abided by the terminfo regarding what a vt100 terminal could do.

What I want is my terminals to work with their existing themes.

Сonfigure pwsh and terminal yourself. This is the easiest way in the current situation. Or wait for the PSReadline developers to listen to our wishes.

Сonfigure pwsh and terminal yourself. This is the easiest way in the current situation. Or wait for the PSReadline developers to listen to our wishes.

The other option is to not use PowerShell.  I have to open a new session explicitly different to use PowerShell.

I could not launch bash in monochrome mode, all of TERM=mono|vt100, NO_COLOR=1|true do not work at all. only for one-line syntax and not for all applications and utilities. Even in TTY (TERM=linux).

If I launched bash:
bash --noprofile --norc

It still started with full set of configurations from ~/.profile and ~/.bashrc

Write-host strips ansi when No_color is true, but -Foreground is not affected.

Rather than "stripping ansi", I suggest code should not be writing ANSI colour escape sequences to the output in the first place when NO_COLOR is present or OutputRendering = PlainText.

I would suggest that authors need to think REALLY hard about use of any kind of colour, and a lot harder than any actually do.
I think the chance of getting authors to write if  no_color around their not terrible well thought out colours isn't great.
If a script that doesn't check the users preference usesWrite-host  "$($PSStyle.Foreground.Yellow) Something $($PSStyle.Reset)"  in place of the frankly easier and longer established Write-host  "Something" -Foreground  yellow   then PowerShell will apply NO_COLOR as the user wishes.

What I want is my terminals to work with their existing themes
There is always the scope to create  terminal themes which become nonsensical . Terminal themes are a set of "slots" which represent colours by convention, and we can set exactly what R,G,B values are used for "Red-on-this-terminal" VS Code's default red and / Windows Terminal default Red are not automatically the same or something simple like 255,0,0.  Windows PowerShell re-defined Dark Magenta in the default terminal settings for PowerShell.exe so that it isn't any kind of magenta but a dark blue and then set the background not to black but dark magenta.

Absolutely any assumption that an application theme makes about the terminal has in each slot can be wrong footed:
what you or I set for winterm / vs code - because of taste or address an eyesight issue we have - can result in PSReadline displaying unhelpfully.
I think there is an argument that at start-up PSreadline should look at No_color and if found set all it's colours to the default foreground colour. That's a different branch from this though.

I'm a bit lost in this thread.  The original problem doesn't repro for me:
$env:NO_COLOR=1
pwsh -noprofile -command '$psversiontable;get-it'

@SteveL-MSFT, I don't think there's anything actionable here, as I've tried to express in my previous summary:

We have no repro for the original problem, except on a no longer supported Windows version, 8.1, which makes it a moot point. (Also, the OP appears to have deleted their account.)

The conceptually related, but separate question of PSReadLine's behavior with respect to coloring of interactive user input at the command prompt should be discussed in:

https://github.com/PowerShell/PSReadLine/issues/3918

Landed here while searching the same issue.  My use case is that I'm starting pwsh with redirected stdin, stdout, and stderr.  With NO_COLOR environment variable set, I'd expect there to be no color escape sequences in neither stdout nor stderr.
Maybe I'm wrong to assume this, but it seems like a logical conclusion based on the docs here:  https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_ansi_terminals?view=powershell-7.4#disabling-ansi-output
Here's a quick repro:
using System.Diagnostics;

var psi = new ProcessStartInfo
{
    FileName = "pwsh.exe",
    RedirectStandardError = true,
    RedirectStandardOutput = true,
    RedirectStandardInput = true,
};
psi.Environment.Add("NO_COLOR", "1");

var process = Process.Start(psi)!;
process.OutputDataReceived += (sender, e) =>
{
    Console.WriteLine(e.Data);
};
process.ErrorDataReceived += (sender, e) =>
{
    Console.WriteLine(e.Data);
};

process.BeginOutputReadLine();
process.BeginErrorReadLine();

await process.StandardInput.WriteLineAsync("$env:NO_COLOR;$psversiontable;get-it;exit;");
await process.WaitForExitAsync();

I set NO_COLOR:1 as a machine environment variable but still get color codes in Azure Automation logs

This also happens when redirecting stdin, stdout, stderr with Win32. Setting the __SuppressAnsiEscapeSequences environment variable makes it work as expected.

Thanks for mentioning the __SuppressAnsiEscapeSequences environment-variable workaround, but it's not a real solution, given the comment made by a team member in https://github.com/PowerShell/PowerShell/issues/19188#issuecomment-1601562724:

we're a little concerned about your use of __SuppressAnsiEscapeSequences as it is a __* variable which are generally (and traditionally) internal variables. (which is one of the reasons it's not documented)

Another related issue:

#19961

As noted in the linked issue, it seems that the bug surfaces when redirecting stderr output, specifically, and I've posted a simple repro there.
Since redirecting stderr is also necessary to make the repro in this issue's initial post produce the symptom, I suggest closing this issue in favor of the aforementioned #19961.


drop down list

do

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 24 Oct 2014 at 10:23

use OverlayContainer

Draft version of DropDownSelector implemented

initial implementation done


Nautobot Docs Revamp

What's Changed

Changed indent to 2 spaces in mkdocs.yml and made it yamllint compliant
Added assets folder for theme related assets and overrides

Leaving nautobot logos in docs for now to avoid any breakage if they are linked to

Changed theme to "Material for mkdocs" with NTC customization and branding
Revamped landing page (root index.md) to include an overview of the documentation structure.
Added the Nautobot Apps section to showcase and link to other documentation sites.
Added new docs dependencies to poetry dev-dependencies
Tested that docs still build in the development environment - hosted docs look good, model docs look good.
Fixed markdownlint errors, both old and new.
Updated example_plugin docs to match the same theme and tested embedded hosting.

A live demo of how this looks like can be found here.
The goal of this first PR is to port the docs without any structural changes or major updates to content, mostly to prevent any possible link breakage. The following step after this PR is merged will be to swap the project domain over to docs.nautobot.com/projects/core/. Once the dust has settled I think we can do larger content changes.

TODO

[x] Explanation of Change(s)
[x] Linked live build for review
[x] Example Plugin Updates (when adding/changing features)
[ ] App logos to be updated before merge with consistent sizing and style

While this PR addresses #2345, it won't apply to any tagged versions that are older than when this gets released. Any new deployments won't start working until en/stable is updated (which is considered the canonical version), even if en/latest or other tagged builds are already live with the new theme.

In golden-config, @itdependsnetworks added a basic test to compare the docs/requirements.txt versions against the ones added to pyproject.toml. It works with exact pinned versions and is great for preventing any drift due to the duplication of the docs dependencies. Is this something we can fit in here as well? If so, what's the best place to put it?
Test: https://github.com/nautobot/nautobot-plugin-golden-config/blob/develop/nautobot_golden_config/tests/test_basic.py
CC @glennmatthews @bryanculver @lampwins

In golden-config, @itdependsnetworks added a basic test to compare the docs/requirements.txt versions against the ones added to pyproject.toml. It works with exact pinned versions and is great for preventing any drift due to the duplication of the docs dependencies. Is this something we can fit in here as well? If so, what's the best place to put it?

I'm not opposed to this. Probably putting it somewhere like nautobot/core/tests/test_docs_requirements.py would be appropriate.

In golden-config, @itdependsnetworks added a basic test to compare the docs/requirements.txt versions against the ones added to pyproject.toml. It works with exact pinned versions and is great for preventing any drift due to the duplication of the docs dependencies. Is this something we can fit in here as well? If so, what's the best place to put it?

I'm not opposed to this. Probably putting it somewhere like nautobot/core/tests/test_docs_requirements.py would be appropriate.

@cmsirbu I won't hold up this PR on this because we already get some consistency for these files via Renovate. Feel free to open a housekeeping issue to add said test.


2 add gcloud trivy/vv/insights 410

This PR fixes #
Checklist

[X] I have signed the CLA
[X] I have updated/added any relevant documentation

Description
What's the goal of this PR?
Use gcloud auth in case it's available - this approach will use gloud auth command to set up dockerconfig
What changes did you make?
What alternative solution should we consider, if any?

gcloud auth configure-docker requires docker installed, and we are trying to avoid that -- so we are going to implement using tokens for now


px-timeline Time Zone Conversion

Application time changed in preferences will change entire application but the px-timeline is not able parse FIJI timezone.
Other than FIJI it works fine displaying time zone

I'm not sure how your application is passing timezone data to your components, but the timeline has a property called timeZone and moment.js contains the timezone Pacific/Fiji, so you should either use <px-timeline time-zone='Pacific/Fiji'> or get a reference to the timeline component and set it with JavaScript, e.g. myTimeline.set('timeZone', 'Pacific/Fiji'); - hope that helps!

Thank You Randy,
We need to change based on the time pref. per our application USER PREF. Fiji is one the time zone we are trying to test it, per below screenshot we are passing time zone to this  component but its not able to parse it.

I'm not following. It seems to be parsing it fine, and +12 is the correct timezone for Fiji. I tested on our demo page by switching from my local time zone (-8) to Fiji (+12) and it only jumped forward by 19 hours, so I'm not sure if there's a daylight savings difference or something?
By "not parsing" do you mean the time in the timeline is different than the one in your case details? I find it odd that the difference is not just the timezone - it's not off by an even hour, the minutes and seconds are different as well.

Thanks for quick response,  Instead of +12 we wanted to see FJT timezone,  like we can see GMT or EDT when changed in time preferences.

That is a limitation of the moment timezone library, not of this component. I tried z, zz, Z, and ZZ, none of them translate to "FJT" unfortunately.


[region-isolation] Add a test case to tree that now succeeds on main.

Just to make sure the error doesn't happen again.
rdar://128195659

@swift-ci smoke test

Did this in a different PR.


Add way to pan camera across map at start of level

 Issue by Sean-Nishi
Monday Dec 11, 2023 at 19:49 GMT
Originally opened as https://github.com/Sean-Nishi/M1Wengine/issues/268

The way to inform the user how the level's map looks for strategy.

 Comment by Sean-Nishi
Monday Dec 18, 2023 at 19:48 GMT

When loading level, after displaying level prompt #271, start camera on top of player, then move camera across map to ending/end part of level. Then move camera back on top of player

 Comment by owalk
Monday Dec 18, 2023 at 19:49 GMT

Create cam manager method for these controls., add way to reset
reset just reverts to a saved state of all the positions before panning.
Panning will move all positions in the panning direction, including the player position.
Camera move adjusts must stop for this to work


Created initial b2ai script with file load and STFT

All the relevant changes are in b2ai_script.ipynb.

Right now I am hard-resampling audio to 16khz
I am curious if the way I am storing the data in a dictionary is the best way or not ( {filename: {signal: signal, metadata: metadata}} )
If so, I will make the scaled functions for the other features we want (thanks for the initial code Satra

can we make these python scripts and not notebooks?

also add a minimal requirements.txt


Confirmed working on DS920+

DS920+
WD Red SN700 1TB in slot 2
DSM 7.1.1-42962 Update 5
Thanks !

Quick question... My NVME worked with DSM 7.1 and has data on it.
After update to DSM 7.2 the NVME is not recognized anymore.
With this script, will DS920+ recognize my NVME with old data again?


Cannot Disable IPv6

Having all kinds of issues with apt-get and such and what I am seeing is it defaults to wanting to use IPv6.  When running, for instance, apt-get update I get:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/multiverse/binary-amd64/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::19). - connect (13: Permission denied) [IP: 2001:67c:1562::19 80]
I have disabled IPv6 on my network adapter but I can't seem to disable it on the Linux subsystem, or BoW (interesting acronym).
So what I did is add the following items to /etc/hosts:
<IP_ADDRESS> archive.ubuntu.com
<IP_ADDRESS> archive.ubuntu.com
<IP_ADDRESS> archive.ubuntu.com
<IP_ADDRESS> archive.ubuntu.com
<IP_ADDRESS> archive.ubuntu.com
<IP_ADDRESS> security.ubuntu.com
<IP_ADDRESS> security.ubuntu.com
<IP_ADDRESS> security.ubuntu.com
<IP_ADDRESS> security.ubuntu.com
<IP_ADDRESS> security.ubuntu.com
And here's what I got:
Err http://archive.ubuntu.com trusty-updates/universe Translation-en
Cannot initiate the connection to archive.ubuntu.com:80 (<IP_ADDRESS>). - connect (13: Permission denied) [IP: <IP_ADDRESS> 80]
I have turned off my AVG anti-virus and firewall and ran with elevated Windows privs; no luck.  Without the hosts entries it wants to default back to IPv6 so something is definitely wonky.

try this workaround: http://askubuntu.com/questions/32298/prefer-a-ipv4-dns-lookups-before-aaaaipv6-lookups/38468#38468

I did end up getting it working before seeing your post by disabling IPv6 on the host (Windows 10) and rebooting.  Now it is working as long as I disable the firewall for a short time; that really needs to be fixed.

@tibmeister There's now a workaround to not overwrite /etc/hosts if you remove the header at the top.  Could you try that and see if that works for you?

Closing since the IPv6 issues should now be fixed.
Firewall & AV issues persist and will be working with several ISV's to remedy these issues.