how to start using proxyFS with our existing saio?

hello everyone,
I would like to use proxyFS with my existing saio, but unfortunately I did'nt find any useful documentation describing how  proxyFS works or what the architecture is look like. I do'nt know how to start . I appreciate any suggestions.

Hi tabeamiri...and welcome to ProxyFS!
Sounds like you already have Swift up and running...and want to add ProxyFS...great!
First things first...we have a Slack channel set up (proxyfs.slack.com) that might be more conversational to get going. Please join.
Next, it is important to understand what ProxyFS is doing. In a nutshell, ProxyFS provides POSIX functionality on top of a Swift Account. Those Containers you normally see inside an Account become top level directories in the root of a file system. Objects in those Containers are files in those (Container) directories...with a very important distinction. If two Objects, for example, are named "foo/cat" and "foo/dog", they will actually appear as files named "cat" and "dog" inside a subdirectory named "foo". Indeed, in aligning Container/Object with POSIX hierarchical file system concepts, there is the added restriction that you can't also have an Object named "foo" as that would collide with the subdirectory named "foo" in the example. Finally, ProxyFS also happens to support symbolic links as well...and these should not be considered equivalent to the symbolic link capability recently added to Swift.
Next up, it might be helpful to understand what protocols you intend to use to access a ProxyFS volume. There are, in fact, SIX distinct protocols simultaneously supported (i.e. all six can be used to talk to the files/objects in a ProxyFS volume):
SMB
NFS
Swift
S3
"Server" FUSE mount
"Client" FUSE mount
In a nutshell, ProxyFS presents it's file system io each of these in subtly different ways:
SMB - Samba "VFS" plug in binds a volume shared via SMB to a ProxyFS volume.
"Server" FUSE - ProxyFS presents the file system locally (i.e. on the node where ProxyFS is running) via FUSE.
NFS - By listing the "Server" FUSE mountpoint in that node's /etc/exports, the node where ProxyFS is running can NFS serve the volume.
Swift & S3 - a special middleware (pfs_middleware) is inserted in the Swift Proxy pipeline. This enables both Object APIs to present the file system as if it were a normal Swift Account.
"Client" FUSE - PFSAgent is a program built for a client environment that remotely accesses Swift and ProxyFS. It presents the file system locally (i.e. on the client) via FUSE. PFSAgent is particularly effective at gaining back the true scale out performance of Swift...at least for reads and writes (i.e. metadata operations are still carried out by a central ProxyFS instance). This is much more scalable than for the SMB and NFS mounting options.
Architecturally, ProxyFS sits along side a Swift Proxy. Indeed, there is a SECOND Swift Proxy instance - referred to as the NoAuth Swift Proxy since it lacks the "auth" middleware typical in Swift Proxy pipelines). This NoAuth Swift Proxy has a very short pipeline...that importantly includes the other middleware included in the repo: meta_middleware.
Configuring ProxyFS (and, indeed, PFSAgent as well) is accomplished with a .INI styled file passed as the lone argument to the proxyfsd program. As a rule, there are many config parameters...and there are no defaults. So it's very helpful to look at some examples (e.g. proxyfsd/defaults.conf). And, of course, for SMB & NFS, you'll want to provide smb.conf and /etc/exports as well.
File systems must be formatted on top of an empty Account. There is no mechanism to convert an existing Swift Account to a ProxyFS Volume. The mkproxyfs binary is used to format a volume based on the same config file ultimately passed to proxyfsd.
I'm sure this sounds like a lot...so where to start. Let me suggest you look in the saio subdirectory of the repo. There you will see a proxyfs.conf file with a handy example. There are scripts in a subdirectory of saio/ that will start everything up for you. Look particularly at  start_and_mount_pfs...it starts Swift, then uses mkproxyfs to format "CommonVolume", then launches proxyfsd to serve it locally (via FUSE), then launches Samba and nfsd, then mounts via SMB & NFS. It will even launch pfsagentd and serve the file system that way as well (look in the pfsagentd directory for pfsagent.conf to see it's config).
Well that's a start. There is a lot to ProxyFS...plenty to dig into based on your interest & needs.
Feel free to reach out via the Slack channel I posted above...and good luck!

Many thanks 🙏 for your comprehensive response. Sorry I could not join to proxyfs.slack.com due to some technically reasons. I'll try it later again.
About the proxyfs and how it stores data, does data store like a file system and not like swift structure, right? And proxy node with some middleware converts this structure to swift like?

Excellent questions...let me explain some inner workings a but.
First the real problems of putting a file system on Swift:
Eventual Consistency is a real challenge. We need to be sure that when we PUT an Object and later GET it, the data returns is what we last wrote. To accomplish this, ProxyFS literally always writes new Objects under the cover. That way, the GET either worjs as desired or we simply retry it.
We cannot modify portions of an Object either (ignoring Eventual Consistency concerns above even if there were none). So we write to new Objects only the changes. We call the sequence of Objects that make up the file's data LogSegments. Much like for Swift Static Large Objects ("SLO"), each file is described by the equivalent of a SLO Manifest that pieces together ranges of these LogSegment Objects to form the contents of a file.
Finally, POSIX file system users expect to be able to rename files and hard link multiple references to a file. With Swift, a rename must necessarily copy the file to its new path such that a subsequent GET (of the new path) can find it. Hard links would be possible by creating a new clone of the SLO Manifest...but subsequent modifications would be challenging to ensure updating of each SLO Manifest. The closest Swift can come to hard links is actually more akin to symbolic links.
So...under the covers, ProxyFS is basically maintaining each file as a SLO Manifest-like sequence of references to LogSegment Objects...and had to garbage collect those LogSegment Objects as they become unreferenced when files are modified or deleted.
In addition, ProxyFS must maintain directories...which simply provide a mapping from a basename to a file. Each "file" referenced is actually termed an Inode. The Inode has a type (file, directory, or symbolic link). In the case of a FileInode, it includes that SLO Manifest to describe each "extent" of the file. Indeed, we call this SLO Manifest equivalent an ExtentMap.
You can actually see what an ExtentMap looks like via HTTP. Issue a command like this:
curl http://:15346/volume//extent-map/foo/bar/cat
That port :15346 is to reach the embedded HTTP Server inside ProxyFS...and is, of course, configurable.
The /foo/bar/cat is the path to the file (equivalently, an Object named "bar/cat" inside a Container/Bucket named "foo").
As you can imagine, a file's ExtentMap can get very elaborate...e.g. if a file is written randomly. To scale up to this complexity, the ExtentMap is stored as a B+Tree....much more powerful than the SLO Manifest that is merely a linear list of "extents".
Directories, too, might get extremely large and yet to support lookups and modifications quickly. So they are implemented as B+Trees as well. Finally, all those directory entries point to Inodes...that table of Inodes is, itself, a B+Tree as well.
As the file system is modified, all that really happens is updates to those three B+Tree types (FileInode ExtentMaps, DirectoryInode lists of dir-entries, and the InodeTable) are merely "logged" in a special Container: ".checkpoint". Since we record such B+Tree changes in logged form, ProxyFS even supports snapshots!
Finally, you asked about the way that middleware (pfs_middleware) enables Object API access to the file system. Let's take the easiest example: GET. As the GET moves thru the Swift Proxy pipeline, it hits pfs_middleware that determines (based on a Header on the Account) that the Account is "BiModal"...meaning it is managed by ProxyFS. So instead of simply passing thru the GET path, it asks (via a JSON RPC) ProxyFS for a copy of the ExtentMap for the range requested. Then, it "plays" the ExtentMap much like the "slo" middleware does...issuing GETs for each extent. PUTs are kind of the reverse...where pfs_middleware places the content of the PUT in some Object...and then tells ProxyFS where it PUT it. ProxyFS then constructs the ExtentMap equivalent, fills in a FileInode, and inserts the appropriate directory entry to point to the new FileInode.
I hope I've addressed your questions...

You can actually see the innards of how ProxyFS stores data by using the "NoAuth" Swift Proxy serving localhost:8090. Do an Account GET and you will see the checkpoint Container (sorry...having a hard time avoiding markdown translating my underscores as requesting boldface) as well as the other Containers used to hold file data. The Objects in each Container are merely 64-bit Hex named...so somewhat hard to know what file data is stored within them. That's what that extent-map URL is helpful for.
Finally, if you do a HEAD on that Checkpoint Container, you should notice a four valued Header. Those four values make up what file system people would call the "superblock"...basically...the root of the file system. The four 64-bit Hex numbers are:
Version/Format# - in this case "3"
LastCheckpointObject - the name of the Object in the Checkpoint Container where  we wrote the last Checkpoint of the file system.
LastCheckpointSize - the length ...at the "tail" end, where we wrote the Checkpoint
ReservedToNonce - the next 64-bit number to use when uniquely naming an Object. Recall that we never want to reuse Object names lest we be unsure a successful GET actually returned the most recent Object Contents? This is how we do that.
If you do an Account GET, you should also see that Header that tells pfs_middleware this Account is "BiModal".
That's it in a nutshell...

oh great, now I know the basics about proxyFS.
and one more question, which is the difference between pfs_middleware and meta_middleware? you described about pfs_middleware and I got it completely. but meta_middleware, no details. should they be both activated in proxy along side proxyFS ?
by the way, would you mind sending me an invitation email for your Slack channel? It seems there is no direct way to join, except with invitation link.
thank you dear edmc-ss

When ProxyFS is deployed in a Swift cluster, there are actually TWO Swift Proxy pipelines. One is the full Swift Proxy that includes all selected middleware filters as desired... typically including some auth filter that restricts access to Accounts to only authorized Users. To this Swift Proxy, we insert the pfs_middleware previously described. This way, Swift & S3 requests that arrive at this client visible Swift Proxy gain support for accessing ProxyFS volumes. Indeed, if you don't add _pfs_middleware, users will be exposed to the underlying raw storage of ProxyFS. That's akin to letting users open up a raw disk device and read/write to the sectors of the drive. Not a good idea. Importantly, this pfs_middleware needs to be placed in every client accessible Swift Proxy to prevent such raw access lest clients corrupt the file system.
Meanwhile, ProxyFS needs to perform its own GETs and PUTs to Swift. Necessarily, it needs this very raw access. Indeed, ProxyFS is a trusted element so also does not need the auth filter present in the client visible Swift Proxy. As such, we refer to this second class of Swift Proxy as the NoAuth Swift Proxy. Given that the NoAuth Swift Proxy does not require authentication, it necessarily cannot be visible to clients. So it should be configured to only listen on localhost. We only need a NoAuth Swift Proxy on each of the nodes that also host a ProxyFS instance.
So that's the difference between the two Swift Proxy types... but you also asked about the meta_middleware filter. Well, it turns out that ProxyFS wants to place a special Header in the Account of the ProxyFS volume. Importantly, we don't want normal users to be able to modify this Header as it is what indicates that the Account is managed by ProxyFS lest, again, users gain raw access to the Account and potentially corrupt the file system. To protect this particular Header, we need to work around the namespace protections provided in the normal Swift Proxy pipeline. To implement this workaround, we insert the meta_middleware in the NoAuth Swift Proxy pipeline.
So, to be clear & summarize:
You need both the normal Swift Proxy and this new NoAuth Swift Proxy in your deployment.
You insert pfs_middleware in the client visible / normal Swift Proxy pipeline.
You insert meta_middleware in the localhost-only NoAuth Swift Proxy pipeline.
If you look in the saio/ directory, you will see examples of all this. In particular, check out the normal Swift Proxy config and the NoAuth Swift Proxy config. In the "normal" Swift Proxy config, notice how pfs_middleware (i.e. filter pfs) is inserted right after dlo. Similarly, in the NoAuth Swift Proxy config we insert meta_middleware (i.e. filter meta) right after dlo.
Hoping this clears up these great questions.

I also invited you<EMAIL_ADDRESS>to the ProxyFS Slack Channel.
Not quite sure why an invite was necessary, but hopefully it makes the process simple now...

I really appreciate it, that solved my problem and I joined to your channel. :)


Enhancement optimize the default configuration related to statistics, etc.

Proposed changes
This pr is mainly to optimize statistical tasks. Includes the following:

No longer generate statistics tasks for empty tables, and move the logic of skipping empty partitions to the process of task generation.
Adjusted the default configuration related to statistics to improve the efficiency of statistics collection, parameters include cbo_concurrency_statistics_task_num,statistic_job_scheduler_execution_interval_ms  and statistic_task_scheduler_execution_interval_ms.
Optimize the display of statistical tasks.
In addition, some org.apache.parquet.Strings packages are changed to com.google.common.base.Strings to avoid the exception that Strings cannot be found in local debug.

etc.
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[x] Yes
[ ] No
[x] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[x] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow review plz


how to stop video when swiping to another tab?

I'm looking for an event trigger that I can use in order to pause a react-native-video when I leave the screen. One of the tabs contains a video component.
I tried using the onTabChange in scrollable-tab-view. The video component does receive a this.props.tabChanged=true, but that moment seems to be too late to set the video to pause.
Any ideas?

Hi. What is onTabChange? How it receive this.props.tabChanged=true I don't know anything about such prop? What do you mean that it is too late?

it's actually called onChangeTab which is a property of ScrollableTabView.
I tried triggering my video (child) component with the onChangeTab event.
OnChangeTab event, changes state of ScrollableTabView which I can push downwards to child component(Video), but that doesn't trigger a Video update.
I guess I'm looking for some simple event like onSwipe, onHide or whatever in order to stop my video.
Maybe there's a way to add an event listener to my Video component that detects onHide?

onChangeTab (Function) - function to call when tab changes, should accept 1 argument which is an Object containing two keys: i: the index of the tab that is selected, ref: the ref of the tab that is selected

try onScroll. Not sure that I understood when exactly you want to stop video? During transition between tabs?

I also tried onScroll without any luck. (maybe my fault)
Anyway, the next solution works very well, using emitter and listener.
Video stops now  when onChangeTab event is triggered.
Main Component, import EventEmitter
`var EventEmitter = require('EventEmitter');`

Add eventEmitter object before mounting
  componentWillMount() {
    this.eventEmitter = new EventEmitter();
  }

render() {
    return (
      <ScrollableTabView renderTabBar={() => <DefaultTabBar />}
                         style={styles.scroll}
                         tabBarUnderlineColor={COLORS.GREEN}
                         tabBarActiveTextColor={COLORS.GREEN}
                         tabBarInactiveTextColor={COLORS.MID_GREY}
                         tabBarBackgroundColor={COLORS.DARK_GREY}
                         tabBarPosition="top"
                         onChangeTab={this.onTabChange.bind(this)}>

Method that's linked to onChangeTab.
  onTabChange(e) {
//2nd parameter object only used or logging
    this.eventEmitter.emit('tabChange', { tabName: e.ref.props.tabLabel});
  }

Attach eventEmitter to the component 'Demo' inside one of the Tabs
(pushes eventEmitter down to Child Component(s) )
<ScrollView tabLabel="Demo"
                    style={styles.tabView}>
          <Demo skill={this.props.skill} events={this.eventEmitter} />
        </ScrollView>

Arriving in the bottom VideoComponent
Import Subscribable (to create listeners)
var Subscribable = require('Subscribable');
after mounting, add listener that catches the 'tabChange' emit from Main Component
and calls this.stopVideo()
    componentDidMount() {
      this.addListenerOn(this.props.events, 'tabChange', this.stopVideo);
    },

@beebase how did you stop react-native-video? i don't see any prop or method to pause/play react-native-video? i am done with tabChange things but stuck to pause/play video.
Thanks


libphonenumber: fix reproducible builds patch

Description of changes
Now actually sorts the directory listings
Fixes https://github.com/NixOS/nixpkgs/issues/276443
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@ofborg build libphonenumber


Augment InvalidTargetDataCheck to check for mismatched indices in target and training data

This fell out of https://github.com/alteryx/evalml/issues/1723. My guess is that this would fit best in InvalidTargetDataCheck but open to suggestions.
We've seen a few cases where indices cause bugs (imputer, encoder, etc). It could be a good idea to warn users if their data has mismatched indices, since that could leave to unexpected behavior. For example, pandas could try to backfill some indices and introduce NaN and cause errors while modelling.
@freddyaboulton FYI 😀

Let's talk about this when we discuss #1723

Let's talk about this when we discuss #1723

Oops I filed a duplicate (#1779). One little piece of context from that issue:

For future compatibility with unsupervised problem types, I suggest we don't raise an error if y=None. I think the invalid target data check covers that case.

@freddyaboulton Correct me if I'm wrong, but I don't think this is is blocked on #1723, which I noticed you picked up? If that's true, I can pick this up and work on it concurrently :))

@angela97lin Yea, this isn't blocked by #1723!

@freddyaboulton Amazingggg, thank you 😁


httpx: use surrogateescape to allow for binary data

This allows recording httpx requests with binary data in the request
body.
Fixes: https://github.com/kevin1024/vcrpy/issues/656

This PR has 2 workflows awaiting approval.

I suggest we close this one in view of #784


Does this still work and update?

Hi @w4lle ,
I have 2 issues that is needed to fix/handle in order for this to work again:

https://github.com/pncsoares/youtube-video-creator-robot/issues/53
https://github.com/pncsoares/youtube-video-creator-robot/issues/60


helm chart cert-manager doesn't apply ingressShim.defaultIssuerName and ingressShim.defaultIssuerKind parameters to ingress-shim container

Describe the bug:
helm chart cert-manager doesn't apply ingressShim.defaultIssuerName and ingressShim.defaultIssuerKind parameters to ingress-shim container leading to this error of ingress-shim:
E1025 22:15:03.953848       1 controller.go:156] ingress-shim controller: Re-queuing item "default/products-ssl" due to error processing: issuer.certmanager.k8s.io "" not found
Expected behaviour:
Adding
--set ingressShim.defaultIssuerName=letsencrypt-staging \
--set ingressShim.defaultIssuerKind=ClusterIssuer \

to helm upgrade cert-manager I expect them to be applied to cert-manager deploy, but for some reason, it doesn't happen. See steps below:
Steps to reproduce the bug:
Creating helm chart and upgrading it following the instructions:
helm install \
    --name cert-manager \
    --namespace kube-system \
    stable/cert-manager

kubectl create -f cluster-issuer.yaml

helm upgrade cert-manager \
		--set ingressShim.defaultIssuerName=letsencrypt-staging \
		--set ingressShim.defaultIssuerKind=ClusterIssuer \
		--namespace kube-system \
    stable/cert-manager

Describing cert-manager deploy after it's been created:
kubectl describe deploy cert-manager-cert-manager --namespace kube-system
Name:			cert-manager-cert-manager
Namespace:		kube-system
CreationTimestamp:	Fri, 26 Oct 2018 11:14:02 +1300
Labels:			app=cert-manager
			chart=cert-manager-0.2.10
			heritage=Tiller
			release=cert-manager
Annotations:		deployment.kubernetes.io/revision=1
Selector:		app=cert-manager,release=cert-manager
Replicas:		1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:		RollingUpdate
MinReadySeconds:	0
RollingUpdateStrategy:	25% max unavailable, 25% max surge
Pod Template:
  Labels:		app=cert-manager
			release=cert-manager
  Service Account:	cert-manager-cert-manager
  Containers:
   cert-manager:
    Image:	quay.io/jetstack/cert-manager-controller:v0.2.5
    Port:	<none>
    Environment:
      POD_NAMESPACE:	 (v1:metadata.namespace)
    Mounts:		<none>
   ingress-shim:
    Image:		quay.io/jetstack/cert-manager-ingress-shim:v0.2.5
    Port:		<none>
    Environment:	<none>
    Mounts:		<none>
  Volumes:		<none>
Conditions:
  Type		Status	Reason
  ----		------	------
  Available 	True	MinimumReplicasAvailable
  Progressing 	True	NewReplicaSetAvailable
OldReplicaSets:	<none>
NewReplicaSet:	cert-manager-cert-manager-7494cc55db (1/1 replicas created)
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----			-------------	--------	------			-------
  4m		4m		1	deployment-controller			Normal		ScalingReplicaSet	Scaled up replica set cert-manager-cert-manager-7494cc55db to 1

Anything else we need to know?:
If I manually edit cert-manager deploy adding:
args:
          - --default-issuer-name=letsencrypt-staging
          - --default-issuer-kind=ClusterIssuer

it actually works.
I actually wonder why stable/cert-manager helm chart in my case shows 0.2.1 version while it should be 0.5 at least?
Environment details::

Kubernetes version: v1.11.3
Install method: helm v2.11.0
Version: cert-manager-0.2.10 (image: v0.2.5)

/kind bug

You need to update your helm repositories before running the upgrade - it looks like you have an old version of the chart locally.
You can see more details on upgrading here: https://cert-manager.readthedocs.io/en/latest/admin/upgrading/index.html
That version of the chart does not support the option, so updating should sort it out!

@munnerz Thanks for your help. You're right. In my case for some reason helm repo 'stable' was pointing to an old version of the chart.
helm repo update doesn't resolve the problem but it's not a point for this discussion. As a workaround, I just added --version 0.5.0 to helm install command.


Folder instructions in filename?

I would like to include path/folder-instructions into the filename because I have a lot of different files (and there are always new categories added) I don't want create rules for. For example my filename is '2019_Jobs_CategoryA_TagB_A-Media_content-name_V01_draft_eng.docx' which means: Move the file to the folder '2019/Jobs/CategoryA/TagB/Media/drafts/eng' whereby 'A-' is an additional instruction and should be removed from the filename afterwards ('2019_Jobs_CategoryA_TagB_content-name_V01_draft_eng.docx').
I have a rough idea to figure it out with python but I'm new to it (see below a sketch). Is there a possibility to use such variables, conditions etc. with organizer natively? If no, is it possible to do it with Python in Organizer at all?

Transform file-string into array
Search for 'A-...', 'V...' and 'content-name' and get index of values
remove value 'A-... and 'content-name' of array
build new filename string
remove value 'V...' and 'A-' of array
build folder-path string (convert _ to /) etc.

Yes, this can be achieved with organize. You can use a python-filter to parse the instructions and generate the new path.
For example:
rules:
- folders: "~/Documents/"
  filters:
    - extension:
      - pdf
      - docx
    - filename:
        contains: "_"
    - python: |
        import os
        parts = []
        instructions = dict()
        for part in path.stem.split("_"):
            if part.startswith("A-"):
                instructions["A"] = part[2:]
            elif part.startswith("V-"):
                instructions["V"] = part[2:]
            elif part.startswith("content-name"):
                instructions["content"] = part[12:]
            else:
                parts.append(part)
        return {
            "new_path": os.path.join(*parts),
            "instructions": instructions,
        }
  actions:
    - echo: "New path: {python.new_path}"
    - echo: "Instructions: {python.instructions}"
    - echo: "Value of A: {python.instructions.A}"
    - move: "~/Documents/{python.new_path}/{path.name}"

I'm closing this issue - if you have any issues with this feel free to reopen

No issues at the moment. Thanks!


iframe-apprtc page gives JS Error appr.tc/r/454648382:1 Uncaught (in promise) DOMException: Not a top-level browsing context."

Browsers and versions affected
Chrome 61.0.3124.4 (Mac), Chrome 61.0.3124.10 (Windows 7, Windows 10)
Steps to reproduce

Browse to https://test.webrtc.org/manual/iframe-apprtc/index.html
Open the javascript console (alt+cmd+j or ctrl+shift+j)
Enter a enter a room name and press "JOIN" and watch for any JS errors

Expected results

No JS Errors

Actual results

JS Error "appr.tc/r/454648382:1 Uncaught (in promise) DOMException: Not a top-level browsing context."

I can re-produce this bug every time

Could be Chrome tightening the security for iframes. @guidou @samdutton do you know anything about this?

I can't reproduce this in Chrome 59.0.3071.86 or 61.0.3129.0 on Mac.
I don't have a Windows box handy, but I can check if there's been a change.

After doing some quick tests, I do NOT see the bug on Mac or Linux using Chrome 61.0.3130.0 (canary)
However, on Linux , I do still see the bug using Chrome 61.0.3129.0

@KaptenJansson https://groups.google.com/forum/? utm_medium=email&utm_source=footer#!msg/discuss-webrtc/32natnrbP6U/0dgpWLMkAwAJ -- possibly related?


video.ibm.com/embed/ not working.

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

youtube-dl --verbose https://video.ibm.com/embed/24972048
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://video.ibm.com/embed/24972048']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.11.3 (CPython) - macOS-11.6.2-x86_64-i386-64bit
[debug] exe versions: ffmpeg 6.0, ffprobe 6.0, rtmpdump 2.4
[debug] Proxy map: {}
[ustream] 24972048: Downloading webpage
ERROR: Unable to extract content video IDs; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.11/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.11/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
                ^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.11/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.11/site-packages/youtube_dl/extractor/ustream.py", line 179, in _real_extract
    content_video_ids = self._parse_json(self._search_regex(
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.11/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract content video IDs; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

I have tried downloading this video, however youtube-dl keeps giving me an error...

video link isn't working

Maybe we should ask this guy: https://clay.earth/profile/d3a08af1-7c19-4757-9576-943b7cb0cdd0.
The problem page, and others, don't have a value assigned to ustream.vars.offAirContentVideoIds; instead a list (of 1 ID) is assigned to ustream.vars.uumfOffAirContentVideoIds. but that ID then leads to:
...
[ustream] 132715210: Downloading JSON metadata
[ustream] 132715210: Downloading connection info
[ustream] 132715210: Downloading stream info
[ustream] 132715210: Downloading connection info (try 2)
[ustream] 132715210: Downloading stream info (try 2)
[ustream] 132715210: Downloading connection info (try 3)
[ustream] 132715210: Downloading stream info (try 3)
ERROR: No video formats found; ...

Ustream extractor test results: 4 failed, 1 skipped.

✘ artem@ARTEM-PC  ~/github/yt   main  youtube-dl --no-check-certificate https://www.youtube.com/watch\?v\=EVUFwyOGMSY\&ab_channel\=NubeColectivaDevs [youtube] EVUFwyOGMSY: Downloading webpage [youtube] EVUFwyOGMSY: Downloading MPD manifest ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.  ✘ artem@ARTEM-PC  ~/github/yt   main  youtube-dl --no-check-certificate https://youtu.be/QRSO44Yd1as     [youtube] QRSO44Yd1as: Downloading webpage ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.  ✘ artem@ARTEM-PC  ~/github/yt   main  youtube-dl --no-check-certificate https://www.youtube.com/watch\?v\=QRSO44Yd1as [youtube] QRSO44Yd1as: Downloading webpage ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.  ✘ artem@ARTEM-PC  ~/github/yt   main  youtube-dl --no-check-certificate QRSO44Yd1as                      [youtube] QRSO44Yd1as: Downloading webpage ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.  ✘ artem@ARTEM-PC  ~/github/yt   main 

@michaelnoe,

are you able to play the video?
do you have any other video links that the Ustream extractor should handle?

It may be that Ustream has been broken or withdrawn in one of the reorganisations or retrenchments for which the modern IBM is rightly notorious.

@dirkf: yes, i am able to play the video... using the url: video.ibm.com/embed/24972048 the video is normally located behind a pay-wall, but by using this url, it works even on a pc I never logged in on. I use chrome...

video is now playing for me now, yesterday it was playing then all of a sudden it would
no longer load.

...using the url: video.ibm.com/embed/24972048 the video is normally located behind a pay-wall, but by using this url,...

Which URL? Do you mean not the one you mentioned originally?

The problem remains unsolved.

Perhaps this issue should be closed as the URL provided in the original issue doesn't seem valid anymore.... it seems like maybe IBM changed their URL format.  My new issue #32996 contains a valid URL that is alive today and perhaps can be used to further investigate this.

Continued in #32996.


The Find/Replace feature does not work

Prerequisites

[x] Put an X between the brackets on this line if you have done all of the following:

Yep — Reproduced the problem in Safe Mode: https://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
Does not fix — Followed all applicable steps in the debugging guide: https://flight-manual.atom.io/hacking-atom/sections/debugging/
Found no solutions — Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
Does not match: #4154, #16390, #6963, or #7936 — Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
Assuming problem in core package (N/A) — Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description
When using the Ctrl + F or Ctrl + Shift + F (Find & Replace) with Regex enabled, the feature fails.
Steps to Reproduce

Press Ctrl + F or Ctrl + Shift + F
Use a valid Regex in the "Find" area, such as --\b or (--)\b
Use a valid string in the "Replace" area, such as ? or $1?
Notice that nothing will be replaced, or the replacement will be literal (such as --apple → --$1apple; but should be --?apple)

Expected behavior:
Replace the current buffer(s) with the valid replacement string.
Actual behavior:
Either replaces the buffer(s) will literals, or does not replace anything.
Reproduces how often:
100% of the time: Safe Mode, fresh install, etc.
Versions

OS
Atom
APM

Windows 10 x64
1.43.0
2.4.3

Additional Information

Example of what I was trying to do:
Original index.html:
...
<meta --user="title. last-name"/>
...

Desired output:
...
<meta ?user="title. last-name"/>
...

Options used (Find & Replace):

[x] Use Regex
[ ] Match Case
[ ] Only In Selection
[ ] Whole Word

Find in current buffer: --\b
Replace in current buffer: ?

I can't just use -- due to HTML comments <!-- ... -->

I think this is a problem with your regex matching instead of the atom editor.
Try this regex match:
(^<!\1+)


Coverity issue seen with CID: 178064

In File: /tests/net/lib/coap/src/main.c
Category: Memory - corruptions
Function: test_block_size
Component: Tests
Please fix or provide comments to square it off in coverity in the link: https://scan9.coverity.com/reports.htm#v32951/p12996

would it be possible to give more descriptive names to these bugs?
like include the path to the affected file in the subject line

Fix in #4413 seems correct.


Bad lighthouse performance score

I was building the tutorial app (https://reflex.dev/docs/tutorial/intro/) and was curious about the performance of Reflex.
I ran the Google Lighthouse diagnostics test which came out pretty bad, especially considering this is a very simple app. Are there any improvements that can be made on that front?

Were you running in dev mode or prod mode (--env prod)? I tried to reproduce this result against https://github.com/reflex-dev/reflex-chat/tree/bf80610ba1ab04cda67b6c2c8ed89c0469943150 and get the following.
Dev Mode

Prod Mode

It makes a difference due to the tree-shaking and static rendering that happens when running in prod mode.


install error in ubuntu 16.04

dear team
when i try to install mxnet on ubuntu 16.04 with R language, i encounter some problems as follows
~/mxnet/R-package$ Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cran.rstudio.com')); install_deps(dependencies = TRUE)"
Error in download.file(url, destfile, method, mode = "wb", ...) :
cannot download all files
此外: Warning messages:
1: In download.file(url, destfile, method, mode = "wb", ...) :
downloaded length 131072 != reported length 3505013
2: In download.file(url, destfile, method, mode = "wb", ...) :
URL 'https://cran.rstudio.com/src/contrib/DiagrammeR_0.8.4.tar.gz': status was 'Failure when receiving data from the peer'
Warning in download.packages(x$name, destdir = dest_dir, repos = x$repos,  :
download of package ‘DiagrammeR’ failed
Error in download.packages(x$name, destdir = dest_dir, repos = x$repos,  :
下标出界
Calls: install_deps ... FUN -> remote_download -> remote_download.cran_remote
停止执行
best!

From the error message, the error has nothing to do with mxnet.

URL 'https://cran.rstudio.com/src/contrib/DiagrammeR_0.8.4.tar.gz': status was 'Failure when receiving data from the peer'

Please check your network setting.


Difficutly populating nested data

I'm trying to 'join' some collection references from a MongoDB stream using Highland.js and I'm having some difficulty populating the id references with the actual document using _.map. Am I not using map correctly? Should I be using a different method altogether? Any help would be much appreciated.
MongoClient = require('mongodb')
_ = require('highland')

MongoClient.connect 'mongodb://localhost/blog', function(err, db) {

	_( db.collection('posts').find(query).stream() )
		.each(function(post) {
			console.log(post)
			// {
			// 	_id: 557cbf50060f9e90521de109
			// 	title: "My Blog Post"
			// 	comments: [
			// 		"55a40877d804d87b433b2696",
			// 		"55a40894d804d87b433b2697",
			// 		"55a40894d804d87b433b2698",
			// 	]
			// }
			post.comments = _(post.comments).map(function(commentId) {
				_.wrapCallback( db.collection('comments').findOne )({_id: commentId}).apply(function(comment){
					return comment
				});
			});
			console.log(post)
		}).done(function() {
			console.log("THUNK!");
		});
}

post.comments ends up a Higland stream object:
{
	_id: 557cbf50060f9e90521de109
	title: "My Blog Post"
	comments: {
		__HighlandStream__: true,
		...
	},

}

...but I'm looking for something more along these lines:
{
	_id: 557cbf50060f9e90521de109
	title: "My Blog Post"
	comments: [
		{
			_id "55a40877d804d87b433b2696"
			body: "Curabitur sit amet turpis eget."
		}
		{
			_id "55a40894d804d87b433b2697"
			body: "Cras ullamcorper magna urna, ut."
		}
		...
	]

}

map returns a stream. You're assigning that stream to post.comments. Instead of using each, map and apply, you should be using flatMap, and treating streams as opaque containers rather than things that you modify. I actually answered a very similar Stack Overflow question recently, which goes into more detail: http://stackoverflow.com/a/31314941/805227

I initially tried using flatMap
_( db.collection('posts').find(query).stream() )
    .flatMap(function(post){})

I get Uncaught Error: Expected Stream, got undefined

The function you pass to flatMap has to return a Stream. flatMap runs once on each item in the stream, takes the streams you return, and returns another stream that will contain the results you the streams you returned, joined together. Because it encapsulates and waits for the return streams, you can use it to sequence asynchronous actions.
Here's a concrete example, using your example above, annotated with what's in the stream before and after each action:
MongoClient = require('mongodb')
_ = require('highland')

MongoClient.connect 'mongodb://localhost/blog', function(err, db) {
    _( db.collection('posts').find(query).stream() ) // this is a Stream of Posts
        .flatMap(function(post) {
            return _(post.comments).flatMap(function(commentId) { // Stream of Ids
                var collection = db.collection('comments'); // need to bind
                return _.wrapCallback(collection.findOne.bind(collection))({_id: commentId}) // Stream of Comments
                  .collect() // Stream of Array of Comments
                  .map(function(comments){
                      post.comments = comments;
                      return post;
                  }); // Stream of single Post, with .comments as array of comments
            }); // Stream of Posts, with .comments as array of comments
        }).each(function(post) {
            // do something with the post
            console.log("THUNK!");
        });
}

It might seem tedious or unnecessary, but thinking about streams in terms of the types of object they contain, and map and flatMap as transforming between different types of streams (map by a single value, flatMap joining streams together) can be really helpful. It's also helpful to try and stay in Highland-land as long as possible, chaining together async operations with flatMap, and only causing a thunk when you really need the value outside the context of the stream (it's counterproductive to use e.g. flatMap to cause side-effects, as it almost never runs when you expect).
Hope this helps!

Yes, that helps... the fog is lifting! One last question. Since you're returning the post for each comment the result duplicates the post for each individual comment. Is there a way to group the comments into a single array and return the post once?

That's what the collect is supposed to do. It groups everything in the stream into an array and sends the array down the returned stream. Is that not what your seeing?

No, it's unwinding the Post duplicating it for each comment and adding one comment per post. So instead of getting 1 post with 8 comments I get 8 posts with 1 comment each. I think this is because we're returning the post inside the comments map:
return _.wrapCallback(collection.findOne.bind(collection))({_id: commentId}) // Stream of Comments
    .collect() // Stream of Array of Comments
    .map(function(comments){
        post.comments = comments;
        return post; // <------ HERE
    });

Ah, my bad. The .collect().map(...) should be outside the flatMap. Currently it's collecting a single comment, then mapping that comment, attaching it to the post, then returning that to flatMap which concats them all together. So, it should be:
MongoClient = require('mongodb')
_ = require('highland')

MongoClient.connect 'mongodb://localhost/blog', function(err, db) {
    _( db.collection('posts').find(query).stream() ) // this is a Stream of Posts
        .flatMap(function(post) {
            return _(post.comments).flatMap(function(commentId) { // Stream of Ids
                var collection = db.collection('comments'); // need to bind
                return _.wrapCallback(collection.findOne.bind(collection))({_id: commentId}) // Stream of single Comment
            }) // Stream of Comments
              .collect() // Stream of Array of Comments
              .map(function(comments){
                  post.comments = comments;
                  return post;
              }); // Stream of single Post, with .comments as array of comments
           }); // Stream of Posts, with .comments as array of comments
        }) // Stream of Posts, with .comments as array of comments
        .each(function(post) {
            // do something with the post
            console.log("THUNK!");
        });
}

That did it! Thanks for the primer.

Happy to help. I think this makes a good basis for a "Highland cookbook". I might collect this (and that SO answer) into a blog post.

I'll going to be working with this for the next couple of months making JSON, CSV, Excel, Word, PDFs files... I'll try and contribute the the cookbook along the way.

I'm trying to follow this issue to build a processor that reads documents from a Mongo database, adds a computed field, and updates the record back in the database.
In the example above, the connection is not being closed - I'm having trouble trying to close the database in the .done() callback, because the connection is closed before the updates finish.
Is my scenario an appropriate use case for Highland? I've been able to build scripts which stream from database, transform the records, and write to stdout or to other streams, but I'm struggling with performing async operations on the stream without either using a clunky Stream.consume implementation or bringing in async for the queue method. I'm fine with those approaches; they just seem like they're not the most efficient way of doing things.

@saitheexplorer This is definitely something that Highland supports.
Can you create a new issue with some sample code that illustrates what you're trying to do? I can help you there. From your description, you shouldn't need to use Stream.consume or async. I can't tell from your description why the .done() callback is being executed early; I'll be able to tell you more if I see your code.

@vqvu Sure - I started building a simplified use case, and my code worked :) I'm going to compare the two versions and see where I'm going wrong.
On a tangentially related note - is there a Highland cookbook in the works? In building the use cases mentioned above, I wrote two blocks which achieved the same objective - once using async and once using Highland. Given the relative ubiquity of async, having a 'translation guide' may be helpful for people like me who are new to streams / Highland.


Central Analyzer Disabled error

I'm trying to configure dependency-check ant for one of my project. I followed following steps, but I'm getting "Central analyzer disabled" error when I run build.xml of the my project.

Downloaded the zip for release 1.3.1
Unzipped it.
Defined taskdef and target for dependency-check.(http://jeremylong.github.io/DependencyCheck/dependency-check-ant/index.html & http://jeremylong.github.io/DependencyCheck/dependency-check-ant/configuration.html)
Prior to this I created target to update  the local repository and it created data directory at the place where I unzipped the jar and has dc.h2.db file as its content.

Any help would be appreciated.

Thanks for pointing this issue out. I just looked at the ant task and the
Central analyzer is disabled by default (this is incorrect). This will be
fixed in the next release.  In the meantime you can enable this via
configuration by adding the attribute:
centralAnalyzerEnabled="true"
--Jeremy
On Wed, Sep 30, 2015 at 2:59 AM, rahulvyadav85<EMAIL_ADDRESS>wrote:

I'm trying to configure dependency-check ant for one of my project. I
followed following steps, but I'm getting "Central analyzer disabled" error
when I run build.xml of the my project.

Downloaded the zip for release 1.3.1
Unzipped it.
Defined taskdef and target for dependency-check.(
http://jeremylong.github.io/DependencyCheck/dependency-check-ant/index.html
&
http://jeremylong.github.io/DependencyCheck/dependency-check-ant/configuration.html
)
Prior to this I created target to update the local repository and it
created data directory at the place where I unzipped the jar and has
dc.h2.db file as its content.

Any help would be appreciated.
—
Reply to this email directly or view it on GitHub
https://github.com/jeremylong/DependencyCheck/issues/362.

@jeremylong Thanks for your prompt reply. Could you just point exactly how should I make this change?

Sorry for the delay. As indicated in the documentation pages you linked to above you can simply add the centralAnalyzerEnabled attribute to the dependency-check node:
<target name="dependency-check" description="Dependency-Check Analysis">
    <dependency-check projectname="Hello World"
                      reportoutputdirectory="${basedir}"
                      centralAnalyzerEnabled="true"
                      reportformat="ALL">

        <fileset dir="lib">
            <include name="**/*.jar"/>
        </fileset>
    </dependency-check>
</target>


Simplify Utils.ToManagedString

Marshal.PtrToStringAnsi does the same as what we were doing manually,
but without extra allocation for byteArray.

As the name says PtrToStringAnsi reads the string as Ansi but we need to treat them as Utf8

You're right, sorry about that.
PtrToStringAnsi does the correct thing on Linux (converts from Utf8), but on Windows it treats bytes according to system codepage.
There's a better fitting PtrToStringUTF8 but unfortunately it's not part of netstandard.

It's part of netstandard2.1 but not 2.0

I want to keep 2.0 for now.

Sure.


Improve require for plugins

To require the mapthing lib in plugins you currently need to specify the relative path to the js file, which can be horrendous. We could provide a global function ( or another solution) to make this easy @yitang

A user can now just use mtRequire(filename) instead of the regular require function


Batch Edit Mix Forms

User Story
As a Protocol Designer user, I would like the ability to batch edit Mix steps, so that I can speed up my protocol design process.
Description
This epic is a fast follow to Batch Edit MVP, where we introduced multi-select and batch edit Transfer forms functionality.
Similar to batch edit Transfer forms, enabled advanced settings for Mix steps will be batch editable (see advanced settings enabled conditions here
Requirements

All advanced settings for Mix are batch editable across Mix steps, as long as requirements to enable changes are met (e.g., pipette type must be the same) See Advanced Settings enabled conditions for enabled/disabled rules

Display editable advanced settings that share values, and display a mixed state "-" for values that differ between steps. If enabled conditions are not met, per "Advanced Settings Enabled Conditions," fields were display as Disabled and will not be editable. Tooltip explaining why will show upon hover, similar to Transfer steps

Remove the "No editable settings" message for Mix + Mix steps

Save button displays and is enabled when changes are made to batch edit mix form.

Mixpanel analytics tags for Batch Edit Mix advanced settings events (saveStepsMulti for Mix, advanced settings within mix step)

Modals for Batch Edit Mix settings should be consistent with Transfer (cannot add a step while batch editing)

QA & Test steps (Neha)

Cutting release candidate tix

Announcement modal (new copy needed)

when we start this, consider the tip postion troubles

when we start this, consider the tip postion troubles

#7470 for reference 🙃


[Android] Handling unsubscribe event while scanning for devices

We should consider if we want to send an event/error when unsubscribing from scanning subscription.
Related comment in PR

Cleanup - closing old Issues.
Please open new one if this is still valid.


[TC-WIFIDIAG-3.1] reset-counts  command not supported in wifi diagnostics Cluster in TE7

Problem
reset-counts  command not supported in wifi diagnostics Cluster
IMClusterCommandHandler.cpp doesnot have support for WiFiNetworkDiagnostics reset-counts
namespace WiFiNetworkDiagnostics {
void DispatchServerCommand(CommandHandler * apCommandObj, const ConcreteCommandPath & aCommandPath, TLV::TLVReader & aDataTlv)
{
// We are using TLVUnpackError and TLVError here since both of them can be CHIP_END_OF_TLV
// When TLVError is CHIP_END_OF_TLV, it means we have iterated all of the items, which is not a real error.
// Any error value TLVUnpackError means we have received an illegal value.
// The following variables are used for all commands to save code size.
CHIP_ERROR TLVError = CHIP_NO_ERROR;
bool wasHandled     = false;
{
switch (aCommandPath.mCommandId)
{
case Commands::ResetCounts::Id: {
Commands::ResetCounts::DecodableType commandData;
TLVError = DataModel::Decode(aDataTlv, commandData);
if (TLVError == CHIP_NO_ERROR)
{
wasHandled = emberAfWiFiNetworkDiagnosticsClusterResetCountsCallback(apCommandObj, aCommandPath, commandData);
}
break;
}
default: {
// Unrecognized command ID, error status will apply.
ReportCommandUnsupported(apCommandObj, aCommandPath);
return;
}
}
}
Logs: LOCK-TC-WIFIDIAG-3.1.txt

@praveenCY Is this still an issue?

dupe of #18321. Closing


Array of tuple literal not assignable to iterable of tuple with --lib es2015 -t es5

TypeScript Versions:  3.5.1, 3.5.3 (currently typescript@latest), 3.6.0-beta (currently typescript@beta), 3.6.0-dev.20190808 (currently typescript@next)

Search Terms:
Code
// Works:
declare function foo<T>(i: Iterable<T>): T
const x = foo([1]); // infers a: number

// Fails:
declare function bar<T, U>(j: Iterable<[T, U]>): [T, U]
const y = bar([[1, 'a']]);
//             ^^^^^^^^
//             Argument of type '(string | number)[][]' is not assignable to
//             parameter of type 'Iterable<[string | number, string | number]>'.

// workaround:
declare function qux<T, U>(j: Array<[T, U]> | Iterable<[T, U]>): [T, U]
const z = qux([[1, 'a']]); // infers z: [number, string]

Expected behavior: variable y is inferred to be of type [number, string] and parameter j is inferred to be of type Array<[number, string]>
Actual behavior: error TS2345: Argument of type '(string | number)[][]' is not assignable to parameter of type 'Iterable<[string | number, string | number]>'.
Playground Link
Related Issues: maybe #29311 ?

Cloned repo, ran  tsc .\hello.ts -target es2015, no error.
Are you sure you don't have another tsc installed on your system? Run a tsc- -v

Are you able to reproduce the error with tsc --lib esnext hello.ts, as suggested in the README?
I am finding that -target es2015 makes the error go away. Thanks, that explains the difference from Playground; if -target is made to match the Playground "Target" option, then the behavior of tsc matches Playground exactly.
However, I'm targeting ES5 (and polyfilling ES2015 features). And the library I'm using, Facebook's ImmutableJS, presumably has to target a range of ES versions, so they want to reference Iterable in their type declaration: https://github.com/immutable-js/immutable-js/blob/master/type-definitions/Immutable.d.ts#L1420
Is this expected behavior with --lib?

If target is ES5 and you're polyfilling Symbol.iterator, then you should turn on downlevelIteration to cause arrays to appear to have the iterator symbol

But that makes for...of loops much slower at runtime? This is a typings issue, I'm not going to sacrifice runtime performance to fix it.
It looks like adding Array<[K, V]> as an additional type overload besides Iterable<[K, V]> fixes the issue. I'll submit a PR to ImmutableJS. Thanks for all your help and the ultra-fast response time!

@RyanCavanaugh Actually, I'm confused why --downlevelIteration matters. lib.es2015.iterable.d.ts declares a Symbol.iterator on Array: https://github.com/microsoft/TypeScript/blob/master/src/lib/es2015.iterable.d.ts#L40
Why isn't that sufficient to cause arrays to appear to have the iterator symbol?
The docs for --downlevelIteration only mention for..of, spread and destructuring, they don't mention anything about the iterator symbol either (Compiler Options).

Actually no, this seems like an issue with type inference.
In most cases (not for..of, obviously), --lib es2015 causes Array<T> to be assignable to Iterable<T>, but not if a tuple type is involved; in some cases, the error message doesn't even mention incompatibility between Array and Iterable, it says:
Type '(string | number)[]' is not assignable to type '[string | number, string | number]'.

In this Playground Link, the first 4 examples here all show that Array<T> is normally assignable to Iterable<T>, the next 4 demonstrate the issue when a tuple type is involved.
Both --downlevelIteration and using Array<[T, U]> as an alternative to Iterable<[T, U]> do workaround the issue, though.

@RyanCavanaugh just updated title, code sample, and Playground link to succinctly illustrate that downlevelIteration should be unnecessary, since array literals are already assignable to iterables if the contents aren't tuples.

@RyanCavanaugh Can you unmark this as a Question? I think the updated code sample succinctly illustrates that it's a bug

Looking

@rbuckton can you explain what's happening? Bug?


Questions for local development

Some questions regarding the development process with t2wml:
First things first, reload app from the debug menu totally works but is missing a shortcut
(I was under the impression that cmd+R reloads the app, that's not the case right now - can we make that happen?)
Other questions:
Can we make the app open in full screen every time? (not a feature request for production but would be helpful to me in dev)
Can we have the app open dev tools/console by default every time?
Can we open one of the projects by default (as to skip going through the list and clicking one on reload)
If this is something I can just change in the app for the time being, let me know (does not have to be a dev feature)

+1 to everything except making the app open dev tools console every time. That's a developer feature, not something that a user would like to see.

How about adding options in the settings menu to control:

open in full screen mode: yes, no, last setting
open in dev mode: yes, no, last setting
open last project automatically: yes no

t2wml now saves its last window state - position on the screen, size, maximized/not and the dev tools state. On startup it restores this state (in production it does not open the dev tools automatically, though). I don't think we need to bother the user with these settings, remembering the last window state is something that people expect, and I doubt they'll want to turn it off. Let's see if someone asks for it before we implement that.
As for opening the last open project - this is a little bit trickier to implement, as it requires moving code around from the renderer to the main process, or adding more interaction between the two. I'll leave that to Devora and Talya for next week, if you think it's necessary. I think that just like Word doesn't open the last project, t2wml doesn't have to, either.

I think this is a good compromise.

Thanks for making the window size and position persistent on re-open!
These questions were mostly related to the development process (not really a feature request)
e.g. opening a project on load is to save me from clicking on one in the list view every time I reload the app :)
I've added a quick ⌘⇧R shortcut to reload the app for development purposes, see https://github.com/usc-isi-i2/t2wml/pull/258
I think we can close this issue now


All Unit Tests (xUnit) are returning -2147450749 code and no visible Tests showing up in Visual Studio Test Explorer Window

Using the latest version of source code from this GitHub repository (Dated: 10/26/2018) & following step to reproduce this bug report.
The returned negative number -2147450749 is the error code coming from .NET CLI.

Open solution into VS-2015
Build (or rebuilt) solution
Error message below from the Output window & no visible tests showing up in Test Explorer (VS-2105).

------ Discover test started ------
========== Discover test finished: 0 found (0:00:00.1676838) ==========
------ Discover test started ------
Discovering tests in 'X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.Memory.Tests\project.json' ["C:\Program Files\dotnet\dotnet.exe" test "X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.Memory.Tests\project.json" --output "X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.Memory.Tests\bin\Debug\netcoreapp1.1" --port 37084 --parentProcessId 43680 --no-build]
'test-xunit' returned '-2147450749'.
Discovering tests in 'X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.Redis.Tests\project.json' ["C:\Program Files\dotnet\dotnet.exe" test "X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.Redis.Tests\project.json" --output "X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.Redis.Tests\bin\Debug\netcoreapp1.1" --port 37088 --parentProcessId 43680 --no-build]
'test-xunit' returned '-2147450749'.
Discovering tests in 'X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.SqlServer.Tests\project.json' ["C:\Program Files\dotnet\dotnet.exe" test "X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.SqlServer.Tests\project.json" --output "X:\Playground\Caching-dev\test\Microsoft.Extensions.Caching.SqlServer.Tests\bin\Debug\netcoreapp1.1" --port 37093 --parentProcessId 43680 --no-build]
'test-xunit' returned '-2147450749'.
========== Discover test finished: 0 found (0:00:01.0875744) ==========
Windows 10
Visual Studio Enterprise 2015
Version 14.0.25431.01 Update 3

They probobly require a later version of the CLI than you have installed.
https://github.com/dotnet/core/blob/master/release-notes/preview-download.md
Alternatively, try running build.cmd from the command line. It pulls the needed dependencies to a user dir (which won't fix VS, but at least you can run the tests one way).

Interesting.  I ran this build.cmd command line and it works fine.   No changes in VS as you said.  Would have be nice to do the step-into & step-over debugger in visual studio.  :-/

Yeah, build.cmd manages it's own dependencies. It installs a pre-release SDK to C:\Users\myuser\AppData\Local\Microsoft\dotnet\ and adds it to your PATH. You can have VS pick this one up by going to your system environment variables and remove the primary SDK from your PATH: C:\Program Files\dotnet. Restart VS and then the tests should work again.

We are closing this issue because no further action is planned for this issue. If you still have any issues or questions, please log a new issue with any additional details that you have.


update KerbalRenamer to use .version file for versioning

this is my first NetKAN PR, so please let me know if I did something wrong, but this should be similar to what was just don for RealHeat here https://github.com/KSP-CKAN/NetKAN/pull/6693

Looks good! You can click "Show all checks" and then "Details" to see the validation log.
https://ci.ksp-ckan.org/job/NetKAN/7620/
Merging...


I tried making a few quick edits with the web editor but I keep missing things here and there.


Segregate Test Workflows by Paths

Currently, all testing workflows are run on creating a pull request.
https://github.com/idea-fasoc/OpenFASOC/blob/cdcd24ff1e8d20e2994cfe88b7b71ba4f7d20e27/.github/workflows/tempSense_sky130hd.yml#L3-L8
This is both inefficient and unnecessary. It would be better if paths are specified in the workflow events so that only the affected generators are tested in a pull request.

@harshkhandeparkar how do you know which generators are affected because of changes?

Some paths would affect all generators - common, for example. But generators should not interfere with each other. So we could run the tests when any common path is changed or any file under the generator's directory is changed.

sounds good. Can you modify the yml files accordingly?

sounds good. Can you modify the yml files accordingly?

Sure. I'll open a PR soon.


added ui interaction and ui element schema info

this request adds the ui_interaction event and the ui_element entity to track ui interactions
it also refactors the index to build the appropriate contexts within the context_helper.ts, let me know if there's a cleaner way to determine whether an object has data. I picked the most important attribute in each object but we can discuss a better way.

Updated the diff based on the review comments, thanks Mason!


Grammar and punctuation changes

There was inconsistent use of periods on listed statements and changes "Some common flags is" to "Some common flags are".

[ x] I have read freeCodeCamp's contribution guidelines.
[x ] My pull request has a descriptive title (not a vague title like Update index.md)
[x ] My pull request targets the master branch of freeCodeCamp.
[x ] None of my changes are plagiarized from another source without proper attribution.
[x ] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Thank you for your contribution to the page! 👍
We're happy to accept these changes, and look forward to future contributions. 📝


PHP8 update confilct

updating to PHP8, now i get an error.
Problem 1
- Root composer.json requires laravel/cashier-mollie 1.14.1 -> satisfiable by laravel/cashier-mollie[v1.14.1].
- laravel/cashier-mollie v1.14.1 requires php ^7.2 -> your php version (8.0.7) does not satisfy that requirement.
composer.json:
"require": {
"php": "^7.3|^8.0",
"akaunting/money": "^1.1",
..
..

Please update cashier mollie to the latest release (or at least 1.15).


Update dependencies

Deploy preview processing.
Built with commit 2e465a82f36185551dbb25ac9f65c597b8af6587
https://app.netlify.com/sites/pixelsonly-production/deploys/59f49f550b79b70d00449136


chrome:scale blurred

scale(0.85)

scale(1)

Environment

Leaflet version:
Browser (with version):chrome62
OS/Platform (with version):

I don't really know what you mean here. Can you please be more specific?

I think this can be closed due to a lack of response. It may also be a duplicate of https://github.com/Leaflet/Leaflet/issues/6232.


Fix some edge cases around bracket tokenization in the Nim syntax definition for monaco. Closes #1781

cheers 😄

travis build failed is unreleated to the changes. Thanks for taking the time to fix the syntax definition :)


Performance Issue when default H2 DB is moved to mysql

The DB is moved from H2 (wso2 default) to mysql and it seems like there is a major drop in performance.
All the API calls made from UI takes a lot of time and response time is high when compared to data stored in wso2 default DB (h2) .
Mysql Instance is running on aws with :
8gb RAM.
50gb mem.
WSo2 is hosted on aws with:
4gb RAM.
60gb mem.
Can someone please help me with this .
Thanks in advance

Hi,
The default H2 database is embedded within the APIM server itself and the connections to the database are established locally. Hence, network latency is considerably low. But when the database has been pointed to an external mysql server, the performance is affected by connection status from APIM server to mysql database server. If there is high network latency, when establishing the connection with the database, the performance can be degraded. Thus, please check the connection status from APIM server to the database server and fix the latency issues.


Allow use of the preferred_username OIDC claim.

Previously, Headscale would only use the email OIDC claim to set the Headscale user. In certain cases
(self-hosted SSO), it may be useful to instead use the preferred_username to set the Headscale username. This also closes the existing issue #938.
This adds a config setting to use this claim instead. The OIDC docs have been updated to include this entry as well. In addition, this adds an Authelia OIDC example to the docs.
I didn't see any existing unit or integration tests for OIDC, so wasn't sure where to add such tests. If I'm wrong, I'm happy to add them!
I did test this against my SSO server, and confirm that the preferred_username grant if you set it works as expected.

[x] read the CONTRIBUTING guidelines
[x] raised a GitHub issue or discussed it on the projects chat beforehand (one already exists)
[ ] added unit tests
[ ] added integration tests
[x] updated documentation if needed
[ ] updated CHANGELOG.md

I think this looks sensible, the tests for oidc are located here: https://github.com/juanfont/headscale/blob/main/integration/auth_oidc_test.go
Would be good if we could have one.

Thanks. It looks like the current integration tests don't explicitly set claims, but the underlying mockoidc does let you queue up successive users with potentially different email/preferred_username claims. I'll try my hand at creating an integration test for this.

Were you able to implement a testcase? I'd love to see this feature since it is blocking me from using OIDC for headscale.

@vbrandl I'm working on it, but I'm new to Go and this codebase, and still figuring out how to more efficiently run the integration tests locally without just pushing to Github. The current OIDC tests don't actually check the claim responses at all, so I also need to thread those checks in.
On my stuff, I'm running from these docker images if you want to try them out: https://github.com/meson800/headscale/pkgs/container/headscale

I just tested your image and it does work as expected, but I'd prefer to run the official image for my infrastructure. So I'll wait until this gets merged.

Thank you for this work @meson800

I did some testing on this and it appears to work great.  I noticed the email is still cited on the "congrats you've registered your machine" page or whatever it is which might cause some confusion.
That should probably also reflect the username if the setting is enabled.

I'll look into changing that welcome page template @jonathanspw  to respect the setting as well. On my local machine I have the mockoidc up and running with new mock users. this will be mergable hopefully within a week or so.

@kradalby I added integration tests as requested, so this should be good for review. In sum, this PR includes:

Added the option to use the username claim as above, and updated the documentation.
Updates to the mockoidc runner; now you can give it a series of users in the format username_claim<EMAIL_ADDRESS>and have the mockoidc report those users.
Using that functionality, there are now integration tests that register clients under test-user<EMAIL_ADDRESS>with the different claim options. I do this in a kind of sketchy way, by checking the FQDN string of TailscaleClients (e.g. makes sure they end in test-user.headscale.net or test-email.headscale.net, depending on the setting. I picked this method of validating since in the existing TailscaleClient interface that HSIC implements, there isn't a user field and I didn't want to add them.
As suggested by @jonathanspw, I also updated the post-login HTML template to say "Authenticated as username, you can now close this window", where username is filled in from the preferredUsername or email claim depending on the setting.

Even with a full nix develop on both actual Linux and WSL on Windows, I was unable to run one of the go formatters as suggested in the README, and couldn't figure out the path munging needed to install the tool on my own. If I'm failing some format or lint check, I can try to fix on my end if it's easier.

Otherwise, looks great!

Isn't it better if we add an internal mail domain for this case, i think this would be easier to deal with in the api and al taht stuff :
ex:
preferred_username : loprima would make an email like<EMAIL_ADDRESS>It would be more accurate with the Tailscale way of managing sso I guess

I think that is something that should be up to your OIDC provider. If
you're self hosting, you can return those types of fake/internal email
addresses with the email grant.
Other self hostable software (Nextcloud, Matrix) deals with the possible
non-uniqueness of the preferred username claim by binding usernames to the
OIDC provider, which is something Headscale doesn't currently do. If
Headscale allows multiple OIDC providers in the future that will have ti be
fixed.
On Thu, Apr 27, 2023, 3:14 PM loprima-l @.***> wrote:

Isn't it better if we add an internal mail domain for this case, i think
this would be easier to deal with in the api and al taht stuff :
ex:
preferred_username : loprima would make an email like
@.***
It would be more accurate with the Tailscale way of managing sso I guess
—
Reply to this email directly, view it on GitHub
https://github.com/juanfont/headscale/pull/1287#issuecomment-1526216390,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOEWD7LKA676I4VSPD4DO3XDLASHANCNFSM6AAAAAAWIJSWKI
.
You are receiving this because you were mentioned.Message ID:
@.***>

I agree with that, but aren't we there to implement best practices to follow ? I think using multiple ways of storing a user "id" will be a mess that will complicate the interconnection between Headscale and other pieces of software.
My advice is that we should have one way to identify a user, if we already have email based user distinction, wouldn't it be bad to implement other ways that doesn't fit with the Tailscale way ?

The default OIDC settings prior to this PR already strips the domain from
the email claim unless you configure it not to. If you e.g. point Google's
OIDC provider at it, you'll get a Headscale user with @gmail.com removed.
There's still only one type of user ID with or without this PR.
I think there is an advantage for allowing the option of using the
preferred_username claim to create users; this is how OIDC supports letting
SSO users/administrators pick stable usernames despite changing a backing
email address. There is a unique source of truth on which user is which,
the OIDC provider.
I've only used Headscale; can you elaborate on what the Tailscale way to do
this is?
On Thu, Apr 27, 2023, 3:31 PM loprima-l @.***> wrote:

I agree with that, but aren't we there to implement best practices to
follow ? I think using multiple ways of storing a user "id" will be a mess
that will complicate the interconnection between Headscale and other pieces
of software.
My advice is that we should have one way to identify a user, if we already
have email based user distinction, wouldn't it be bad to implement other
ways that doesn't fit with the Tailscale way ?
—
Reply to this email directly, view it on GitHub
https://github.com/juanfont/headscale/pull/1287#issuecomment-1526253118,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOEWDZVBYP5IX2JMG2UO3TXDLCR5ANCNFSM6AAAAAAWIJSWKI
.
You are receiving this because you were mentioned.Message ID:
@.***>

As you can see here :
Tailscale is using the provider domain, I think that's a better way to do it, maybe we must implement this way also in Headscale, but it would be necessary to use a headscale prefix, so we don't break up the compatibility with native Tascale APP, I don't know how they refer to the login methods.

Thanks for the screenshot!
My feeling is that I think this probably belongs with multi-provider
support, not in this PR. I could work on it though in a future PR though.
I would imagine that with multiple providers, the headscale admin could
configure a suffix for each OIDC provider. E.g. Github could get a @github
prefix, self-hosted Authelia could get a @internal.sso prefix, whatever.
On Thu, Apr 27, 2023, 4:03 PM loprima-l @.***> wrote:

As you can see here : [image: See here]
https://user-images.githubusercontent.com/69201633/234977645-4f4fbc59-d59e-465a-8bce-ca7594fb82f6.png
Tailscale is using the provider domain, I think that's a better way to do
it, maybe we must implement this way also in Headscale, but it would be
necessary to use a headscale prefix, so we don't break up the compatibility
with native Tascale APP, I don't know how they refer to the login methods.
—
Reply to this email directly, view it on GitHub
https://github.com/juanfont/headscale/pull/1287#issuecomment-1526328046,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOEWDZMQ6ZUBNQVGUSQWQLXDLGKXANCNFSM6AAAAAAWIJSWKI
.
You are receiving this because you were mentioned.Message ID:
@.***>

I think that would be AMAZING <3
Like it will help a lot in the infinite quest of having a standardized system, I'm really busy to work on it but if you need help let me know I could write the docs if you want, I'm already planing to open a PR to add a gitbook.

I merged against main so this can run tests again.
@kradalby do you want me to change the environment variables as in the review above? I named them to be consistent with the surrounding code.
@juanfont do you want me to update/edit this PR in some way/can you review? I see that there are other OIDC PRs that this might conflict with this (#1435), I can incorporate those changes if needed.

When will this PR be merged to main branch, that feature is acutely I want.

Yes, please. Immutable identifier to be used by headscale would be nice to have, as most IDPs allows users to change their emails, and thus break their headscale setup/login.

I can try to update this branch to get it considered again; a lot has changed since I submitted the PR.

I can try to update this branch to get it considered again; a lot has changed since I submitted the PR.

Please do.

Hi! as part of #1473, we have reorganised a lot of the code.
To clear PRs that needs to be rebased or redone, we are closing open PRs that will require significant code change to be merged.
In addition, the issue of the PR might in some cases have been fixed, change or no longer relevant, so it would be great if this is considered as well.
Thank you for your contribution!
If it is still relevant and the PR is reopened, we will aim at getting the changes into the next release after the reorg if accepted.

@kradalby I can't reopen the PR myself, but this is ready to be considered again; I rebased and tested meson800:oidc_username_claim on top of the re-orged main.
Summary

[x] read the CONTRIBUTING guidelines
[x] raised a GitHub issue or discussed it on the projects chat beforehand (one already exists)
[ ] added unit tests
[x] added integration tests
[x] updated documentation if needed
[x] updated CHANGELOG.md

This PR adds the ability to use the preferred_username claim, closing #938. I added documentation, updated the CHANGELOG.md, and added integration tests; I didn't add a unit test but can if needed.
The short version is that you can now set the use_username_claim config setting inside the oidc category. If set to true, the preferred_username claim is used to set the Headscale username instead of the email claim. I updated the way mockoidc is called to let integration tests set these claims.
Integration tests
In particular, the integration tests are implemented by setting the mock OIDC user to test-user<EMAIL_ADDRESS>then it checks that the FQDN of the user is test-user.headscale.net or test-email.headscale.net.
Test failures
Of note, I ran all of the integration tests in a stub PR in my repo; some integration tests appear flaky so I had to rerun them, but one still fails despite reruns, TestPingAllByIP. This one also seem to be failing in other PR runs in this repo so I don't think it's related (example 1)
Real tests
I generated a Docker image for this PR and tested it out in my tailnet, and it works, correctly using the preferred_username claim:

Looks awesome! Would really help my auth setup.

Any reason this wasn't merged? I see it's closed. I think it's a very welcome improvement.

I think that one of the concerns is it wasn't generalizable enough. See this comment: https://github.com/juanfont/headscale/issues/938#issuecomment-1952697228


Add diagnostic for KT-65542 compatibility.json

I've run ./gradlew allTests locally - all tests passed.
These tests passed as well. (they should be the most interesting in our case)


added placeholders

Current coverage is 42.97%

Merging #217 into master will not affect coverage as of 1a4a825

@@            master    #217   diff @@
======================================
  Files           21      21
  Stmts          370     370
  Branches         0       0
  Methods          0       0
======================================
  Hit            159     159
  Partial          0       0
  Missed         211     211

Review entire Coverage Diff as of 1a4a825

Powered by Codecov. Updated on successful CI builds.


Add new element (Raw Image, some delegates events and methods) to avoid use of byte[]

To be coherent, IVideoSource.ExternalVideoSourceRawSample must also use RawImage
Bump to version <IP_ADDRESS>

PR:
Bump version to 1.2.0
Same delegates, event and methods than version 1.1.0 (no renaming done)
Add sone new elements:
new Delegates:
public delegate void RawVideoSampleFasterDelegate(uint durationMilliseconds, RawImage rawImage);
public delegate void VideoSinkSampleDecodedFasterDelegate(RawImage rawImage);

new Object:
RawImage

IVideoEncoder:
new methods:
byte[] EncodeVideoFaster(RawImage rawImage, VideoCodecsEnum codec);
IEnumerable<RawImage> DecodeVideoFaster(byte[] encodedSample, VideoPixelFormatsEnum pixelFormat, VideoCodecsEnum codec);

IVideoSource:
new event:
event RawVideoSampleFasterDelegate OnVideoSourceRawSampleFaster;

**_new method:_**

void ExternalVideoSourceRawSampleFaster(uint durationMilliseconds, RawImage rawImage);

IVideoSink:
new event:
event VideoSinkSampleDecodedFasterDelegate OnVideoSinkDecodedSampleFaster;


yaml: Unexpected indent in simple key: value case

Describe the bug
After entering in a key: value entries in yaml, the next line indent is increased by shiftwidth.
To Reproduce
Detailed steps to reproduce the behavior:

Open a new yaml file.
Enter "a: b" and hit enter.

Expected behavior
Next line is equal indent.
Environment (please complete the following information):

Vim version: homebrew (8.2 1-3300)
OS: mac 10.15.7
Terminal: iTerm2

Additional context
The culprit is here: https://github.com/vim/vim/blob/eed9616120f60ad17d5e133f505bba5c559aee89/runtime/indent/yaml.vim#L145-L148
I suspect that final * is not desirable.

why do you think the * is not desirable?

Because it makes the thing do nothing. As noted in the original report, this line matches every key: value pair, not just the ones that have multi-line indications.

Should it be +   (backslash-plus) instead?

That would make a certain amount of sense.

OK, let me make that change, please report again if that doesn't work.

Changing the "*" into "\+" actually makes the test fail.  The indent for "value" goes missing in
key: something
    value

Yup, ok, so apparently the indent fixes itself after you type another :, it's just surprising when it initially indents. But still correct.

This problem doesn't occur in Vim 8.1.
I confirmed diff between codes in 8.1 and 8.2.
The following fixes affect this problem.
https://github.com/vim/vim/commit/acc224064033e5cea21ef7f1eefb356ca06ff11d#diff-723cc09b2d0f8f9383c670a8846979363fbc67a741aad1aae657242b8c2667a3L56
https://github.com/vim/vim/commit/acc224064033e5cea21ef7f1eefb356ca06ff11d#diff-723cc09b2d0f8f9383c670a8846979363fbc67a741aad1aae657242b8c2667a3L65
In 8.1, process doesn't enter any if/elseif prevline block in GetYAMLIndent function.
In 8.2, process enters the last elseif prevline block in GetYAMLIndent function. (https://github.com/vim/vim/blob/master/runtime/indent/yaml.vim#L145)
@brammool
Would you please check these fixes?

can you please create a PR for that?

@chrisbra
I created the PR.
(and confirmed the problem is resolved)


pod install fails

Podfile
target 'TestApp'
use_frameworks!

pod 'ReactiveKit', '~> 2.0'
pod 'Alamofire', '~> 3.3'
pod 'JSONCodable', '~> 2.1'
pod 'AlamofireReactive', '>= 1.0'
pod 'AlamofireImage', '~> 2.0'
pod 'ReactiveAppKit', '~> 2.0'

pod --version
1.0.1

pod install --verbose
  Preparing

Analyzing dependencies

Inspecting targets to integrate
  Using `ARCHS` setting to build architectures of target `Pods-TestApp `: (``)

Finding Podfile changes
  A ReactiveAppKit
  - Alamofire
  - AlamofireImage
  - AlamofireReactive
  - JSONCodable
  - ReactiveKit

Resolving dependencies of `Podfile`
[!] Unable to find a specification for `ReactiveAppKit (~> 2.0)`

/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/resolver.rb:442:in `handle_resolver_error'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/resolver.rb:65:in `rescue in resolve'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/resolver.rb:57:in `resolve'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/installer/analyzer.rb:563:in `block in resolve_dependencies'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/user_interface.rb:63:in `section'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/installer/analyzer.rb:561:in `resolve_dependencies'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/installer/analyzer.rb:76:in `analyze'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/installer.rb:227:in `analyze'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/installer.rb:151:in `block in resolve_dependencies'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/user_interface.rb:63:in `section'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/installer.rb:150:in `resolve_dependencies'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/installer.rb:114:in `install!'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/command/install.rb:37:in `run'
/Library/Ruby/Gems/2.0.0/gems/claide-1.0.0/lib/claide/command.rb:334:in `run'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/lib/cocoapods/command.rb:50:in `run'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-1.0.1/bin/pod:55:in `<top (required)>'
/usr/local/bin/pod:23:in `load'
/usr/local/bin/pod:23:in `<main>'

+1 - Still broken

ReactiveAppKit has been converted in Bond almost a year ago :)
https://github.com/ReactiveKit/Bond :)


This library look great but don't know how to use this library in angular2.0 (final) project?

Here's some examples that might help!

https://www.thepolyglotdeveloper.com/2016/01/include-external-javascript-libraries-in-an-angular-2-typescript-project/
<EMAIL_ADDRESS>https://coryrylan.com/blog/angular-2-cli-adding-third-party-libraries

Then it's just a matter of configuring the clients to point to your hub! :)

Thank a lot @danielstjules


Include multi-agent system on front page and io site

Something a long the lines of: JALSE is a multi-agent system framework etc.
With a link to http://en.wikipedia.org/wiki/Multi-agent_system

This entire front page should be re-written to talk about what we offer now


upgrade to java 8

Refer to this link for build results (access rights to CI server needed):
http://sdvjenkins.slub-dresden.de/job/Github PR GDM Model (Graph Json)/9/
Test FAILed.

tests ran successfully @ jenkins, see http://<IP_ADDRESS>/view/Github PR Builds/job/Github PR GDM Model (Graph Json)/10/

Refer to this link for build results (access rights to CI server needed):
http://sdvjenkins.slub-dresden.de/job/Github PR GDM Model (Graph Json)/11/
Test PASSed.


Fix issue with multiple "PermitRootLogin" lines in Ubuntu sshd_config

Now edits the already-existing PermitRootLogin line in place to ensure it's uncommented, and then ensures it has the correct value as set by the "access_ssh_root_mode" parameter (if existing)

This is addressed by #34


how to get ecc reads from Circle-map outputs file

After running Circle-map, I can get a bed file of ECC locations. But how can I get the ECC reads? I think not all reads that in the ECC locations are ECC reads.

Dear @panxiaoguang
You are right. Circle-Map Realign provides a summarized output of the eccDNA reads.BUt it does not provide direct access to the reads.
I have a beta release of that reports the output realigned in reads in BAM format, following the standard SAM format to report the realigned reads as split reads. I think that this is exactly what you are looking for. You can execute it as follows:
Circle-Map bam2bam -i sorted_circular_dna_candidates.bam -qbam qname.bam -fasta /path/to/hg38.fa -o output.bam

Where:
sorted_circular_dna_candidates.bam: Is the same input as Circle-Map Realign
qname.bam is a query name sorted bam file
Best,
Inigo

Thank you very much for your reply

Hi, Inigo,
after running Circle-Map bam2bam, I successfuly got a bam file. But when I run samtools sort, I got an error message "[E::bgzf_read_black] Invalid BGZF header at offset<PHONE_NUMBER>". Have you ever encountered this error, and what are the possible reasons?


Chart X-Axis tick limiting

Hi,
Is it possible to set the number of items in the x axis at a given time? for example.. currently for the chart, the x axis contains the same amount of values..:
Small screen size

Medium screen size

Large screen size

Regardless of the chart size, there is 7 items on the x-axis... is there an option in the API to limit this such as:
axis: {
x: {
values: 3
}
}
this would produce the following:

axis: {
x: {
values: 5
}
}
would produce the following:

Hi
I am facing one issue in X axis values. See my code below
<script src="http://d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src="c3.js"></script>
<script>
var chart = c3.generate({
    data: {
        x : 'x',
        columns: [
            ['x', 'Ho Chi Minh City', 'Tanjung Priok (Jakarta)', 'Shenzhen', 'Haiphong', 'Antwerp', 'Chennai', 'Memphis Rail Ramp', 'Los Angeles / Long Beach / San Pedro', 'Shanghai', 'Shanghai'],
            ['Your lowest rate', 30, 200, 100, 400, 300, 25, 1000, 40000,20000,23000],
            ['Benchmark rate', 400, 200, 100, 4000, 1900, 100, 1400, 250,16000,19000],
            ['Lowest rate on route', 350, 15000, 19000, 16210, 16000, 12040, 12510, 18800,19500,11150],
        ],
        type: 'bar'
    },
    axis: {
        x: {
            type: 'category' // this needed to load string x value
        },
        y: {
          label: {
            text: 'Your Y Axis',
            position: 'outer-middle'
          },

        tick: {
          values: [1000, 5000, 10000, 15000, 20000, 25000, 30000, 35000, 40000, 45000, 50000]
        }

        }
    },

});

I have used "Shanghai" at two places in X axis value. Now its showing the position of Shanghai in array not name when I am plotting the graph. See attached graph image. How can I show the duplicate name in X - axis ..... because in my case there can be duplicate records in X- axis.


Buffer duration control?

Is there any way to control the buffer duration with the media-kit backend? For Android there's AndroidLoadControl.minBufferDuration and AndroidLoadControl.minBufferDuration, and for iOS there's DarwinLoadControl.preferredForwardBufferDuration in just_audio.
It appears like currently the player is using the AndroidLoadControl on Windows, but that could just be a coincidence.
Either way, it seems like the buffer always stops at the end of the current track, and does not pre-fetch the next track, contrary to what happens on the mobile platforms. Setting the buffer duration to 10 minutes should always fetch the next 10 minutes of the current audio source (which is a ConcatenatingAudioSouce in this case), even if the current track is shorter than that.
Is that a limitation of media-kit?

No, it's not a limitation, you can edit the buffer size with the JustAudioMediaKit.bufferSize static parameter.
Please check the README: https://github.com/Pato05/just_audio_media_kit?tab=readme-ov-file#plugin-specific-configuration-settings-for-media_kits-player-instance

Either way, it seems like the buffer always stops at the end of the current track, and does not pre-fetch the next track

That is true. I wanted to create a proxy server to handle this, but that would be too bloat-y to include in the main plugin, so I'll probably never actually do this. However, feel free to create a new plugin for it!
Actually this might not be necessary as it seems mpv already supports this via the --prefetch-playlist option (see media_kit#200). I'll test for a possible solution in the next days, as today I may not be able to.

So there's no way to set the buffer size based on a duration somehow?

Not as far as I'm aware...

The --prefetch-playlist option actually doesn't seem to solve the issue. It expects a static queue, and only fetches the next track shortly before the current track ends. So it's more like a gapless option?

depends on the demuxer cache settings [...]. This merely opens the URL of the next playlist entry as soon the current URL is fully read. (source)

According to docs it's kinda similar to what just_audio provides in Android. Though docs also say it's highly experimental, so I may make this available under a flag that is false by default.
Also I'm not really sure how "gapless" would be different than pre-fetching the next track..

I am a bit confused by the part about it not pre-fetching video data. It sounded like it would simply open a connection and fetch the header of the next item, instead of actually loading any media data. I guess we'll have to try it out.
Where would I need to add this option to try things out?

It sounded like it would simply open a connection and fetch the header of the next item, instead of actually loading any media data.

You may actually be right, that's why it says "merely", I guess...

Where would I need to add this option to try things out?

You could add it right after _player = Player(...); in mediakit_player.dart, you'd have to call _player.nativePlayer.setProperty('prefetch-playlist', 'yes'); or something along these lines.. There's more info in the media_kit issue though...

Ok, so I did some testing by using a python HTTP server (from scratch, so that I can check whenever the connection is open), and it seems that --prefetch-playlist just makes the loading a little bit less lazy by starting to load the track moments before it has to be played (see attached video)
Screencast from 2024-04-09 14-25-14.webm
I don't think increasing the demuxer buffer will actually do anything, as by default it should be about 150MB, and, well, the song doesn't even go close to that...

So there's no way to set the buffer size based on a duration somehow?

I was looking at the mpv docs, and yes, you can specify a duration instead of specifying bytes, with the --cache-secs option. It may make sense to use the AndroidLoadControl.maxBufferDuration option as the value for this option, or just let it be.
If https://github.com/ryanheise/just_audio/issues/1222 gets fixed, I could export the Player instance, so that more options can be set via setProperty, thus allow more freedom on controlling the player without editing just_audio_media_kit's code..

While testing the new option, it seems that the actual behaviour in media_kit is slightly different (and way closer to what we want in the first place). My example app is just a tweak of the example in just_audio, where I removed the ClippingAudioSource, and basically changed the URL for the third entry to my local HTTP server.
I'll cut to the chase: it started loading the track when there were about 13 minutes left from the podcast, and it didn't only stop when it received the headers, but the entire song was downloaded way before it was required for playback!
prefetch.webm

Sorry for not responding to this, I'm currently sitting on a huge stack of unread notifications and I'm spread way to thin. So thanks for all your work, and the nice write-ups.
Looks like everything ended up working out great!
I see that you've already published your changes. I'll give this a try immediately and report back!

Yeah, that seems to work well! Sometimes the buffer doesn't show up in our UI immediately, but the next track is definitely starting to play immediately.
Skipping ahead two tracks doesn't seem to work yet, but that's fine. Could be a cache size issue, although you mentioned a size of 150 MB, which should be more than enough.
Either way, I also consider this issue closed. Providing the buffer duration in seconds would be nice, but if you want to wait for ryanheise/just_audio#1222 for that, that's also fine with me!
Thanks again!

I'm still a but fuzzy with the relation between media-kit and mpv. I thought media-kit was a stand-alone player that doesn't use platform-specific implementations, but your comment makes me think it's just an additional wrapper around the "native" players for all platforms, that exposes a unified API?

I'm still a but fuzzy with the relation between media-kit and mpv. I thought media-kit was a stand-alone player that doesn't use platform-specific implementations, but your comment makes me think it's just an additional wrapper around the "native" players for all platforms, that exposes a unified API?

media-kit uses libmpv internally. There would be only two "platform implementations", being the native platform (platforms with dart:io) and the web platform (flutter web, which implements an HTML player)


Make Strict.traverseWithKey actually strict

Fixes #192

@pierric, do you think you could take a look at this and see if it works for you?

Note: this needs tests before it can be merged.