Provide opt-in cfg to use precompiled blob

Closes #2538
Supercedes #2579
Follows my recommendation: https://github.com/serde-rs/serde/pull/2579#issuecomment-1684934840
Makes it opt-in choice to use the precompiled binary with cfg(serde_derive_build = "precompiled") and leaving it to the top-level binary to use the precompiled blob if it explicitly opts in to - by default people will compile from source always.
This also demonstrates the cfg based approach that frees dependency chain from messing up with the features in the middle leaving it to the top level binary as informed choice.
This is used elsewhere e.g.: https://github.com/dalek-cryptography/curve25519-dalek/tree/main/curve25519-dalek#manual-backend-override - that was result of this discussion: https://github.com/dalek-cryptography/curve25519-dalek/issues/414 providing background why / how this approach has been used elsewhere.
Cc/ @sagudev I cherry picked
Default - my host is x86_64-linux-gnu
$ cargo build
   Compiling proc-macro2 v1.0.66
   Compiling unicode-ident v1.0.11
   Compiling quote v1.0.33
   Compiling syn v2.0.29
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
    Finished dev [unoptimized + debuginfo] target(s) in 3.20s

Precompiled  opt-in at top-level binary - host is x86_64-linux-gnu
$ env RUSTFLAGS="--cfg serde_derive_build=\"precompiled\"" cargo build
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
    Finished dev [unoptimized + debuginfo] target(s) in 0.28s

@simonsan Could add a feature in addition but cfg is the easiest to handover the top-level binary the choice.
Please note that this PR makes it opt-in only where the top-level binary would need to want the precompiled one.
By default you would get it built from the source if this PR is merged.

I will repeat my comment from #2579.

I don't believe this adequately resolves https://github.com/serde-rs/serde/issues/2538. This would still cause unknown, unsigned, and unverifiable executables to placed on any systems which use serde as a dependency. That's still a notable security risk.

While I'm obviously in favor of this, as it's a massive improvement, this still leaves an unverifiable executable sitting on the system.
I'd state the binary should only be downloaded on opt-in.

@kayabaNerv it is opt-in ? it doesn't download or does it ? Or did I miss the download somewhere ? let me check

serde-derive-bin which has the binary

That would certainly make it more explicit and easier to reject in a crate graph. But it doesn't actually solve too much, as afaik it will still be downloaded, just never built. Or at least show up in a Cargo.lock

As someone who works in the medical field with Rust I want to express my support for this

Two things

I don't think changing the default to opt-in will be something that we'll land right now. I rather think that starting out with making it opt out will be something we'll be able to land while figuring out the entire situation.
Please document how to use this in the readme.

I rather think that starting out with making it opt out will be something we'll be able to land while figuring out the entire situation.

Why? This feature didn't even exist until recently, is considered experimental and is widely considered a problem. This should be opt-in until the Rust community agrees as a whole that it is good (safe, reproducible, ...) enough to be opt-out.

Why?

Because this crate is owned by dtolnay and he may do with it as he wishes. My personal opinions on the topic are irrelevant. While I would prefer fundamental crates to have entire (funded!) teams working on them, that's not the reality.
I am reviewing this from the perspective of getting the PR landed.

Because this crate is owned by dtolnay and he may do with it as he wishes.

Serde has nearly 200 Million downloads on crates.io, and is a staple of the entire Rust ecosystem. It's downright disrespectful to the entire Rust community to say their wants and needs are less important just because the one and only benevolent dictator has the publish rights on crates.io. There's a good argument that this crate should be adopted by the Rust org to avoid such bottlenecks. At this scale the need to appeal to someone's feelings is irrelevant. We need this to be opt-in until the day things upstream are safe enough to guarantee prebuilt binaries will be safe, perhaps using projects like Watt.

@pinkforest While I agree anyone being able to include any file is a distinct issue, it being in a separate crate which isn't actually downloaded would solve that. I do see a comment claiming optional dependencies, even if unused, will appear in the crate graph and be downloaded though, which may void my comment's validity.

It's downright disrespectful to the entire Rust community to say their wants and needs are less important just because the one and only benevolent dictator has the publish rights on crates.io

I am not saying that. I am saying what the reality is. Not what I wish reality were.

@oli-obk ok - I will add another commit to opt-out - can always re-base to previous commit if opt-in is chosen instead so choice will be with maintainers - I will also add doc how to use it - thanks

New behaviour with cfg(serde_derive_build = "source")
Default - host x86_64-linux-gnu
$ cargo build
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
    Finished dev [unoptimized + debuginfo] target(s) in 0.16s

Override - host x86_64-linux-gnu
$ env RUSTFLAGS='--cfg serde_derive_build="source"' cargo build
   Compiling proc-macro2 v1.0.66
   Compiling unicode-ident v1.0.11
   Compiling quote v1.0.33
   Compiling syn v2.0.29
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
    Finished dev [unoptimized + debuginfo] target(s) in 3.17s

@simonsan

How would I specify that I don't want the precompiled binary when I only pull in serde with the serde_derive feature like this?

The most flexible option is to use:
serde = { version = "1.0.x, <1.0.172", features = ["derive"] }

where x is minimum version required by your project.
@oli-obk

But it doesn't actually change too much, as afaik it will still be downloaded, just never built. Or at least show up in a Cargo.lock

IIRC unused crates do not get downloaded during compilation, but, yes, they are get mentioned in Cargo.lock.
From my experience, having a separate crate for pre-compiled version of the crate will be quite useful when dependencies get vendored. Even though it will be mentioned in Cargo.lock, we will not have a semi-random blob as part of our vendored source code.

It is really easy to accidentally override the rustflags and thus opt back into the precompiled version. RUSTFLAGS will override ~/.cargo/config.toml and a per-project .cargo/config.toml will override the global one. As such putting the opt-out once in your ~/.cargo/config.toml may accidentally use the precompiled binary again if the project you build uses .cargo/config.toml or you or the project needs RUSTFLAGS (eg in rust's build system)

Using rustflags as opt-out makes it easy to accidentally stop opting out. RUSTFLAGS and the per-project .cargo/config.toml take precedence over ~/.cargo/config.toml and RUSTFLAGS takes presedence over the per-project .cargo/config.toml, so if you or the project needs to use either the opt-out will not be passed by cargo. This is for example the case in rust's build system.

I've added a simplified gating but then I've noticed when testing there is a bug related to cross compliation
The cross-compilation bug is here:

#2582

It seems this is always building on host TARGET configuration for some reason or maybe it's just me ?

Current CI errors are github fault as they fail on checkout.

Upon closer inspection from dependency PoV - I've seemingly replicated the cross-compilation issue from dep side:

https://github.com/serde-rs/serde/issues/2582#issuecomment-1685226535

(But this should work as-is regardless when compiling on host and GitHub gets itself together)

I just remember another issue with RUSTFLAGS opt-outs. Cargo ignores RUSTFLAGS for host dependencies like proc-macros when cross-compiling.

@bjorn3 which level / version ? In stable / nightly the [target.'cfg(..)'.dependencies] gating seems to work for both manifest: https://github.com/serde-rs/serde/pull/2580#issuecomment-1685227828 and relaying to lib.rs ?

CI :heavy_check_mark: :tada:

I'll be able to try out the PR locally in 8 days when I'm back to having something more powerful than a phone 😅
I'll give it another code review some time later today

This seems to have a fatal flaw as I both found and finally realized the hard way after @bjorn3 pointed out namedly RUSTFLAGS gets ignored to proc-macros upon cross-compiling :woman_facepalming:
So it would have to be by feature (as much as difficulties those pose) per #2579 (+ documentation) :thinking: :thinking:  ?

Do you remember what the issue# was ?

https://github.com/rust-lang/cargo/issues/4423

Ok this is now by feature as option when used with from_source
x86_64-linux-gnu Host default
$ rm -rf target ; cargo build --features from_source
   Compiling proc-macro2 v1.0.66
   Compiling unicode-ident v1.0.11
   Compiling quote v1.0.33
   Compiling syn v2.0.29
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
    Finished dev [unoptimized + debuginfo] target(s) in 3.18s

wasm32-u-u cross-compile on x86_64-linux-gnu Host default
[foobar@localhost serde_derive]$ rm -rf target ; cargo build --target wasm32-unknown-unknown
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
    Finished dev [unoptimized + debuginfo] target(s) in 0.29s

wasm32-u-u cross-compile on x86_64-linux-gnu Host - Using from_source Feature
[foobar@localhost serde_derive]$ rm -rf target ; cargo build --features from_source --target wasm32-unknown-unknown
   Compiling proc-macro2 v1.0.66
   Compiling unicode-ident v1.0.11
   Compiling quote v1.0.33
   Compiling syn v2.0.29
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
    Finished dev [unoptimized + debuginfo] target(s) in 3.46s

@bjorn3 How do you feel this approach ? feature=from_source would be unified

There's no env var, but maybe you can use the config settings described in https://doc.rust-lang.org/cargo/reference/config.html#buildrustflags
target.CFG.rustflags may apply to the host?

@bjorn3 @oli-obk How do you feel this approach ? feature=from_source would be unified

How does this work if used as a dependency?

How does this work if used as a dependency?

Would need to use the dependency with optional feature included - cargo adds them together so one dependency adding it would make it build from source in whole build as a sum.
But I will test the build.rustflags w/ CARGO_ENCODED_RUSTFLAGS - that might be an option too to provide.

If serde_derive is used for a build dependency afaik with the v2 resolver it wouldn't get feature unified with serde_derive used by a non-build dependency or the root crate.

Hey, initially an opt-in version was pitched. Is opt-in off the table now?

Hey, initially an opt-in version was pitched. Is opt-in off the table now?

You're right, that was the initial pitch. The discussion around why it was changed has been marked as "not relevant" (not at all sure why) but you can find it in this thread by expanding those comments.

Hey, initially an opt-in version was pitched. Is opt-in off the table now?

I personally would also prefer an opt-in solution. But if any crate you used has opt-in to use the binary, you can not disable it anymore. So opt-out would probably the better solution.

Opt-in through RUSTFLAGS doesn't have that problem.

@bjorn3 I just tested using commit #585482 the resolver="2" and it seems to work regardless ?
Repro here: https://github.com/pinkforest/serde_optout_unification_proof
Source by opt-out
top-bin]$ rm -rf ../target ; cargo build --features from_source
   Compiling proc-macro2 v1.0.66
   Compiling unicode-ident v1.0.11
   Compiling quote v1.0.33
   Compiling syn v2.0.29
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
   Compiling some-lib v0.1.0 (/home/foobar/code/serde_optout_unification_proof/some-lib)
   Compiling top-bin v0.1.0 (/home/foobar/code/serde_optout_unification_proof/top-bin)
    Finished dev [unoptimized + debuginfo] target(s) in 2.97s

Default precompiled
top-bin]$ rm -rf ../target ; cargo build
   Compiling serde_derive v1.0.183 (/home/foobar/code/serde/precompiled/serde_derive)
   Compiling some-lib v0.1.0 (/home/foobar/code/serde_optout_unification_proof/some-lib)
   Compiling top-bin v0.1.0 (/home/foobar/code/serde_optout_unification_proof/top-bin)
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s

Anything I did wrong here ?

I'd like to mention that the decision of using precompiled binaries for Linux x64 platforms is already causing significant ecosystem fragmentation. That might be useful to keep in mind for reviewing this PR: https://github.com/time-rs/time/discussions/611

IMHO Opt-in for the precompiled binary should be the desired outcome. It is backwards compatible with the ecosystem's current serde dependency specifications, and its expectations before learning about the precompiled binary.

As followed this issue, just mention to care about upcoming cyber resillience act,
only/or by default just binary would drop out serde of projects.

@bjorn3 fwiw - I just tested parked at commit #585482 the resolver="2" and it seems to work regardless ?

Maybe it only applies to non-proc-macro deps? In any case great that it works.

@AlexTMjugador wrote:

I'd like to mention that the decision of using precompiled binaries for Linux x64 platforms is already causing significant ecosystem fragmentation. That might be useful to keep in mind for reviewing this PR: time-rs/time#611

Regarding this - do you think it would be worth putting forward a PR that fully reverts the precompiled PR until this or #2579 land (or whatever is the real fix)? To me that approach seems like it will give some breathing room to work out the correct fix.

Regarding this - do you think it would be worth putting forward a PR that fully reverts the precompiled PR until this or #2579 land (or whatever is the real fix)? To me that approach seems like it will give some breathing room to work out the correct fix.

I'd support that, but I'm not a serde repository maintainer to decide.

I'd support that, but I'm not a serde repository maintainer to decide.

Yep - I understand that. I read your reply over in the time issue and figured you might have a reasonable perspective on this one way or another.

from_source should really be on by default, making it not include the binary in the default feature set.

Thanks will do - appreciate looking over - I can just override this existing branch in this PR no problem

Nice. I'd prefer a new PR to preserve the final working approach that was settled on in this one for opt-out.

Ok great no worries :heart:


Improve uptime during agent upgrades (with L7 policy)

Proposal / RFE
It is known that agent upgrades will disrupt connectivity if L7 policy is in place. I'm wondering if there's an opportunity to improve this though, perhaps by leveraging the pre-flight check process and the SO_REUSEPORT socket option.
Today, the preflight-check runs a dummy agent: perhaps, that agent could take over handling the traffic (HTTP, DNS, etc.), listening on the same port(s) as the "real" cilium-agent. An upgrade process could look like this:

run the pre-flight check (and leave it running)
upgrade/restart the cilium-agent
un-install the pre-flight check

I haven't really looked at what this would mean from code perspective (cilium-agent, envoy etc), but in concept, it seems it may work and get rid of the downtime altogether, so I thought I'd throw this here and get some feedback.

This is a great question. In recent versions we have begun to use SO_REUSEPORT already to minimize downtime (#12801). However there still remains a period where kubernetes shuts down the existing pod and launches the new pod. This can last typically for several seconds. Last time I looked, this was more influenced by k8s scheduling of the new pod and less about Cilium startup time (assuming preflight check is in use for pre-loading the cilium docker images).
If we were to run another instance of regular Cilium like this, in practice we would likely need to disable certain other functionality to avoid having two Cilium agents attempting to manage the BPF programs and maps at the same time. So we would likely want to load the preflight version with a flag to disable various features to avoid such conflicts.
Taking this idea a bit further, another design is to look at moving the lifecycle of the proxies entirely out of the agent. This way the lifecycle of the L7 dataplane would be entirely decoupled from the agent. I agree that this is an interesting direction to investigate.

If we were to run another instance of regular Cilium like this, in practice we would likely need to disable certain other functionality to avoid having two Cilium agents attempting to manage the BPF programs and maps at the same time. So we would likely want to load the preflight version with a flag to disable various features to avoid such conflicts.

Totally: I wouldn't expect the full functionality and definitely no bpf changes.

Taking this idea a bit further, another design is to look at moving the lifecycle of the proxies entirely out of the agent. This way the lifecycle of the L7 dataplane would be entirely decoupled from the agent.

I like it! Out of curiosity, is the data used by the proxies strictly local to the node where the traffic originates? I wonder if service topology would help here....

is the data used by the proxies strictly local to the node where the traffic originates?

For egress L7, yes. More broadly speaking, it's not necessarily for where the traffic 'originates', but local yes. In the case of ingress L7, it's applying policy for the destination pod and is locally configured by the agent at the destination.

duh! I forgot about ingress! ok.

ping!
(keeping this alive: a control-plane restart shouldn't create downtime on the data-plane)

We are affected by this issue in our project for which it is critical to have zero downtime upgrades. After some initial testing, I saw that we have failed ingress/egress requests if we use one of the following features in Cilium network policies (I'm sure there are more):

ingress policy w/ a HTTP path rule (spec.ingress.toPorts.rules.http.path)
egress policy w/ a FQDN rule (spec.egress.toFQDNs.matchName)

I have the following questions:

Is it still expected to have downtimes when using Cilium network policies like the ones above? We are using Cilium 1.16.4.
Are there any known workarounds for achieving zero downtime upgrades of the cilium agent?


Fixed irritating typo in ResetPasswordAsync()-docs

@lkempf,
Thanks for having already signed the Contribution License Agreement. Your agreement has not been validated yet. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

@lkempf The new version also contains a typo 😄, changed word should be succeeds, not succeedes.

I think I'll just create a new pull request now


crtags is giving 'TypeError: ('init() takes exactly 2 arguments (1 given)', <class 'clang.cindex.LibclangError'>, ())' on ubuntu 14.04

stack trace
$ crtags -v ~/curl-7.41.0/

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 380, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class 'clang.cindex.LibclangError'>, ())

I got the same error on Ubuntu 12.04. I tried harshal-dhumal's fork, too, but it also got the same error :(

I haven't been able to figure out how to capture Keyboard interrupt from child processes. One way I see is to remove all multiprocessing code.

I got the same error on several systems (CentOS 6, Fedora 22 etc.) where no clang was installed.
Later I figured out that, it actually has a dependency with "clang", which cannot be resolved by "pip install crange".
BTW. the "python setup.py install" didn't work under python 2.

Note that the pypi versions of clang since 3.7 declare themselves to be "Obsolete" according to the package descriptions on https://pypi.python.org/pypi/clang/ (3.8, 3.9.post1, 4.0.post1):

OBSOLETE. LLVM-CLANG NOW PUBLISHES PYTHON PACKAGE.
JUST ADD THE OFFICIAL llvm-3.7 repo in your apt.

A workaround for me was to uninstall the pip package for clang (which got installed when I did "pip install crange") and install python-clang-3.9.  Since I'm using Linux Mint 17 (rosa), I did that by
sudo apt-get install python-clang-3.9
(The same package exists on Ubuntu 14 (trusty) and Ubuntu 16 (xenial).)
After doing that, crange works fine for me.  I hope that workaround helps someone else too.
I'm not sure if the fundamental issue is that clang in the official repos is somehow "better" than in pypi, or that something has changed in clang 4.0 vs 3.9 and earlier.  When I tried to install using only pip, it naturally grabbed the latest clang in pypi, which is "4.0.post1".  When I installed clang using "apt", 3.9 was the latest available, so I used that.


context in the tree

If I add comment in the first submission then I switch to another submission and then come back to first submission, my comment not saved:(

need not


Regarding the number of data in ReVeal dataset

Hi, thanks for the awesome project and dataset.
I have some question for the number of data in dataset. In your paper, it says that there are 18,169 programs in the ReVeal dataset, while I found 2,240 + 20,494 items in the two json file https://drive.google.com/drive/folders/1KuIYgFcvWUXheDhT--cBALsfy1I4utOy. I wonder what causes the difference, and how should I use the data in the  json file?
Thanks :)

My recommendation is going straight to the NVD and SARD test cases, and not using released partial datasets from authors. This is no critique to the ReVeal authors.


Windows Defender Application Control/AppLocker: typo correction & remove blanks

Proposed changes:

Correction: change "how to used" to read "how to use".
Removal: remove 2 redundant blank lines at the end of the document.

Ref. issue ticket:

#1230 (No data?)

Please note:

if the Microsoft Docs Team already has got material for this page,
feel free to close this Pull Request without merging it, especially
if your material contains improvements to the same page
(and may be ready to be pushed quite soon).

@illfated Thanks Trond!


[question] Can I change image DPI?

Now I set the DPI, the image seems to change the width and height, instead of changing the value of DPI, my question is: is it possible to directly modify the value of DPI without changing the width and height?

Instead of the DrawImage function, you should use the FitImage which allows
you to set a rectangle where to draw the image and what filling method to
use: ImageContain, ImageCover, or ImageFill. Let me know if that works!
El mié., 1 de mar. de 2023 05:37, overbeck @.***>
escribió:

Now I set the DPI, the picture seems to change the width and height,
instead of changing the value of DPI, my question is: is it possible to
directly modify the value of DPI without changing the width and height?
—
Reply to this email directly, view it on GitHub
https://github.com/tdewolff/canvas/issues/201, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKOGHTAZPNWRFRPCBJOA4TWZ4KG3ANCNFSM6AAAAAAVLZWKG4
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

My code is similar to the following, it seems to be invalid, maybe there is a problem with the way I use it, please guide me, thank you

func TestKnife_ChangeDPI(t *testing.T) {
	genDir()
	dpi := 150.0
	file := "img/2023-03-01/17_00_09.png"
	imgSource, err := os.Open(file)
	if err != nil {
		panic(err)
	}
	defer imgSource.Close()
	img, err := canvas.NewPNGImage(imgSource)
	if err != nil {
		panic(err)
	}
	width := float64(img.Image.Bounds().Dx())
	height := float64(img.Image.Bounds().Dy())
	c := canvas.New(width, height)
	ctx := canvas.NewContext(c)
	ctx.FitImage(img, canvas.Rect{
		X: 0,
		Y: 0,
		W: width,
		H: height,
	}, canvas.ImageFill)

	// save
	filename := saveDir + "/" + time.Now().Format("15_04_05") + ".png"
	if err = renderers.Write(filename, c, canvas.DPI(dpi)); err != nil {
		panic(err)
	}

	fmt.Printf("生成图片成功: %s", filename)
}

My code is similar to the following, it seems to be invalid, maybe there is a problem with the way I use it, please guide me, thank you
func TestKnife_ChangeDPI(t *testing.T) {
	genDir()
	dpi := 150.0
	file := "img/2023-03-01/17_00_09.png"
	imgSource, err := os.Open(file)
	if err != nil {
		panic(err)
	}
	defer imgSource.Close()
	img, err := canvas.NewPNGImage(imgSource)
	if err != nil {
		panic(err)
	}
	width := float64(img.Image.Bounds().Dx())
	height := float64(img.Image.Bounds().Dy())
	c := canvas.New(width, height)
	ctx := canvas.NewContext(c)
	ctx.FitImage(img, canvas.Rect{
		X: 0,
		Y: 0,
		W: width,
		H: height,
	}, canvas.ImageFill)

	// save
	filename := saveDir + "/" + time.Now().Format("15_04_05") + ".png"
	if err = renderers.Write(filename, c, canvas.DPI(dpi)); err != nil {
		panic(err)
	}

	fmt.Printf("生成图片成功: %s", filename)
}

Here is the result of running with canvas

Its DPI has not changed (it has always been 72), but the width and height have been changed. What I need is the following effect. Can it be achieved?
DPI is modified from 72 to 150, and the width and height remain the same

I don't understand what you want.  An image within a vector
image does have a DPI, since vector images are described in real distances,
such as inches or millimeters. Second: changing an image's DPI within a
vector image always changes it's width and height. DPI stands for
dots-per-inch and canvas does not resample images, i.e. the amount of
pixels (width x height) always stays the same.
If you want to resample an image, you should resample before drawing. If
you want to set a PNG's metadata field for DPI (if that exists) you should
do so manually (that requires no change to the image data, only the
metadata).
El jue., 2 de mar. de 2023 02:58, overbeck @.***>
escribió:

My code is similar to the following, it seems to be invalid, maybe there
is a problem with the way I use it, please guide me, thank you
func TestKnife_ChangeDPI(t *testing.T) {
genDir()
dpi := 150.0
file := "img/2023-03-01/17_00_09.png"
imgSource, err := os.Open(file)
if err != nil {
  panic(err)

}
defer imgSource.Close()
img, err := canvas.NewPNGImage(imgSource)
if err != nil {
  panic(err)

}
width := float64(img.Image.Bounds().Dx())
height := float64(img.Image.Bounds().Dy())
c := canvas.New(width, height)
ctx := canvas.NewContext(c)
ctx.FitImage(img, canvas.Rect{
  X: 0,

  Y: 0,

  W: width,

  H: height,

}, canvas.ImageFill)
// save
filename := saveDir + "/" + time.Now().Format("15_04_05") + ".png"
if err = renderers.Write(filename, c, canvas.DPI(dpi)); err != nil {
  panic(err)

}
fmt.Printf("生成图片成功: %s", filename)
}
Here is the result of running with canvas
[image: image]
https://user-images.githubusercontent.com/44406033/222353527-8289a336-f100-4175-9753-3f53c6ee10db.png
Its DPI has not changed (it has always been 72), but the width and height
have been changed. What I need is the following effect. Can it be achieved?
DPI is modified from 72 to 150, and the width and height remain the same
[image: image]
https://user-images.githubusercontent.com/44406033/222353949-404653cd-5b86-46dc-b3ed-220fe1b57034.png
—
Reply to this email directly, view it on GitHub
https://github.com/tdewolff/canvas/issues/201#issuecomment-1451386831,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKOGHVQHFJQSAUQUSL5FG3W2BAIDANCNFSM6AAAAAAVLZWKG4
.
You are receiving this because you commented.Message ID:
@.***>

I don't understand what you want.  An image within a vector image does have a DPI, since vector images are described in real distances, such as inches or millimeters. Second: changing an image's DPI within a vector image always changes it's width and height. DPI stands for dots-per-inch and canvas does not resample images, i.e. the amount of pixels (width x height) always stays the same. If you want to resample an image, you should resample before drawing. If you want to set a PNG's metadata field for DPI (if that exists) you should do so manually (that requires no change to the image data, only the metadata). El jue., 2 de mar. de 2023 02:58, overbeck @.*> escribió:
…
My code is similar to the following, it seems to be invalid, maybe there is a problem with the way I use it, please guide me, thank you func TestKnife_ChangeDPI(t testing.T) { genDir() dpi := 150.0 file := "img/2023-03-01/17_00_09.png" imgSource, err := os.Open(file) if err != nil { panic(err) } defer imgSource.Close() img, err := canvas.NewPNGImage(imgSource) if err != nil { panic(err) } width := float64(img.Image.Bounds().Dx()) height := float64(img.Image.Bounds().Dy()) c := canvas.New(width, height) ctx := canvas.NewContext(c) ctx.FitImage(img, canvas.Rect{ X: 0, Y: 0, W: width, H: height, }, canvas.ImageFill) // save filename := saveDir + "/" + time.Now().Format("15_04_05") + ".png" if err = renderers.Write(filename, c, canvas.DPI(dpi)); err != nil { panic(err) } fmt.Printf("生成图片成功: %s", filename) } Here is the result of running with canvas [image: image] https://user-images.githubusercontent.com/44406033/222353527-8289a336-f100-4175-9753-3f53c6ee10db.png Its DPI has not changed (it has always been 72), but the width and height have been changed. What I need is the following effect. Can it be achieved? DPI is modified from 72 to 150, and the width and height remain the same [image: image] https://user-images.githubusercontent.com/44406033/222353949-404653cd-5b86-46dc-b3ed-220fe1b57034.png — Reply to this email directly, view it on GitHub <#201 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKOGHVQHFJQSAUQUSL5FG3W2BAIDANCNFSM6AAAAAAVLZWKG4 . You are receiving this because you commented.Message ID: @.>

I just want: DPI changed from 72 to 150, width and height remain the same
For example: image width and height: 100*100 DPI: 72
I want to turn the image into: 100*100 DPI: 150
In Python just need the following code
I don't know if it can be implemented in Canvas, or my usage is wrong, please advise me, thank you
import datetime

from PIL import ImageFile, Image

def image():
    ImageFile.LOAD_TRUNCATED_IMAGES = True
    Image.MAX_IMAGE_PIXELS = None

    # Known file width and height are: 100 * 100
    file = "17_00_09.png"
    fp = open(file, "rb")
    img = Image.open(fp)

    # Resize image size, the entry parameter is a tuple, the new image size
    # Keep the width and height of the original image
    img_new = img.resize((100, 100))

    new_file = datetime.datetime.now().strftime("%H_%M_%S") + "_dpi.png"
    # Modify the image dpi to 150
    img_new.save(new_file, dpi=(150, 150))
    print("Image processed: {}".format(new_file))

if __name__ == "__main__":
    image()

Open the new file in Photoshop and observe that the DPI has changed from 72 to 150 , width and height: 100*100 (keep the original value)

DPI is simply metadata if saved at all and is next to meaningless outside of printing. What is the reason you need a specific DPI value?
It's use in Photoshop and other image editors is as a convenience for printing specification. If an image has a DPI setting of 72 (the default) and your printing an image that is 600×600px it will print out as 8.3×8.3″ on the page. If the DPI is 150 and is still 600×600px it will print out as 4×4″ on the page. For any other usage I can think of DPI is completely ignored.

DPI is simply metadata if saved at all and is next to meaningless outside of printing. What is the reason you need a specific DPI value?
It's use in Photoshop and other image editors is as a convenience for printing specification. If an image has a DPI setting of 72 (the default) and your printing an image that is 600×600px it will print out as 8.3×8.3″ on the page. If the DPI is 150 and is still 600×600px it will print out as 4×4″ on the page. For any other usage I can think of DPI is completely ignored.

I need to draw a picture and use it for printing purposes
DPI should be noted in the program that draws images

Ah, it is for printing. Yes, it absolutely should be part of the program. It just sounded like you may have been trying to do something with DPI that wasn't print based. Thanks for the clarification.  👼

Please note that a PNG image (or any raster image like TIFF, GIF, JPEG, etc.) has no DPI setting.  Canvas uses vector operations to define the content of an image, and then rasterizes using a DPI to define the scale into pixels (such as PNG). I understand the PNGs have a metadata field that defines the scale as which it should be printed, but this has nothing to do with the DPI at which Canvas rasterizes!
In other words, redrawing the image in Canvas using a different DPI only changes the scale of the image, it does not set the DPI metadata field! Actually, the image writer library (image/png) that Go provides to write PNG files does not support setting this field from what I see. Anyways, I understand the confusion between DPI of Canvas and DPI of the metadata field of PNGs, but these are different things! What you need is not Canvas, you need to load your PNG and change the DPI metadata field, and then save it. I'm not sure if someone has written a library to set that...
Does this make sense?

Closing as this should be set in the image writer library, not in Canvas


Create edu.xyz

@BOGDAN-DJ This domain does not belong to a university, nor a college. I have to close the request without merging. Please ensure you are familiar with this repository rules located at the bottom of this page before submitting a new request. Thank you for the understanding!


docker: use exec form for CMD directive

the CMD start is equal with CMD sh -c start, so pid of graph node is not 1

@lutter Hi, can you please take a look?

@islishude there seem to be CI issues, could you please rebase the branch?


feat: enable mypy pre-commit check

Description
This PR enables mypy typing checks for pre-commit.

[ ] You added unit testing when necessary
[ ] You added functional testing when necessary

I see a bunch of ignore and also a lot of force casting to string and I wonder if it does not introduce complexity for what it brings (the str(...) part)

especially if the code operates with AWS API, typing makes developer's life much better I guess 😄


Disable Today

Hi,
It seems impossible to prevent an user to pick today.
With these settings :
showTodayButton : false, disabledDates : [     new Date() // today ],
the default date (when open) is today and the user can use the Select button to pick today.
I tried with inputDate, but it lock to only the inputed date.

You can use the option inputDate and set it to another day apart from today.
This is the option that sets the default date.

Yes, this was the solution. To be more specific, it didn't work the same depending the Date format I used.
It worked with :
inputDate: new Date(2017, 2, 2)  but not using a Date from moment.js


Use double quotes in code

Mixed style is used everywhere, so it is time to use only one.

This was intentionally not set.
While I value enforcing conventions for the sake of readability and consistency, I don't feel like mixing single and double quotes makes an significant impact in readability to justify enforcing a style an incur in the cost of introducing extra friction to regular developers and new contributors.


Fixed recovering from old data removal error

bug: sometimes when joining new network run old network data was not removed and caused issues


[BUG]Matter-window-covering update operational state incorrectly

I found that the operational state was displayed incorrectly when the window covering stopped. It always showed "opening" or "closing" unless I refreshed it manually. Maybe there is something that needs to be fixed in the functions current_pos_handler and current_status_handler.
In this case, it would cause the bug:
When the blind starts to move, the blind only updates the operational status, setting IS_MOVING to true and EVENT_STATE to OPERATIONAL_STATE_EVENT. Until the blind stops, the operational status and current position update simultaneously (the message packet is added at the end). The message of the current position will be processed first. Because EVENT_STATE is OPERATIONAL_STATE_EVENT but IS_MOVING is true, it will set EVENT_STATE to NO_EVENT. Thus, when it processes the message of operational status, the code will not execute device::emit_event_for_endpoint to update the status, because EVENT_STATE is NO_EVENT. Finally, it just sets IS_MOVING to false and shows "opening" or "closing".
If  just replace line 159 with device:set_field(EVENT_STATE, WindowCoveringEventEnum.CURRENT_POSITION_EVENT), could it cause other problems?
Looking forward to your reply. Thanks.
The message packet of the matter subscription response showed the following. It always reports the current position first:

[1718162472.140788][34296:34298]` CHIP:EM: >>> [E:4155r S:20551 M:67760107] (S) Msg RX from 1:0000000000000001 [5F03] --- Type 0001:05 (IM:ReportData)
[1718162472.140841][34296:34298] CHIP:EM: Handling via exchange: 4155r, Delegate: 0xaaaabdd5ba30
[1718162472.140986][34296:34298] CHIP:DMG: ReportDataMessage =
[1718162472.141022][34296:34298] CHIP:DMG: {
[1718162472.141053][34296:34298] CHIP:DMG: 	SubscriptionId = 0xc267cab4,
[1718162472.141085][34296:34298] CHIP:DMG: 	AttributeReportIBs =
[1718162472.141127][34296:34298] CHIP:DMG: 	[
[1718162472.141159][34296:34298] CHIP:DMG: 		AttributeReportIB =
[1718162472.141208][34296:34298] CHIP:DMG: 		{
[1718162472.141242][34296:34298] CHIP:DMG: 			AttributeDataIB =
[1718162472.141279][34296:34298] CHIP:DMG: 			{
[1718162472.141319][34296:34298] CHIP:DMG: 				DataVersion = 0x79ff2eea,
[1718162472.141358][34296:34298] CHIP:DMG: 				AttributePathIB =
[1718162472.141400][34296:34298] CHIP:DMG: 				{
[1718162472.141441][34296:34298] CHIP:DMG: 					Endpoint = 0x1,
[1718162472.141486][34296:34298] CHIP:DMG: 					Cluster = 0x102,
[1718162472.141530][34296:34298] CHIP:DMG: 					Attribute = 0x0000_000E,
[1718162472.141573][34296:34298] CHIP:DMG: 				}
[1718162472.141617][34296:34298] CHIP:DMG:
[1718162472.141660][34296:34298] CHIP:DMG: 				Data = 0,
[1718162472.141699][34296:34298] CHIP:DMG: 			},
[1718162472.141742][34296:34298] CHIP:DMG:
[1718162472.141776][34296:34298] CHIP:DMG: 		},
[1718162472.141824][34296:34298] CHIP:DMG:
[1718162472.141855][34296:34298] CHIP:DMG: 		AttributeReportIB =
[1718162472.141897][34296:34298] CHIP:DMG: 		{
[1718162472.141930][34296:34298] CHIP:DMG: 			AttributeDataIB =
[1718162472.141999][34296:34298] CHIP:DMG: 			{
[1718162472.142039][34296:34298] CHIP:DMG: 				DataVersion = 0x79ff2eea,
[1718162472.142079][34296:34298] CHIP:DMG: 				AttributePathIB =
[1718162472.142120][34296:34298] CHIP:DMG: 				{
[1718162472.142162][34296:34298] CHIP:DMG: 					Endpoint = 0x1,
[1718162472.142251][34296:34298] CHIP:DMG: 					Attribute = 0x0000_000A,
[1718162472.142293][34296:34298] CHIP:DMG: 				}
[1718162472.142336][34296:34298] CHIP:DMG:
[1718162472.142378][34296:34298] CHIP:DMG: 				Data = 0,
[1718162472.142417][34296:34298] CHIP:DMG: 			},
[1718162472.142459][34296:34298] CHIP:DMG:
[1718162472.142493][34296:34298] CHIP:DMG: 		},
[1718162472.142533][34296:34298] CHIP:DMG:
[1718162472.142563][34296:34298] CHIP:DMG: 	],
[1718162472.142640][34296:34298] CHIP:DMG: 	InteractionModelRevision = 1
[1718162472.142669][34296:34298] CHIP:DMG: }
[1718162472.142810][34296:34298] CHIP:TOO: Endpoint: 1 Cluster: 0x0000_0102 Attribute 0x0000_000E DataVersion:<PHONE_NUMBER>
[1718162472.142861][34296:34298] CHIP:TOO:   CurrentPositionLiftPercent100ths: 0
[1718162472.142948][34296:34298] CHIP:TOO: Endpoint: 1 Cluster: 0x0000_0102 Attribute 0x0000_000A DataVersion:<PHONE_NUMBER>
[1718162472.142985][34296:34298] CHIP:TOO:   OperationalStatus: 0

I had

a similar problem where when the motor was running to the 0% or 100% position, the motor percentage slider position was successfully updated, but the label was still opening or closing.


Array with references to definitions requires array of pointers to be returned

Problem statement
When defining responses as an array whose items are refs to definitions the generated code is an array of pointers to model structs.
This seems reasonable, but it forces me to convert my array of structs to an array of pointers to those structs.
Is there any way to avoid making that conversion?
Thanks for your help!
I really appreciate your work in this library! :clap:
Swagger specification
      responses:
        200:
          schema:
            type: array
            items:
              $ref: "#/definitions/digitalizacion"

Environment
swagger version: 0.12.0
go version: 1.9
OS: linux

Is there any updates or workarounds? Now it's possible to make array of structs only if not using ref.


Infinite downloads

Dev Home version
0.401.226.0
Windows build number
10.0.23536.1000
Other software
OS Build Version: 10.0.23536.1000.amd64fre.ni_prerelease.230826-1546
.NET Version: .NET 6.0.20
Steps to reproduce the bug
estaba descargando aplicaciones y lleva ya mucho tiempo descargandolas
Expected result
una descarga normal
Actual result
sigo esperando la descarga
Included System Information
Physical Memory: 31.85GB (25.57GB free)
Processor Architecture: x64
Included Extensions Information
Extensions:
Microsoft.Windows.DevHome_0.401.226.0_x64__8wekyb3d8bbwe
Microsoft.Windows.DevHomeGitHubExtension_0.400.226.0_x64__8wekyb3d8bbwe

Bing Translate:
Infinite downloads
I was downloading apps and has been downloading them for a long time
a normal download
I'm still waiting for the download

@Chucho0, thank you for opening this issue and sorry for the delayed response. This issue was resolved in this PR:

https://github.com/microsoft/devhome/pull/2089

Added recovery when RPC exceptions occur.

And further enhancements have also been applied in the PRs below:

https://github.com/microsoft/devhome/pull/3134

Elevated process now uses the same WinGet logic as Dev Home's main process (enhanced implementation overall + recovery)

https://github.com/microsoft/devhome/pull/3307

Mitigated an issue that occurs when Dev Home's elevated process attempts to install multiple packages at the same time.

Would it possible to help validate that this issue no longer repros on latest Dev Home?


Ellipse MakerJS

I draw ellipse and have this problem (smaller ellipses inside are not affected). can anyone please help me solve it.

My code:
`var ellipse = new makerjs.models.Ellipse(0, 0, 200, 500);
model = {
models: {
ellipse: ellipse
}
}
dxf = makerjs.exporter.toDXF(model);
`

Overview Image

Overview image


concrete_descendents clobbers descendents

ALL software version info
param v1.12.2, Colab notebook
Description of expected behavior and the observed behavior
Simply put, classes with the same name get clobbered in get_concrete_descendents().
I came across this bug when trying to run some unit tests with duplicated code (and class definitions) were in different files.
Complete, minimal, self-contained example code that reproduces the issue
import sys
import param

from param.parameterized import descendents

print(param.__version__)

class A(object):
  pass

class B(A):
  pass

b = B

print(b in param.concrete_descendents(A).values())

class B(A):
  pass

print(b in param.concrete_descendents(A).values(), b in descendents(A))
print(B in param.concrete_descendents(A).values(), B in descendents(A))
print(issubclass(b, A), issubclass(b, A))

Colab

I can reproduce the problem. And looking at the code, it is easy to see why it happens, as there is no uniqueness to the name of a class. The correct fix would properly be to use the class as key and name as value, though this is a breaking change.
Maybe the best thing is to test if there are duplicate class names in param.concrete_descendents and raise an exception if there are. This could raise exceptions in existing code, but at least it will indicate that something is gone wrong.
https://github.com/holoviz/param/blob/76c7026346b73951dcf4308b6302f0742b0e83e7/param/__init__.py#L1090-L1101

Hm yes the data structure used should probably not have been a dictionary but a list of tuple/namedtuple?
@philippjfr param.concrete_descendents is used quite a lot in Panel. What change would you be willing to accept, if any, to fix the reported issue?

Deprecating concrete_descendents with a warning and having a new function with a better data structure could be a solution.
It would allow people to shift to the new function and not break anyone using the current implementation.

Yep that's another valid way to approach that 👍


React Native with Genymotion causes a bug

[ ] Review the documentation: https://facebook.github.io/react-native
[ ] Search for existing issues: https://github.com/facebook/react-native/issues
[ ] Use the latest React Native release: https://github.com/facebook/react-native/releases

Environment
Environment:
OS: Windows 10
Node: 8.11.3
Yarn: 1.7.0
npm: 5.6.0
Watchman: Not Found
Xcode: N/A
Android Studio: Version  <IP_ADDRESS> AI-173.4819257
Packages: (wanted => installed)
react: 16.3.1 => 16.3.1
react-native: ~0.55.2 => 0.55.4
Description
First, I created new app with create-react-native-app and tested it with expo on my android phone. And that worked fine. But I wanted to test it on my windows. So I decided to install Genymotion and Genymotion came with Virtual Box. After I installed Genymotion with Virtual Box everything did not work properly. I think there are too many issues with it. At least I had to set REACT_NATIVE_PACKAGER_HOSTNAME with proper ip address from ipconfig command. And I still don't know what to fix next. Is there any easier way to test react native app on my windows?
Reproducible Demo
Let us know how to reproduce the issue. Include a code sample, share a project, or     share an app that reproduces the issue using https://snack.expo.io/. Please follow the guidelines for providing a MCVE: https://stackoverflow.com/help/mcve

@facebook-github-bot stack-overflow


Helm Chart: Pulling it all together

Hi,
Thought of creating a new issue to discuss how we can pull all this work together in time for GCC, for integration with CloudMan. The original issues have become a bit stale now but for reference: https://github.com/galaxyproject/galaxy-kubernetes/issues/1, https://github.com/galaxyproject/galaxy-kubernetes/issues/2 and https://github.com/galaxyproject/galaxy-kubernetes/issues/4.
Have taken a stab at assembling some tasks/questions that (I think) remain, but may be off base on a few, so please correct/edit.

Merge final chart into master and remove simple chart etc. so we have a final set of charts to work on.
Currently, the Docker images come from various sources, including pcm32 and phnmnl registries. Consolidate all docker images for the helm charts into galaxyproject's registry.
Currently, the chart is using a custom Postgres container+service definitions. Transition to the official Postgres helm chart? This seems safer since it's likely to get HA modes and better health checks and maintenance as we go along.
Move proftpd to separate, self-contained chart?
New chart for Pulsar? (probably not for this iteration though).
Have a standardized mechanism for exposing all Galaxy config settings.
Rolling upgrade support for the Galaxy charts. Do upgrades work smoothly at present? We seem to be needing helm post-install/upgrade jobs for database migration.
Galaxy pod scale up/down support. Can we also have multiple Galaxy pods running for a zero-downtime rolling upgrade of Galaxy?
Add an option to switch between the HTCondor and Kubernetes Job runners. (that is, integrate the work contributed by Microsoft/AnswerALS), with possible support for future options (e.g. Pulsar, Slurm).
Perhaps have a separate chart for Galaxy alone, and an umbrella chart that pulls everything together?

ping: @abhi2cool @afgane @bgruening @jmchilton @natefoo @rc-ms @pcm32

Thanks for this update Nuwan.  Great suggestions.  I'm going to figure out if there's a way to fund this work in time for GCC. I'll keep you posted on progress.

Hi everyone,
First, let me tell you that both PhenoMeNal Public instance and Publicdev instances have been running with the new helm chart and community images based containers (plus our init flavouring) since around February or so. I also managed to make a deployment on a completely Google-native Kubernetes cluster (as opposed to kubenow or kubespray deployed on GCP).
Let me go point by point:

Merge final chart into master and remove simple chart etc. so we have a final set of charts to work on.
I was about to do this, but my last trials with Galaxy community images that I haven't generated did go well, those containers need to be built with special environment variables being past. I will add to the codebase a bash script to build the community images with adequate settings so that they can be used. I can do a merge to develop now, and possibly a release (to master) when that documentation and script has been tested by others.

Currently, the Docker images come from various sources, including pcm32 and phnmnl registries. Consolidate all docker images for the helm charts into galaxyproject's registry.
The script that I mention sorts that out.

Currently, the chart is using a custom Postgres container+service definitions. Transition to the official Postgres helm chart? This seems safer since it's likely to get HA modes and better health checks and maintenance as we go along.
This has a reason, the container includes an init script so that the schema is loaded through a postgres dump load instead of the very slow Galaxy migration scripts. We could move to a Postgres chart, but this is not currently urgent and since man power is limited, I would leave this for later. Pull requests are welcomed of course. Now, more than moving to the default postgres chart (which AFAIK doesn't support HA), I have been toying with the idea of moving to an HA postgres setup like the one provided by stolon. My current issue with it is that it uses shared filesystem for the different instances, which defies a bit the purpose of performance.

Move proftpd to separate, self-contained chart?
I would give this even less priority. This makes sense when you either have someone else building the chart (such as in the postgres case) or you have a separate use for it, if neither is the case, I fail to see a good justification for this.

New chart for Pulsar? (probably not for this iteration though).
What do you mean exactly here?

Have a standardized mechanism for exposing all Galaxy config settings.
Yes, I have thought of these but haven't implemented. There are some construct on Helm that we could use to iterate on values within a specific scope so that they are exposed automatically as env variables that Galaxy can read.

Rolling upgrade support for the Galaxy charts. Do upgrades work smoothly at present? We seem to be needing helm post-install/upgrade jobs for database migration.
If there are no schema changes in the DB, this should work out of the box, provided that you have a container of course for the following version. I think that there is a env var that you can set for Galaxy to do auto-migrations, with that it could work as it is. Having said that, there are sometimes breaking changes in Galaxy in terms of how it deals with its config, like the pass from .ini to .yaml for config or the change in the main log file, which required adjustment to the chart at some point.

Galaxy pod scale up/down support. Can we also have multiple Galaxy pods running for a zero-downtime rolling upgrade of Galaxy?
Given the current setup of the Galaxy community images, this is currently infeasible (the pod scale). We need to move on the community images to separate containers for web and handlers (or the ability to run them with these roles) and avoiding the supevisor layer, so that the container orchestrator becomes the process manager. Then you should be easily able to scale on the web servers or the handlers as needed (as they run on individual pods). Until this is all secluded on the same container, this idea won't fly. What should happen is that the handlers are all left behind a Kubernetes service, instead of web/master having to be aware of each handler individually. In understand that there is support for this now on 18.01. My move of the charts to be 18.01 compatible was a bit hampered by some missing wheels that we needed for some modules that we use in PhenoMeNal, but I expect to resume that soon.

This ^^^ is in my view the most pressing thing to do from the architecture point of view, and if you have man power, I would put it here, in the form of pull requests to the docker-galaxy-stable/compose repo. This would truly make Galaxy completely scalable within Kubernetes (currently is scalable in terms of dispatching jobs, but if the Galaxy instance reaches the limits of the amount of jobs that it can track/manage, we are in trouble).

Add an option to switch between the HTCondor and Kubernetes Job runners. (that is, integrate the work contributed by Microsoft/AnswerALS), with possible support for future options (e.g. Pulsar, Slurm).
The option is currently there, it is configurable through an environment variable. However I have personally no use for it and haven't tested it nor I haven't received any PRs with people integrating the HTCondor containers in the helm chart.

Perhaps have a separate chart for Galaxy alone, and an umbrella chart that pulls everything together?
I would suggest to leave this for later, I think it is over-engineering it a bit and we have other more pressing things to do with limited man power.

@rc-ms Thanks RC, that'd be awesome, even if it's after GCC. For GCC, we may be able to get by just by limiting scope.
@pcm32 Thanks for the detailed update. A few comments.

I can do a merge to develop now, and possibly a release (to master) when that documentation and script has been tested by others.

This would be great. We can start issuing PRs on a common code base afterwards.

Pull requests are welcomed of course. Now, more than moving to the default postgres chart (which AFAIK doesn't support HA), I have been toying with the idea of moving to an HA postgres setup like the one provided by stolon.

I'm happy to give this a go. I've seen stolon, but as you said, no postgres chart with HA support has made it to the official repo yet. Still, we'll gain some advantages by going with the official chart for now to benefit from any improvements to liveness/readiness probes or any other community contributed improvements.

I would give this even less priority. This makes sense when you either have someone else building the chart (such as in the postgres case) or you have a separate use for it, if neither is the case, I fail to see a good justification for this.

I think that makes sense for this iteration. In future, there will likely be modularity benefits and eventually, the possibility of replacement with an official, third-party maintained chart. We could use a kind of "monorepo" till that happens.

New chart for Pulsar? (probably not for this iteration though).

What do you mean exactly here?

Having a separate chart for Galaxy-Pulsar has more to do with future applications, and we can/should drop it for this immediate iteration.

Yes, I have thought of these but haven't implemented. There are some construct on Helm that we could use to iterate on values within a specific scope so that they are exposed automatically as env variables that Galaxy can read.

This is an important piece, so we'd be happy to pitch in and make this happen. However, I guess the challenge is - which mechanism is best? We've discussed this issue before, but I don't think we ever settled on a final mechanism? A few possibilities that come to mind:

Use a configmap and map each value as an envvar to the galaxy container, prefixed by GALAXY_CONFIG, which the Galaxy stable container recognizes.

Use a configmap to map galaxy.ini and indeed, any other ini files we want, directly as volumes. That way, we can just map them into the appropriate location. However, this may not be compatible with the current operation of Docker-Galaxy-stable?

Install confd in docker-galaxy-stable?

Given the current setup of the Galaxy community images, this is currently infeasible (the pod scale). We need to move on the community images to separate containers for web and handlers (or the ability to run them with these roles) and avoiding the supevisor layer, so that the container orchestrator becomes the process manager.

If we simply duplicate the pod, and point it at the same database, will that not work?

Closing because most of the work is done.


Step Functions: Support mutable variables outside of For and Map scopes.

new StepFunction(stack, 'sfn', async () => {
   let a = "";
   for(const x in [1,2,3]) {
      a = `${a}${x}`
   }
   return a;
});

Partially implemented in #326 - for supports mutating variables.


Emit a completion event from OnboardingPage with all the captured data

@voltrevo @kautukkundan Is this issue still needed?

I think we can close this one now.


cli: unable to run action defined in 3rd level of a repo directory

An action defined in user/repo/dir@master runs but if an action is defined as in user/repo/dir/subdir@master fails to run.
Popper is only able to run actions from up to the 2nd level directory and not more than that.

this will be subsumed by #623


chore(pages): component content cleanup

Related Ticket(s)
https://github.com/carbon-design-system/carbon-for-ibm-dotcom/issues/6941
https://github.com/carbon-design-system/carbon-for-ibm-dotcom/issues/7041
Description
Waiting for #114 to merge
This updates all test page content to match current React Storybook and Codesandbox examples. Also changes example page names (URLs) to be more generic.
Note: Many component style fixes are in v1.25.0 and will not be reflected until package.json is updated. Current application is still using v1.24.0.
Changelog
New

FPO images

Changed

update all page content to match latest React Storybook/Codesandbox
learn.js, solutions.js, services.js ⭢ example-page-a.js, example-page-b.js, example-page-c.js respectively

Removed

node-fetch

Thanks @kennylam for the cleanup!
Not sure if I see updated e2e tests, can you update so they point to the new pages?

Not sure if I see updated e2e tests, can you update so they point to the new pages?

@jeffchew Oh good call, I didn't realize they were added to this app.

@kennylam there's still some missing imagery, is that supposed to be covered in this PR?

@kennylam there's still some missing imagery, is that supposed to be covered in this PR?

Strange, none of the images are coming through in the deploy preview. But they're all loading locally no problems.


Is there any pre-trained model for source:de-target:en language pair

Hi,
I've seen that there are a couple of pre-trained  NMT models exist in the repository for (source:en, target:de) language pair. However, I wonder is there any pre-trained model for the reverse direction, i.e. from source:de to target:en ?

We haven't released any De-En models yet. @edunov, do you have any De-En models that would make sense to release?

Hi, yes we have just released WMT'19 winning submissions that include De->En models, please check here: https://github.com/pytorch/fairseq/tree/master/examples/wmt19

Could you also add smaller de en models, such as ones from WMT 2018?

In case this is of use to anyone, I trained a de-en model based following the Scaling NMT readme but reversing the translation direction. The model is not particularly great with a sacrebleu score of 31.0 on the newstest2014 test set, but it served my purposes sufficiently.
In case anyone wants to use it, the download link/instructions are available at the repo's readme


The use of runtime associated collection fails in workflow service

Nighlty nugets @ https://xpandnugetserver.azurewebsites.net and nightly installers @ https://dev.azure.com/eXpandDevOps/eXpandFramework/_build?definitionId=3
It might be a good idea to search the archived forums before posting.\
Issues are autoclosed after a period of inactivity, If you are the owner consider closing them on resolve.
Hi there
We are using eXpand in a Win application and a workflow service. We have created runtime members using the win application. This works fine. We have created the following AssociatedCollection:
<RuntimeNonPersistentMember Name="CustomTariffs" CollectionType="Persistent.Custom.CustomType002" AssociatedMember="CustomTransportRegion"  CreatedAtDesignTime="False" AssociationName="CustomType001-CustomTariffs" IsNewNode="True" />
The win application can use this member with no problems.
The workflow service fails to create this runtime member. And it states that the CollectionType and or AssociatedMember are null
We did some research and found that the model that is loaded in the win application differs from the model of the service application.
The win application loads the given xml as a OneToManyCollection member. The service however loads this xml as a RuntimeNonPersistentMember. So again: The xml is the same but it is loaded differently in memory.
When the runtime member builder tries to create the member when starting the service, the cast of the modelMemberEx fails to the OneToManyCollection interface.
Can you help us resolve this problem? Can you give us advise to solve this? If you can't do this without a sample project we will try to create one next week.
Thanks,
Jacob

Hi Jacob,
If not much trouble amd given the fact that there is a lot of work to support the new GitHub workflow I would appreciate a sample. It will for sure produce a fast response from me.
Thanks for the understanding

Hi Tolis
I have some trouble to get this to work. I have used the workflow demo, but I think the workflow service doos not load runtime members from the model. When I create a new workflow service project, it creates a new application model. This model stays empty. How can I get this to work to test this? Or what can be the cause of my original question?  Sorry for the additional questions
Thanks,
Jacob

still need help? I see you closed the issue

Hi Tolis
I accedentily closed the issue. But I still need help. I tried to create a sample project, but eXpand doos not seem to support my situation out of the box.
We try to do all the commits that you do on our own code. We do this because we have done some customizations. One of this customizations is that we have modified the modeldiffernece module so that we can specify a fixed application name so that we can run serveral applications on the sampe application model.
We have done this for a workflow service project and a windows project. This project use the same application model.
The problem that we experiencing now is that the windows application can handle runtime collection members and the workflow service can't handle them. I tried restoring code of the modeldifference modules to earlier versions but this did not help. It seems to be related to the Xpand.Persistent.Base or Xpand.Persistent.BaseImpl projects.
It seems that when the Application.Model property is used for the first time that it is different for the win and service application.
Can you please help me with finding the cause of this? So why doos the win application translate the xml of a modelaspect different from the service application?
If you can provide me with some tips it would help me alot.
Thanks,
Jacob

where u store the model? do u use Modeldifference module? how you create your members

Hi Tolis
Model is stored in db. I do use the modeldiffernece module. The members are created in the win application in model designer
I tried making a sample project. Attached you will find a project containing your workflow demo. I added a custom collection member on the Issue type. This is a collection of Tasks. The member is called CustomTasks. I alse created a workflow definition that updates every issue with a subject with the value '2' It will update the subject with the count of the customtasks collection of that issue.  (workflow definition is called test5).
This workflow fails to execute because it cannot find the customTasks collection.
I did not get this to work so that the workflow service loads the runtime members.
So the difference between our product and this demo is that in our product the service runs in a seperate project (using the xpandworkflowapplication). Our product uses complex strategy and standard authentication. And we modified the modeldiffernece module so that different projects can use the same applicaiton model (using a fixed application name instead of the name of the running executable).
Can you get this attachment to work (the workflow definition test5). Or can you help me with my Original question using the additional info I provided?
P.s. database is included in the zip
Thanks,
Jacob
WorkflowDemo.zip

thanks for the sample, I tested and your workflow works fine for me. see my screencast.
https://www.screencast.com/t/WgDt3GXPEsm

Hi Tolis
It will work in the WF designer. But it won't work using the workflow service. Please watch my attached screencast
Tanks,
Jacob
Gravar #2.zip

it looks lie the ModelDifferenceModule gets confused because the model is bound to WorkflowDemoWindowsFormsApplication. For now you can force loading by

In you server load WorkflowDemoWindowsFormsModule instead of WorkflowDemoModule (Program.cs line 98)
add the next code to WinModule.cs

        public override void Setup(XafApplication application){
            base.Setup(application: application);
            if (application is XpandServerApplication serverApplication)
                serverApplication.SetupComplete += (sender, args) => {
                    XafTypesInfo.Instance.RegisterEntity(typeof(XPObjectType));
                    using (var objectSpace = application.CreateObjectSpace()){
                        var modelDifferenceObjects = objectSpace.GetObjectsQuery<ModelDifferenceObject>().Where(o => o.DifferenceType == DifferenceType.Model);
                        foreach (var modelDifferenceObject in modelDifferenceObjects){
                            modelDifferenceObject.GetModel((ModelApplicationBase)application.Model);
                        }
                    }
                    application.Modules.OfType<ModelDifferenceBaseModule>().First().LoadModels();

                };
        }

this is a temp solution and it wont be reuired from the next minor. The module will guide you by throwing exceptions.

Installer lab build <IP_ADDRESS> includes commit ModelDifferenceModule: preconfigure loading models list on server scenarios #234,ModelDifferenceModule: preconfigure loading models list on server scenarios #234 that relate to this task. Please test if it addresses the problem. If you use nuget add our NugetServer as a nuget package source in VS

Hi Tolis
The WorkflowDemo works with your changes. My own project however doos not. I realise that I need to research this further by myself.
Thanks for your time and effort.
Jacob

I suggest u have the symbols loaded and check why the MDO module does not load the differences.

feel free to ask if you have concrete questions

Hi Tolis
The workflow service in our product loads the differences fine, but only runtime associated collections are interpretated differently in the service application. This results in an exception (see Original post) when starting the service. Other runtime members are working fine.
What can be the cause of this?
Thanks,
Jacob

I do not make sense, in my tests I found that the service did not load the model at all because the model is bound to the WinApplication and not the server application. I posted a workaround already can you verify that your model is loaded from the service?
is this line called for the model you define the members?
modelDifferenceObject.GetModel((ModelApplicationBase)application.Model);

The service application puts it in a RuntimeNonPersistentMember object

you can enable symbols and step into
https://github.com/eXpandFramework/eXpand/blob/84e8e3e037bae7a4a1460c6657d7bb40a3fb49ca/Xpand/Xpand.Persistent/Xpand.Persistent.Base/RuntimeMembers/RuntimeMemberBuilder.cs#L35-L36

Hi Tolis
When I run the Win application my code gets in the if on line 176. When I run the service the code doos not get in this if. I gets to end of this function and crashes on line 188 in the CreateCustomMembers function, because some parameters are empty.
I think this is caused by the fact that the service application puts my runtime collection members in other objects when it translates the xml. The service puts them in RuntimeNonPersistentMember objects and the win application puts them in OneToManyCollection objects.
I'm looking for the cause of this. Why is the service putting the xml in onther objects?
Thanks,
Jacob

I cannot understand from the info you posted however you mentioned

We are using eXpand in a Win application and a workflow service. We have created runtime members using the win application. This works fine. We have created the following AssociatedCollection:
<RuntimeNonPersistentMember Name="CustomTariffs" CollectionType="Persistent.Custom.CustomType002" AssociatedMember="CustomTransportRegion" CreatedAtDesignTime="False" AssociationName="CustomType001-CustomTariffs" IsNewNode="True" />

which does not look good to me, have a look how an associated collection should look like.
https://github.com/eXpandFramework/eXpand/blob/84e8e3e037bae7a4a1460c6657d7bb40a3fb49ca/Demos/Modules/SystemTester/SystemTester.Module/FunctionalTests/RunTimeMembers/RunTimeMembers.xafml#L15

Hi Tolis
The Xml you posted is not looking the same like the xml that is generated by the sample project. So I think the problem is somewhere else. I will look into this further
Thanks,
Jacob

Hi Tolis
I solved the problem adding
if (typeof(IModelMember).IsAssignableFrom(parameters.NodeType))             {                 if (parameters.Values.ContainsKey("AssociatedMember"))                     parameters.NodeType = typeof(IModelMemberOneToManyCollection);             }
to the ConvertXml method in XpandModuleBase.
Do you know why this is working for you without this code in there and why it is not working for me?
Thanks,
Jacob


Function to decompose URI to an object

Is your feature request related to a problem? Please describe.
Various RPs in their properties expose url of the service. however they do it inconsistently - sometimes with trailing slash, sometimes without it, sometimes it's only a domain name and proper protocol needs to be added.
On the other hand, some resources expect full url, and some just a domain name.
example:
Storage account exposes primary endpoints in form of a full url (https://name.blob.core.windows.net/) but front door backend requires only a domain name and protocol is set via other property. So to extract domain name only, we need to use substring from 9th character and limit to endpoint length minus 9 (8 for https and 1 for trailing slash)
Describe the solution you'd like
I'd like to have a runtime function that will take a string and make it an URI object (https://learn.microsoft.com/en-us/dotnet/api/system.uri?view=net-7.0) perhaps not with all properties but the most common one:  scheme, host, port, path, query).
also, but not sure if possible, calling string function on the object would give back the absolute URI or have a function that could build one by providing separately scheme, host, port, path and query.

I'd really like this function too.


Deferred Save of Index

This change gives the user an option when creating a LocalIndex to not write the changes to file on every update.
The new option named deferredSave when enabled will not write changes to the index.json file on every update. The user must manually call commitChanges() to write the data to the index.json file.

@Stevenic thanks for the great project, I love the simplicity. I am using it for my AI Mod for The Sims 4. I want to keep all the data in memory until the player has saved their game, then write the changes to a file.

@guspuffygit can I get you merge your changes into main? Sorry I missed this but I'll get it approved an merged this week

+1


305 maildir expansion

This should fix https://github.com/soywod/himalaya/issues/305
Please note that I'm pretty overwhelmed by the new code so I was pretty unsure how to test it.
I'd be happy if some of you guys could quickly try this branch out and give a submission :)

wait, this PR wants to merge with master. Whoops.

Oh, I didn't knew that, haha. Thanks for the information!


[feature request] improve sort / filter ability on ls commands

i'd really like to have a way to sort / filter the list there, I have like 5 domains and 10 apps and it's already getting very tedious to understand what's going on.
i suggest having something along the following lines, which is extremely valuable imho:

now alias ls --sort url would sort by url column instead of age column
now alias ls --sort source would sort by source column instead of age column
now alias ls --sort domain would sort by domain (currently doesn't have its own column)
now ls --sort url would sort by url column instead of age column
now ls --sort app would sort by app column instead of age column
now ls --sort domain would sort by domain (currently doesn't have its own column)

additionally you could also have a --filter flag which would do exactly the same but filtering by the passed value (you already have it partly in place for now ls where you can do now ls <appname>). i don't think there is a need to also have a sort direction though.

I would do a little research in what should be the default sorting, in terms of UX.
For teams the output of now ls can be lots of lines, and a sorting by app name should be better. For individuals it can be interested to sort by age, like today. Just giving my input...
Let me see if I can do at least now ls --sort in a PR,

For this purpose, we deployed support for metadata! 😊


Test-cafe browser GUI is not launching when we triggered tests from Jenkins and console messages are displaying properly.

Hi,
Today we have created jenkins job to trigger TestCafe scripts, we triggered the TestCafe scripts. But Browser GUI not all displayed while scripts are running, but we can console messages.
Note: Jenkins has permission to interact with my desktop.
Hence could you please suggest what would be the solution here. Do we require any additional plugin to trigger TestCafe scripts in Jenkins?
Jenkins version is: 2.164.1

Thank you for your inquiry. This issue looks like a question that would be best asked on StackOverflow - an amazing platform for users to ask and answer questions. Also, refer to the Enable Jenkins to Start the Browser UI article to find information on how to configure Jenkins.
We try to keep the GitHub issues tracker for bugs and feature requests only (see also: Contributing). If you think that this issue is a bug, feel free to open a new issue, but please make sure to follow the bug template. Thank you in advance.


Tick the line to display the avatar button, but the avatar is not displayed

GitLens Version:11.2.1
Git Version: git version 2.27.0.windows.1

VSCode Version: 1.53.2
OS Version:

Steps to Reproduce:
Tick the line to display the avatar button, but the avatar is not displayed. I had it before. I don't know why it suddenly disappeared. I tried to uninstall and reload it, and install other versions, but it still doesn't display

Is your avatar the only one that doesn't show up, or does it affect others as well? Is this repo on GitHub? If so, did you allow GitLens access to GitHub?

I don't know what happened. Before, the avatar can be displayed. Plug-in functions can be used, but the avatar is not displayed. It's very uncomfortable @eamodio

If you visit https://www.gravatar.com/avatar/00000000000000000000000000000000.jpg?s=16&d=robohash do you see a small avatar?
Can you try to see if the issue is still reproducible in the insiders edition?
You can install the insiders edition from here. Be sure to disable/uninstall the stable version of GitLens first.
I've also added a new GitLens: Reset Avatar Cache command that can be run from the command palette to hopefully reset any bogus image that might have been cached.

由于总所周知的原因，https://gravatar.com/这个网址应该打不开了，过一段时间应该就可以了，或者找个梯子。

@eamodio Thank you. Run gitlens: reset avatar cache. My problem has been solved


Fixed typo which wouldn't let me compile

um, would you please create a new release on npm ? The default npm install doesn't work without this pr ;)

@jmls  I just update the release on npm, thanks.


Crash when all tabs are closed

Steps to Reproduce

Open any amount of tabs, or utilize the single default tab
Close all tabs (CTRL+SHIFT+W)
Attempt to open a new tab (➕ or CTRL+SHIFT+T)
⚰

stacktrace(s):
https://paste.ivr.fi/catemuryvu // this one happens when pressing ➕ or CTRL+SHIFT+T
https://paste.ivr.fi/sijuxunegu.r // this one happens when pressing CTRL+T in the same scenario
Explicitly tested on 2ba4da02aecf937c740ef9b0a9f7bd7b8c3ef256 but I doubt we've fixed this issue since last week

Crash introduced in https://github.com/Chatterino/chatterino2/pull/4187


Virtually Every Utube video i download d/ls at approx 50kb/s

but when i try a mainstream supported utube video i get full speed.  this has been the same after every update since early 2020.     what is going on?  am i doing something incorrect?  d/l full quality better than 480p is a thing of my past.

about to swap to teleG youtube downloader

Try suggestions from this post https://github.com/ytdl-org/youtube-dl/issues/30715
Using the latest version from repo worked for me


Suggestion: Insert link in AsciiDoc format when the document itself is in AsciiDoc

Hello @usernamehw, thank you for this handy extension! What do you think about supporting AsciiDoc in addition to Markdown? I believe Markdown could remain the default since it's much more widespread, but I myself frequently use AsciiDoc for documentation and other purposes. It would be great if the extension recognized AsciiDoc and inserted links in the appropriate format, so that I wouldn't have to reformat them manually. If you like the idea, I'd be happy to take it on and submit a PR.

Would it require to some external dependencies (detecting/parsing text) or
simply using https://asciidoctor.org[Asciidoctor] format when file extension is .adoc or .asciidoc?

The latter, only using a different format

You can try it in 0.2.0 version.

Works as intended. Thank you very much!

Hello @usernamehw! Sorry for bothering you again, but I hit on another idea. What do you think of supporting HTML as well, that is insert an HTML link ($urlTitle) when the document being edited is HTML, similarly to what you did for AsciiDoc? (Please let me know if you'd like to me to file a separate issue. I decided not to do this to avoid spamming.)


Lint fix

Codecov Report

Merging #1 (4118c24) into main (63a6b20) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main       #1      +/-   ##
==========================================
- Coverage   71.62%   71.58%   -0.05%
==========================================
  Files        1816     1816
  Lines      115144   115144
  Branches    16678    16660      -18
==========================================
- Hits        82474    82426      -48
- Misses      31552    31599      +47
- Partials     1118     1119       +1

Flag
Coverage Δ

e2e-tests
66.57% <ø> (+0.04%)
:arrow_up:

unit-tests
63.43% <100.00%> (-0.05%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

ghost/collections/src/CollectionsService.ts
100.00% <100.00%> (ø)

... and 14 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Update pt-kill patch to dynamically adjust dynamic idle/busy time adjustments on a per-user basis.

Hi @pseudomorph, thanks for the pull request. Before we can merge it, we need you to sign our Contributor License Agreement. You can do so electronically here: http://opensource.box.com/cla
Once you have signed, just add a comment to this pull request saying, "CLA signed". Thanks!

👍

CLA signed.

Verified that @pseudomorph has just signed the CLA. Thanks, and we look forward to your contribution.


animation loops request

Hello, I'm not sure how hard this would be, the literal only "complaint" that I have for this site/code is that the kind of animations that I often see are about 1 to 6 seconds, and I would love to see an option to have them continually loop. Seems like a potential addition to links ending in mp4
Here is a wonderful example: https://twitter.com/Kekeflipnote/status/1584464315662602240

Will work on this 👍

Tweets with GIFs should now loop(ish), ish because they're repeated until they are 30 seconds long.
I wanted to convert them to raw GIF but the file sizes are just too big to make the experience fast for users. Maybe in the future


Support 411 and 410 subsampling rations for go1.5+

This adds support for the YCbCrSubsampleRatio411 and YCbCrSubsampleRatio410 added in Go 1.5.

@nfnt This commit has been updated to reflect your comments - specifically the duplicate *_go15.go files have been removed.  The logic in ycc.go is now capable of supporting multiple versions of Go without compile errors.  The test still requires go1.5+.

Accepting this PR allows https://github.com/nfnt/resize/pull/51 to be closed.

Thanks! I'll look into it later today, have to first get familiar with my code again.


AWS::EFS::FileSystem -> ThroughputMode E3030: You must specify a valid value for ThroughputMode

CloudFormation Lint Version
0.72.1
What operating system are you using?
Windows
Describe the bug
There's an error detected in the AWS::EFS::FileSystem resource type when using ThroughputMode: elastic
E3030: You must specify a valid value for ThroughputMode (elastic). Valid values are ["bursting", "provisioned"]
According to https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-efs-filesystem.html#cfn-efs-filesystem-throughputmode the supported types are bursting | elastic | provisioned and this configuration should not cause an error.
Expected behavior
No error then using ThroughputMode: elastic into an AWS::EFS::FileSystem resource type
Reproduction template
AWSTemplateFormatVersion: 2010-09-09
EFS:
Type: 'AWS::EFS::FileSystem'
Properties:
BackupPolicy:
Status: ENABLED
Encrypted: true
LifecyclePolicies:
- TransitionToIA: AFTER_30_DAYS
FileSystemTags:
- Key: Name
Value: efs-file-system
- Key: ApplicationRole
Value: role-efs-file-system
KmsKeyId: 'alias/aws/elasticfilesystem'
ThroughputMode: elastic

Automated maintenance picked it up:
https://github.com/aws-cloudformation/cfn-lint/pull/2493/files#diff-a5a6f378b19287fd419ef6c9edf44d4493163add0a727420af026573a02f165a


Python3 compatibility; better Unicode support

This PR includes a set of changes to allow the t2t-datagen program to run to completion under Python 3.x. It fixes imports of urllib and cPickle and removes the assumption that strings contain bytes. It also makes minor bug fixes and efficiency enhancements. Functionality-wise, it modifies the subword vocabulary builder so that all (Unicode) characters that occur in the source text are automatically included in the vocabulary, vs. the previous design that included all characters with ordinals 0..255 inclusive, regardless of whether they occurred in the vocabulary or not. Also, in the ByteTextEncoder, Unicode characters with ordinals >= 256 are replaced by a marker character (currently ×, chr(215)).
This fixes issue #2
Caveat: the tests are not updated by this PR.

Having Python 3 give exactly the same results as Python 2 sounds like the best thing, great thanks for this effort Villi!

I modified ByteTextEncoder to be fully compatible between Python2 and Python3. Both of them encode and decode to an internal UTF-8 representation, although the Python3 one is more explicit about it. This removes the need for special handling of ordinals >= 256.

I think I see the appeal of mapping any character code outside of 0-255 into an "unknown" character, but that throws out a lot of easily handled cases--which can be done automatically with a package like unidecode: it converts en dashes into hyphen-minuses, ellipses into three periods, etc.  That seems preferable for machine-translation data.  Thus, I modified the original tokenizer.py very simply with:
from unidecode import unidecode
...

token_start = 0

if six.PY2:
    raw_text.decode("utf-8", "replace")
raw_text = unidecode(raw_text)

for pos in range(0, len(raw_text)):
...

and then modified text_encoder.py only to get rid of the string-escape encoding (following this PR).   I haven't tested under python 2.7 but this appears to work under 3.6.

@jgmakin We ended up with a different solution that uses UTF-8 encoding both on Python2 and 3, with no out-of-bound or "unknown" characters. This seems to work fine on the character generator models. T2T has no difficulty learning correct UTF-8 encoding, apparently.

Fair enough.  I suppose this issue is shading into questions of problem definition: how many types of characters does a translator have to master?  Maybe someone from the community (with a lot of GPUs) can check how much performance varies under the different conditions: (1) all characters forced into ASCII representation; (2) UTF-8 but with punctuation only from the ASCII set (i.e., the current state of affairs); (3) UTF-8 with properly identified punctuation from the entire UTF-8 set.  My intuition is that (1) should be much easier, but I could be wrong.

I'm afraid (1) is not really an option if you are translating to or from languages where no ASCII representation exists for important characters; Chinese and other Asian languages are obvious examples and the same applies for Turkish, Icelandic, etc.

No kidding!  But the popular benchmarks are German-English and French-English.  (I realize that French and German use diacritics, but they are readable without them.    Anyway, I concede that using the whole UTF-8 set is the more general solution.

We run a large number of experiments like that with LSTM models, and it looks like this basically doesn't matter as long as you have at least a bit of various encodings in your training data. So yes -- writing extra code to treat emdash same as dash may give you a little improvement if your training data has very very few emdashes. But it almost doesn't matter in any other setting, and you're putting on yourself the engineering burden and the risk of bugs that come from it. If you want a resilient model, it looked to us to be better to gather some training data with different encodings (or even generate it).

Alright, I'm convinced! and thanks, "basically doesn't matter..." is interesting.