Include a prop definition for to next to href in ButtonBase's TypeScript types

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯
ButtonBase accepts a LinkComponent which it will use in place of <button> if either a href or to is used.
https://github.com/mui-org/material-ui/blob/75f27074b89381d25a1fab85fc2098af58e89cb0/packages/material-ui/src/ButtonBase/ButtonBase.js#L286-L288
https://github.com/mui-org/material-ui/blob/75f27074b89381d25a1fab85fc2098af58e89cb0/packages/material-ui/src/ButtonBase/ButtonBase.js#L295-L297
However the TypeScript types only include a href so if you include a to you will get a TypeScript error, despite the fact that the code functions.
https://github.com/mui-org/material-ui/blob/75f27074b89381d25a1fab85fc2098af58e89cb0/packages/material-ui/src/ButtonBase/ButtonBase.d.ts#L98-L101
Expected Behavior 🤔
ButtonBase should include a to in the props type as well, since the JS it represents actually handles that.
Ideally it would be nice if it was defined as a type somewhere that could be overridden with a module declaration in a project.
Steps to Reproduce 🕹

Context 🔦
The Routing libraries documentation includes a Global theme Link example where code does not need to pepper component={Link} everywhere and can easily use <Button> and MUI's <Link> with React Router.
The example works with the caveat that it only uses href. So you can't swap between internal React Router links and external links. And the href needs to be a string so you cannot use React Router's To object form.
Despite this limitation in the example it appears that ButtonBase is already coded to allow you to use to instead of href. So neither of those limitations is necessary. And the ideal setup where to uses Link and href uses a seems entirely possible already. The only problem is that MUI's TypeScript definition conflicts with how the JS is actually written.
Your Environment 🌎

`npx @material-ui/envinfo`
  System:
    OS: Linux 5.10 Ubuntu 20.10 (Groovy Gorilla)
  Binaries:
    Node: 16.5.0 - ~/.nvm/versions/node/v16.5.0/bin/node
    Yarn: 1.22.5 - /mnt/c/Program Files (x86)/Yarn/bin/yarn
    npm: 7.19.1 - ~/.nvm/versions/node/v16.5.0/bin/npm
  Browsers:
    Chrome: Not Found
    Firefox: Not Found
  npmPackages:
    @emotion/react: ^11.4.0 => 11.4.0
    @emotion/styled: ^11.3.0 => 11.3.0
    @material-ui/core: ^5.0.0-beta.2 => 5.0.0-beta.2
    @material-ui/icons: ^5.0.0-beta.1 => 5.0.0-beta.1
    @material-ui/lab: ^5.0.0-alpha.41 => 5.0.0-alpha.41
    @material-ui/private-theming:  5.0.0-beta.2
    @material-ui/styled-engine:  5.0.0-beta.1
    @material-ui/styles: ^5.0.0-beta.2 => 5.0.0-beta.2
    @material-ui/system:  5.0.0-beta.2
    @material-ui/types:  6.0.1
    @material-ui/unstyled:  5.0.0-alpha.41
    @material-ui/utils: ^5.0.0-beta.1 => 5.0.0-beta.1
    @types/react: ^17.0.15 => 17.0.15
    react: ^17.0.2 => 17.0.2
    react-dom: ^17.0.2 => 17.0.2
    styled-components:  5.2.1
    typescript: ~4.2.0 => 4.2.4

ButtonBase should include a to in the props type as well, since the JS it represents actually handles that.

That's not a sufficient reason for us to expose it. We're just assuming a specific intent of that prop. But we're not actually implementing the prop. The component needs to implement to instead.

What is the point of assuming specific intent for to if it's not possible to pass a LinkComponent that uses to without a TypeError?
It's not possible to override ButtonBase's props in the current type definition. Can't we at least make that possible so the real behaviour of LinkComponent, overrides, and to can at least be defined by the user.

What is the point of assuming specific intent for to if it's not possible to pass a LinkComponent that uses to without a TypeError?

That should work already. If not please include a complete repro.

That should work already. If not please include a complete repro.

https://codesandbox.io/s/linkcomponent-to-demo-h1u9m?file=/demo.tsx
Basically, what's the point of having a LinkComponent prop that supports to if actually using it just gives you a ts error. I know component works, but then what's the point of the LinkComponent prop. Being able to at least have something like a ButtonBasePropsOverrides (or was it "Extends" that other components were using) would be preferable.

ButtonBase has not variant prop as well.
<Button
  component={RouterLink}
  to="/somewhere"
  variant="contained"
>
  Link
</Button>

is probably what you wanted and passes type-checking.

I'm only using ButtonBase simply because it's the common component that other components are based on and the component where LinkComponent is defined and the to behaviour is handled. My example still fails if I use Button instead of ButtonBase.
And as I said I know that you can use component instead. However if the solution is always to use component then what is the point in having LinkComponent at all? Especially if it's coded specifically so it could be useful as a defaultProp override but if you actually try using it that way in a real project you instead get a TypeScript error you have no way to fix or override.


Reload Method?

Hi Alexis - I am trying to use the new version on an Angular2 project.   I am not sure how to use it.   The new version looks to be using a different reload mechanism than previously (below).
Do you have any sample?
Thanks for your incredible work here!
Best regards,
John
let HotReloader: any = <any>window[<any>'hr'];
HotReloader && HotReloader.on('change', (fileName: string) => {
    if (fileName.indexOf('html') !== -1) {
        var newBody = document.createElement('body')
        newBody.appendChild(document.createElement('sparq-app'))
        document.body = newBody;
        platformBrowserDynamic().bootstrapModule(AppModule);
    }
});

Hi @jcmoore0.
Yeah, the eventing model has been stripped away (I'm planning on replacing it with an Rx Observable). I'm guessing here you're wanting to to save yourself a page reload?

Thanks for the quick reply.  Your observable plan sounds terrific.
You got it.   I am wanting to save some reload time :)
Hot reload goals:

html on html change detected  (html is outside js before production bundling)
module/component reload on js file change detected
module recompile reload on ts file change detected

I was doing #1 with the previous version.
I am looking for a sample to make at #1 work with the new version and eventually do 2 and 3.
Best regards,
John

If I understand you correctly, then you should have 2 and 3 out of the box. 1 is trickier at the moment since theres no eventing api yet. Are you just saving yourself the full page refresh? Because that shouldn't be happening that often.
If you can give me an example repository I might be able to help you better.


No UI text on Linux (AUR f3d-bin) when no file is loaded

Describe the bug
When opening f3d with no files, Linux & macOS versions show different UIs.
On macOS, it displays help text and XYZ gizmo; on Linux, it's blank.

Linux
Mac

To Reproduce
Steps to reproduce the behaviour:

Start f3d on Linux with no arguments
Start f3d on Mac with no arguments
Observe the difference

Expected behaviour
Linux and Mac versions show the same behaviour with a help text.
The help text is valid for both platforms, as the Linux version accepts drag-and-drop files too.
You might show the XYZ gizmo on all platforms too because it hints about the nature of the app
System Information:

OS: macOS 13.3.1 (22E261) / EndeavourOS latest

F3D Information
Paste the content of f3d --version:
F3D 2.0.0

F3D - A fast and minimalist 3D viewer
Version: 2.0.0.
Build date: 2023-04-20 17:06:45.
Build system: Darwin 64-bits.
Compiler: AppleClang <IP_ADDRESS>00029.
External rendering module: ON.
Raytracing module: OFF.
VTK version: 9.2.6 (build 20230224).

Additional context
Add any other context about the problem here.

on Linux, it's blank.

That is definitely unexpected, how di you install F3D in that case ?

f3d-bin AUR is not managed by the F3D team, please report the issue on the aur package.
FYI even f3d-git may be broken, we need to update it.
FYI @kylosus

Sorry, my bad. It was Arch Linux AUR package f3d-bin.
I was in hurry and didn't wait for community/f3d package update, and didn't use your official flatpak.
I've tried community/f3d (which is updated now) and it shows no UI either.
Now I'm trying to install Flatpak package and check it

I've tried community/f3d (which is updated now) and it shows no UI either.

We reworked the configuration files logic, so an update is probably needed  (@sventaro)

@epsimatic : I just update the f3d-git package, if that helps.

The text seems to disappear when an empty config.d directory is present in /etc/f3d/, which the f3d-bin package creates for convenience. I suppose the package not create this directory? I had assumed all config dirs would be read in order and the packaged configs were meant to be examples/defaults, but the documentation says F3D only uses the first one found.

the documentation says F3D only uses the first one found.

Yes, this is correct. This could be changed but it involve a lot of work and I could not really justify it.

The text seems to disappear when an empty config.d directory is present in /etc/f3d/, which the f3d-bin package creates for convenience.

I've been looking at the package and it needs to be updated with all the recent changes to configuration files logic.

To summarize the issue:

Arch community/f3d : config broken, no UI
AUR/f3d-bin (not managed by you) : config broken
AUR/f3d-git : config valid, UI available after https://github.com/f3d-app/f3d/issues/752#issuecomment-1518577091
Flatpak : config broken, no UI
macOS downloadable .dmg : config valid, UI available

Maybe it's reasonable to reopen the issue until community/f3dand Flatpak packages are fixed?

Why not just grabbing the portable tarbal ?

It's a worse decision for the long run — no updates, no dependency management, etc.

AUR/f3d-bin is updated now.

Thanks a lot @kylosus

Arch community/f3d : config broken, no UI

Not managed by the F3D team either, I've logged an issue on arch bugreports system, I hope @svenstaro can take a look soon

Flatpak : config broken, no UI

Not managed by the F3D team either, I hope @kevinsmia1939 can take a look, no idea how to report an issue there.

reopen the issue until community/f3dand Flatpak packages are fixed?

I'd rather not since this is out of the hands of the F3D team and developers.

@mwestphal
I'm working on the update.
The issue can be report here: https://github.com/flathub/io.github.f3d_app.f3d/issues

Arch package should now be fixed in rel-3.

@svenstaro tested and confirmed to work perfectly now ! Thanks !


Fix display issues

found the issue i was not calling a page size  changed event so it was not triggering how many tiles to display

Fixed in latest release  v3.0.5


Fixing build error

rake aborted!
NoMethodError: undefined method start_with?' for #<Array:0x007ffb81278c40> /.rvm/gems/ruby-2.0.0-p481/bundler/gems/Flow-70322b758ce8/lib/cocoa.rb:27:in block (3 levels) in <top (required)>'

I'm curious, how did you manage to have an Array inside app.files? Could you print the value of app.files in lib/cocoa.rb before the delete_if loop so that we can see what's inside?

I have 2 rather large arrays:
[["/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/managed_object.rb"], ["/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/collection_proxy.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/config.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/context.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/deprecation.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/model.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/object.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/object_proxy.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/partial_predicate.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/query.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/relationship_query.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/store.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/cdq-1.0.8/lib/../motion/cdq/targeted_query.rb"], ["/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/action_sheet_provider.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/alert_action.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/alert_constants.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/alert_controller_provider.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/alert_field.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/alert_view_provider.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/button_templates.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/field_templates.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/ios9_fix_uialertcontroller.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/RedAlert-1.4/lib/project/red_alert.rb"], ["/Users/mbonar/.rvm/gems/..."./app/stylesheets/video_controls_stylesheet.rb", "./app/stylesheets/video_screen_stylesheet.rb", "./app/views/transparent_modal.rb", "./app/views/video_controls.rb", "/Users/mbonar/.rvm/gems/ruby-2.0.0-p481/gems/motion-yaml-1.4/lib/project/yaml.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/actions.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/authorization.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/config.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/expectation.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/header.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/net.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/response.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/session.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/stubbable.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/cocoa/reachability.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/cocoa/request.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/net/cocoa/response_proxy.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/json/cocoa/json.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/digest/cocoa/digest.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/store/cocoa/store.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/base64/cocoa/base64.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/location/location.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/location/cocoa/location_services.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/task/task.rb", "/Users/mbonar/dev/rubymotion/Flow/lib/../flow/task/cocoa/task.rb"]
then the above repeats one more time
  From: Laurent Sansonetti<EMAIL_ADDRESS>
To: HipByte/Flow<EMAIL_ADDRESS>Cc: marlabz6<EMAIL_ADDRESS>Sent: Thursday, February 4, 2016 11:58 AM
Subject: Re: [Flow] Fixing build error (#6)
I'm curious, how did you manage to have an Array inside app.files? Could you print the value of app.files in lib/cocoa.rb before the delete_if loop so that we can see what's inside?—
Reply to this email directly or view it on GitHub.

I get the same error occurs.
I do not know why that create a redundant array.
I'm monkey-patching !
lib/cocoa.rb
  app.files.flatten!

  samples = %w(android ios osx).delete_if {|t| t == template}
  samples.each do |sample|
    app.files.delete_if { |path| path.start_with?("./app/#{sample}") }
  end


How to exclude generation every time when we do the compilation?

Hi,
I have changed the generatedSourceDestinationDir to src directory. Now I have having a problem that every time when I compile the project, it is getting executed. How can I stop this behavior?

Isn't this what you'd expect from annotation processors?
If you want to do things in several steps, you'd have to run a first compilation with -proc:only to generate sources (without actually compiling anything), then another run with -proc:none (or without the annotation processors) and the sources generated by the first run to actually compile everything.
Now, depending on what workflow you actually want, you could also configure aptOptions.annotationProcessing to true/false and add the generated sources as a source directory (or delete it in a doFirst on the compileJava task) depending on which behavior you want. In any case this is beyond the scope of this plugin.
Duplicate of #71


A magnificent work, thank you very much for sharing (Need Pik-Fix Data)

I sent you a mail a few days ago, I'm still waiting for an answer from you sir for Pik-Fix Data (real_old_data, real_old_ref, and texture2), I would like to do a test on black and white images/video, I hope to have a positive answer from you, Best, NASS

Have you received Pik-Fix data? I have the same question.


Preselect and disable already added variant combinations

When a variant combination already is added, it would make since to preselect it and disable those buttons (or remove them from the options).
Furthermore, based on the starterkit when choosing Color: Black, Green they are submitted, but you get an error:
Some of your variants does not have the same variant types

Maybe the "Add variant options" button should be disabled in that case? but I am not sure how I will work when you only have a single variant group like Color - in that case it should be okay to add variants only from one variant group.
I also thought about if you can configurate the product to use different variant groups? e.g. if you sell a shoe you might want to use variant groups for Color and Shoe Size .. but for a T-shirt Color and Size ...
so often you want to add variants based on the product type.
Finally it would also be great to demonstrate in demo site, that if you e.g. select Color:Black it is available in Size:S,M,L .. but for Color:Green it is only available in Size:S,M on frontend (just for demonstration purpose)

All of what you mention here is actually something we either chose by design or it already does it.

"Some of your variants does not have the same variant types"
We have chosen to do it like that to give you guys the most possible ways to use the system. There are actually some products out there that have croocked variant groups. You can have that.

Also if you want to ensure that ALL variant groups are chosen by the editor you should tick the "Force editor to choose all variant groups" on the variant editor data type. That will disable the add button until at least one of each variant group have been selected.

If you sell t-shirts and shoes you can make different variant editor data types and add them to different product document types. Each variant editor data type can have different variant group filters meaning one product could have colors and sizes and others have colors and "Shoe size".

The demo site already demonstrates options being disabled after choosing a color. If a size is not in the color it will be disabled. You guys can just change the javascript to remove the ones that can't be chosen if that's the best choise of design for your shop.

/Rune

Hi Rune

Okay, I haven't really tried configuration the "Product" futher, but I guess I could create child document types and composition document types to configurate different product types with different variant groups if you have a store with very different product types. It was also answer for 3.

Okay, didn't notice the "Force editor to choose all variant groups" property. However with this checked it only has diabled the add button until one value from one of the groups has been chosen - not both. Any reason why not require one value from each variant group before enable the button as you get an error anyway when only choosing one value (when using multiple variant groups) - see previously posted image.
Futhermore when selection af Color (e.g. Green) "- Select all variant types -" change to "- Deselect all variant types -"  ... it doesn't happen when selection just a Size (e.g. Medium).

Ahh okay, I see it now. The contrast between enabled and disabled text i the dropdown options is just low (at least in Chrome). Maybe change to a lighter grey text color for disbled options.

I still believe the  "Force editor to choose all variant groups" works:

I do see that there's a small bug with "- Select all variant types -". I will create a new issue for that.

Okay, it seems to work now when I check it.
Just when I tried yesterday from my home PC on the demo site, the "Add variant options" button was enabled when I just selected one of the variant groups - also with "Force editor to choose all variant groups" enabled.
Therefore I was able to submit the variants for Color: Black, Green without a Size selected - I got the "Some of your variants does not have the same variant types" though.
But it seems to work fine now :)

You might have change the other property editor. Umbraco 7.4 might create a new editor (Pretty anoying by the way)

Yeah, maybe I had just changed the config for the wrong datatype instance - it did notice there is two instances of Variant Editor in the demo site.


Bug: Increasing size of blacklist leads to questionable report output

I added 30 subgraph deployments to the blacklist (making it a total of ~45 subgraphs) and with the new, longer list the generated reports contained questionable strategies.
With the longer blacklist I am now consistently getting a single strategy returned with only 1 subgraph being allocated to. The expected profit is very low, so it doesn't appear to be a valid suggestion to only allocate to 1 subgraph.

Could you post your config file that you used for this experiment?

Hmm... I'm unable to reproduce now, so it's possible I was mixing this experiment setup with the pinnedList experiment I was running.
Hold off on this issue for now? I'll comment again here and tag you if I can reproduce again.

Hmm... I'm unable to reproduce now, so it's possible I was mixing this experiment setup with the pinnedList experiment I was running.
Closing the issue now to avoid confusion. I'll revive it if I can reproduce again.

Sounds good!


Person detection training doesn't converge. Evaluation showing accuracy of 52% after 1M steps

@tensorflow/micro
System information

Host OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
TensorFlow installed from (source or binary): Docker installation
Tensorflow version (commit SHA if source): 1.14.0
Target platform (e.g. Arm Mbed OS, Arduino Nano 33 etc.): None

Describe the problem
I have done the training as exactly as in the document https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/examples/person_detection/training_a_model.md with some minor command changes which is not updated in the original document. The training was done for 1Million steps and when evaluated the accuracy was 52%.
The minor modifications were

use the script download_and_convert.py instead of build_visualwakewords_data.py.
Changed arguments --input_grayscale to --use_grayscale

Please provide the exact sequence of commands/steps when you ran into the problem
The commands i used are as follows all from the path models/research/slim

python slim/download_and_convert_data.py
--dataset_name=visualwakewords
--dataset_dir=/tmp/visualwakewords

python slim/train_image_classifier.py     --train_dir=vww_96_grayscale     --dataset_name=visualwakewords     --dataset_split_name=train     --dataset_dir=coco_dataset/     --model_name=mobilenet_v1_025     --preprocessing_name=mobilenet_v1     --train_image_size=96     --use_grayscale=True     --save_summaries_secs=300     --learning_rate=0.045     --label_smoothing=0.1     --learning_rate_decay_factor=0.98     --num_epochs_per_decay=2.5     --moving_average_decay=0.9999     --batch_size=96     --max_number_of_steps=1000000

python slim/eval_image_classifier.py --alsologtostderr checkpoint_path=vww_96_grayscale/model.ckpt-1000000 --dataset_dir=coco_dataset/ --dataset_name=visualwakewords --dataset_split_name=val --model_name=mobilenet_v1_025 --preprocessing_name=mobilenet_v1 --use_grayscale=True --train_image_size=96

Can someone help me in identifying why it is not converging as stated in the paper?

Sorry my mistake there was a typo in checkpoint_path argument. Now I am getting 81%.
One additional change to training_a_model.md. Changed the argument --train_image_size to --eval_image_size.


Is there a polymorphic way to encode/decode ?

Sorry I am fairly new to elixir.
Is there something like a protocol implemented by all protobuf modules?
It would be awesome to be able to do:
Protobuf.encode(MyModule.Msg.new())

I tried something like:
defmodule Objects do
  use Protobuf, from: 'file.proto'
  defprotocol Serializable do
    def serialize(object)
    def deserialize(binary)
  end

  defimpl Serializable, for: Msg do
    def serialize(object) do
      Msg.encode(object)
    end

    def deserialize(binary) do
      Msg.decode(binary)
    end
  end

end

But of course it fails because it does not know Msg.
Could this be achieve with macros?

What you sketched out should work just fine, however I think the problem is that you are not using the fully-qualified name for Msg. I can't tell you what it should be exactly without knowing what is in file.proto, but I assume it should be something like Objects.Msg.
In any case, this is a good idea, and I'm pretty sure this can be implemented fairly easily within exprotobuf by injecting the defimpl and defining the protocol.

Ok, with my morning coffee in full effect, I realize why I'm totally wrong about the above.
Protocols don't work for decoding because you have to explicitly define what type you are decoding anyway (protocols require type information to delegate to the correct defimpl, and a plain binary isn't enough).
For encoding, we could provide a protocol abstraction, however it would be a very thin protocol around Protobuf.Encoder.encode(my_struct, MyStructModule.defs).

@ibizaman I've added a Protobuf.Serializable protocol, which defines a serialize function. It is automatically implemented for any exprotobuf generated module/struct, so you shouldn't have to do anything. Just note that if you do not compile protocols, it will be slow to use versus the more direct call to the specific module's encode function. If you compile protocols, there should be no difference.

Wow thanks a lot, such reactivity!
Indeed, to decode you must specify the type. At least it works for encoding.


[SPARK-47495][CORE] Fix primary resource jar added to spark.jars twice under k8s cluster mode

What changes were proposed in this pull request?

In SparkSubmit, for isKubernetesClusterModeDriver code path, stop appending primary resource to spark.jars to avoid duplicating the primary resource jar in spark.jars.
Why are the changes needed?

Context:
To submit spark jobs to Kubernetes under cluster mode, the spark-submit will be called twice. The first time SparkSubmit will run under k8s cluster mode, it will append primary resource to spark.jars and call KubernetesClientApplication::start  to create a driver pod. The driver pod will run spark-submit again with the updated configurations (with the same application jar  but that jar will also be in the spark.jars). This time the SparkSubmit will run under client mode with spark.kubernetes.submitInDriver  as true. Under this mode, all the jars in spark.jars will be downloaded to driver and jars' urls will be replaced by the driver local paths. Later SparkSubmit will append primary resource to spark.jars again. So in this case, spark.jars will have 2 paths of duplicate copies of primary resource, one with the original url user submit with, the other with the driver local file path. Later when driver starts the SparkContext it will copy all the spark.jars to spark.app.initial.jar.urls, and replace the driver local jars paths in spark.app.initial.jar.urls with driver file service paths, with which the executor can download those driver local jars.
Issues:
The executor will download 2 duplicate copies of primary resource, one with the original url user submit with, the other with the driver local file path, which leads to resource waste.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Unit test added.
Was this patch authored or co-authored using generative AI tooling?

No

retest this please

cc @dongjoon-hyun

Ack, @dbtsai .

@dongjoon-hyun Updated the PR and associated it with the new JIRA https://issues.apache.org/jira/browse/SPARK-47495. Please let me know if this looks good to you. Thanks!

+CC @zhouyejoe

Thank you for updating, @leletan .
I'll resume the review this weekend.

I just assigned this to me in order not to forget. It doesn't block any community reviews.

Merged to master because the last commit only changes JIRA ID in the test case name.

Welcome to the Apache Spark community, @leletan .
I added you to the Apache Spark contributor group (in JIRA) and assigned SPARK-47495 to you.
Congratulations for your first commit, @leletan .


Create built-in deadman switch in Ruler

TBD

I guess still valid. We still wait for the description though.

Is this still valid?

I do have a use case for this, atm I've had to manually define always firing alerts. So to have this build in ruler would be nice. If @bwplotka can write criteria/design etc I would love to have a look at implementing it.
My idea: check if downstream prom API's are reachable and if so continue to fire the alert. If it's not reachable, stop firing the alert. In case AM is down the heartbeat also fails. So with this method, you have the whole alerting chain.


Need to be rewritten using async/.await feature

Need to be rewritten using async/.await feature, see https://rust-lang.github.io/async-book/

Done.


PyQt issue after deployment 2020-2

This problem arose at the deployment 2020-2 visit in the GUI program /home/vescovo/arpes_viewer/main.py
Qt.ControlModifier should be used instead of Qt.KeyboardModifier.ControlModifier.
/opt/conda_envs/collection-2020-2.0rc7-1) xf21id1@xf21id1-ws1:~/.ipython/profile_collection/startup$ python /home/vescovo/arpes_viewer/main.py
libGL error: failed to open drm device: Permission denied
libGL error: failed to load driver: nouveau
QXcbConnection: XCB error: 3 (BadWindow), sequence: 1133, resource id: 31783985, major code: 40 (TranslateCoords), minor code: 0
Traceback (most recent call last):
  File "/home/vescovo/arpes_viewer/main_window.py", line 1267, in onMouseMoved
    if modifier == Qt.KeyboardModifier.ControlModifier:
AttributeError: type object 'KeyboardModifier' has no attribute 'ControlModifier'
Aborted

Versions:
# Does not work in this env:
(/opt/conda_envs/collection-2020-2.0rc7-1) root@xf21id1-ws1:~# conda list | grep -i qt
pyqt                      5.9.2            py37h05f1152_2    defaults
pyqtgraph                 0.10.0                   pypi_0    pypi
qt                        5.9.7                h5867ecd_1    defaults
qt5reactor                0.6.1                      py_0    https://conda.anaconda.org/nsls2forge
qtconsole                 4.7.5                      py_0    defaults
qtpy                      1.9.0                      py_0    defaults
sphinxcontrib-qthelp      1.0.3                      py_0    defaults
(/opt/conda_envs/collection-2020-2.0rc7-1) root@xf21id1-ws1:~# pip list | grep -i qt
pyqtgraph                     0.10.0
qt5reactor                    0.6.1
qtconsole                     4.7.5
QtPy                          1.9.0
sphinxcontrib-qthelp          1.0.3
(/opt/conda_envs/collection-2020-2.0rc7-1) root@xf21id1-ws1:~# conda deactivate

# Works in this env:
root@xf21id1-ws1:~# conda activate collection-2019-2.0-esm
(/opt/conda_envs/collection-2019-2.0-esm) root@xf21id1-ws1:~# conda list | grep -i qt
pyqt                      5.9.2            py36h22d08a2_1    http://alexandria/conda/defaults
pyqtgraph                 0.10.0                   pypi_0    pypi
qt                        5.9.6                h8703b6f_2    http://alexandria/conda/defaults
qtconsole                 4.4.2                    py36_0    http://alexandria/conda/defaults
qtpy                      1.5.2                    py36_0    http://alexandria/conda/defaults
(/opt/conda_envs/collection-2019-2.0-esm) root@xf21id1-ws1:~# pip list | grep -i qt
pyqtgraph           0.10.0
qtconsole           4.4.2
QtPy                1.5.2

That is odd, the PyQt layer didn't change version, and Qt was only a bug fix release. The qtpy layer moved up quite a few versions - are they going through that?

cc @bsobhani, we observed this issue with the new collection-2020-2.0rc7-1 conda env.


Fix function overloads (part 1 of 2)

Fixes #32, fixes #13.
Parse and include function signatures for breathe directives.
Part 2 of 2 will occur when fixed (?) in Breathe, or how to use it is clarified.

@avilchess @mquinson
The proposed fixes here should rectify the problems Exhale has with your project's function overloads.
If you have time would you be willing to quickly install and run this branch?  Function overloads ended up being much more complicated than I anticipated.
$ pip install git+http://github.com/svenevs/exhale.git@fix/function-overloads#egg=exhale

I would like to release a new version with the fixes in this branch.  I can verify that the codebase for #13 succeeds with these fixes (finally...), but it would be really helpful to get confirmation / feedback from users if you have time :)

Functions in the full api listing include their fully qualified name.  So namespace foo { void bar(); } used to have its title as Function bar and now is Function foo::bar.

Would you rather this not be the case?  Would you rather it be configurable via yet-another-config-variable?
Motivation: "flat" naming is not very useful in the main Library API page, which is where all of the functions get enumerated (they aren't in any hierarchies).

When a function overload is detected, its parameters are included in the title of the function page.
void bar(int);         // => Function bar(int)
void bar(float, float);// => Function bar(float, float)
void foo(int);         // => Function foo ---> No parameters when no overload

Warnings will be emitted for overload groups (group := function name) that cannot currently be documented correctly.  As far as I can tell, Breathe currently only accepts func_name(list, of, parameters).  When templates are different but function parameters are the same, I cannot get Breathe to do this.  So these
https://github.com/svenevs/exhale/blob/0a0ae56c29ebc7c8e838ffa8ce5a81d303aa6146/testing/projects/cpp_func_overloads/include/overload/overload.hpp#L79-L99
All produce .. doxygenfunction:: overload::bar(typename C::type), which will lead to it documented three times on the Library API page, also resulting in warnings about multiple declarations from Sphinx.  Exhale produces a very umm noticable warning about this and proposes temporary alternatives (particularly how to skip with Doxygen).

Thank you for any thoughts / suggestions / feedback!


distrobox: <IP_ADDRESS> -> <IP_ADDRESS>

Description of changes
https://github.com/89luca89/distrobox/releases/tag/<IP_ADDRESS>

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

Just tried to create a container
$ distrobox create --name test --image debian
Creating 'test' using image debian       [ OK ]
Distrobox 'test' successfully created.
To enter, run:

distrobox enter test

$ distrobox enter test
 Error: could not start entrypoint.
{"msg":"exec container process (missing dynamic library?) `/usr/bin/entrypoint`: No such file or directory","level":"error","time":"2023-11-22T00:15:15.341593Z"}

Yeah I noticed this as well. My best guess: the issue is that distrobox-init is a symlink.

https://github.com/89luca89/distrobox/blob/3435f4d27070a99668bfa29a3e508db4ecc09009/distrobox-create#L77-L80

Been able to pinpoint the regression to https://github.com/89luca89/distrobox/commit/5dd7b00e3f095e1976c0e2aa4f972efdcd6f4f3a. Not sure why that is though as I don't think those mounts should matter?

I don't get the rationale behind that comment, i believe resolving the symlink is what we actually want here.

So it always mounts /run/current-system/sw/bin/distrobox-init as /usr/bin/entrypoint in the container rather than a specific store path which might eventually get garbage collected, no?

Been able to pinpoint the regression to https://github.com/89luca89/distrobox/commit/5dd7b00e3f095e1976c0e2aa4f972efdcd6f4f3a. Not sure why that is though as I don't think those mounts should matter?

My guess is that there are just too many symlinks in the entire chain here:
» where distrobox-init
/etc/profiles/per-user/pschmitt/bin/distrobox-init
» ls -l /etc/profiles/per-user/pschmitt/bin/distrobox-init
lrwxrwxrwx 2 root root 80  1. Jan 1970  /etc/profiles/per-user/pschmitt/bin/distrobox-init -> /nix/store/gwmsb1lpc1a0jvb2dc6yf7g27lv0662a-home-manager-path/bin/distrobox-init
» ls -l /nix/store/gwmsb1lpc1a0jvb2dc6yf7g27lv0662a-home-manager-path/bin/distrobox-init
lrwxrwxrwx 2 root root 80  1. Jan 1970  /nix/store/gwmsb1lpc1a0jvb2dc6yf7g27lv0662a-home-manager-path/bin/distrobox-init -> /nix/store/cci7i1s0pfdnpwva68plrbspcnza37bv-distrobox-<IP_ADDRESS>/bin/distrobox-init
» ls -l /nix/store/cci7i1s0pfdnpwva68plrbspcnza37bv-distrobox-<IP_ADDRESS>/bin/distrobox-init
-r-xr-xr-x 20 root root 64968  1. Jan 1970  /nix/store/cci7i1s0pfdnpwva68plrbspcnza37bv-distrobox-<IP_ADDRESS>/bin/distrobox-init

The CRI might not handle this all too well, as in if the symlink does get bind-mounted directly into the container that would explain why it not having access to /nix makes this fail.

So it always mounts /run/current-system/sw/bin/distrobox-init as /usr/bin/entrypoint in the container rather than a specific store path which might eventually get garbage collected, no?

These paths wouldn't get gc'd away since the package would still be installed.
The following patch makes <IP_ADDRESS> usable:
diff --git a/distrobox-create b/distrobox-create
index 64e9911..21c707b 100755
--- a/distrobox-create
+++ b/distrobox-create
@@ -74,14 +74,15 @@ unshare_ipc=0
 unshare_netns=0
 unshare_process=0
 unshare_devsys=0
+
+self_dir="$(dirname "$(realpath "$0")")"
+nix_config_file="$(realpath "${self_dir}/../share/distrobox/distrobox.conf")"
+
 # Use cd + dirname + pwd so that we do not have relative paths in mount points
-# We're not using "realpath" here so that symlinks are not resolved this way
-# "realpath" would break situations like Nix or similar symlink based package
-# management.
-distrobox_entrypoint_path="$(cd "$(dirname "${0}")" && pwd)/distrobox-init"
-distrobox_export_path="$(cd "$(dirname "${0}")" && pwd)/distrobox-export"
-distrobox_genentry_path="$(cd "$(dirname "${0}")" && pwd)/distrobox-generate-entry"
-distrobox_hostexec_path="$(cd "$(dirname "${0}")" && pwd)/distrobox-host-exec"
+distrobox_entrypoint_path="${self_dir}/distrobox-init"
+distrobox_export_path="${self_dir}/distrobox-export"
+distrobox_genentry_path="${self_dir}/distrobox-generate-entry"
+distrobox_hostexec_path="${self_dir}/distrobox-host-exec"
 # In case some of the scripts are not in the same path as create, let's search
 # in PATH for them.
 [ ! -e "${distrobox_entrypoint_path}" ] && distrobox_entrypoint_path="$(command -v distrobox-init)"
@@ -98,6 +99,7 @@ version="<IP_ADDRESS>"
 # priority over system defaults
 # leave priority to environment variables.
 config_files="
+	$nix_config_file
 	/usr/share/distrobox/distrobox.conf
 	/usr/share/defaults/distrobox/distrobox.conf
 	/usr/etc/distrobox/distrobox.conf
diff --git a/distrobox-enter b/distrobox-enter
index 6d8998a..bb05437 100755
--- a/distrobox-enter
+++ b/distrobox-enter
@@ -75,10 +75,14 @@ skip_workdir=0
 verbose=0
 version="<IP_ADDRESS>"

+self_dir="$(dirname "$(realpath "$0")")"
+nix_config_file="$(realpath "${self_dir}/../share/distrobox/distrobox.conf")"
+
 # Source configuration files, this is done in an hierarchy so local files have
 # priority over system defaults
 # leave priority to environment variables.
 config_files="
+	$nix_config_file
 	/usr/share/distrobox/distrobox.conf
 	/usr/share/defaults/distrobox/distrobox.conf
 	/usr/etc/distrobox/distrobox.conf
diff --git a/distrobox-generate-entry b/distrobox-generate-entry
index 3243691..6a7910a 100755
--- a/distrobox-generate-entry
+++ b/distrobox-generate-entry
@@ -45,10 +45,14 @@ icon_default="${HOME}/.local/share/icons/terminal-distrobox-icon.svg"
 verbose=0
 version="<IP_ADDRESS>"

+self_dir="$(dirname "$(realpath "$0")")"
+nix_config_file="$(realpath "${self_dir}/../share/distrobox/distrobox.conf")"
+
 # Source configuration files, this is done in an hierarchy so local files have
 # priority over system defaults
 # leave priority to environment variables.
 config_files="
+	$nix_config_file
 	/usr/share/distrobox/distrobox.conf
 	/usr/share/defaults/distrobox/distrobox.conf
 	/usr/etc/distrobox/distrobox.conf
diff --git a/distrobox-list b/distrobox-list
index aaec85e..235e529 100755
--- a/distrobox-list
+++ b/distrobox-list
@@ -44,10 +44,14 @@ verbose=0
 version="<IP_ADDRESS>"
 container_manager="autodetect"

+self_dir="$(dirname "$(realpath "$0")")"
+nix_config_file="$(realpath "${self_dir}/../share/distrobox/distrobox.conf")"
+
 # Source configuration files, this is done in an hierarchy so local files have
 # priority over system defaults
 # leave priority to environment variables.
 config_files="
+	$nix_config_file
 	/usr/share/distrobox/distrobox.conf
 	/usr/share/defaults/distrobox/distrobox.conf
 	/usr/etc/distrobox/distrobox.conf
diff --git a/distrobox-rm b/distrobox-rm
index 702c1dd..2e37538 100755
--- a/distrobox-rm
+++ b/distrobox-rm
@@ -54,10 +54,14 @@ rm_home=0
 response_rm_home="N"
 version="<IP_ADDRESS>"

+self_dir="$(dirname "$(realpath "$0")")"
+nix_config_file="$(realpath "${self_dir}/../share/distrobox/distrobox.conf")"
+
 # Source configuration files, this is done in an hierarchy so local files have
 # priority over system defaults
 # leave priority to environment variables.
 config_files="
+	$nix_config_file
 	/usr/share/distrobox/distrobox.conf
 	/usr/share/defaults/distrobox/distrobox.conf
 	/usr/etc/distrobox/distrobox.conf
diff --git a/distrobox-stop b/distrobox-stop
index fd17cc1..e0dbc8f 100755
--- a/distrobox-stop
+++ b/distrobox-stop
@@ -52,10 +52,14 @@ non_interactive=0
 verbose=0
 version="<IP_ADDRESS>"

+self_dir="$(dirname "$(realpath "$0")")"
+nix_config_file="$(realpath "${self_dir}/../share/distrobox/distrobox.conf")"
+
 # Source configuration files, this is done in an hierarchy so local files have
 # priority over system defaults
 # leave priority to environment variables.
 config_files="
+	$nix_config_file
 	/usr/share/distrobox/distrobox.conf
 	/usr/share/defaults/distrobox/distrobox.conf
 	/usr/etc/distrobox/distrobox.conf
diff --git a/distrobox-upgrade b/distrobox-upgrade
index ab5e96f..dc8d295 100755
--- a/distrobox-upgrade
+++ b/distrobox-upgrade
@@ -39,10 +39,14 @@ rootful=0
 verbose=0
 version="<IP_ADDRESS>"

+self_dir="$(dirname "$(realpath "$0")")"
+nix_config_file="$(realpath "${self_dir}/../share/distrobox/distrobox.conf")"
+
 # Source configuration files, this is done in an hierarchy so local files have
 # priority over system defaults
 # leave priority to environment variables.
 config_files="
+	$nix_config_file
 	/usr/share/distrobox/distrobox.conf
 	/usr/share/defaults/distrobox/distrobox.conf
 	/usr/etc/distrobox/distrobox.conf

{ stdenvNoCC, lib, fetchFromGitHub, makeWrapper, wget }:

stdenvNoCC.mkDerivation (finalAttrs: {
  pname = "distrobox";
  version = "<IP_ADDRESS>";

  src = fetchFromGitHub {
    owner = "89luca89";
    repo = finalAttrs.pname;
    rev = finalAttrs.version;
    hash = "sha256-UWrXpb20IHcwadPpwbhSjvOP1MBXic5ay+nP+OEVQE4=";
  };

  dontConfigure = true;
  dontBuild = true;

  nativeBuildInputs = [ makeWrapper ];

  patches = [ ./resolve-symlinks.patch ];

  installPhase = ''
    runHook preInstall

    # https://github.com/89luca89/distrobox/issues/408
    substituteInPlace ./distrobox-generate-entry \
      --replace 'icon_default="''${HOME}/.local' "icon_default=\"$out"
    ./install -P $out

    runHook postInstall
  '';

  # https://github.com/89luca89/distrobox/issues/407
  postFixup = ''
    wrapProgram "$out/bin/distrobox-generate-entry" \
      --prefix PATH ":" ${lib.makeBinPath [ wget ]}

    mkdir -p $out/share/distrobox
    echo 'container_additional_volumes="/nix:/nix"' > $out/share/distrobox/distrobox.conf
  '';

  meta = with lib; {
    description = "Wrapper around podman or docker to create and start containers";
    longDescription = ''
      Use any linux distribution inside your terminal. Enable both backward and
      forward compatibility with software and freedom to use whatever distribution
      you’re more comfortable with
    '';
    homepage = "https://distrobox.privatedns.org/";
    license = licenses.gpl3Only;
    platforms = platforms.linux;
    maintainers = with maintainers; [ atila ];
  };
})

The quickest win for the nixpkg might be to ship a static config that enables mouting of /nix in all the containers.

Appears to work fine from brief testing after applying your changes. Thanks!

Please also change meta.homepage from https://distrobox.privatedns.org/ to https://distrobox.it/ . Currently the former is a redirect page.

Replaced finalAttrs.pname with distrobox for the repo name as per this suggestion from the other PR.

Fell out of sight for me, but yeah, let's do this


websocket connected,ios app crash run in background 180second

2020-12-08 18:31:42.368421+0800 thash[2061:928409] dnssd_clientstub read_all(8) DEFUNCT
2020-12-08 18:31:42.368613+0800 thash[2061:928409] [VERBOSE-2:FlutterObservatoryPublisher.mm(160)] Could not register as server for FlutterObservatoryPublisher. Check your network settings and relaunch the application.
Message from debugger: Terminated due to signal 13

m/public/Thash/notice4d1e07738bf84f9da65b9c34f45b43f2.png
2020-12-09 09:22:02.858115+0800 thash[2162:1075193] dnssd_clientstub read_all(8) DEFUNCT
2020-12-09 09:22:02.858239+0800 thash[2162:1075193] [VERBOSE-2:FlutterObservatoryPublisher.mm(160)] Could not register as server for FlutterObservatoryPublisher. Check your network settings and relaunch the application.
Message from debugger: Terminated due to signal 13

 [VERBOSE-2:profiler_metrics_ios.mm(184)] Error retrieving thread information: (ipc/send) invalid destination port

  dnssd_clientstub read_all(8) DEFUNCT
2020-12-09 14:59:25.083276+0800 thash[2251:1147117] [VERBOSE-2:FlutterObservatoryPublisher.mm(160)] Could not register as server for FlutterObservatoryPublisher. Check your network settings and relaunch the application.
Message from debugger: Terminated due to signal 13

Hi @deboLing
Can you please provide your flutter doctor -v and a minimal complete reproducible code sample?
Thank you

Hi @deboLing
Are you using any third-party packages? Can you please provide your flutter doctor -v your flutter run --verbose and a complete reproducible minimal code sample
Thank you

Without additional information, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now.
Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact description of what happens, logs, and the output of flutter doctor -v.
All system setups can be slightly different, so it's always better to open new issues and reference related issues.


Fix-binaries

Codecov Report

Merging #16 (447ed5c) into main (a6b3182) will not change coverage.
The diff coverage is 100.0%.

Additional details and impacted files

Files
Coverage Δ

src/lib.rs
96.6% <100.0%> (ø)


db_hdf5_open error

Describe the bug
A clear and concise description of what the bug is.
Attempted to open globe.silo (from visit data directory) with VisIt running in serial mode and i am getting the error: db_hd5_Open: specified driver cannot open this file. Error appears in the engine_ser.5.vlog when using visit -debug 5
When opening the file, all i see is a white screen. The error was found by using the debug flag to generate logs.
The OS running as RHEL7. Machines are virtual and accessed via remote desktop and XRDP. Error was reproduced with 24 bit and 32 bit color settings in remote desktop. HDF5 is not native to the system, so that could be a cause.

Which version of VisIt?
Did you install a pre-built version of VisIt, or did you build VisIt yourself from sources using the build_visit script?
If you built it yourself, what was your build_visit command line?
It may be the case that the options you used with build_visit didn't include hdf5, which means silo was built without the hdf5 driver, and that would explain the error.

@bigas,
We are using VisIt 3.1.4. It was the pre-built version from the VisIt website. I did write the custom launcher, and confirmed that this issue existed for the serial version. I suspect we are likely missing a system dependency since it's an isolated lab, I just would love some help determining what the dependency was...

Does the installation of VisIt include 'libhdf5.so' in the 'lib' directory?
You could call 'ldd' on that library to see if a dependency cannot be found.
In your custom launcher do you modify LD_LIBRARY_PATH at all?

We do have libhdf5.so in the lib directory. The custom launcher does add intel mpi to the LD_LIBRARY_PATH and lntelMPI Libfabric (which always had shown up as missing...
VisIt 3.1.4 does work fine in our standard environment - we are just seeing this error in one of the labs...

@markcmiller86 Intel MPI should not be causing this issue - we ran in serial and took hosts out of the mix. This is what we are seeing:
    libsiloh5.so => not found
    libsiloh5.so => not found
    libsiloh5.so => not found

Erin

Do you know which binary was downloaded for install on that machine?

Also,  is 'libsiloh5.so' in the VisIt installation lib directory, or is it just 'libsilo.so'

@biagas libsiloh5.so is in the Visit lib directory. Also interestingly enough - VisIt 3.1.2 works beautifully. VisIt 3.1.4 will not load the host profiles... (3.1.4 was a port of 3.1.2 as far as the setup went on our end...). I would have downloaded the VisIt binary for RHEL7...

(3.1.4 was a port of 3.1.2 as far as the setup went on our end...).

Not quite sure what you meant by that statement.
The install script used for installing from pre-built changes (possibly) with each version of VisIt, so you should always use the visit-install script that matches the version being installed.
When your internal launcher sets LD_LIBRARY_PATH, are you appending the new values? Because VisIt itself sets some values. Just want to make sure VisIt's values aren't being overwritten.

Ok - so by port - the custom launcher and host profiles for 3.1.2 were used as a starting point for 3.1.4. Each install is a fresh one and we typically just open the tarballs that are downloaded from the VisIt site.

When we set the LD_LIBRARY_PATH we prepend - to get libfabric and libmpi. The VisIt values are captured and placed at the end.
lib_fabric_path = ["/apps/intel_oneapi_hpc_toolkit_2021.2/mpi/latest/libfabric/lib/prov:/apps/intel_oneapi_hpc_toolkit_2021.2/mpi/latest/libfabric/lib"]
17         mpi_ld_library_path = ["/apps/intel_oneapi_hpc_toolkit_2021.2/mpi/latest/lib/release:/apps/intel_oneapi_hpc_toolkit_2021.2/mpi/latest/lib"]
16         ld_path = self.splitpaths(GETENV("LD_LIBRARY_PATH"))
15         mpi_path = "/apps/intel_oneapi_hpc_toolkit_2021.2/mpi/latest/bin"
14         SETENV("PATH",self.joinpaths([mpi_path, GETENV("PATH")]))
13         SETENV("LD_LIBRARY_PATH",self.joinpaths(lib_fabric_path + mpi_ld_library_path + ld_path ))

This code snippet worked fine in 3.1.2

So - i've taken the custom launcher out of the picture (moved it to .customlauncher) and have tried to load host profiles and that isn't working...

I've looked at source code differences between 3.1.2 and 3.1.4, and nothing of significance jumps out at me that would cause the problem.
Can you try appending the intel additions to LD_LIBRARY_PATH instead of prepending. See if that helps?

@biagas  - i've removed the custom launcher entirely. The fact that 3.1.4 isn't copying host files over concerns me...

The fact that 3.1.4 isn't copying host files over concerns me...

Are you referring to the installation of host profiles during the install of VisIt?  Those are generally only profiles that have been added to our repository.
If you are referring to host profiles you created by hand for 3.1.2:  where did you 'install' them?
Generally host profiles not installed by VisIt can be stored in ~/.visit/hosts for use by any version of VisIt that is run on that system.  If you placed them in the 3.1.2 install location, then 3.1.4 won't know about them.

We build our own and update the networks.dat etc. files to show them. They are in a location that 3.1.4 should know about....

HDF5 errors:  my only thought is to use ldd on the siloh5.so library and see what is listed as 'not found'.
temporarily set LD_LIBRARY_PATH to point to visit's lib directory before running ldd.
Then try ldd on any dependency of libsiloh5.so.

We build our own and update the networks.dat etc. files to show them. They are in a location that 3.1.4 should know about....

networks.dat isn't read by VisIt when it is being run, it is used during creation of the installer, to inform the installer of which host-profiles to offer during installation.  VisIt only searches for host profiles in its own installation or in ~/.visit/hosts.

@biagas  - OK - i've taken the custom launcher out of the picture, and am ignoring the host issue for now. We still see the failure to load with the library error - so will work that issue first. How can i export LD_LIBRARY_PATH to the screen from Visit. Does it appear in one of the debug logs?

If you run 'visit -env' it will print out the environment variables it sets up, including LD_LIBRARY_PATH.

Interestingly enough, VISIT_MESA_LIB is not getting set in the mesa version of the code.

VISIT_MESA_LIB is only set when VisIt is using OSMesa substitute for Mesa/GL. It should show up if you run 'visit  -nowin -env'.
AFAICT, the LD_LIBRARY_PATH looks fine.
Again, I suggest you start looking at any NOT FOUND listings in the results of 'ldd' for  libsiloh5.so, and libhdf5.so in /apps/visit_3.1.4-wmesa/3.1.4-linux-x86_64/lib.

OK - when i add the path specifically to LD_LIBRARY_PATH everything is found. When run through VisIt - LD_LIBRARY_PATH does not seem to be updating correctly

FYI - we are seeing this with all of our VisIt versions at this point - so i am thinking there may be a patch or driver update issue - but i do need to identify the problematic piece


[FLINK-5714] [table] Use a builder pattern for creating CsvTableSource

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

[x] General

The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
The pull request addresses only one issue
Each commit in the PR has a meaningful commit message (including the JIRA id)

[x] Documentation

Documentation has been added for new functionality
Old documentation affected by the pull request has been updated
JavaDoc for public methods has been added

[x] Tests & Build

Functionality added by the pull request is covered by tests
mvn clean verify has been executed successfully locally or a Travis build has passed

Add a builder to create CsvTableSource. And I have also updated the documentation.

Thank you for your advice. I will update the PR soon.

updated!

Looks good to merge. Merging...

I found another issue: We should use a LinkedHashMap otherwise the order of fields and types is not guaranteed. I fixed it and will merge this now.


Codelens for interactive window show in notebook

Environment data

VS Code: Codespace
Jupyter Extension version (available under the Extensions sidebar): v2022.3.1000851004

Seems to be caused by changes in VS Code, we can no longer use a Document Filter to exclude notebook document cells.
Seems that all notebook document cells are automatically included

Yeah, that was the change I was wondering about. Rich had to update intellisense for that change. I guess we just missed this filter.

This should be fixed.

Fixed and code has been mereged

Verified. Looks good now.


Unable to sync data on first time

[root@dev ~]# mysqld -V /usr/sbin/mysqld  Ver 8.0.39-30 for Linux on x86_64 (Percona Server (GPL), Release 30, Revision 41ebc5d9) [root@dev ~]# clickhouse-server -V ClickHouse server version <IP_ADDRESS>.altinitystable (altinity build).
I use Percona and when run sync first time, it show like this
[root@dev ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db db_source(inMySQL) --target_db db_dst(inClickHouse) --initial_only=True Traceback (most recent call last):   File "/usr/local/bin/mysql_ch_replicator", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main     run_db_replicator(args, config)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 78, in run_db_replicator     db_replicator = DbReplicator(   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 108, in __init__     self.mysql_api = MySQLApi(   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/mysql_api.py", line 15, in __init__     self.reconnect_if_required()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/mysql_api.py", line 25, in reconnect_if_required     self.db = mysql.connector.connect(   File "/usr/local/lib64/python3.9/site-packages/mysql/connector/pooling.py", line 322, in connect     return CMySQLConnection(*args, **kwargs)   File "/usr/local/lib64/python3.9/site-packages/mysql/connector/connection_cext.py", line 151, in __init__     self.connect(**kwargs)   File "/usr/local/lib64/python3.9/site-packages/mysql/connector/abstracts.py", line 1426, in connect     self._open_connection()   File "/usr/local/lib64/python3.9/site-packages/mysql/connector/connection_cext.py", line 335, in _open_connection     self._cmysql.connect(**cnx_kwargs) TypeError: argument 3 must be str or None, not int

It looks like you don't use quotes for mysql password in config file:
This is wrong:
mysql:
  host: 'localhost'
  port: 9306
  user: 'root'
  password: 12345

This is correct:
mysql:
  host: 'localhost'
  port: 9306
  user: 'root'
  password: '12345'

I will add validation and more meaningful error message.

My config is
`mysql:
host: 'localhost'
port: 3306
user: 'ch_rep'
password: '123'
clickhouse:
host: 'localhost'
port: 8123
user: 'default'
password: 'clickhouse'
binlog_replicator:
data_dir: '/var/data/mysql/'
records_per_file: 100000
databases: 'database(InMySQL'
tables: 'source_table(inMySQL)`
After run the scipt
`[root@dev ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db regina --target_db regina_clickhouse --initial_only=True
Traceback (most recent call last):
File "/usr/local/lib64/python3.9/site-packages/mysql/connector/connection_cext.py", line 335, in _open_connection
self._cmysql.connect(**cnx_kwargs)
_mysql_connector.MySQLInterfaceError: Access denied for user 'ch_rep'@'::1' (using password: YES)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/mysql_ch_replicator", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main
run_db_replicator(args, config)
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 78, in run_db_replicator
db_replicator = DbReplicator(
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 108, in init
self.mysql_api = MySQLApi(
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/mysql_api.py", line 15, in init
self.reconnect_if_required()
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/mysql_api.py", line 25, in reconnect_if_required
self.db = mysql.connector.connect(
File "/usr/local/lib64/python3.9/site-packages/mysql/connector/pooling.py", line 322, in connect
return CMySQLConnection(*args, **kwargs)
File "/usr/local/lib64/python3.9/site-packages/mysql/connector/connection_cext.py", line 151, in init
self.connect(**kwargs)
File "/usr/local/lib64/python3.9/site-packages/mysql/connector/abstracts.py", line 1426, in connect
self._open_connection()
File "/usr/local/lib64/python3.9/site-packages/mysql/connector/connection_cext.py", line 340, in _open_connection
raise get_mysql_exception(
mysql.connector.errors.ProgrammingError: 1045 (28000): Access denied for user 'ch_rep'@'::1' (using password: YES)`
The user ch_rep able to access to MySQL
[root@dev ~]# mysql -u ch_rep -p Enter password: Welcome to the MySQL monitor.  Commands end with ; or \g. Your MySQL connection id is 18 Server version: 8.0.39-30 Percona Server (GPL), Release 30, Revision 41ebc5d9

Please try to set:
host: '<IP_ADDRESS>'
Probably user doesn't have permissions to connect with IPv6 address, and by default localhost resolving to IPv6

<IP_ADDRESS> is work @bakwc
Not sure if i need to open new thread or follow here
i have a table in MySQL then i sync it to Clickhouse and with Engine is MySQL, after i create another database in CH and create a table with CH structure with ENGINE MergeTree.
I do insert from the table with MySQL on CH to the table on different database on CH with ENGINE MergeTree, it works perfect
I test with mysql_ch_replicator then it show error ==> Exception: unknown mysql type "tinyint"
[root@dev ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db db(inMySQL) --target_db db(InCH) --initial_only=True [dbrepl regina 2024-10-30 14:58:40,488     INFO] launched db_replicator [dbrepl regina 2024-10-30 14:58:40,488     INFO] recreating database [dbrepl regina 2024-10-30 14:58:40,527     INFO] last known transaction None [dbrepl regina 2024-10-30 14:58:40,593    ERROR] unhandled exception Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 150, in run     self.create_initial_structure()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 160, in create_initial_structure     self.create_initial_structure_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 170, in create_initial_structure_table     clickhouse_structure = self.converter.convert_table_structure(mysql_structure)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 119, in convert_table_structure     clickhouse_field_type = self.convert_field_type(field.field_type, field.parameters)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 110, in convert_field_type     clickhouse_type = self.convert_type(mysql_type)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 104, in convert_type     raise Exception(f'unknown mysql type "{mysql_type}"') Exception: unknown mysql type "tinyint" Traceback (most recent call last):   File "/usr/local/bin/mysql_ch_replicator", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main     run_db_replicator(args, config)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 84, in run_db_replicator     db_replicator.run()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 150, in run     self.create_initial_structure()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 160, in create_initial_structure     self.create_initial_structure_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 170, in create_initial_structure_table     clickhouse_structure = self.converter.convert_table_structure(mysql_structure)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 119, in convert_table_structure     clickhouse_field_type = self.convert_field_type(field.field_type, field.parameters)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 110, in convert_field_type     clickhouse_type = self.convert_type(mysql_type)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 104, in convert_type     raise Exception(f'unknown mysql type "{mysql_type}"') Exception: unknown mysql type "tinyint"
I already create the table structure with CH standard, do i need to drop it then the script auto create or ?
Any suggest?

The problem was due to missing tinyint type support. I added tinyint, please upgrade mysql_ch_replicator (pip install --upgrade mysql_ch_replicator) and try again
No need to create structure manually, mysql_ch_replicator will create structure himself (existing tables will be removed)

Could you please share the full exception?

[root@abh-dev04 ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db regina --target_db regina_clickhouse --initial_only=True [dbrepl regina 2024-10-30 15:28:22,353     INFO] launched db_replicator [dbrepl regina 2024-10-30 15:28:22,353     INFO] recreating database [dbrepl regina 2024-10-30 15:28:22,376     INFO] last known transaction None [dbrepl regina 2024-10-30 15:28:22,430     INFO] running initial replication [dbrepl regina 2024-10-30 15:28:22,430     INFO] running initial replication for table employee_positions [dbrepl regina 2024-10-30 15:28:22,431     INFO] replicating from scratch [dbrepl regina 2024-10-30 15:28:24,518    ERROR] Error serializing column modified_dateinto data typeDateTime64(3) Traceback (most recent call last):   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/transform.py", line 104, in chunk_gen     col_type.write_column(data, output, context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 209, in write_column     self.write_column_data(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 224, in write_column_data     self._write_column_binary(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in _write_column_binary     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column]   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in <listcomp>     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column] AttributeError: 'NoneType' object has no attribute 'timestamp' [dbrepl regina 2024-10-30 15:28:24,533    ERROR] unhandled exception Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 151, in run     self.perform_initial_replication()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 193, in perform_initial_replication     self.perform_initial_replication_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in perform_initial_replication_table     self.clickhouse_api.insert(table_name, records)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/clickhouse_api.py", line 144, in insert     self.client.insert(table=full_table_name, data=records_to_insert)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 625, in insert     return self.data_insert(context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 269, in data_insert     response = self._raw_request(block_gen, params, headers, error_handler=error_handler, server_wait=False)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 459, in _raw_request     error_handler(response)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 254, in error_handler     raise ex   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/transform.py", line 104, in chunk_gen     col_type.write_column(data, output, context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 209, in write_column     self.write_column_data(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 224, in write_column_data     self._write_column_binary(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in _write_column_binary     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column]   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in <listcomp>     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column] AttributeError: 'NoneType' object has no attribute 'timestamp' Traceback (most recent call last):   File "/usr/local/bin/mysql_ch_replicator", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main     run_db_replicator(args, config)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 84, in run_db_replicator     db_replicator.run()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 151, in run     self.perform_initial_replication()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 193, in perform_initial_replication     self.perform_initial_replication_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in perform_initial_replication_table     self.clickhouse_api.insert(table_name, records)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/clickhouse_api.py", line 144, in insert     self.client.insert(table=full_table_name, data=records_to_insert)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 625, in insert     return self.data_insert(context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 269, in data_insert     response = self._raw_request(block_gen, params, headers, error_handler=error_handler, server_wait=False)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 459, in _raw_request     error_handler(response)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 254, in error_handler     raise ex   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/transform.py", line 104, in chunk_gen     col_type.write_column(data, output, context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 209, in write_column     self.write_column_data(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 224, in write_column_data     self._write_column_binary(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in _write_column_binary     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column]   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in <listcomp>     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column] AttributeError: 'NoneType' object has no attribute 'timestamp'

Thank you, exception related to DateTime handling. We support DateTime, but probably there are some non supported values, investigating.

Prepared the fix, please update mysql_ch_replicator and try again.

When i ran the script first time, it create a table xxx_tmp but the script i point the db to right db that i already create.
I drop the xxx_tmp db and update the final one and it fail because it unable to find xxx_tmp db.
It should write to target_db on the script, right? and i believed it cache last run somewhere to cause this issues
`[root@abh-dev04 ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db regina --target_db regina_clickhouse --initial_only=True
[dbrepl regina 2024-10-30 16:35:53,626     INFO] launched db_replicator
[dbrepl regina 2024-10-30 16:35:53,627     INFO] running initial replication
[dbrepl regina 2024-10-30 16:35:53,627     INFO] running initial replication for table employee_positions
[dbrepl regina 2024-10-30 16:35:53,627     INFO] continue from primary key None
[dbrepl regina 2024-10-30 16:35:55,499    ERROR] unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 136, in run
self.perform_initial_replication()
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 193, in perform_initial_replication
self.perform_initial_replication_table(table)
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in perform_initial_replication_table
self.clickhouse_api.insert(table_name, records)
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/clickhouse_api.py", line 144, in insert
self.client.insert(table=full_table_name, data=records_to_insert)
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 614, in insert
context = self.create_insert_context(table,
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 713, in create_insert_context
describe_result = self.query(f'DESCRIBE TABLE {full_table}')
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 218, in query
return self._query_with_context(query_context)
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 228, in _query_with_context
response = self._raw_request(body,
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 461, in _raw_request
self._error_handler(response)
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 384, in _error_handler
raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
clickhouse_connect.driver.exceptions.DatabaseError: HTTPDriver for http://localhost:8123 received ClickHouse error code 81
Code: 81. DB::Exception: Database regina_clickhouse_tmp does not exist. Maybe you meant regina_clickhouse?. (UNKNOWN_DATABASE) (version <IP_ADDRESS>.altinitystable (altinity build))
Traceback (most recent call last):
File "/usr/local/bin/mysql_ch_replicator", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main
run_db_replicator(args, config)
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 84, in run_db_replicator
db_replicator.run()
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 136, in run
self.perform_initial_replication()
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 193, in perform_initial_replication
self.perform_initial_replication_table(table)
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in perform_initial_replication_table
self.clickhouse_api.insert(table_name, records)
File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/clickhouse_api.py", line 144, in insert
self.client.insert(table=full_table_name, data=records_to_insert)
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 614, in insert
context = self.create_insert_context(table,
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 713, in create_insert_context
describe_result = self.query(f'DESCRIBE TABLE {full_table}')
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 218, in query
return self._query_with_context(query_context)
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 228, in _query_with_context
response = self._raw_request(body,
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 461, in _raw_request
self._error_handler(response)
File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 384, in _error_handler
raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
clickhouse_connect.driver.exceptions.DatabaseError: HTTPDriver for http://localhost:8123 received ClickHouse error code 81
Code: 81. DB::Exception: Database regina_clickhouse_tmp does not exist. Maybe you meant regina_clickhouse?. (UNKNOWN_DATABASE) (version <IP_ADDRESS>.altinitystable (altinity build))`

It will remove tmp database itself when finish replication. It works following way:

Replicate everything into a _tmp database
Rename _tmp database into a main database.
(it's done this way to prevent removing main database in production).

You need to do following:

Remove binlog_replicator data_dir on your file system.
Run script the second time. It will start from scratch instead of continue from saved position.

Thanks @bakwc for explanation
I unable to find binlog_replicator
[root@abh-dev04 ~]# find / -name binlog_replicator* /usr/local/lib/python3.9/site-packages/mysql_ch_replicator/binlog_replicator.py /usr/local/lib/python3.9/site-packages/mysql_ch_replicator/__pycache__/binlog_replicator.cpython-39.pyc [root@abh-dev04 ~]# find / -name *binlog_replicator* /usr/local/lib/python3.9/site-packages/mysql_ch_replicator/binlog_replicator.py /usr/local/lib/python3.9/site-packages/mysql_ch_replicator/__pycache__/binlog_replicator.cpython-39.pyc

data_dir: '/var/data/mysql/'
records_per_file: 100000```

It looks like it is '/var/data/mysql/'
I would propose to use another directory for the script, better dont mix it with mysql.

I though we need to point to MySQL foldr where to store the bin. Already fix and another issues come out
[root@abh-dev04 ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db regina --target_db regina_clickhouse --initial_only=True [dbrepl regina 2024-10-30 18:24:51,846     INFO] launched db_replicator [dbrepl regina 2024-10-30 18:24:51,847     INFO] recreating database [dbrepl regina 2024-10-30 18:24:51,861     INFO] last known transaction None [dbrepl regina 2024-10-30 18:24:51,906     INFO] running initial replication [dbrepl regina 2024-10-30 18:24:51,906     INFO] running initial replication for table employee_positions [dbrepl regina 2024-10-30 18:24:51,907     INFO] replicating from scratch [dbrepl regina 2024-10-30 18:24:53,923    ERROR] Error serializing column modified_dateinto data typeDateTime64(3) Traceback (most recent call last):   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/transform.py", line 104, in chunk_gen     col_type.write_column(data, output, context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 209, in write_column     self.write_column_data(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 224, in write_column_data     self._write_column_binary(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in _write_column_binary     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column]   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in <listcomp>     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column] AttributeError: 'NoneType' object has no attribute 'timestamp' [dbrepl regina 2024-10-30 18:24:53,926    ERROR] unhandled exception Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 151, in run     self.perform_initial_replication()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 193, in perform_initial_replication     self.perform_initial_replication_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in perform_initial_replication_table     self.clickhouse_api.insert(table_name, records)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/clickhouse_api.py", line 144, in insert     self.client.insert(table=full_table_name, data=records_to_insert)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 625, in insert     return self.data_insert(context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 269, in data_insert     response = self._raw_request(block_gen, params, headers, error_handler=error_handler, server_wait=False)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 459, in _raw_request     error_handler(response)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 254, in error_handler     raise ex   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/transform.py", line 104, in chunk_gen     col_type.write_column(data, output, context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 209, in write_column     self.write_column_data(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 224, in write_column_data     self._write_column_binary(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in _write_column_binary     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column]   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in <listcomp>     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column] AttributeError: 'NoneType' object has no attribute 'timestamp' Traceback (most recent call last):   File "/usr/local/bin/mysql_ch_replicator", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main     run_db_replicator(args, config)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 84, in run_db_replicator     db_replicator.run()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 151, in run     self.perform_initial_replication()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 193, in perform_initial_replication     self.perform_initial_replication_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in perform_initial_replication_table     self.clickhouse_api.insert(table_name, records)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/clickhouse_api.py", line 144, in insert     self.client.insert(table=full_table_name, data=records_to_insert)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 625, in insert     return self.data_insert(context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 269, in data_insert     response = self._raw_request(block_gen, params, headers, error_handler=error_handler, server_wait=False)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 459, in _raw_request     error_handler(response)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 254, in error_handler     raise ex   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/transform.py", line 104, in chunk_gen     col_type.write_column(data, output, context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 209, in write_column     self.write_column_data(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 224, in write_column_data     self._write_column_binary(column, dest, ctx)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in _write_column_binary     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column]   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 215, in <listcomp>     column = [((int(x.timestamp()) * 1000000 + x.microsecond) * prec) // 1000000 for x in column] AttributeError: 'NoneType' object has no attribute 'timestamp'

It was already fixed. Please check that you upgraded to the latest version (0.0.22)

another issues
TypeError: unsupported operand type(s) for -: 'NoneType' and 'datetime.date'

Please share the full stack trace.

Fixed this one in version 0.0.23, also added some simple test for all types you mentioned.
Please check, if other errors happen you could probably share somewhere a mysql_dump file with your date so I could load it and check myself everything works.
Or just report any other issues with stacktrace and I will fix them.

That table is works well @bakwc, i saw all records but yet to check in detail. I will test with few other complex table and keep you post if need.
On the other hand, I have few tables in my CH databases and after ran the script it auto delete other table.
May i suggest to point the destination db & table in the param so that if they dont exist, then create it, otherwise juts insert more data

I just changed to another table for syn and it said, no database. Of course i know how to fix because you did show me but just want to let you know, it should not work like that, right?
[root@abh-dev04 ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db regina --target_db regina_clickhouse --initial_only=True [dbrepl regina 2024-10-30 20:50:43,525     INFO] launched db_replicator [dbrepl regina 2024-10-30 20:50:43,526     INFO] running initial replication [dbrepl regina 2024-10-30 20:50:43,526     INFO] running initial replication for table employee_positions [dbrepl regina 2024-10-30 20:50:43,527     INFO] skip table employee_positions - not matching any allowed table [dbrepl regina 2024-10-30 20:50:43,527     INFO] continue from primary key 1297369 [dbrepl regina 2024-10-30 20:50:45,896    ERROR] unhandled exception Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 136, in run     self.perform_initial_replication()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 193, in perform_initial_replication     self.perform_initial_replication_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in perform_initial_replication_table     self.clickhouse_api.insert(table_name, records, table_structure=clickhouse_table_structure)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/clickhouse_api.py", line 155, in insert     self.client.insert(table=full_table_name, data=records_to_insert)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 614, in insert     context = self.create_insert_context(table,   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 713, in create_insert_context     describe_result = self.query(f'DESCRIBE TABLE {full_table}')   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/client.py", line 218, in query     return self._query_with_context(query_context)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 228, in _query_with_context     response = self._raw_request(body,   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 461, in _raw_request     self._error_handler(response)   File "/usr/local/lib64/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 384, in _error_handler     raise OperationalError(err_str) if retried else DatabaseError(err_str) from None clickhouse_connect.driver.exceptions.DatabaseError: HTTPDriver for http://localhost:8123 received ClickHouse error code 81  Code: 81. DB::Exception: Database regina_clickhouse_tmp does not exist. Maybe you meant regina_clickhouse?. (UNKNOWN_DATABASE) (version <IP_ADDRESS>.altinitystable (altinity build))

If you want that data being updated you need to run script in continuous mode:
mysql_ch_replicator --config config.yaml run_all
In this mode it will do following:

Make initial copy
Watch for changes (new inserts or deletes), and insert / delete all new changes automcatically
But in this mode you should not turn off script, you should keep it running.

I just changed to another table for syn and it said, no database.

Yeah, I will fix one. Currently it save state for the previous run and try to continue, but if you changed the table it should not try to continue but instead start a new one.

ee.zip
I think this is relate to the comment on the MySQL table with unicode
[root@abh-dev04 ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db regina --target_db regina_clickhouse --initial_only=True [dbrepl regina 2024-10-30 22:22:50,957     INFO] launched db_replicator [dbrepl regina 2024-10-30 22:22:50,957     INFO] recreating database [dbrepl regina 2024-10-30 22:22:50,973     INFO] last known transaction None [dbrepl regina 2024-10-30 22:22:51,180    ERROR] unhandled exception Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 170, in run     self.create_initial_structure()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 180, in create_initial_structure     self.create_initial_structure_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 190, in create_initial_structure_table     clickhouse_structure = self.converter.convert_table_structure(mysql_structure)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 121, in convert_table_structure     clickhouse_field_type = self.convert_field_type(field.field_type, field.parameters)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 112, in convert_field_type     clickhouse_type = self.convert_type(mysql_type)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 106, in convert_type     raise Exception(f'unknown mysql type "{mysql_type}"') Exception: unknown mysql type "dụ:" Traceback (most recent call last):   File "/usr/local/bin/mysql_ch_replicator", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main     run_db_replicator(args, config)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 84, in run_db_replicator     db_replicator.run()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 170, in run     self.create_initial_structure()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 180, in create_initial_structure     self.create_initial_structure_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 190, in create_initial_structure_table     clickhouse_structure = self.converter.convert_table_structure(mysql_structure)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 121, in convert_table_structure     clickhouse_field_type = self.convert_field_type(field.field_type, field.parameters)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 112, in convert_field_type     clickhouse_type = self.convert_type(mysql_type)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 106, in convert_type     raise Exception(f'unknown mysql type "{mysql_type}"') Exception: unknown mysql type "dụ:"

Fixed in 0.0.25

[root@abh-dev04 ~]# mysql_ch_replicator --config /etc/clickhouse-server/syn_mysql.yaml db_replicator --db regina --target_db regina_clickhouse --initial_only=True [dbrepl regina 2024-10-31 06:31:29,661     INFO] launched db_replicator [dbrepl regina 2024-10-31 06:31:29,661     INFO] recreating database [dbrepl regina 2024-10-31 06:31:29,683     INFO] last known transaction None [dbrepl regina 2024-10-31 06:31:30,254    ERROR] unhandled exception Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 170, in run     self.create_initial_structure()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 180, in create_initial_structure     self.create_initial_structure_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 190, in create_initial_structure_table     clickhouse_structure = self.converter.convert_table_structure(mysql_structure)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 130, in convert_table_structure     clickhouse_field_type = self.convert_field_type(field.field_type, field.parameters)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 121, in convert_field_type     clickhouse_type = self.convert_type(mysql_type)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 115, in convert_type     raise Exception(f'unknown mysql type "{mysql_type}"') Exception: unknown mysql type "varbinary(128)" Traceback (most recent call last):   File "/usr/local/bin/mysql_ch_replicator", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 120, in main     run_db_replicator(args, config)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/main.py", line 84, in run_db_replicator     db_replicator.run()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 170, in run     self.create_initial_structure()   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 180, in create_initial_structure     self.create_initial_structure_table(table)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/db_replicator.py", line 190, in create_initial_structure_table     clickhouse_structure = self.converter.convert_table_structure(mysql_structure)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 130, in convert_table_structure     clickhouse_field_type = self.convert_field_type(field.field_type, field.parameters)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 121, in convert_field_type     clickhouse_type = self.convert_type(mysql_type)   File "/usr/local/lib/python3.9/site-packages/mysql_ch_replicator/converter.py", line 115, in convert_type     raise Exception(f'unknown mysql type "{mysql_type}"') Exception: unknown mysql type "varbinary(128)"

For the last one - could you please share what is the data type of payment_period in MySQL (mysql create statement for the whole table)?

Fixed in 0.0.26, there was an issue handling unsigned numeric types.

It works for now. I will try to sync all tables and see how it goes.
One more thing, i feel the migration speed is abit slow

One more thing, i feel the migration speed is abit slow

Yeah, the issue is mysql slowly extracting data (even providing pretty large batches, eg. 50k entries per batch).
But if you use it in continuous mode it will use binlog and have better performance (still initial replication will be slow).

Closing this one as initial replication works. Let me know if any other issues appears.