Draft: oneVPL for intel gpu

Description of changes
I got an Intel Arc dedicated gpu and want to test hardware decoding with ffmpeg. This requires using libvpl instead of libmfx to interface with hardware, and is a configure flag in ffmpeg (newer sources than currently in nixpkgs). I didn't get ffmpeg to build with the new configure flag, due to a seemingly unrelated error during ffmpeg's configurePhase:
ERROR: cuvid requested, but not all dependencies are satisfied: ffnvcodec
What is here, is intel's new dispatch library, libvpl, and the library that should be loaded by libvpl when an Arc discrete gpu is detected: oneVPL-intel-gpu. It's marked Draft until I can test with an updated ffmpeg.
I also have some TODO comments I'm hoping to get feedback on, which will be much appreciated
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

amazing work, thanks for taking a look! I'll be back home next week but until then I don't have time to test.

changed the PR title to oneVPL from libvpl, and I intend on changing the package name as well

These are the derivations and overlays I currently use as part of my NixOS configuration. The oneVPL repos are non-flake inputs to my configuration flake.

My oneVPL-intel-gpu derivation:
{
  lib,
  stdenv,
  cmake,
  pkg-config,
  gtest,
  libdrm,
  libva,
  self,
}:
stdenv.mkDerivation rec {
  pname = "onevpl-intel-gpu";
  version = "custom";
  src = self.inputs.onevpl-intel-gpu;

  nativeBuildInputs = [cmake pkg-config];
  buildInputs = [
    libdrm
    libva
  ];
  nativeCheckInputs = [gtest];

  cmakeFlags = [
    "-DBUILD_TESTS=${
      if doCheck
      then "ON"
      else "OFF"
    }"
    "-DUSE_SYSTEM_GTEST=ON"
  ];

  doCheck = true;

  meta = with lib; {
    description = "Intel oneVPL GPU Runtime";
    license = licenses.mit;
    platforms = ["x86_64-linux"];
  };
}

My oneVPL derivation:
{
  self,
  lib,
  stdenv,
  cmake,
  pkg-config,
  gtest,
  libdrm,
  libffi,
  libpciaccess,
  libva,
  libX11,
  libXau,
  libXdmcp,
  wayland,
  wayland-protocols,
  ...
}:
stdenv.mkDerivation rec {
  pname = "onevpl";
  version = "custom";
  src = self.inputs.onevpl;

  nativeBuildInputs = [cmake pkg-config];
  buildInputs = [
    libdrm
    libffi
    libpciaccess
    libva
    libX11
    libXau
    libXdmcp
    wayland
    wayland-protocols
  ];
  nativeCheckInputs = [gtest];

  cmakeFlags = [
    "-DBUILD_TESTS=${
      if doCheck
      then "ON"
      else "OFF"
    }"
    "-DUSE_SYSTEM_GTEST=ON"
    "-DINSTALL_EXAMPLE_CODE=OFF"
  ];

  doCheck = false;

  meta = with lib; {
    description = "Intel oneVPL GPU Runtime";
    license = licenses.mit;
    platforms = ["x86_64-linux"];
  };
}

My jellyfin-ffmpeg overlay:
final: prev: {
  jellyfin-ffmpeg = final.ffmpeg_6-full.overrideAttrs (finalAttrs: prevAttrs: rec {
    src = final.fetchFromGitHub {
      owner = "jellyfin";
      repo = "jellyfin-ffmpeg";
      rev = "v6.0-4";
      sha256 = "sha256-o0D/GWbSoy5onbYG29wTbpZ8z4sZ2s1WclGCXRMSekA=";
    };

    version = "custom";
    buildInputs = prevAttrs.buildInputs ++ [final.onevpl final.onevpl-intel-gpu];
    configureFlags = (final.lib.lists.remove "--enable-libmfx" prevAttrs.configureFlags) ++ ["--enable-libvpl"];
  });
  jellyfin = prev.jellyfin.override {
    ffmpeg = final.jellyfin-ffmpeg;
  };
}

I think I've got it working in Nixpkgs. Could you check the changes in my fork? I based those on this PR. See https://github.com/PJungkamp/nixpkgs/tree/onevpl
Feel free to pull those into your branch. I'm not quite settled on NixOS yet, so I won't fill myself in as a maintainer and thus won't create a PR myself.

@PJungkamp hey, thanks for documenting everything here and in that linked issue. I tried to use your fork, and it compiles successfully but I'm still getting the Device creation failed error. Anything else you might have set on your nixos config to get this all working (specifically with Jellyfin)?

@J-Swift thanks for sharing your findings! Because of the nuance with environment variables to actually get these libraries to work at runtime, I think I should also define a nixos module like hardware.intelGPU.enable = true; that sets these by default. That can be a follow up PR
I'm hoping to test myself this week, and finally mark this PR ready. I'm not sure if the ffmpeg maintainers will like having a change like this so close to 23.11 feature freeze, so I'll probably leave the mfx -> vpl change out of the ffmpeg package. @Atemu any thoughts?

The ffmpeg part of this is probably rather trivial. As long as it builds and still works for everything else, I don't see why we shouldn't add this to at least ffmpeg-full. Once it's been enabled there for a release, we can move it to regular ffmpeg should some other package (jellyfin?) require it. Closure size might be a concern at that point, have you tested that?

@evanrichter yeah there is some stuff in the nixos-hardware configs (https://github.com/NixOS/nixos-hardware/blob/master/common/cpu/intel/default.nix and https://github.com/NixOS/nixos-hardware/blob/master/common/gpu/intel/default.nix), I added my own jellyfin service wrapper to do as you describe. Here is what I've got for reference (intelDriverVersion is just a variable I'm using to indicate which kernel driver to use, but I've only verified it works with iHD locally):
{
  # https://jellyfin.org/docs/general/aministration/hardware-acceleration/intel/#linux-setups
  users.users."${config.services.jellyfin.user}".extraGroups = [ "render" ];

  hardware.opengl = {
    enable = true;
    extraPackages = with pkgs; [
      (if intelDriverVersion == "iHD" then intel-media-driver else interl-vaapi-driver)
      intel-ocl
      vaapiVdpau
      libvdpau-va-gl
    ];
  };

  boot.initrd.kernelModules = [ "i915" ];
  environment.sessionVariables = {
    LIBVA_DRIVER_NAME = if intelDriverVersion == "iHD" then "iHD" else "i965";
    VDPAU_DRIVER = "va_gl";
    ONEVPL_SEARCH_PATH = lib.strings.makeLibraryPath (with pkgs; [oneVPL-intel-gpu intel-media-sdk]);
  };

  systemd.services.jellyfin.environment = {
    LIBVA_DRIVER_NAME = if intelDriverVersion == "iHD" then "iHD" else "i965";
    VDPAU_DRIVER = "va_gl";
    ONEVPL_SEARCH_PATH = lib.strings.makeLibraryPath (with pkgs; [oneVPL-intel-gpu intel-media-sdk]);
  };

  # NOTE(jpr): useful debugging tools you can enable
  # environment.systemPackages = [
  #   pkgs.pciutils
  #   pkgs.libva-utils
  #   pkgs.intel-gpu-tools
  #   pkgs.jellyfin-ffmpeg
  # ];
}

@atemu closure size goes down by a few MB, seemingly because intel-media-sdk is not a direct dependency, but it would be found at runtime by the new onevpl dependency. Which now that I think about it, that will probably cause problems for uses depending on intel-media-sdk playback or rendering due to library search paths?
here's the output of nix path-info to get closure sizes of the two builds
@PJungkamp I pulled in your commit for ffmpeg, hope that's ok!
also looks like I rebased too far once... need to remove that first commit

Awesome!
My Nix Build Machine that was supposed to run jellyfin is currently down, so I lost sight of this issue...
I also took a look into intel-media-sdk, which can also discover the onevpl backend at runtime. This seems to be done by searching a path specified at build time (MODULES_PATH I think), where the media-sdk .so is installed into and the onevpl .so is searched for. Maybe we should also pull onevpl as an optional, default enabled dependency into the media sdk derivation. That would make all media sdk remaining consumers get the proper GPU acceleration on Alder Lake and newer.

https://github.com/Intel-Media-SDK/MediaSDK/blob/7a72de33a15d6e7cdb842b12b901a003f7154f0a/api/mfx_dispatch/linux/mfxloader.cpp#L194
Maybe putting it into the buildInputs is enough, it seems to search for onevpl in the normal library path.

uh oh this is a big rebuild now :grimacing:
@PJungkamp I think runtimeInputs is what we need for media sdk to be able to dispatch to onevpl depending on the hardware it detects. Based on this, I think this environment variable can be used to force the newer api if needed:
environment.sessionVariables.INTEL_MEDIA_RUNTIME = "ONEVPL";

@midchildan should intel-media-sdk be marked deprecated?

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2843

If you want to avoid rebuilds, keep libmfx and put the vpx enable behind an optionals (with separate patch to staging removing the optionals again).

I think I know what you mean. I'll try it. Can those two commits be in the same Pr?
Also ofborg tagging may be delayed, but I thought it was my intel-media-sdk commit that added all the rebuilds. As a user, it makes more sense for ffmpeg to be the root of the rebuilds however

I think I know what you mean. I'll try it. Can those two commits be in the same Pr?

No, that's the whole point. The mass-rebuild is triggered by the --disable-libvpl addition and --enable-libmfx configure flag removal.
Given this PR's uncriticality, I'd rather just target staging here.

Please follow the contributing guide on how to change the merge base.


spoken language understanding -> Spoken Language Understanding

What
Changed "spoken language understanding" to "Spoken Language Understanding"
Why
To align with our other online assets

Changed Spoken Language Understanding to capital letters


Swapnil patle01 patch 2

Fixes Issue

Changes proposed

Check List (Check all the applicable boxes)

[ ] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[ ] All new and existing tests passed.
[ ] This PR does not contain plagiarized content.
[ ] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

You have created two prs for the same thing, could you close one of them


New version: taers232c.GAMADV-XTD3 version 6.71.18

Created by 🥟 Dumplings in Run #194.
Microsoft Reviewers: Open in CodeFlow

For moderators
This account is automated by Github Actions and the source code was created by CoolPlayLin. If you have any questions about any pull request, don't hesitate to ping Exorcism0666.

[!warning]
With the new version of Komac, numerous Pull Requests may contain bugs or unwanted elements. If such issues arise, please tag russellbanks so that these bugs can be addressed in future updates.

[!important]
Please carefully review these Pull Request before merging. If it is a Pull Request for removing incorrect content and the URLs are issue, free upon manual checking, please close this Pull Request directly. (It is best to inform Exorcism0666 of the closure.)

良い一日をお過ごしください！


StrictYamlMapping + unit tests

PR for #22
Implemented StrictYamlMapping and unit tested it.

Coverage decreased (-7.9%) to 83.217% when pulling 1cbe0690765c8e7beb522ad4fcdffe078b21e67d on 22 into c5192376a088324f43f5491b3e0520bf330f14dd on master.

Coverage decreased (-7.9%) to 83.217% when pulling 1cbe0690765c8e7beb522ad4fcdffe078b21e67d on 22 into c5192376a088324f43f5491b3e0520bf330f14dd on master.

Coverage decreased (-7.9%) to 83.217% when pulling 1cbe0690765c8e7beb522ad4fcdffe078b21e67d on 22 into c5192376a088324f43f5491b3e0520bf330f14dd on master.

Coverage increased (+1.5%) to 92.537% when pulling 2bc2c283eea3ddeeff22dbe5a77825b2881368e5 on 22 into c5192376a088324f43f5491b3e0520bf330f14dd on master.

Coverage increased (+1.5%) to 92.537% when pulling 2bc2c283eea3ddeeff22dbe5a77825b2881368e5 on 22 into c5192376a088324f43f5491b3e0520bf330f14dd on master.

@rultor merge pls

@rultor merge pls

@amihaiemil OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge pls

@amihaiemil Done! FYI, the full log is here (took me 1min)


Revert request parameters and update form options.

Could I get more info on this PR - there's no description?


DC Forward Modelling Image has spell check red underline

on the page: http://em.geosci.xyz/en/latest/content/maxwell2_steady_state/Forward_Modelling.html

@DanielBild-Enkin , @lacmajedrez : you guys created the original, right? could you help out here? Thanks!

closed with #59


An error occurred while using ConvertGroup.List  HV000124: Found multiple group conversions for source group

From the error message, it appears that I have multiple group conversions, but in reality, I only have one
I have a simple Spring Boot project
import jakarta.validation.Valid;
import jakarta.validation.groups.ConvertGroup;
import jakarta.validation.groups.Default;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Size;
This is my JavaBean class
@ NotBlank and @ Size were set for field01 and field respectively, and groups were established
public class MyTestBean {

	@NotBlank(groups = {NotBlankCheck.class})
	private String field01;

	@Size(max = 3, min = 0 ,groups = {SizeCheck.class})
	private String field02;

	public String getField01() {
		return field01;
	}

	public void setField01(String field01) {
		this.field01 = field01;
	}

	public String getField02() {
		return field02;
	}

	public void setField02(String field02) {
		this.field02 = field02;
	}

}

This is my model, using @ Valid for testBean01 and testBean02
All types are MyTestBean
ConvertGroup has been set up, hoping to perform NotBlank check only on field01 for testBean01 and Size check only on field02 for testBean02
public class MyModel {

	interface Action01{

	}

	@Valid
	@ConvertGroup.List({
		@ConvertGroup(from=Default.class,to=NotBlankCheck.class)
	})
	private MyTestBean testBean01;

	@Valid
	@ConvertGroup.List({
		@ConvertGroup(from=Default.class,to=SizeCheck.class)
	})
	private MyTestBean testBean02;

	public MyTestBean getTestBean01() {
		return testBean01;
	}

	public void setTestBean01(MyTestBean testBean01) {
		this.testBean01 = testBean01;
	}

	public MyTestBean getTestBean02() {
		return testBean02;
	}

	public void setTestBean02(MyTestBean testBean02) {
		this.testBean02 = testBean02;
	}

}

This is my Controller
@Controller
@RequestMapping("/")
public class MyController {

	@ModelAttribute
	public MyModel setUpForm() {
		MyModel myModel = new MyModel();
		MyTestBean bean01 = new MyTestBean();
		bean01.setField01("");
		bean01.setField02("abcdef");
		MyTestBean bean02 = new MyTestBean();
		bean02.setField01("");
		bean02.setField02("abcdef");
		myModel.setTestBean01(bean01);
		myModel.setTestBean02(bean02);
		return myModel;
	}

	@GetMapping(value = "test")
	public String test(@Validated MyModel myModel, BindingResult result) {
		if (result.hasErrors()) {
			return "null";
		}
		return "success";
	}

}

But when I accessed the controller, an error occurred,Regardless of the situation, I will encounter this error when using @ ConvertGroup.List, even if I delete other @ ConvertGroups and only keep one
jakarta.validation.ConstraintDeclarationException: HV000124: Found multiple group conversions for source group jakarta.validation.groups.Default: com.example.validatedemo2.NotBlankCheck.
	at org.hibernate.validator.internal.metadata.provider.AnnotationMetaDataProvider.getGroupConversions(AnnotationMetaDataProvider.java:567) ~[hibernate-validator-8.0.1.Final.jar:8.0.1.Final]
	at org.hibernate.validator.internal.metadata.provider.AnnotationMetaDataProvider.getGroupConversions(AnnotationMetaDataProvider.java:547) ~[hibernate-validator-8.0.1.Final.jar:8.0.1.Final]

But when I don't use List, but instead use @ ConvertGroup, everything works fine..But I do need to use List
	@Valid
	@ConvertGroup(from=Default.class,to=NotBlankCheck.class)
	private MyTestBean testBean01;

	@Valid
	@ConvertGroup(from=Default.class,to=SizeCheck.class)
	private MyTestBean testBean02;

Is there a misunderstanding about my use of @ConvertGroup.List? Or does he have some problems with himself
validatedemo2.zip

Hi @xg578962913, is there a reason you're using @ConvertGroup.List instead of multiple @ConvertGroup annotations, like this:
public class User {
    @Valid
    @ConvertGroup(from=Default.class, to=BasicPostal.class)
    @ConvertGroup(from=Complete.class, to=FullPostal.class)
    Set<Address> getAddresses() { [...] }
}

From looking at the Hibernate Validator implementation, I think the issue is both the @ConvertGroup and @ConvertGroup.List annotations are being picked up as separate annotations, and the logic is including the values in the one @ConvertGroup annotation twice.
See the two getGroupConversion methods here
I'm not sure if using @ConvertGroup.List instead of multiple @ConverGroup annotations is valid (That's beyond my knowledge of repeateable annotations), but if it is, this would need to be fixed by Hibernate Validator. I would recommend closing this issue and opening an issue in the Hibernate Validator github.

@xg578962913 you are not supposed to use the @ConvertGroup.List annotation directly. Here you found a small lenient case of the JVM.
If you had done
        @ConvertGroup.List({
		@ConvertGroup(from=Default.class,to=NotBlankCheck.class),
		@ConvertGroup(from=Foo.class,to=Bar.class),
	})

You would have seen a Java lang error
java.lang.annotation.AnnotationFormatError: Duplicate annotation for class: interface jakarta.validation.groups.ConvertGroup: @jakarta.validation.groups.ConvertGroup(from=java.io.Foo.class, to=com.example.validatedemo2.Bar.class)
        at java.base/sun.reflect.annotation.TypeAnnotationParser.mapTypeAnnotations(TypeAnnotationParser.java:387)
        at java.base/sun.reflect.annotation.AnnotatedTypeFactory$AnnotatedTypeBaseImpl.<init>(AnnotatedTypeFactory.java:152)
        at java.base/sun.reflect.annotation.AnnotatedTypeFactory.buildAnnotatedType(AnnotatedTypeFactory.java:67)
        at java.base/sun.reflect.annotation.TypeAnnotationParser.buildAnnotatedType(TypeAnnotationParser.java:78)
        at java.base/java.lang.reflect.Field.getAnnotatedType(Field.java:1235)

It's because the JVM itself will transform
	@ConvertGroup(from=Default.class,to=NotBlankCheck.class)
	@ConvertGroup(from=Foo.class,to=Bar.class)
        private MyTestBean testBean01;

into
        @ConvertGroup.List({
		@ConvertGroup(from=Default.class,to=NotBlankCheck.class),
		@ConvertGroup(from=Foo.class,to=Bar.class),
	})
        private MyTestBean testBean01;

So you using @ConvertGroup.List messes with this.
We technically could fix it in Hibernate Validator but your usage is incorrect, just use the "flat" version @mswatosh showed.

closing


If monster is a transforming card, make it use the current version of the transforminfo embed

As created in #754:

Do you want the "Avail. killers (pre-xform)" to come back if it's going back in regular id, or do you suppose it's already understood?

Do you want the "Avail. killers (pre-xform)" to come back if it's going back in regular id

good call, yeah, but only for the regular id screen version


url Mod Rewrite

Mod Rewrite
exemple your forum https://forkbb.ru/topic/28/Установка (Installation)
to
https://forkbb.ru/topic/28/Installation

Url depends on the topic name. Name the topic "Installation" -> get the url https://forkbb.ru/topic/28/Installation
%D0%A3 and so on are UTF-8 characters encoded in accordance with the specification other than standard Latin. I didn't come up with this ;)

ok

https://github.com/forkbb/forkbb/commit/8e3c74367e9f0fbd1adf2332d04b0cf8a53b4b28

how to update?

yes yes I understand thanks

https://github.com/forkbb/forkbb/releases/tag/0.0071.2

This change doesn't require a database update right?

This change doesn't require a database update right?

Yes, only one file has been changed.

what is?

Administration -> Index -> View server statistics -> Show info

Current ICU version = 74 if I'm reading the page correctly.
You can ask your hosting support to update this library if it is old. Whether they will update or not is another question.
But you don’t have to ask if there is no noticeable drop in ForkBB execution speed when transliteration is enabled in main.php:
    'FRIENDLY_URL' => [
        'lowercase' => true,
        'translit' => 'Russian-Latin/BGN;Any-Latin;Latin-ASCII;',
        'WtoHyphen' => true,
    ],

compared to when transliteration is disabled:
    'FRIENDLY_URL' => [
        'lowercase' => true,
        'translit' => '',
        'WtoHyphen' => true,
    ],

my icu 64,2

main.php? folder?

see https://forkbb.ru/topic/63/app-config-main-php


Configurable mime type

Basically because I prefer application/javascript over javascript.
I've added it as an option keeping javascript as default for compatibility reasons.

Thanks for contributing :) Looks good to me (although I would have called the option mimetype,  but let's just leave it as it is)


Feature dimensions of BoQ wth DinoV2 backbone

Hi, @amaralibey
BoQ is a wonderful work.
My questions are:

What is the feature dimension of BoQ with DINOV2 used in the experiment reported in README?
What is the feature dimension in Table 3 of your paper?
Looking forward to your reply!

Are there any experimental results in Tokyo or SF-XL?
Is it because excessive dimensionality can cause memory overflow?

Hello @BinuxLiu,
Thank you for your interest!

The feature dimension of BoQ when using the DinoV2 backbone is 12,288 (384x32).
In Table 3 of our paper, we are using the ResNet50 backbone, where the dimension is 16,384.

I will run experiments on the SF-XL dataset and report the results once the tests are completed. I did not have any problem with memory when I tested on San Francisco (more than 1M images). 12288 is still 3x smaller than NetVLAD's 32,768 dim.
I currently do not have the Tokyo dataset on my computer. I will need to ask the authors for the download link, and I may add it later.
Best,
Amar

Thank you for your answer.

Awesome!
2024-07-12 22:53:09   Test set: < BaseDataset, tokyo247 - #database: 75984; #queries: 315 >
2024-07-12 22:59:39   Recalls on < BaseDataset, tokyo247 - #database: 75984; #queries: 315 >: R@1: 98.10, R@5: 98.10, R@10: 98.73, R@100: 99.68

Hello @BinuxLiu,
Thank you for taking the time to test on Tokyo247. Are these results with DinoV2-BoQ with images resized to 322x322? If so, do you permit that I put these performance on the README?

2024-07-12 23:16:09   Recalls on < BaseDataset, tokyo247 - #database: 75984; #queries: 315 >: R@1: 96.51, R@5: 97.78, R@10: 98.41, R@100: 100.00
2024-07-12 23:16:09   Finished in 0:02:59

This is the Tokyo247 results with DinoV2-BoQ with images resized to 322x322.
Of course.

Nice, thank you.
Could you please tell which model generated these results?

Awesome!
2024-07-12 22:53:09   Test set: < BaseDataset, tokyo247 - #database: 75984; #queries: 315 >
2024-07-12 22:59:39   Recalls on < BaseDataset, tokyo247 - #database: 75984; #queries: 315 >: R@1: 98.10, R@5: 98.10, R@10: 98.73, R@100: 99.68

I used adaptive resolution to evaluate the dataset and also used the DINO-BoQ model to test and get the first set of results.
The resolution has a great impact on the results of Tokyo247 dataset.
class VPRModel(torch.nn.Module):
    def __init__(self,
                 backbone,
                 aggregator):
        super().__init__()
        self.backbone = backbone
        self.aggregator = aggregator

    def forward(self, x):

        if not self.training:
            b, c, h, w = x.shape
            h = round(h / 14) * 14
            w = round(w / 14) * 14
            x = transforms.functional.resize(x, [h, w], antialias=True)

        x = self.backbone(x)
        x, attns = self.aggregator(x)
        return x, attns

@BinuxLiu,
Okay I see, thanks, Tokyo247 has high-resolution queries with varying image sizes, so it makes sense that performance improves when maintaining the original aspect ratio (we discussed this aspect in the Supplementary).
I'm glad you enjoyed our work and find it useful :)
By the way, I will be launching a new framework for VPR in the coming days. It'd be great to get some feedback. I'll keep you in touch.
Best,
Amar.

Sorry, I forgot to test the performance of ResNet-50 last time.
At a resolution of 322x322:
Recalls on < BaseDataset, tokyo247 - #database: 75984; #queries: 315 >: R@1: 90.48, R@5: 94.29, R@10: 96.51, R@100: 97.78

At the original resolution:
Recalls on < BaseDataset, tokyo247 - #database: 75984; #queries: 315 >: R@1: 94.29, R@5: 96.51, R@10: 96.51, R@100: 98.41

Welcome to use my results, I guess there shouldn't be any errors.
If the experimental results of SF-XL datasets are available, please let me know.
Tips: Save the features of the SF-XL database during the first run to speed up experiments with multiple query sets.

Hi，for ResNet-50 ,I get the result of 90.8 95.6  96.5 at a resolution of 384 ×384. Am I testing something wrong? @BinuxLiu

Hi, @LKELN
I think your results are consistent with mine.
First, different machines have some influence (very small).
Second, the denominator used to calculate the recall of Tokyo 247 is smaller than that of other datasets (making it easier to notice the difference).
Please compare the details of result, the difference is reasonable.

You may not have noticed that I'm reporting results at two resolutions.

I agree that slight fluctuations are normal,  the original resolution you mention is 480×640?  If your original resolution is also 384 x 384, then the fluctuation is anomalous I think.


[VS Code] Collaboration session not found.

<!--
For Visual Studio problems/feedback, please use the Report

This error generally indicates that the session you're trying to join has ended. That said, I'm going to close this as a duplicate of #1003, so that we can discuss it via a single issue. Thanks!


feat: remove ack from pulsar reader

This removes the ack when reading messages from pulsar. The reason being that if we run a query twice, we want it to run over the same data in the stream instead of consuming the data.
For materializations, we should rely on the subscription and broker to decide the offset at which to start reading (once we start saving state). Otherwise, each materialization should start at the beginning of the stream and produce deterministic results. Of course, this does depend on the configured lifetime of events in the stream, but that is expected.
However, there is a small chance this breaks the pulsar demo that worked by preparing messages into parquet files then querying over those - since we're not acking, if we try to prepare multiple times, I wonder if we would be duplicated input data into the prepared files. But, unlikely this demo is used anymore, and we most likely want to phase this code path out anyway in favor of reading directly from the stream. Preparing streams to files has its place, but there's more work involved there regardless to have that a user-ready feature.

agree that we should not worry about the prepare from pulsar path.

This removes the ack when reading messages from pulsar. The reason being that if we run a query twice, we want it to run over the same data in the stream instead of consuming the data.
For materializations, we should rely on the subscription and broker to decide the offset at which to start reading (once we start saving state). Otherwise, each materialization should start at the beginning of the stream and produce deterministic results. Of course, this does depend on the configured lifetime of events in the stream, but that is expected.
However, there is a small chance this breaks the pulsar demo that worked by preparing messages into parquet files then querying over those - since we're not acking, if we try to prepare multiple times, I wonder if we would be duplicated input data into the prepared files. But, unlikely this demo is used anymore, and we most likely want to phase this code path out anyway in favor of reading directly from the stream. Preparing streams to files has its place, but there's more work involved there regardless to have that a user-ready feature.

This seems weird to me. For cases like this, what I generally expect:

If we need to start over, we start a new subscription, which starts at a given offset, etc.
If we restart (after reading a message, but before committing to state), we should reprocess the same message.

To accomplish this, systems like Beam and Flink send an ack for a given message once results derived from that message are persisted in state. But, it is generally important to send the ack...

This removes the ack when reading messages from pulsar. The reason being that if we run a query twice, we want it to run over the same data in the stream instead of consuming the data.
For materializations, we should rely on the subscription and broker to decide the offset at which to start reading (once we start saving state). Otherwise, each materialization should start at the beginning of the stream and produce deterministic results. Of course, this does depend on the configured lifetime of events in the stream, but that is expected.
However, there is a small chance this breaks the pulsar demo that worked by preparing messages into parquet files then querying over those - since we're not acking, if we try to prepare multiple times, I wonder if we would be duplicated input data into the prepared files. But, unlikely this demo is used anymore, and we most likely want to phase this code path out anyway in favor of reading directly from the stream. Preparing streams to files has its place, but there's more work involved there regardless to have that a user-ready feature.

This seems weird to me. For cases like this, what I generally expect:
1. If we need to start over, we start a new subscription, which starts at a given offset, etc.

2. If we restart (after reading a message, but before committing to state), we should reprocess the same message.

To accomplish this, systems like Beam and Flink send an ack for a given message once results derived from that message are persisted in state. But, it is generally important to send the ack...

I'm not sure if it's different in Beam or Flink, but if we ack a message on a particular Pulsar stream, that means that we are forever removing that message from the stream, preventing other queries/consumers from accessing items from that stream.
This certainly shouldn't be done if we are to support regular queries (exploration/training) from a stream, and I don't think it makes sense in the materialization case either, as we may want multiple materializations producing different results from the same input stream.

I believe that the ack only applies to your consumer. Eg., when a pulsar stream has N consumers, each message will be delivered to each consumer. If a consumer acks a message, the other consumers will still see the message. It is only after all N consumers have acked the message that the message will be considered acked. Additionally, Pulsar has options for retaining messages so that future consumers will still see a message.
We shouldn't be thinking of this as "acking on the stream", but rather "this consumer acknowledge that it has consumed the message from the stream", but it doesn't speak to other consumers.

From https://pulsar.apache.org/docs/3.0.x/concepts-messaging/#acknowledgment

The consumer sends an acknowledgment request to the broker after it consumes a message successfully. Then, this consumed message will be permanently stored, and deleted only after all the subscriptions have acknowledged it.

Each materialization should be a separate subscription. Our materialization consumes a message (and acknowledges it) once it has been consumed successfully -- for us that means the result of that message is persisted in state. The message will continue to be stored until "all other subscriptions have acknowledged it."

From https://pulsar.apache.org/docs/3.0.x/concepts-messaging/#acknowledgment

The consumer sends an acknowledgment request to the broker after it consumes a message successfully. Then, this consumed message will be permanently stored, and deleted only after all the subscriptions have acknowledged it.

Each materialization should be a separate subscription. Our materialization consumes a message (and acknowledges it) once it has been consumed successfully -- for us that means the result of that message is persisted in state. The message will continue to be stored until "all other subscriptions have acknowledged it."

https://pulsar.apache.org/docs/3.0.x/concepts-messaging/#message-retention-and-expiry
"By default, Pulsar message brokers:
immediately delete all messages that have been acknowledged by a consumer"
There is a message retention policy that allows the topic to keep messages that are acknowledged. This means that if there is only a single subscriber on a topic at a time, then we'll delete the messages once they're acknowledged, based on the documentation. Doesn't this read as:

Create topic
Create materialization reading from topic
Ack messages 1-10 from topic
Messages 1-10 are acked by only existing subscriber, so delete them
Create materialization 2 reading from same topic
Materialization 2 sees messages 10...n

I don't think we plan to rely on our first subscription existing and thus causing retention.
Consider a few things:

Any message sent before we start a materialization will be sent when there are no subscriptions, so will be deleted / dropped.
If we want Pulsar to retain things forever, we should use Pulsar tiered storage, which is designed for this. Not sending the ACK will cause Pulsar to keep it in the "active" path forever, rather than being able to reclaim that memory, etc.

I don't think we plan to rely on our first subscription existing and thus causing retention.
I'm not sure what you mean by this.

Any message sent before we start a materialization will be sent when there are no subscriptions, so will be deleted / dropped

You mean if it is deleted/dropped due to it not being read before its lifetime expires?

we should use Pulsar tiered storage

if all users enabled this, we can query over historical data that is stored in long-term storage as if it were still in the stream. As long as we're okay with the current ack and drop behavior. I think that I'm unclear on to what your main pushback on this change is - is it because we're not acting as a "consumer" on a stream should act like if we don't ack messages when we read them?
We won't be able to support querying over pulsar streams unless tiered storage (or some other mechanism) is enabled, if we leave it as is.

I'm also confused by Ben's most recent comments, but I'll start a different thread...
Everytime we make a new materialization, we should use a new subscription-id (ideally based off the materialization id), and then starting reading from the start of the topic.  If we need to restart the materialization, we should restore state from the previous compute snapshot, and then start re-consuming messages.
I think this means that we only want to ack messages that have been successfully persisted to a compute snapshot.
Until we this code path supports resuming from snapshots, it might be best to not ack read messages.  Therefore when we restart processing a materialization, we can re-read from the beginning of the topic to restore state.

every query needs to use a different subscription id.

So, closing this for now. I agree now that we shouldn't be removing the ack, since that's contrary to the behavior of a consumer from a stream. But, I do still believe that we would get the behavior we desire by removing the ack (i.e. queries can read from the streams independently and read all messages without removing any from the stream)

i think maybe you misunderstood: i think we should remove the ack for
now... but probably put it back when we start snapshotting compute state.
On Thu, Jun 22, 2023 at 6:30 PM jordanrfrazier @.***>
wrote:

So, closing this for now. I agree now that we shouldn't be removing the
ack, since that's contrary to the behavior of a consumer from a stream.
But, I do still believe that we would get the behavior we desire by
removing the ack (i.e. queries can read from the streams independently and
read all messages without removing any from the stream)
—
Reply to this email directly, view it on GitHub
https://github.com/kaskada-ai/kaskada/pull/424#issuecomment-1602976417,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUE5RVI3U63D5BET2JEGSDXMRXLBANCNFSM6AAAAAAY4XYFWM
.
You are receiving this because your review was requested.Message ID:
@.***>

I'm fine either way for now, but I think in general the above discussion is aligned.

The expectation for consumer behavior is to acknowledge messages.
Every query should use a different subscriber ID -- so other queries acknowledging a message or not shouldn't matter (beyond the affect it has on retention).
Until the first query is run (and thus the first subscriber ID is created) and/or if all subscribers disappear, message won't be retained. As a result, it is not correct for us to rely on "because we don't acknowledge the message will still be there". That is only true of messages that appear (or exist) after our subscriber ID is created, and until our subscriber ID is deleted.

That said -- it's OK to remove the ACK for now if we need for demos, etc. but we should figure out something that lets us implement computation as a "proper" consumer.


NCCL Error on multi-gpu setup with Nvidia driver 545

System Info
OS: Ubuntu 22.04.3 LTS
Kernel: 5.15.0-91-generic
text-generation-inference:
{
"model_id": "/model",
"model_sha": null,
"model_dtype": "torch.float16",
"model_device_type": "cuda",
"model_pipeline_tag": null,
"max_concurrent_requests": 128,
"max_best_of": 2,
"max_stop_sequences": 4,
"max_input_length": 3072,
"max_total_tokens": 4096,
"waiting_served_ratio": 1.2,
"max_batch_total_tokens": 38544,
"max_waiting_tokens": 20,
"validation_workers": 2,
"version": "1.4.0",
"sha": "c2d4a3b5c7bb6a8367c00f7c797bf87f4b2fcef9",
"docker_label": "sha-c2d4a3b"
}
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        Off | 00000000:01:00.0 Off |                  N/A |
|  0%   25C    P8              10W / 350W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off | 00000000:21:00.0 Off |                  N/A |
|  0%   25C    P8              13W / 350W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
Information

[X] Docker
[ ] The CLI directly

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
On a system with more than 1 GPU, with Nvidia driver 545.29.06 (the latest 545 driver in the ubuntu repo), I run this command:
docker run --gpus '"device=0,1"' --shm-size 1g -v $(pwd):/model -p 8080:80 ghcr.io/huggingface/text-generation-inference:1.4 --model-id /model --quantize bitsandbytes-fp4 --trust-remote-code --port 80 --json-output --hostname <IP_ADDRESS> --max-input-length 3072 --max-total-tokens 4096
I receive the following console output:
{"timestamp":"2024-02-13T22:17:59.304845Z","level":"INFO","fields":{"message":"Args { model_id: \"/model\", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(BitsandbytesFP4), speculate: None, dtype: None, trust_remote_code: true, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 3072, max_total_tokens: 4096, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"<IP_ADDRESS>\", port: 80, shard_uds_path: \"/tmp/text-generation-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, tokenizer_config_path: None, env: false }"},"target":"text_generation_launcher"} {"timestamp":"2024-02-13T22:17:59.304895Z","level":"WARN","fields":{"message":"trust_remote_codeis set. Trusting that model/modeldo not contain malicious code."},"target":"text_generation_launcher"} {"timestamp":"2024-02-13T22:17:59.304907Z","level":"INFO","fields":{"message":"Sharding model on 2 processes"},"target":"text_generation_launcher"} {"timestamp":"2024-02-13T22:17:59.305006Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]} {"timestamp":"2024-02-13T22:18:01.846572Z","level":"INFO","fields":{"message":"Files are already present on the host. Skipping download.\n"},"target":"text_generation_launcher"} {"timestamp":"2024-02-13T22:18:02.407969Z","level":"INFO","fields":{"message":"Successfully downloaded weights."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]} {"timestamp":"2024-02-13T22:18:02.408183Z","level":"INFO","fields":{"message":"Starting shard"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:02.408583Z","level":"INFO","fields":{"message":"Starting shard"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:12.415417Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:12.417509Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:22.421204Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:22.425835Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:32.427004Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:32.435364Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:42.432782Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:42.444084Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:52.438547Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:18:52.452346Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:19:02.444730Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:19:02.460619Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:19:07.248791Z","level":"ERROR","fields":{"message":"Shard complete standard error output:\n\nThe argumenttrust_remote_codeis to be used with Auto classes. It has no effect here and is ignored.\nThe argumenttrust_remote_codeis to be used with Auto classes. It has no effect here and is ignored.\nYou are using a model of type llama to instantiate a model of type . This is not supported for all configurations of models and can yield errors.\n[E ProcessGroupNCCL.cpp:475] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=60000) ran for 60273 milliseconds before timing out.\n[E ProcessGroupNCCL.cpp:489] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.\n[E ProcessGroupNCCL.cpp:495] To avoid data inconsistency, we are taking the entire process down.\n[E ProcessGroupNCCL.cpp:916] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=60000) ran for 60273 milliseconds before timing out.\nterminate called after throwing an instance of 'std::runtime_error'\n  what():  [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=60000) ran for 60273 milliseconds before timing out."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:19:07.248832Z","level":"ERROR","fields":{"message":"Shard process was signaled to shutdown with signal 6"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-02-13T22:19:07.258333Z","level":"ERROR","fields":{"message":"Shard 0 failed to start"},"target":"text_generation_launcher"} {"timestamp":"2024-02-13T22:19:07.258360Z","level":"INFO","fields":{"message":"Shutting down shards"},"target":"text_generation_launcher"} {"timestamp":"2024-02-13T22:19:07.402113Z","level":"INFO","fields":{"message":"Shard terminated"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} Error: ShardCannotStart
The container then exits.
The exact command works on the same exact system when I downgrade the drivers to 535.  It also works with the workaround I found here: https://github.com/vllm-project/vllm/issues/1801#issuecomment-1847110199 (adding the NCCL_P2P_DISABLE=1 environment variable)
Expected behavior
I'm not sure if this is the right place for the issue, but it's the first place I encountered this (seems like it might be NVIDIA's problem more than anyone).  Other model training and inference tests passed with the new driver version (although I'm unsure how many of them actually utilize NCCL).  My expected behavior is that the TGI server would start without error on the latest NVIDIA driver version.

Sorry to say but I don't think it's linked to the drivers. Currently running the same drivers without any issue.
Disabling P2P is odd...
I'm guessing I'm using an NVLink which you don't have, or maybe something is broken in the drivers in the detection ?
https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
In any case we can't really disable P2P since it's quite necessary on NVLink machines.

Thanks a lot for sharing the solution
NCCL_P2P_DISABLE=1

really helpful for others that might stumble here.

'm guessing I'm using an NVLink which you don't have

You guessed right, I do not have an NVLink on that system.  Presumably then it's a problem with NCCL using/negotiating PCI-based P2P?  Do you have a suggestion on how I can follow up on that with NVIDIA (I don't really have an enterprise support contract, it's just a gaming machine I'm using locally)?


I have created https server on node server and after that i am not able to get the socket.io.js here is my code:

var fs 		= require('fs');
var server 	= require('https');
var express = require('express');
var app 	= express();
// This is needed if the app is run on heroku:
var port = process.env.PORT || 8181;
//var port = process.env.HTTPS_PORT || 8181;
// Initialize a new socket.io object. It is bound to
// the express app, which allows them to coexist.
httpsServer = server.createServer({
key: fs.readFileSync('my.agentmx.com.key'),
cert: fs.readFileSync('my.agentmx.com.crt'),
ca: fs.readFileSync('my.agentmx.com.ca-bundle')
}, app).listen(port);
//var io = require('socket.io').listen(app.listen(port));
var io = require('socket.io')(httpsServer);
// Require the configuration and the routes files, and pass
// the app and io as arguments to the returned functions.
require('./config')(app, io);
require('./routes')(app, io);
var url = require('url');
console.log('Your application is running on http://my.agentmx.com:' + port);

I think you're in the wrong github repo.  Try this one: https://github.com/socketio/socket.io

Thank you thedustinsmith.


Only evaluate on hard set

Good work! I noticed in the leaderboard that the Hard Score is a lot lower. So i wanna just evaluate LMM performances on the HARD set, i wanna know how to do it. (Do i need to only run evaluations on the questions with visual-input == '2'?) Thanks!

Correct. And both easy and hard questions are calculated in separate using the original script.


Font-Awesome Simple Tutorial

A basic Font-Awesome integration tutorial.

What's the reason why to delete our files?

Hm. That's in error.


UIT Of Douala created

Hello,
I have added the University Institute of Technologies Ot the University of Douala in Cameroun in the list.

@lanokims Pull request merged. Thank you!


Diagonal, diagflat, trace, qr, rq, svd in experimental_ncon

Merges master into experimental_ncon
Adds wrappers and tests for diagonal, diagflat, trace, qr, rq and svd.

Tried to merge into the wrong checkout by mistake


Rationalize incremental materialization

resolves #140
Description
This PR implements the first four (4) recommendations from #140 (much fuller description there) to rationalize the 'incremental' materialization:

What's actually supported across file formats + strategies
Better consistency with other adapters' "append-only" behavior when a functional config (unique_key, partition_by) is not specified

TODO

[x] ~Fix failing test_dbt_incremental: I think this PR represents a genuine change in behavior!~ It brings it in line with the default adapter tests, which can only be a step in the right direction :)

[ ] I'm not sure about how to implement recommendation (5), which proposed that we should set file_format: delta + incremental_strategy: merge as defaults if method == 'odbc' (replacing parquet + insert_overwrite, respectively). A bit more on this below:

We could add this logic just within the Jinja macros (e.g. within file_format_clause, dbt_spark_validate_get_file_format, dbt_spark_validate_get_incremental_strategy), but then it wouldn't be consistent with the values in manifest.json, since the file format default is set in python:
https://github.com/fishtown-analytics/dbt-spark/blob/2f7c2dcb419b66d463bb83212287bd29280d6964/dbt/adapters/spark/impl.py#L32-L33
@kwigley Any chance you could advise on if it would be possible to set that config default dynamically, based on the connection method? Or if that's a terrible idea? :)
Even if we can't get that last piece working, I still think we should move forward with 4/5, and advise Databricks users to simply set in dbt_project.yml:
models:
  +file_format: delta
  +incremental_strategy: merge

This is what we advise today, anyway.
Checklist

[x] I have signed the CLA
[x] I have run this code in development and it appears to resolve the stated issue
[ ] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

Any chance you could advise on if it would be possible to set that config default dynamically

@jerco unfortunately, I don't think it is possible with the current impl. The adapter config does not know anything about the connection when it is created :(

Any chance you could advise on if it would be possible to set that config default dynamically

@jerco unfortunately, I don't think it is possible with the current impl. The adapter config does not know anything about the connection when it is created :(

Thanks for looking into it @kwigley!
I suppose this would be a compelling thing that we could do, if we split off the Databricks pieces of dbt-spark into a new adapter plugin, dbt-databricks, that inherited from the current plugin, added certain functionality, and overrode certain defaults.

Thanks for looking into it @kwigley!
I suppose this would be a compelling thing that we could do, if we split off the Databricks pieces of dbt-spark into a new adapter plugin, dbt-databricks, that inherited from the current plugin, added certain functionality, and overrode certain defaults.

Thanks for the review @Fokko! I just reorganized these macro files a bit as well.

I think the most important thing is that we make sure that we test the edge cases.

I completely agree. I have a "suite" of "test cases" in a local project that I used to work through all the edge cases:

CASE
Apache Spark
Databricks Cluster
Databricks Endpoint

bad_file_format
❌
❌
❌

bad_strategy
❌
❌
❌

bad_merge_not_delta
❌
❌
❌

bad_insert_overwrite_delta
❌
❌
❌

default_append
✅
✅
❔

insert_overwrite_no_partitions
✅
✅
❌

insert_overwrite_partitions
✅
✅
❌

delta_append
❔
✅
❔

delta_merge_no_key
❔
✅
❔

delta_merge_unique_key
❔
✅
❔

Keys:

✅ Works
❌ Raises explicit, clear compilation error
❔ dbt is unopinionated: no explicit error, may work or may return database error (e.g. using delta in Apache Spark, forthcoming support for create temp view via endpoint)

I'm struggling a bit with how to implement this in a more-automated fashion:

It's slightly more involved than a unit test. I tried extending unit/test_macros.py without success, macros are a bit more interlocking and require dbt-specific Jinja features like return
It's slightly more granular that our suite of adapter integration tests. Ideally we'd have a way to iterate through a sequence, changing one config each time, and catch exceptions where we hope for them.

FWIW We don't have automated tests today for incremental strategies, either, so I think this still constitutes a step in the right direction.

Thanks for the review @Fokko! I just reorganized these macro files a bit as well.

I think the most important thing is that we make sure that we test the edge cases.

I completely agree. I have a "suite" of "test cases" in a local project that I used to work through all the edge cases:

CASE
Apache Spark
Databricks Cluster
Databricks Endpoint

bad_file_format
❌
❌
❌

bad_strategy
❌
❌
❌

bad_merge_not_delta
❌
❌
❌

bad_insert_overwrite_delta
❌
❌
❌

default_append
✅
✅
❔

insert_overwrite_no_partitions
✅
✅
❌

insert_overwrite_partitions
✅
✅
❌

delta_append
❔
✅
❔

delta_merge_no_key
❔
✅
❔

delta_merge_unique_key
❔
✅
❔

Keys:

✅ Works
❌ Raises explicit, clear compilation error
❔ dbt is unopinionated: no explicit error, may work or may return database error (e.g. using delta in Apache Spark, forthcoming support for create temp view via endpoint)

I'm struggling a bit with how to implement this in a more-automated fashion:

It's slightly more involved than a unit test. I tried extending unit/test_macros.py without success, macros are a bit more interlocking and require dbt-specific Jinja features like return
It's slightly more granular that our suite of adapter integration tests. Ideally we'd have a way to iterate through a sequence, changing one config each time, and catch exceptions where we hope for them.

FWIW We don't have automated tests today for incremental strategies, either, so I think this still constitutes a step in the right direction.

As discussed with @kwigley yesterday, the ideal type of test to write for this kind of functionality is a "true" integration test, like the ones we have in Core (dbt/test/integration).
To that end, it would be great to include and expose base.py as an available module in pytest-dbt-adapter. It's relatively stable boilerplate code that provides the foundation for writing integration or functional tests on any adapter. Breaking it out into its own module is also a prerequisite to someday achieving true modularity for the dbt-core/-postgres/-redshift/-snowflake/-bigquery tests.

As discussed with @kwigley yesterday, the ideal type of test to write for this kind of functionality is a "true" integration test, like the ones we have in Core (dbt/test/integration).
To that end, it would be great to include and expose base.py as an available module in pytest-dbt-adapter. It's relatively stable boilerplate code that provides the foundation for writing integration or functional tests on any adapter. Breaking it out into its own module is also a prerequisite to someday achieving true modularity for the dbt-core/-postgres/-redshift/-snowflake/-bigquery tests.

@kwigley I gave a go at writing custom integration tests for the changes in this PR, using the work in https://github.com/fishtown-analytics/dbt-adapter-tests/pull/13. I'd love to get your help here—perhaps a bit of pair-programming? :)
Once we can manage to get these tests running, I'll feel really good about merging and releasing the additions to the adapter testing suite, even if it's missing some of the bells and whistles. (E.g. I've decided that use_profile leveraging the .dbtspec target, while a very cool idea, is nice but not strictly necessary for this first version—it's the same environment variables, at the end of the day.)

@jtcohen6 can you take a peek at why test_insert_overwrite_apache_spark is failing?

Ugh. Here's where we're at: ODBC connections to Databricks clusters aren't respecting the set spark.sql.sources.partitionOverwriteMode = DYNAMIC parameter. I have no idea why. Does pyodbc generate a new connection for each executed SQL statement, with new default parameters each time?
(Those set statements aren't supported at all by the new SQL Analytics endpoints. This issue is exclusive to method: odbc + cluster.)
One thing we could do: Tell folks to set spark.sql.sources.partitionOverwriteMode = DYNAMIC within their Databricks clusters (docs). I'll try this in a few minutes. If it works, it may be a better idea anyway, rather than having the incremental materialization try to set this parameter each time.


Add certstrap utility

The Genesis Cloud Foundry template deployment uses certstrap.  It would be nice to have it available on the jumpbox.

We are unable to find in the https://github.com/jhunt/jumpbox that it installs it.  Are we not going to add it to jumpbox?

This still needs to happen in the jumpbox BOSH release

Although, given that certstrap doesn't make binary releases available, we may just replace the certstrap usage with openssl usage.  Certs aren't that hard...


Bugs in the SMAP vegetation water content plugin in LVT

The use of this plugin leads to a model crash. There are several typos in the definition of the VWC as a diagnostic variables in LVT

Hi @sujayvkumar,
Looks like these bugs were addressed in #416. Is it ok to close out this issue?

sounds good


Error

root@instance:/home/ubuntu# bash <(curl -fsSL bit.ly/peer2fly)
Swap already enabled
Filename                                Type            Size            Used            Priority
/swapfile                               file            2097148         60808           -2
              total        used        free      shared  buff/cache   available
Mem:           11Gi       6.8Gi       3.6Gi       4.0Mi       1.3Gi       4.7Gi
Swap:         2.0Gi        59Mi       1.9Gi
ERROR: Unsupported architecture: aarch64

Tried on amd64 vps too, but it does nothing after enabling swap

after running second time (also tried to provide --email)


Formatted Vision.hx

i think ill close this, i want to keep the formatting used in the library consistent...

ok


lls: 0.1.0 -> 0.2.0

Description of changes
no change log provided
diff: https://github.com/jcaesar/lls/compare/v0.1.0...v0.2.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 248338 run on x86_64-linux 1

1 package built:

lls


Update workerpool

This is an experiment to try and update workerpool. See https://github.com/babel/broccoli-babel-transpiler/pull/203 for more information.

Hmm, looks like this contains commits that have already landed on master. Can you rebase to drop those out?

@rwjblue actually I just had the wrong base. This would need to be a v6 patch release but depends on https://github.com/babel/broccoli-babel-transpiler/pull/203 also being patch released too 🙈 I was planning to put together a better explanation for you on this but it has been a mad week

Closing as broccoli-babel-transpiler now includes workerpool v6, so it seems this is no longer needed.


[Translation] [russian] - Translate Type As Constraint

changed the name according to the convention so the is:open is:pr russian search line in PR will represent the status.


Fixed #26126 -- Fixed transient failure of test_max_age_expiration

https://code.djangoproject.com/ticket/26126

I think 994d6137a2519436d17d5b3d16cb49a3fa79f93e would help. Sorry I forget about that when accepting the ticket.

Good idea. I've updated the PR, thanks!

Looks okay to me. ;-)
merged in b17a9150a0c3a132e82b53755ede62a45f897875, thanks!


zmin and zmax ignored in Heatmaps

import numpy as np
import plotly.plotly as py
from plotly.graph_objs import Heatmap,Data
data = np.arange(10000).reshape(100,100)
trace = Heatmap(z=np.flipud(data), zmin=1000, zmax=2000)
py.iplot(Data([trace]))

Using version 1.6.6 on Python 3 (miniconda, all updates), IPython 3.0.0dev

Creates this:
https://plot.ly/~michaelaye/106/

You need to add zauto=False.

if i set zmin/zmax, isn't it obvious that I don't want zauto?

i still would call it an API bug then.

Or a doc's bug, as it's not mentioned in the zmin doctext that zauto needs to be False for it to be accepted.

but thanks for the very fast reply!

Thanks for the headups.
I agree. Setting zmin or zmax is obvious enough and zauto=False is redundant.
This is the api we use for layout.range. If layout.range is supplied by a user then layout .autorange is set to false automatically.
Moreover, yes, the docstrings for zmin and zmax should mention somthing about zauto.
Both these bugs are not related to code in the python api directly, so I'll close this issue.
Thanks again.

2021 and still not resolved, not even in the documentation. What is the right place to request that feature or to make a pull request?

Let us revive this. This is a weird behavior. Made me lose 20 mins.

@felixriese, @Alexander-Serov
Heatmap zmin, zmax work very well, for years.
Example:
import numpy as np
import plotly.graph_objects as go

h = 0.05
y, x = np.mgrid[-5 : 5+h : h, -5 : 10+h : h]
z = np.sin(x)**10 + np.cos(10 + y*x) + np.cos(x) + 0.2*y + 0.1*x
fig=go.Figure(go.Heatmap(z=z, colorscale="turbo"))  #no restrictions
fig.update_layout(width=450, height=450)

fig1=go.Figure(go.Heatmap(z=z, colorscale="turbo", zmin=-5, zmax=5))
fig1.update_layout(width=450, height=450)

The same plot (like the last one) you can get setting coloraxis with cmin, cmax, not zmin, zmax:
fig2=go.Figure(go.Heatmap(z=z, coloraxis="coloraxis"))
fig2.update_layout(width=450, height=450, coloraxis=dict(colorscale="turbo", cmin=-5, cmax=5))


A strange phenomenon about quiver sampler

Hi, thank you for your wonderful opensouce works about Quiver. Recently I was trying to look deep into the sampler module, but met a quite strange phenomenon. We can see the following picture. I was testing about the quiver sampler speed and model training speed.
Machine 1: Tesla V100-SXM2-16GB, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, 20 physical CPU cores.
Model: GraphSage, batch_size=128, samples=[25,10], Reddit.

It seems very strange, because the time cost by sampler and model training is less than only using sampler.
I try several methods to figure out the reason, like DataLoader, CUDA Stream, etc, but still confused.  Maybe you can help me?

Well, this is strange

What is your quiver_sampler's mode

UVA. But I remember that GPU mode has the same phenomenon, if my memory is correct.

That's odd, let me check it

hi, have you figured out the reason?

hi, have you figured out the reason?

Hi,  I would like to ask  how do you run the quiver.  I install torch-quiver 0.1.0 from https://github.com/quiver-team/torch-quiver/blob/main/docker/README.md
when i run examples/pyg/reddit_quiver.py,
it shows AttributeError: module 'torch_quiver' has no attribute 'device_quiver_from_csr_array'.  I tried their docker image and it shows AttributeError: Can't get attribute 'DataEdgeAttr' on <module 'torch_geometric.data.data' from '/opt/conda/lib/python3.7/site-packages/torch_geometric/data/data.py'>


kv value too large when using consul backend and acme

What version of Traefik are you using (traefik version)?
v.1.1.2
What is your environment & configuration (arguments, toml...)?
docker swarm 1.13
Containers: 5
 Running: 2
 Paused: 0
 Stopped: 3
Images: 33
Server Version: 1.13.0
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: glusterfs local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: c8rqvh23p2kqqpbyj9ydf5xxy
 Is Manager: true
 ClusterID: n1td1rbeiwcokhp7gglg6z4gy
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: <IP_ADDRESS>
 Manager Addresses:
  <IP_ADDRESS>:2377
  <IP_ADDRESS>:2377
  <IP_ADDRESS>:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: ohmy-rancher1
ID: ZC23:CHZC:AT2V:KFGG:AD7T:NPUZ:H5SY:BA6N:3KIT:ORRK:X6HW:O7PZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 <IP_ADDRESS>/8

traefik is running on all three nodes as load balancer, output below is not perfectly valid json as I cut out the PreviousSpec and UpdateStatus bits.
docker service inspect traefik
[
    {
        "ID": "r2i408yp9lj1t319voglwd4ay",
        "Version": {
            "Index": 146
        },
        "CreatedAt": "2017-01-27T09:38:04.005237112Z",
        "UpdatedAt": "2017-01-27T09:53:47.215408631Z",
        "Spec": {
            "Name": "traefik",
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "traefik:latest@sha256:c81e1a321ec90a41987b1c8cb887606e3e632acadb3b29f5b97f0ad03c59ffd9",
                    "Args": [
                        "--web",
                        "--consul",
                        "--consul.endpoint=consul.service.consul:8500",
                        "--docker",
                        "--docker.swarmmode",
                        "--docker.watch"
                    ],
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/var/run/docker.sock",
                            "Target": "/var/run/docker.sock"
                        }
                    ],
                    "DNSConfig": {}
                },
                "Resources": {
                    "Limits": {},
                    "Reservations": {}
                },
                "RestartPolicy": {
                    "Condition": "any",
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Constraints": [
                        "node.role==manager"
                    ]
                },
                "Networks": [
                    {
                        "Target": "s7xhhuwbexlefkq87eu1jyvvb"
                    },
                    {
                        "Target": "me7eoua9vuvrg1ywy3p3sm1ik"
                    }
                ],
                "ForceUpdate": 0
            },
            "Mode": {
                "Global": {}
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "MaxFailureRatio": 0
            },
            "Networks": [
                {
                    "Target": "traefik"
                },
                {
                    "Target": "ingress"
                }
            ],
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8080,
                        "PublishedPort": 8080,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 443,
                        "PublishedPort": 443,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
	"Endpoint": {
     "Spec": {
         "Mode": "vip",
         "Ports": [
             {
                 "Protocol": "tcp",
                 "TargetPort": 80,
                 "PublishedPort": 80,
                 "PublishMode": "ingress"
             },
             {
                 "Protocol": "tcp",
                 "TargetPort": 8080,
                 "PublishedPort": 8080,
                 "PublishMode": "ingress"
             },
             {
                 "Protocol": "tcp",
                 "TargetPort": 443,
                 "PublishedPort": 443,
                 "PublishMode": "ingress"
             }
         ]
     },
     "Ports": [
         {
             "Protocol": "tcp",
             "TargetPort": 80,
             "PublishedPort": 80,
             "PublishMode": "ingress"
         },
         {
             "Protocol": "tcp",
             "TargetPort": 8080,
             "PublishedPort": 8080,
             "PublishMode": "ingress"
         },
         {
             "Protocol": "tcp",
             "TargetPort": 443,
             "PublishedPort": 443,
             "PublishMode": "ingress"
         }
     ],
     "VirtualIPs": [
         {
             "NetworkID": "me7eoua9vuvrg1ywy3p3sm1ik",
             "Addr": "<IP_ADDRESS>/16"
         },
         {
             "NetworkID": "s7xhhuwbexlefkq87eu1jyvvb",
             "Addr": "<IP_ADDRESS>/16"
         }
     ]
 }

per@host:~$ consul kv get --recurse traefik
traefik/accesslogsfile:
traefik/acme/:
traefik/acme/account/lock:
traefik/acme/account/object:REMOVED
traefik/acme/caserver:
traefik/acme/email:REMOVED
traefik/acme/entrypoint:https
traefik/acme/ondemand:true
traefik/acme/onhostrule:true
traefik/acme/storage:traefik/acme/account
traefik/checknewversion:true
traefik/consul/:
traefik/consul/endpoint:consul.service.consul:8500
traefik/consul/filename:
traefik/consul/prefix:traefik
traefik/consul/watch:true
traefik/debug:false
traefik/defaultentrypoints/0:http
traefik/defaultentrypoints/1:https
traefik/docker/:
traefik/docker/domain:docker.localhost
traefik/docker/endpoint:unix:///var/run/docker.sock
traefik/docker/exposedbydefault:true
traefik/docker/filename:
traefik/docker/swarmmode:true
traefik/docker/usebindportip:false
traefik/docker/watch:true
traefik/entrypoints/http/:
traefik/entrypoints/http/address::80
traefik/entrypoints/http/compress:false
traefik/entrypoints/http/network:
traefik/entrypoints/https/:
traefik/entrypoints/https/address::443
traefik/entrypoints/https/compress:false
traefik/entrypoints/https/network:
traefik/entrypoints/https/tls/:
traefik/entrypoints/https/tls/ciphersuites/0:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
traefik/entrypoints/https/tls/ciphersuites/1:TLS_RSA_WITH_AES_256_GCM_SHA384
traefik/entrypoints/https/tls/minversion:VersionTLS12
traefik/gracetimeout:10
traefik/insecureskipverify:false
traefik/leader:b5850a68-1b48-45e3-94c0-edc3d6eb03ae
traefik/loglevel:DEBUG
traefik/maxidleconnsperhost:200
traefik/providersthrottleduration:2000000000
traefik/web/:
traefik/web/address::8080
traefik/web/certfile:
traefik/web/keyfile:
traefik/web/readonly:false

What did you do?
automatically generated a new certificate through acme
What did you expect to see?
certificate generated correctly
What did you see instead?
error presenting token: StoreConfig error: Unexpected response code: 413 (Value exceeds 524288 byte limit)]
Searching for this I came across this commit https://github.com/hashicorp/consul/commit/8f37f967e0fc677fa54c9beea988dc331e35718f which mentiones the 512kb limit.
My thoughts about a solution is along the lines of serializing the acme account data into keys and values instead of json so they use the kv store "properly".

Maybe the methods used for StoreConfig can be adapted for this use as well, had a brief look at it but not sure if it's general enough or needs modifications.

Wow :'( Bad news.

My thoughts about a solution is along the lines of serializing the acme account data into keys and values instead of json so they use the kv store "properly".

I was thinking of this at first while coding ACME KV store, but it ended almost impossible to make it work... It seems difficult to serialize certificate structs and deserialize them after.
Not sure how we can fix this one easily.

Some more information, reading this https://coreos.com/etcd/docs/latest/dev-guide/limit.html I get the impression etcd will also have the issue a bit later.
I've tried to understand the serialization and the acme.json format but completely new to go. So far it doesn't look like anything special that aren't already handled or needed by the regular settings serialization as that should handle certificates as well.
I'll help in any way possible as this is currently affecting both our test and production environments

@emilevauge

It seems difficult to serialize certificate structs and deserialize them after.

What kind of problem(s) did you run into?

@timoreimann I can't remember exactly (it as a long time ago ^^), but I had issue while unmarshaling from KV store to structs. I think the issue was with acme.RegistrationResource from https://github.com/xenolf/lego. I finally ended up using json marshaling because this struct contains jose.JsonWebKey fields.
I remember that I spent a lot of time trying to not use json marshaling, but I failed :'(

Some updates, I was out sick over the weekend but started testing a solution for this and progress so far
diff --git a/vendor/github.com/containous/staert/kv.go b/vendor/github.com/containous/staert/kv.go
index 528f0a5..67533ef 100644
--- a/vendor/github.com/containous/staert/kv.go
+++ b/vendor/github.com/containous/staert/kv.go
@@ -205,6 +205,12 @@ func collateKvRecursive(objValue reflect.Value, kv map[string]string, key string
        }
        switch kind {
        case reflect.Struct:
+               if m := objValue.MethodByName("MarshalJSON"); m.IsValid() {
+                       val := m.Call([]reflect.Value{})
+                       kv[name] = base64.StdEncoding.EncodeToString(val[0].Bytes())
+                       return nil
+               }
                for i := 0; i < objValue.NumField(); i++ {
                        objType := objValue.Type()
                        if objType.Field(i).Name[:1] != strings.ToUpper(objType.Field(i).Name[:1]) {

This is quite specific in that it checks for a MarshalJSON method which jose.JsonWebKey exposes and if that's the case calls that rather than the regular StoreConfig.
From my initial testing this seems to serialize fine with the changes below
diff --git a/cluster/datastore.go b/cluster/datastore.go
index 538d70b..51996c2 100644
--- a/cluster/datastore.go
+++ b/cluster/datastore.go
@@ -235,11 +235,11 @@ func (s *datastoreTransaction) Commit(object Object) error {
                return fmt.Errorf("Transaction already used, please begin a new one")
        }
        s.Datastore.meta.object = object
-       err := s.Datastore.meta.Marshall()
-       if err != nil {
-               return fmt.Errorf("Marshall error: %s", err)
-       }
-       err = s.kv.StoreConfig(s.Datastore.meta)
+       //err := s.Datastore.meta.Marshall()
+       //if err != nil {
+       //      return fmt.Errorf("Marshall error: %s", err)
+       //}
+       err := s.kv.StoreConfig(s.Datastore.meta)
        if err != nil {
                return fmt.Errorf("StoreConfig error: %s", err)
        }

I haven't started looking at UnMarshalling yet but thought I'd give a status update to see if this can be a workable solution or have no chance of getting accepted.

@Paxxi Did you ultimately find a solution or workaround for this? We're having the same problem trying to figure out how to make a Consul (or etcd) backend work with a 4M+ acme.json file.

The patch I posted above handles the serialization part correctly but I haven't gotten any time to implement deserialization unfortunately. Because of issues unrelated to Traefik with Docker swarm the project was put on hold.

Is there any interest in accepting a patch for this? I'm happy to write one, but only if there's a chance of getting it merged - I'm not sure how easy it would be to do this in a transparent way, so it'd probably have to be a major version release. Thoughts?

@jgwmaxwell We are open to a contribution on this. But beware, this will be a bit tricky. Long story short, there is an issue if don't marshall to json before storing the struct into the kv store. We need to first marshall to json, then we could use https://github.com/mitchellh/mapstructure to then store the json to the kv store. If you need some help, join us on the Slack channel :)

Hi @emilevauge thanks for that. I've been auditing the code a little more, and I've got a couple of concerns about how possible this is without a significant rewrite of the ACME backend, unless I've misunderstood - it seems to want to load all the details about an "account", rather that just retrieving the specific cert being interacted with. What are your thoughts on revisiting this? I'm aware the upgrade path for existing installations could be a little difficult too.

I'll hit you up on slack to discuss this a bit further - I'm not sure our use-case is going to match well with the majority of users' here. Will copy discussion here for discoverability later.

I'm not going ahead with this now, as our use case isn't really in the majority of users best interests (managing 10k+ domains).

We like to see this change and we have only 100 domains when traefik ACME stop working.  We uses Traefik to load balance websites that are deployed with Jenkins from GIT and we want all sites to have https.

In the mean time in order to workaround the issue i made a docker image of consul v1.0.2 accepting 5Mb key's value available on docker hub @ hexeo/consul:big-key.
https://hub.docker.com/r/hexeo/consul/
If no solution can be found on the traefik side, i am willing to make an automated build with the same patch.

Closed by #2814.

@nekuz0r what path did you take? Increasing storage, adding gzip w/o base64 IMHO just postpones the problem, but maybe as @jgwmaxwell pointed out, this is not a standard traefik use case and for most user 100 or so domains is enough... not for us though.

@emilevauge are there any metrics on how many domains can be stored with the gzip compression enabled now?

Has anything been done to actually solve this?
I currently have to setup a Traefik instance that is used to handle redirects on a lot of domains to a single domain. The list is growing to 100+ and I'm hitting the dreaded StoreConfig error: Unexpected response code: 413 (Value exceeds 524288 byte limit) error.
What are my options?
More specific error message (filtered some values):
{
   "level":"error",
   "msg":"Error Saving ACME account &{
     Email:xxx
     Registration:0xc0001e2d20
     PrivateKey:[xxx]
     KeyType:4096
     DomainsCertificate:{
       Certs:[0xc000f940c0 and 108 more entries]
       lock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0}
     }
     ChallengeCerts:map[]
     HTTPChallenge:map[
      JUHY7Qx6un3aEK2dnW6wvA-fLfMucOEtou48dWCK94g:map[domain.tld:map[xxx]]
      75 more entries
     ]}: StoreConfig error: Unexpected response code: 413 (Value exceeds 524288 byte limit)",
   "time":"2019-05-06T09:16:29Z"
}

Feels to me like the acme certificates could be stored on a per domain level rather than one gigantic map/string?

@alcohol You're absolutely right, we ran into this exact problem with a large number of certificates on Traefik.
For what it's worth, we ended up switching to Nginx Ingress with Jetstack Cert Manager and it works amazingly well. Each certificate is stored as its own k8s secret, so it's infinitely scalable.


Feature request: how to fail a build for particular code fixes?

Search Terms
code fix warn error fail build lint tslint
Suggestion
Some code fixes, such as @elizabethdinella's super exciting #25082, are important enough that some projects would want to fail a build if any code actions of that particular type are suggested. Fixes such as async/await conversion and unused variable removals would be helpful, while optional fixes such as converting members to getter+setter pairs likely wouldn't be. Projects would likely need to configure their own preferences.
Use Cases
Suppose you have a repository that's been fully converted to async/await and someone sends a pull request containing a .then can can be converted away using a code fix. We would want to fail the build for the user not having done so. How?
Examples
One option would be for TypeScript to have a setting on the CLI & tsconfig.json for code fixes to emit as diagnostic errors. This would require each code fix to have a friendly name:
{
  "compilerOptions": {
    "codeFixStrictness": {
      "async-await-from-then": "error"
    },
  },
  "include": ["src/**/*.ts"]
}

Alternately, we could write TSLint rules for each code fix (or one rule with settings & AST walker logic?) per code fix. no-unused-variable uses the diagnostics emit to read TS-native warnings; we could do a similar thing with crawling the tree.
Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript / JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. new expression-level syntax)

I definitely get the appeal of this but have some real technical concerns.  There's also the pedestrian problem that the codefix code isn't in tsc.js, but obviously we can change that. We also freely rev the codefix code to include new fix scenarios and it'd be troublesome to have that qualify as a "breaking change".
It'd really make more sense to do this as an out-of-band tool since you'd presumably also want to actually run the fix and then maybe do some formatting afterwards. That's a lot easier to do from a commandline tool that's set up to do it with your team's logic. If someone took this on to make a high-quality high-config tool we could have it be a TS-specific kind of "codemod". We'd probably even use it ourselves to fix stuff on DefinitelyTyped.
I'd like to hear more about what people might use this for, though.

This hasn't gathered much feedback and, empirically, no one has yet bothered to write the tool that would make this happen, so this doesn't look like a good fit. @iisaduan is working on a tool that will do this, though, so cc'ing her for FYI on a possible scenario


How can I make it work for vue files?

Worked with this confign in context_commentstring
            vue = {
                __default= "<!-- %s -->",
                html = "<!-- %s -->",
                javascript = "// %s",
                script = "// %s",
                css = '/* %s */',
                comment = "<!-- %s -->",
            },

Hey @madhurachanna! Something like that should work indeed, but there should be no need for configuring Vue explicitly.
If you don't mind, I would like to figure out why the defaults didn't work for you :)
Are you using the JavaScript and CSS treesitter parsers so that they are injected into Vue files? Or are you using regex highlights inside the script and style sections? If you have installed the js & css parsers, then they should be automatically injected and the default configuration should work.
Could you paste your treesitter config here?

Hey, @JoosepAlviste
Can you add vue = '<!-- %s -->' inside M.config in internal.lua ?
By default comments works  with javascript and css section and not in template or html section of the code.
Adding just vue = '<!-- %s -->' works correctly.
Thank you.

My current treesitter config is
    context_commentstring = {
        enable = true,
        config = {
             -- Languages that have a single comment style
            typescript = '// %s',
            css = '/* %s */',
            scss = '/* %s */',
            html = '<!-- %s -->',
            svelte = '<!-- %s -->',
            vue = '<!-- %s -->',
        }
    },

Oh I see what the problem is. I have set the commentstring in Vue files in my own configuration in a FileType autocmd with setlocal commentstring=<!--\ %s\ -->. So that's why it worked for me.
I added vue = '<!-- %s -->' to the defaults and it seems to work without my extra configuration now.
I'll close this issue for now, but you can reopen it if it doesn't work or something else comes up.
Thanks for the help!

@JoosepAlviste . Will it be good idea to add vue under supported langues? in ReadMe file?
https://github.com/JoosepAlviste/nvim-ts-context-commentstring#adding-support-for-more-languages

@madhurachanna Of course, thanks for the reminder, added! 😄

This is not working for vue files.

Inside script block commentstring should be // %s
Inside style commentstring should be /* %s  */
Inside template block commentstring should be <!-- %s -->

I have this as a config as suggested by @madhurachanna
	context_commentstring = {
		enable = true,
		enable_autocmd = false,
		config = {
			vue = {
				__default = "<!-- %s -->",
				html = "<!-- %s -->",
				javascript = "// %s",
				script = "// %s",
				css = "/* %s */",
				comment = "<!-- %s -->",
			},
		},
	},

Note: I'm using Comment.nvim and Vue 3

Hey @shubham-cpp! It should work in Vue files by default without any additional configuration. Can you try this?
	context_commentstring = {
		enable = true,
		enable_autocmd = false,
	},

If it still doesn't work then could you open a new issue with a minimal config for reproducing it?

the issue is persistent

open a new issue with a minimal config for reproducing it

will do this

Minimal Config
local packer = require("packer")
packer.init({
  enable = true,
  threshold = 0,
})
local use = packer.use
packer.reset()

use("wbthomason/packer.nvim")

use({
  "numToStr/Comment.nvim",
  config = function()
    require('Comment').setup()
  end
})

use({
  "nvim-treesitter/nvim-treesitter",
  run = ":TSUpdate",
  config = function()
    require("nvim-treesitter.configs").setup({
      ensure_installed = {
        "html",
        "css",
        "javascript",
        "typescript",
        "tsx",
        "vue",
      },
      highlight = { enable = true, use_languagetree = true },
      context_commentstring = {
        enable = true,
        enable_autocmd = false,
        --[[ config = {
        vue = {
        __default = "// %s ",
        html = "<!-- %s -->",
        javascript = "// %s",
        script = "// %s",
        css = "/* %s */",
        comment = "// %s",
        },
        }, ]]
      },
    })
  end
})

use({"JoosepAlviste/nvim-ts-context-commentstring"})

Ok everything is working with minimal config, maybe its an issue from my end. Sorry for false issue  😅

ensure_installed config is work for me


[Fix #8035] Fix a false positive for Lint/DeprecatedOpenSSLConstant

Fixes #8035.
This PR fixes the following false positive for Lint/DeprecatedOpenSSLConstant when using double quoted string argument.
% cat example.rb
OpenSSL::Cipher::AES256.new('cbc')
OpenSSL::Cipher::AES256.new("cbc")

% bundle exec rubocop --only Lint/DeprecatedOpenSSLConstant -a
(snip)

Offenses:

example.rb:1:1: W: [Corrected] Lint/DeprecatedOpenSSLConstant: Use
OpenSSL::Cipher.new('AES-256-cbc') instead of OpenSSL::Cipher::AES256.new('cbc').
OpenSSL::Cipher::AES256.new('cbc')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
example.rb:2:1: W: [Corrected] Lint/DeprecatedOpenSSLConstant: Use
OpenSSL::Cipher.new('AES-256-"cbc"') instead of OpenSSL::Cipher::AES256.new("cbc").
OpenSSL::Cipher::AES256.new("cbc")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

% cat example.rb
OpenSSL::Cipher.new('AES-256-cbc')
OpenSSL::Cipher.new('AES-256-"cbc"')

#8035 has another different false positive, which resolves as another PR.

Before submitting the PR make sure the following are checked:

[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Run bundle exec rake default. It executes all tests and RuboCop for itself, and generates the documentation.

Thanks!


Is it lack the vmacc_vv_<ty>_tu intrinsics in intrinsic_funcs.md?

Hi,
Is it lack the vmacc_vv_<ty>_tu intrinsics in intrinsic_funcs.md, but should be supported according to Tail Policy in the Intrinsics (rvv-intrinsic-rfc.md) of multily-add intrinsics? The same of vector reduction intrinsics.
Best regards,
Tin

Yes, you are right. We will fix it later, thanks!

@zakk0610 And the vcompress, vslide[down|up], vmv.s.x intrinsics aslo lack the _tu intrinsic. All missing includes:
# 11.13. Vector Single-Width Integer Multiply-Add Instructions
vmacc
vnmsac
vmadd
vnmsub

# 11.14. Vector Widening Integer Multiply-Add Instructions
vwmaccu
vwmacc
vwmaccsu
vwmaccus

# 13.6. Vector Single-Width Floating-Point Fused Multiply-Add Instructions
vfmacc
vfnmacc
vfmsac
vfmadd
vfmsub
vfnmsub

# 13.7. Vector Widening Floating-Point Fused Multiply-Add Instructions
vfwmacc
vfwnmacc
vfwmsac
vfwnmsac

# 14.1. Vector Single-Width Integer Reduction Instructions
vredsum
vredmaxu
vredmax
vredminu
vredmin
vredand
vredor
vredxor

# 14.2. Vector Widening Integer Reduction Instructions
vwredsumu
vwredsum

# 14.3. Vector Single-Width Floating-Point Reduction Instructions
vfredosum
vfredusum
vfredmax
vfredmin

# 14.4. Vector Widening Floating-Point Reduction Instructions
vfwredosum
vfwredusum

# 16. Vector Permutation Instructions
vmv.s.x
vfmv.s.f
vslideup
vslidedown
vcompress

https://github.com/riscv-non-isa/rvv-intrinsic-doc/issues/47 have updated a new batch of policy intrinsics. I think the issue is resolved now and we can close the issue :)


External helpers

Add an API for a user to customize invoking external helper functions.

coverage: 82.316% (-1.0%) from 83.35%
when pulling 00aea046a9284e5c7907a6bd19dbb5823e7bb662 on hawkinsw:external_helpers
into 2c163d66458c513de5ec54d7fcaed26bf86fcd21 on iovisor:main.