Add resource estimation chem tutorial

Adds a high level overview / tutorial of the chemistry Hamiltonians and how one might obtain / analyze costs from them. Depends on #520 and #508.
I'm thinking of adding another tutorial on how to add an algorithm (like the naive 2nd quantized chem hamiltonian I allude to in this), as a follow up.

while I'm generally a proponent of less formal diction, I think most of the "Ok"s can be removed from the notebook

OK!


[helm] bump era and use get_metadata for test

Testnet era increased during release-1.2
Also to separate helm test from metrics exposed, especially since there have been some state sync metrics changes. Just test via jsonrpc instead, and fail on HTTP error via curl flag.
Tested by deploying PFN locally via minikube, and running the helm test. Also CI passed :) ci-test/helm-test

/land


feat: add extmarks for markdown list markers

#46

I'm okay with adding this, but it should work for all filetypes, not just Markdown.

@lukas-reineke I don't know what we should do about neorg and org.
Neorg's unordered list items look like there and I don't think there is value in substituting them all.
    - Unordered list level 1
    -- Unordered list level 2
    --- Unordered list level 3
    ---- Unordered list level 4
    ----- Unordered list level 5
    ------ Unordered list level 6

Org treesitter doesn't differentiate between ordered and unordered lists, so it will not be possible.

I found a bug that when you horizontally scroll the window and the first column is invisible, the extmark is still shown on the first visible column. Will look into it when I have time.

Neorg's unordered list items look like this and I don't think there is value in substituting them all.

iirc Neorg has it's own conceal module included that substitutes them.


jewel: large resync with kraken content

Signed-off-by: Sébastien Han<EMAIL_ADDRESS>

jenkins test ceph_ansible-kraken-centos7_cluster

Everything is green. I'll fix Travis in another PR. Merging this.


Cleanup files

It wasn't deleting .diff files and old .fail screenshots. Now the screenshots folder will only have the original screenshots and the screenshots of failing tests.
options.screenshots will take full path relative to the root folder of the project.
phantomcss: {
options: {
screenshots: 'autotest/ui-tests/reference-screenshots',

I'm happy to take a look at this again if you want to get it caught up with master (or better yet, the 1.0 branch)


pollock population crashing

I'm getting the pollock population to crash rather quickly...
CAA simplethresh  FmsySim       -30        -1      SIM       -30        -1 hindcastMean     1
It looks like pollock gets initialized around 2.69e6 (mt?) and quickly goes to <100 mt by y=48 (so I think 13 years of fishing). I can't quite tell if this is why I'm getting this warning message:
1: In get_FBRP(parmgt = parmgt, parpop = parpop, parenv = parenv, Rfun_lst = Rfun_BmsySim,
get_FBRP: maximum yield occurs at endpoint of candidate F values
Not sure if this is a known or unknown issue.  Or user error.

I'm going to close this.  I think it's been fixed. If not, I'll open a new issue


ollama 0.1.31 -> 0.1.33

Description of changes

release notes: https://github.com/ollama/ollama/releases/tag/v0.1.33
synchronize ollama patches with ollama/llm/patches in 0.1.33
removed replacement $CXX with g++ in postPatch for compat with ollama/llm/generate/gen_common.sh in 0.1.33

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Will ROCm break with this still @abysssol #304823

I tested this (by running nix-build -A ollama.passthru.tests in the nixpkgs root), and 0.1.33 has the same problem as 0.1.32 (see #304823) in that it breaks the rocm compilation. It seems that everything builds right, until the end when a check in the build script finds that the compiled artifact doesn't have any dependencies on the rocm libraries. I did test what happens if I disable the check and run the compiled ollama binary anyway, but it just crashes almost instantly after running it.
So, I'm against merging this into the current nixpkgs-unstable, since this is a breaking change that removes support for rocm, and breaking changes are restricted since the next stable release will be soon (#303286).
I may be alright with merging this into the next unstable after nixos-24.05 has released, but until then I think it would be a bad idea to merge a breaking change.

ollama 0.1.34 released

If possible please move the package derivation to pkgs/by-name

ollama 0.1.36 has been released
https://github.com/ollama/ollama/releases/tag/v0.1.36

ollama 0.1.37 has been released

I tested this (by running nix-build -A ollama.passthru.tests in the nixpkgs root), and 0.1.33 has the same problem as 0.1.32 (see #304823) in that it breaks the rocm compilation. It seems that everything builds right, until the end when a check in the build script finds that the compiled artifact doesn't have any dependencies on the rocm libraries. I did test what happens if I disable the check and run the compiled ollama binary anyway, but it just crashes almost instantly after running it.

as far as missing dependencies, I was able to get passed this using the rocm-core package in rocmPath. There is a collision with rocmPackages-clr so I removed it for now to test.
  rocmPath = buildEnv {
    name = "rocm-path";
    paths = [
      rocmPackages.hipblas
      rocmPackages.rocblas
      rocmPackages.rocsolver
      rocmPackages.rocsparse
      rocmPackages.rocm-device-libs
      rocmClang
      rocmPackages.rocm-core
    ];
  };

I also had to set the CLBlast_DIR path and append some paths to CMAKE_PREFIX_PATH at the pre-build step for this to work for me.
  preBuild = ''
    # disable uses of `git`, since nix removes the git directory
    export OLLAMA_SKIP_PATCHING=true
    # build llama.cpp libraries for ollama
    # set CLBlast_DIR and append ROCM path to CMAKE_PREFIX_PATH
    ${lib.optionalString enableRocm
      ''export CLBlast_DIR="${clblast}/lib/cmake/CLBlast" \
      export CMAKE_PREFIX_PATH="${rocmPath}:${rocmPackages.rocm-comgr}/lib/cmake:$CMAKE_PREFIX_PATH" ''}
    go generate ${lib.optionalString enableRocm ''-tags rocm''} ./...
  '';

However, this brings me to other errors, specifically with building for gfx1010. It seems like some libraries are still not built correctly. I have played around with the build script but have had no luck for now.

I haven't fully tested, but I was able to get ollama to see my AMD card. It may just work?
Here's what I changed from your patch:
diff --git a/pkgs/tools/misc/ollama/default.nix b/pkgs/tools/misc/ollama/default.nix
index 1934ac80a..b0dc7a9da 100644
--- a/pkgs/tools/misc/ollama/default.nix
+++ b/pkgs/tools/misc/ollama/default.nix
@@ -24,7 +24,7 @@

 , config
   # one of `[ null false "rocm" "cuda" ]`
-, acceleration ? null
+, acceleration ? "rocm"
 }:

 let
@@ -104,6 +104,13 @@ let
   };

   runtimeLibs = lib.optionals enableRocm [
+    rocmPackages.clr
+    rocmPackages.hipblas
+    rocmPackages.rocblas
+    rocmPackages.rocsolver
+    rocmPackages.rocsparse
+    rocmPackages.rocm-device-libs
+    rocmClang
     rocmPackages.rocm-smi
   ] ++ lib.optionals enableCuda [
     linuxPackages.nvidia_x11
@@ -166,6 +173,7 @@ goBuild ((lib.optionalAttrs enableRocm {
   postPatch = ''
     # replace inaccurate version number with actual release version
     substituteInPlace version/version.go --replace-fail 0.0.0 '${version}'
+    substituteInPlace llm/generate/gen_linux.sh --replace-fail 'exit 1' ""
   '';
   preBuild = ''
     # disable uses of `git`, since nix removes the git directory

Obviously changing the default acceleration to rocm was just so I could run
nix build .#ollama

I don't know if there is a way to enable rocm when running that build command.
Also, the substituteInPlace should really be a patch (or even an upstream bug?)
Basically the problem is that the upstream change here
https://github.com/ollama/ollama/pull/3218
made it so that we need the rocm libraries as runtime dependencies instead of build time, since they aren't fully pulled in until runtime. I think anyway, I haven't completely tested this.
Here is it working (at least to the point of seeing my AMD card):
$ HSA_OVERRIDE_GFX_VERSION="11.0.0" result/bin/ollama serve
time=2024-05-13T13:58:50.843-04:00 level=INFO source=images.go:828 msg="total blobs: 0"
time=2024-05-13T13:58:50.843-04:00 level=INFO source=images.go:835 msg="total unused blobs removed: 0"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullModelHandler-fm (5 handlers)
[GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateModelHandler-fm (5 handlers)
[GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushModelHandler-fm (5 handlers)
[GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyModelHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteModelHandler-fm (5 handlers)
[GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowModelHandler-fm (5 handlers)
[GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
[GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListModelsHandler-fm (5 handlers)
[GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListModelsHandler-fm (5 handlers)
[GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
time=2024-05-13T13:58:50.843-04:00 level=INFO source=routes.go:1071 msg="Listening on <IP_ADDRESS>:11434 (version 0.1.33)"
time=2024-05-13T13:58:50.843-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1030226676/runners
time=2024-05-13T13:58:50.987-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 rocm]"
time=2024-05-13T13:58:50.987-04:00 level=INFO source=gpu.go:96 msg="Detecting GPUs"
time=2024-05-13T13:58:50.988-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-13T13:58:50.988-04:00 level=WARN source=amd_linux.go:49 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-05-13T13:58:50.988-04:00 level=INFO source=amd_linux.go:217 msg="amdgpu memory" gpu=0 total="20464.0 MiB"
time=2024-05-13T13:58:50.988-04:00 level=INFO source=amd_linux.go:218 msg="amdgpu memory" gpu=0 available="20464.0 MiB"

@volfyd I tested these changes with 0.1.37, but it doesn't appear to work for me; I'm not sure if I'm doing something wrong. My changes are in this branch. I did test with rocmClang included as well like your diff, that didn't work either.
Ollama does run, but doesn't detect the libraries. This is the relevant section of ollama's debug logs. I also confirmed that only the cpu is used when actually running a model.
time=2024-05-14T06:38:09.748-04:00 level=INFO source=routes.go:1052 msg="Listening on <IP_ADDRESS>:11434 (version 0.1.37)"
time=2024-05-14T06:38:09.753-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama902218830/runners
time=2024-05-14T06:38:09.869-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 rocm]"
time=2024-05-14T06:38:09.869-04:00 level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-05-14T06:38:09.870-04:00 level=WARN source=amd_linux.go:346 msg="amdgpu detected, but no compatible rocm library found.  Either install rocm v6, or follow manual install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual-install"
time=2024-05-14T06:38:09.870-04:00 level=WARN source=amd_linux.go:278 msg="unable to verify rocm library, will use cpu" error="no suitable rocm found, falling back to CPU"
time=2024-05-14T06:38:09.870-04:00 level=INFO source=types.go:71 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="62.0 GiB" available="2.6 GiB"

@abysssol  Maybe you need HSA_OVERRIDE_GFX_VERSION?
I tried 0.1.37 and it is working for me.
diff --git a/pkgs/tools/misc/ollama/default.nix b/pkgs/tools/misc/ollama/default.nix
index fdda6ba3f1e8..73f9b08c2ca0 100644
--- a/pkgs/tools/misc/ollama/default.nix
+++ b/pkgs/tools/misc/ollama/default.nix
@@ -24,28 +24,28 @@

 , config
   # one of `[ null false "rocm" "cuda" ]`
-, acceleration ? null
+, acceleration ? "rocm"
 }:

 let
   pname = "ollama";
   # don't forget to invalidate all hashes each update
-  version = "0.1.31";
+  version = "0.1.37";

   src = fetchFromGitHub {
     owner = "jmorganca";
     repo = "ollama";
     rev = "v${version}";
-    hash = "sha256-Ip1zrhgGpeYo2zsN206/x+tcG/bmPJAq4zGatqsucaw=";
+    hash = "sha256-ZorOrIOWjXltxqOXNkFJ9190EXTAn+YcjZZhDBJsLqc=";
     fetchSubmodules = true;
   };
-  vendorHash = "sha256-Lj7CBvS51RqF63c01cOCgY7BCQeCKGu794qzb/S80C0=";
+  vendorHash = "sha256-zOQGhNcGNlQppTqZdPfx+y4fUrxH0NOUl38FN8J6ffE=";
   # ollama's patches of llama.cpp's example server
   # `ollama/llm/generate/gen_common.sh` -> "apply temporary patches until fix is upstream"
   # each update, these patches should be synchronized with the contents of `ollama/llm/patches/`
   llamacppPatches = [
     (preparePatch "03-load_exception.diff" "sha256-1DfNahFYYxqlx4E4pwMKQpL+XR0bibYnDFGt6dCL4TM=")
-    (preparePatch "04-locale.diff" "sha256-r5nHiP6yN/rQObRu2FZIPBKpKP9yByyZ6sSI2SKj6Do=")
+    #(preparePatch "04-locale.diff" "sha256-r5nHiP6yN/rQObRu2FZIPBKpKP9yByyZ6sSI2SKj6Do=")
   ];

   preparePatch = patch: hash: fetchpatch {
@@ -103,6 +103,13 @@ let
   };

   runtimeLibs = lib.optionals enableRocm [
+    rocmPackages.clr
+    rocmPackages.hipblas
+    rocmPackages.rocblas
+    rocmPackages.rocsolver
+    rocmPackages.rocsparse
+    rocmPackages.rocm-device-libs
+    rocmClang
     rocmPackages.rocm-smi
   ] ++ lib.optionals enableCuda [
     linuxPackages.nvidia_x11
@@ -164,9 +171,10 @@ goBuild ((lib.optionalAttrs enableRocm {
   ] ++ llamacppPatches;
   postPatch = ''
     # replace a hardcoded use of `g++` with `$CXX` so clang can be used on darwin
-    substituteInPlace llm/generate/gen_common.sh --replace-fail 'g++' '$CXX'
+    #substituteInPlace llm/generate/gen_common.sh --replace-fail 'g++' '$CXX'
     # replace inaccurate version number with actual release version
     substituteInPlace version/version.go --replace-fail 0.0.0 '${version}'
+    substituteInPlace llm/generate/gen_linux.sh --replace-fail 'exit 1' ""
   '';
   preBuild = ''
     # disable uses of `git`, since nix removes the git directory

So I run the daemon here:
HSA_OVERRIDE_GFX_VERSION="11.0.0" ~/..code/nixpkgs/result/bin/ollama serve

and then in another window
$ ~/..code/nixpkgs/result/bin/ollama --version
ollama version is 0.1.37

~                                                                                lhuhn@chlorine
$ ~/..code/nixpkgs/result/bin/ollama run llama3
pulling manifest
pulling 00e1317cbf74... 100% ▕███████████████████████████████████████████████████████▏ 4.7 GB
pulling 4fa551d4f938... 100% ▕███████████████████████████████████████████████████████▏  12 KB
pulling 8ab4849b038c... 100% ▕███████████████████████████████████████████████████████▏  254 B
pulling 577073ffcc6c... 100% ▕███████████████████████████████████████████████████████▏  110 B
pulling ad1518640c43... 100% ▕███████████████████████████████████████████████████████▏  483 B
verifying sha256 digest
writing manifest
removing any unused layers
success
>>> Hello.
Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?

>>>

and then checking back with the logs in the first window:
time=2024-05-14T13:38:58.973-04:00 level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0

...
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XT, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.30 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  4155.99 MiB
llm_load_tensors:        CPU buffer size =   281.81 MiB

I made an override. It's really messy because overriding Go builds is really messy.
_: super: {
  ollama = let
    version = "0.1.37";
    hash = "sha256-ZorOrIOWjXltxqOXNkFJ9190EXTAn+YcjZZhDBJsLqc=";
    vendorHash = "sha256-zOQGhNcGNlQppTqZdPfx+y4fUrxH0NOUl38FN8J6ffE=";
    src = super.fetchFromGitHub {
      owner = "jmorganca";
      repo = "ollama";
      rev = "v${version}";
      inherit hash;
      fetchSubmodules = true;
    };
  in
    (super.ollama.overrideAttrs (old: rec {
      inherit version src;
      patches = let
        preparePatch = patch: hash:
          super.fetchpatch {
            url = "file://${src}/llm/patches/${patch}";
            inherit hash;
            stripLen = 1;
            extraPrefix = "llm/llama.cpp/";
          };
      in
        [(builtins.head old.patches)]
        ++ [
          (preparePatch "02-clip-log.diff" "sha256-rMWbl3QgrPlhisTeHwD7EnGRJyOhLB4UeS7rqa0tdXM=")
          (preparePatch "03-load_exception.diff" "sha256-1DfNahFYYxqlx4E4pwMKQpL+XR0bibYnDFGt6dCL4TM=")
          (preparePatch "04-metal.diff" "sha256-Ne8J9R8NndUosSK0qoMvFfKNwqV5xhhce1nSoYrZo7Y=")
          (preparePatch "05-clip-fix.diff" "sha256-rCc3xNuJR11OkyiXuau8y46hb+KYk40ZqH1Llq+lqWc=")
        ];

      postPatch = ''
        # replace inaccurate version number with actual release version
        substituteInPlace version/version.go --replace-fail 0.0.0 '${version}'
        substituteInPlace llm/generate/gen_linux.sh --replace-fail 'exit 1' ""
      '';
    }))
    .override {
      buildGo122Module = args:
        super.buildGo122Module (args
          // {
            inherit version src vendorHash;
            postFixup = let
              runtimeLibs = [
                super.rocmPackages.clr
                super.rocmPackages.hipblas
                super.rocmPackages.rocblas
                super.rocmPackages.rocsolver
                super.rocmPackages.rocsparse
                super.rocmPackages.rocm-device-libs
                (super.linkFarm
                  "rocm-clang"
                  {
                    llvm = super.rocmPackages.llvm.clang;
                  })
                super.rocmPackages.rocm-smi
              ];
              rocmPath = super.buildEnv {
                name = "rocm-path";
                paths = runtimeLibs;
              };
            in ''
              # the app doesn't appear functional at the moment, so hide it
              mv "$out/bin/app" "$out/bin/.ollama-app"
              # expose runtime libraries necessary to use the gpu
              mv "$out/bin/ollama" "$out/bin/.ollama-unwrapped"
              makeWrapper "$out/bin/.ollama-unwrapped" "$out/bin/ollama" --set-default HIP_PATH '${rocmPath}' \
                --suffix LD_LIBRARY_PATH : '/run/opengl-driver/lib:${super.lib.makeLibraryPath runtimeLibs}'
            '';
          });
    };
};

@volfyd I tried setting HSA_OVERRIDE_GFX_VERSION to 10.3.0, but that seemed to have no effect. For reference, I have an rx 6950 xt, and it's worked correctly with previous versions of ollama.
Have you tested the changes on my branch to see if they work for you? If it doesn't work, then I must have just set something up wrong, but if it does work for you then there must be some strange hardware specific incompatibility.

Your nixpkgs works for me if I use the HSA_OVERRIDE_GFX_VERSION="11.0.0" environment variable. (I omited some output):
$ mkdir ~/..code/abysssol
$ cd ~/..code/abysssol
$ git clone --reference ~/..code/nixpkgs https://github.com/abysssol/nixpkgs.git
$ cd nixpkgs
$ git checkout ollama-update-0.1.37
$ sed -i -e 's/acceleration ? null/acceleration ? "rocm"/' pkgs/tools/misc/ollama/default.nix
$ nix build .#ollama
$ sudo systemctl stop ollama
$ result/bin/ollama serve
...
time=2024-05-15T10:21:12.215-04:00 level=INFO source=amd_linux.go:311 msg="no compatible amdgpu devices detected"
^C
$ HSA_OVERRIDE_GFX_VERSION="11.0.0" result/bin/ollama serve
...
time=2024-05-15T10:21:48.861-04:00 level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0
time=2024-05-15T10:21:48.861-04:00 level=INFO source=types.go:71 msg="inference compute" id=0 library=rocm compute=gfx1100 driver=0.0 name=1002:744c total="20.0 GiB" available="20.0 GiB"
^C

So then I got curious, and I removed the change I proposed (add the libraries as runtime dependencies):
$ vi pkgs/tools/misc/ollama/default.nix
$ git diff
diff --git a/pkgs/tools/misc/ollama/default.nix b/pkgs/tools/misc/ollama/default.nix
index dbe46f2fc4c6..69fe04da371c 100644
--- a/pkgs/tools/misc/ollama/default.nix
+++ b/pkgs/tools/misc/ollama/default.nix
@@ -24,7 +24,7 @@

 , config
   # one of `[ null false "rocm" "cuda" ]`
-, acceleration ? null
+, acceleration ? "rocm"
 }:

 let
@@ -107,9 +107,9 @@ let
   };

   runtimeLibs = lib.optionals enableRocm
-    (rocmLibs ++ [
+    [
       rocmPackages.rocm-smi
-    ])
+    ]
   ++ lib.optionals enableCuda [
     linuxPackages.nvidia_x11
   ];
$ nix build .#ollama
$ HSA_OVERRIDE_GFX_VERSION="11.0.0" result/bin/ollama serve
...
time=2024-05-15T10:47:09.055-04:00 level=INFO source=types.go:71 msg="inference compute" id=0 library=rocm compute=gfx1100 driver=0.0 name=1002:744c total="20.0 GiB" available="20.0 GiB"
^C

And that still worked! So my theory may be incorrect. I don't know why this is working on my system and not yours. Perhaps I have something different in my NixOS config? I will try again, reverting even more of the change...

OK, so here I am trying it with most of the "fixes" removed (except the build fix related to the exit 1)
I git reset to peel off your commit, and then unstage the bits I don't want, run git restore to get rid of them. Then the diff below is relative to a plain nixpkgs.
$ git reset --soft HEAD~1
$ lazygit
$ git restore .
$ sed -i -e 's/acceleration ? null/acceleration ? "rocm"/' pkgs/tools/misc/ollama/default.nix
$ git diff HEAD
diff --git a/pkgs/tools/misc/ollama/default.nix b/pkgs/tools/misc/ollama/default.nix
index fdda6ba3f1e8..2cd2792da1f0 100644
--- a/pkgs/tools/misc/ollama/default.nix
+++ b/pkgs/tools/misc/ollama/default.nix
@@ -24,28 +24,30 @@

 , config
   # one of `[ null false "rocm" "cuda" ]`
-, acceleration ? null
+, acceleration ? "rocm"
 }:

 let
   pname = "ollama";
   # don't forget to invalidate all hashes each update
-  version = "0.1.31";
+  version = "0.1.37";

   src = fetchFromGitHub {
     owner = "jmorganca";
     repo = "ollama";
     rev = "v${version}";
-    hash = "sha256-Ip1zrhgGpeYo2zsN206/x+tcG/bmPJAq4zGatqsucaw=";
+    hash = "sha256-ZorOrIOWjXltxqOXNkFJ9190EXTAn+YcjZZhDBJsLqc=";
     fetchSubmodules = true;
   };
-  vendorHash = "sha256-Lj7CBvS51RqF63c01cOCgY7BCQeCKGu794qzb/S80C0=";
+  vendorHash = "sha256-zOQGhNcGNlQppTqZdPfx+y4fUrxH0NOUl38FN8J6ffE=";
   # ollama's patches of llama.cpp's example server
   # `ollama/llm/generate/gen_common.sh` -> "apply temporary patches until fix is upstream"
   # each update, these patches should be synchronized with the contents of `ollama/llm/patches/`
   llamacppPatches = [
+    (preparePatch "02-clip-log.diff" "sha256-rMWbl3QgrPlhisTeHwD7EnGRJyOhLB4UeS7rqa0tdXM=")
     (preparePatch "03-load_exception.diff" "sha256-1DfNahFYYxqlx4E4pwMKQpL+XR0bibYnDFGt6dCL4TM=")
-    (preparePatch "04-locale.diff" "sha256-r5nHiP6yN/rQObRu2FZIPBKpKP9yByyZ6sSI2SKj6Do=")
+    (preparePatch "04-metal.diff" "sha256-Ne8J9R8NndUosSK0qoMvFfKNwqV5xhhce1nSoYrZo7Y=")
+    (preparePatch "05-clip-fix.diff" "sha256-rCc3xNuJR11OkyiXuau8y46hb+KYk40ZqH1Llq+lqWc=")
   ];

   preparePatch = patch: hash: fetchpatch {
@@ -161,10 +163,10 @@ goBuild ((lib.optionalAttrs enableRocm {
     # this also disables necessary patches contained in `ollama/llm/patches/`
     # those patches are added to `llamacppPatches`, and reapplied here in the patch phase
     ./disable-git.patch
+    # TODO: add reason
+    ./disable-lib-check.patch
   ] ++ llamacppPatches;
   postPatch = ''
-    # replace a hardcoded use of `g++` with `$CXX` so clang can be used on darwin
-    substituteInPlace llm/generate/gen_common.sh --replace-fail 'g++' '$CXX'
     # replace inaccurate version number with actual release version
     substituteInPlace version/version.go --replace-fail 0.0.0 '${version}'
   '';
diff --git a/pkgs/tools/misc/ollama/disable-lib-check.patch b/pkgs/tools/misc/ollama/disable-lib-check.patch
new file mode 100644
index<PHONE_NUMBER>00..8ce5fcb04e25
--- /dev/null
+++ b/pkgs/tools/misc/ollama/disable-lib-check.patch
@@ -0,0 +1,10 @@
+--- a/llm/generate/gen_linux.sh
++++ b/llm/generate/gen_linux.sh
+@@ -245,7 +245,6 @@
+     if [ $(cat "${BUILD_DIR}/bin/deps.txt" | wc -l ) -lt 8 ] ; then
+         cat "${BUILD_DIR}/bin/deps.txt"
+         echo "ERROR: deps file short"
+-        exit 1
+     fi
+     compress
+ fi
$ nix build .#ollama
$ HSA_OVERRIDE_GFX_VERSION="11.0.0" result/bin/ollama serve
...
time=2024-05-15T12:10:24.603-04:00 level=INFO source=types.go:71 msg="inference compute" id=0 library=rocm compute=gfx1100 driver=0.0 name=1002:744c total="20.0 GiB" available="20.0 GiB"
^C

And you can see that it still works! I should probably take ollama out of my NixOS config and see if that makes a difference.

So in my NixOS config, I have
hardware.opengl.extraPackages = with pkgs; [
  rocmPackages.clr.icd #following for GPU AI acceleration
  rocmPackages.rocm-smi
  rocmPackages.clr
  rocmPackages.hipblas
  rocmPackages.rocblas
  rocmPackages.rocsolver
  rocmPackages.rocm-comgr
  rocmPackages.rocm-runtime
  rocmPackages.rocsparse

  # unnecessary for OLLAMA ?
  amdvlk
  rocm-opencl-icd #gaming?
  rocm-opencl-runtime #gaming?
  libva #some hardware acceleration for stuff like OBS
  vaapiVdpau
  libvdpau-va-gl
];

If I remove that, I get
time=2024-05-15T14:48:47.092-04:00 level=WARN source=amd_linux.go:278 msg="unable to verify rocm library, will use cpu" error="no suitable rocm found, falling back to CPU"

Higher up @abysssol  you said

I did test what happens if I disable the check and run the compiled ollama binary anyway, but it just crashes almost instantly after running it.

I am not sure if "crash" means it exits with an error or without, but I have never seen it crash really. It either works if I run it with the environment variable and the necessary hardware.opengl.extraPackages or it falls back to CPU without crashing.

I am not sure if "crash" means it exits with an error or without, but I have never seen it crash really. It either works if I run it with the environment variable and the necessary hardware.opengl.extraPackages or it falls back to CPU without crashing.

I seem to remember testing it a few weeks ago and it crashing without any output/logs/error, somewhat like a segfault or linker error (from unavailable libs). However, my current testing has never had this outcome either, so I'm not sure if it's a false memory, or if I confused crashing as an outcome of a different experiment as being the result of removing the lib check, or if a different ollama version had different behavior in this regard. You can probably safely ignore that comment, as it seems to be irrelevant/outdated/false. My apologies for the confusion.
I tried adding the packages from your hardware.opengl.extraPackages into both the ollama package's runtimeLibs and into my hardware.opengl.extraPackages, and I tried this both with and without HSA_OVERRIDE_GFX_VERSION set. Unfortunately, I've still been unable to get ollama to use rocm. It seems that my ability to contribute to figuring this out will be greatly reduced as ollama seems simply unwilling to run rocm on my machine.
However, since this seems to work right for you it may work right for others, so it's probably worth putting these changes in whatever version ends up getting into unstable to at least allow correct functionality for some users.
Could you try removing everything from your hardware.opengl.extraPackages and adding that into the ollama package's runtimeLibs to see if that correctly exposes those libraries to ollama?
   runtimeLibs = lib.optionals enableRocm
     (rocmLibs ++ [
       rocmPackages.rocm-smi
+      rocmPackages.clr.icd
+      rocmPackages.rocm-comgr
+      rocmPackages.rocm-runtime
+      pkgs.rocm-opencl-icd
+      pkgs.rocm-opencl-runtime
+      pkgs.libva
+      pkgs.vaapiVdpau
+      pkgs.libvdpau-va-gl
+      pkgs.amdvlk
     ])
   ++ lib.optionals enableCuda [
     linuxPackages.nvidia_x11
   ];

If that works, then try removing libraries to find the minimum necessary libraries for ollama to function correctly. That way someone can use ollama without having to first add anything into their hardware.opengl.extraPackages, as the package will be self-contained.
Thank you for all your help with this.

I have an explanation!
ollama looks for rocmhipblas and rocmlocblas and requires them to be in the same directory. They can be in the same directory if they are installed by the NixOS system, but otherwise they will each be in their own directory.
https://github.com/ollama/ollama/blob/5bece945090b94a3f1eab03be48fb6f6b25e1e79/gpu/amd_linux.go#L33
	ROCmLibGlobs          = []string{"libhipblas.so.2*", "rocblas"} // TODO - probably include more coverage of files here...

https://github.com/ollama/ollama/blob/main/gpu/amd_common.go#L14
// Determine if the given ROCm lib directory is usable by checking for existence of some glob patterns
func rocmLibUsable(libDir string) bool {
	slog.Debug("evaluating potential rocm lib dir " + libDir)
	for _, g := range ROCmLibGlobs {
		res, _ := filepath.Glob(filepath.Join(libDir, g))
		if len(res) == 0 {
			return false
		}
	}
	return true
}

One thing to note. If I make the check (the check for hipblas and rocblas in the same directory) less strict, to only require that rocblas is found, then ollama appears to work with the graphics card.
diff --git a/pkgs/tools/misc/ollama/default.nix b/pkgs/tools/misc/ollama/default.nix
index 861f0901ea88..66aef69cd10e 100644
--- a/pkgs/tools/misc/ollama/default.nix
+++ b/pkgs/tools/misc/ollama/default.nix
@@ -24,7 +24,7 @@

 , config
   # one of `[ null false "rocm" "cuda" ]`
-, acceleration ? null
+, acceleration ? "rocm"
 }:

 let
@@ -106,24 +106,8 @@ let

   runtimeLibs = lib.optionals enableRocm [
     rocmPackages.rocm-smi
-    rocmPackages.clr.icd #following for GPU AI acceleration
-    rocmPackages.clr
     rocmPackages.hipblas
     rocmPackages.rocblas
-
-    rocmPackages.rocsolver
-    rocmPackages.rocm-comgr
-    rocmPackages.rocm-runtime
-    rocmPackages.rocsparse
-
-    # unnecessary for OLLAMA ?
-    pkgs.amdvlk
-    pkgs.rocm-opencl-icd #gaming?
-    pkgs.rocm-opencl-runtime #gaming?
-    pkgs.libva #some hardware acceleration for stuff like OBS
-    pkgs.vaapiVdpau
-    pkgs.libvdpau-va-gl
-
   ] ++ lib.optionals enableCuda [
     linuxPackages.nvidia_x11
   ];
@@ -187,6 +171,7 @@ goBuild ((lib.optionalAttrs enableRocm {
   postPatch = ''
     # replace inaccurate version number with actual release version
     substituteInPlace version/version.go --replace-fail 0.0.0 '${version}'
+    substituteInPlace gpu/amd_linux.go --replace-fail '"libhipblas.so.2*", "rocblas"' '"rocblas"'
   '';
   preBuild = ''
     # disable uses of `git`, since nix removes the git directory

ollama does not appear to explicitly load hipblas at runtime (but it does use it at build time) so I think the check in the code is just overly strict.

I've finally managed to figure out how to get the rocm build to work for me, all thanks to you @volfyd!
I couldn't have done it without your help; I was stuck, and had entirely given up on getting ollama to work right with rocm.
I've created pr #312608 for further testing and discussion. If all goes well (rocm works correctly for others and no further issues are found), I want to merge it as soon as the next stable nixos version is released.


Kendo Grid not able to select a row using the column header

I'm trying to select a row using Column header Id, but the Table row is failing to identify the element or column.
I'm using Kendo UI MVC with Atata framework (code version).
Login().Wait(PAGE_WAIT_TIME_SEC).ListPage()
.ListGrid.Should.BeVisible()
.ListGrid.Rows[r => r.Id == taskId]
.Click();
Exception:
2019-12-02 18:43:10.3402 ERROR OpenQA.Selenium.NoSuchElementException : Unable to locate visible "List" grid's "Id == "1560"" row element:

Search options: {Visibility=Visible, Timeout=5s, RetryInterval=0.5s, IsSafely=False}
at Atata.ControlListScopeLocator.GetElement(SearchOptions searchOptions, String xPathCondition) in C:\POC\Atata\Atata\Atata\ScopeSearch\ControlListScopeLocator.cs:line 30
at Atata.UIComponent1.OnGetScopeElement(SearchOptions searchOptions) in C:\POC\Atata\Atata\Atata\Components\UIComponent1.cs:line 135
at Atata.UIComponent.GetScopeElement(SearchOptions searchOptions) in C:\POC\Atata\Atata\Atata\Components\UIComponent.cs:line 106
at Atata.UIComponent.get_Scope() in C:\POC\Atata\Atata\Atata\Components\UIComponent.cs:line 74
at Atata.Control1.OnClick() in C:\POC\Atata\Atata\Atata\Components\Control1.cs:line 59
at Atata.TableRow1.OnClick() in C:\POC\Atata\Atata\Atata\Components\TableRow1.cs:line 31
at Atata.Control1.Click() in C:\POC\Atata\Atata\Atata\Components\Control1.cs:line 49
at Atata.Tests.DropdownPageTests.Dropdown_Open_SelectAnRow() in C:\POC\Atata\Atata.Tests\Atata.Tests\DropdownPageTests.cs:line 87

Do you have a column with the header text "Id"? This is what it was looking by default. It may be "ID", for example.
Also, there is an example https://atata.io/examples/kendoui-grid/ which might help a bit.
Otherwise, please provide a sample of HTML of your Kendo Grid with headers and some rows.

Yes. I do have an column header with text "Id". I understand we should not be using common word in naming convention, but thats how the web site is built. I can try to include a class name to the header and try to use FindByClass?

It is actually OK that it is named "Id". Then this column should be found. Could you please share the code sample of your Grid Row implementation class? And if possible, provide a sample of HTML of your Kendo Grid with headers and some rows.

Seems like the issue is on firing the Click event on the row. Using the header text, the row is found.
.ListGrid.Rows[r => r.Id == taskId]
.Click();


Update LICENSE

improvements

The license is already the unlicense and therefore void, might as well be something interesting in it.


bug: docker build workflow taking too much time

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug
After adding arm64 and amd64 docker build support in docker-publish workflow, it's taking almost 17 minutes to build and upload the images. We need to investigate the issue.

Steps To Reproduce
N/A
Expected Behavior
It should build the images quickly; go build step takes this much time. We need to reduce that to 6~7 minutes.
Screenshots
No response
Device Information [optional]
No response
Stack Trace [optional]
No response
Additional context
No response
Are you working on this issue?
Yes

@The-Debarghya PR #51 has been reverted, because it's taking longer than before.

After the latest changes, by adding cache layers and adding --platform=$BUILDPLATFORM  tag in Dockerfile, it's taking only 3 minutes to build amd64 and arm86 images. :rocket:


Native sessions for vs-code

There should be a way for Amalthea to create vs-code native sessions. Similarly to how we have #149 for Rstudio.
However in this case several question are very much open:

what is required to do this - is there a container image or something similar that amalthea can use or do we have to make this from scratch?
how will the session be accessed - over http or ssh?
how will authentication work
how to connect your local vscode (desktop) program to the one running in amalthea

The "what" that amalthea can could be one of:

https://github.com/gitpod-io/openvscode-server
https://github.com/coder/code-server

More useful stuff from Rok playing with gitopod:
but yeah, it seems possible to set up ssh the way you suggested and using certificates w/ oauth - this is the config file they set up
Host davhau-machnix-1uhxc0by1ts
HostName <IP_ADDRESS>
User gitpod
Port 59890
IdentityFile /var/folders/99/jng0jljd13db0h4jwbwtx4mh0000gn/T/gitpod_e8aa7536-1de7-403f-b9d2-450a285ca2f1_id_rsa
IdentitiesOnly yes

so locally they spin up an ssh session presumably to proxy to the remote server - I see this running:
rok              76202   0.0  0.1 409254400  21216   ??  Ss    3:27PM   0:00.11 /var/folders/99/jng0jljd13db0h4jwbwtx4mh0000gn/T/gitpod-local-companion-76011-LIBNQ2F5af7m
rok              76023   0.0  0.0 409085984   5488   ??  S     3:26PM   0:00.08 ssh -v -T -D 59850 -o ConnectTimeout=15 -F /var/folders/99/jng0jljd13db0h4jwbwtx4mh0000gn/T/gitpod_ssh_config-725-TiqFIs38bYmL davhau-machnix-1uhxc0by1ts
rok              76018   0.0  0.2 447054288  55056   ??  Ss    3:26PM   0:00.43 /Applications/Visual Studio Code.app/Contents/MacOS/Electron --ms-enable-electron-run-as-node /Users/rok/.vscode/extensions/ms-vscode-remote.remote-ssh-0.80.0/out/localServer.js {"serverId":1,"ipcHandlePath":"/var/folders/99/jng0jljd13db0h4jwbwtx4mh0000gn/T/vscode-ssh-askpass-1d9fd347a8387a44dd7a276915a723b39e24dfff.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","59850","-o","ConnectTimeout=15","-F","/var/folders/99/jng0jljd13db0h4jwbwtx4mh0000gn/T/gitpod_ssh_config-725-TiqFIs38bYmL","davhau-machnix-1uhxc0by1ts"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/rok/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-190185ec-da15b6fd3ef856477bf6f4fb29ba1b7af717770d-0.80.0/data.json"}
rok              75432   0.0  0.1 409257616  24736   ??  Ss    3:26PM   0:00.47 /var/folders/99/jng0jljd13db0h4jwbwtx4mh0000gn/T/gitpod-local-companion-725-KqaW48Z0sJ6r

If we use the gitopod code server then we are missing the vs code extension that gitopod provides that can be used to connect to a gitopod openvscode-server. This way we do not even have to use the browser to access the vscode sessions but we can connect to them directly from the local vscode instance.
I wonder if their vscode extension is open source - because that’s the missing piece

well, and setting up ssh via oauth in the first place

but I’ve found some good examples of that that don’t seem crazy hard

Here is the vscode extension used to access a vscode server from a local vscode installation: https://github.com/gitpod-io/openvscode-server/tree/gp-code/main/extensions/gitpod/src

Hi,
Seeing this as well as #149, are you exploring a path that would make Renku go in the direction of https://www.eclipse.org/che/ ?


How to Over-ride printer-uri ?

We have a Samsung CLX-6220FX Colour LASER multifunction printer that supports draft IPP v1.1 circa 2008 connecting via Ethernet. It supports IPv4 and IPv6. We operate an IPv6-only network and have discovered a bug in the printer firmware that we need to work around.
Specifically, although the printer listens on its IPv6 addresses it has a bug where it expects the printer-uri embedded in requests to contain its IPv4 URI! When the IPv6 URI is used the response is client-error-not-found.
We have proved, using ipptool -d uri2=$IPv4_URI  ipp://$IPv6_URI/?version=1.1 get-print-attributes-debug.test (our modified test script that uses ATTR uri printer-uri $uri2, that the printer will respond correctly if we embed the IPv4 URI in requests.
We need some advice on how we can over-ride this in CUPS whether through some configuration change or modifying the source and rebuilding. There appears to be code handling printer-uri in cups/dest.c but not being familiar with the codebase it isn't clear precisely what changes we'd need to make to ingest some custom configuration value and use it at this low level.

@iam-TJ There isn't a way to pass a different URI for general CUPS printing. If the printer has a mDNS hostname (something.local) you might be able to use that instead (and that will auto-switch between IPv6 and IPv4) but your better bet with this printer is just to use AppSocket (socket://[ipv6-address]:9100) since it doesn't properly support IPP.


Proposal: Limit minimal roughness

Godot version: 3.0.2
In the current implementation of physically based rendering (PBR) the roughness value is allowed to go very low, causing extreme flickering of the specular highlights when moving. Because of the normalization that's used in the specular calculations (energy conservation) you get tiny highlights with an insane brightness that are prone to aliasing.
This sometimes even results in black artifacts (I assume the values go out of range and bleed into a black halo due to the bloom/glow blurring).
Here's a screenshot of how extreme the differences can be:

Reproduce: Take the materials demo and enable "Glow" in the enviroment node. Then look at the second material "mirror" while moving. You might have to adjust the settings a bit more to get as extreme results as me but it should definitely be visible.
While it's a special case (roughness = 0), this produces too extreme artifacts / flickering in my opinion. Engines usually limit the roughness to a value that's low enough to still allow shiny surfaces while reducing the flickering to a minimum.
Possible solution: Increase the minimum roughness limit (if there alredy is one) or add one.
A limit which is adjustable from the editor if needed would be an alternative to hardcoding it.

Yeah, it should add a minimum roughness limit or add an Anti-Flicker option for Glow.

A roughness of 1 causes major graphical glitches when the specular mode is set to Phong:
GLES3:

GLES2 (all materials have Phong specular mode):

I'm just relaying info from #21513 here, since they both are a result of extreme roughness values being set. Maybe this issue should be renamed to "Limit extreme roughness values" or something, to better accommodate this (so I can close the other one as a duplicate.)

This should be fixed in Godot 3.2, closing. Note that Godot 4.0 also features an optional screen-space roughness limiter to decrease the amount of visible specular aliasing.


Add cssHandlingScheme property support

lgtm, thanks!

Should I update README.md ?

no worries! I took care of it. published as 0.7.6

Thanks


bump nsp to a version that does not include extraneous dependencies

nsp fixed an issue (https://github.com/nodesecurity/nsp/issues/185) in 2.8.1 that bundled extraneous dependencies. This had the effect of pulling in git-validate in one of our projects, which was overriding the husky git hooks.

Coverage remained the same at 20.106% when pulling bcfa8d578b53b3eeec15ae104e2962efe8a935c8 on bump_nsp into 35de2bbe80619f71cc2b5f630e93e30e0fd2f670 on master.

Coverage remained the same at 20.106% when pulling bcfa8d578b53b3eeec15ae104e2962efe8a935c8 on bump_nsp into 35de2bbe80619f71cc2b5f630e93e30e0fd2f670 on master.


coreos SELinux: mount invalid.

After reboot a GCP VM took long to start responding and serial console output was filled with:
SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Also a kontena vagrant plugin created node says this (so far once in the dmesg)
Might be related to https://github.com/coreos/bugs/issues/1580

Took me forever to figure out what the source of it was. Came from kubernetes trying to run weave. Long trail of errors to get here.

The CoreOS issue is very vauge, most of the problems mentioned there seem to have other causes unrelated to the dmesg warning.

Took me forever to figure out what the source of it was. Came from kubernetes trying to run weave. Long trail of errors to get here.

I assume the context for this might be in https://github.com/weaveworks/weave/issues/2881... that seems to be about SELinux in non-permissive mode, which isn't going to be relevant with Kontena ATM.


TypeORM and TS.ed after routes init not started

Informations

Type
Version

Bug/
4.x

Description
Hello.
I have been trying to integrate TS.ed with TypeORM for a long time. I found a potential error that does not allow simultaneous integration and using componentsScan. Ff I add the typeorm and componentsScan options simultaneously when the server is started, the $ afterRoutesInit hook will not be executed in my service.
EXAMPLE:
@ServerSettings({
    rootDir,
    mount: {
        '/v1': `${rootDir}/controllers/**/**.controller.{ts,js}`
    },
    componentsScan: [
        `${rootDir}/middlewares/**/**.{ts,js}`,
        `${rootDir}/services/**/**.{ts,js}`,
    ],
    typeorm: [
        {
            name: "default",
            type: "postgres",
            host: "<IP_ADDRESS>",
            port: 5432,
            username: "postgres",
            password: "postgres",
            database: "postgres",
            synchronize: true,
            logging: false,
            entities: [
                `${__dirname}/entity/*{.ts,.js}`
            ],
            migrations: [
                `${__dirname}/migrations/*{.ts,.js}`
            ],
            subscribers: [
                `${__dirname}/subscriber/*{.ts,.js}`
            ]
        }
    ],
    httpPort: process.env.PORT || 3000,
    httpsPort: false,
    acceptMimes: ['application/json', ''],
    swagger: {
        path: '/doc'
    },
})

@Service()
export class UserService implements AfterRoutesInit {

    private connection: Connection;

    constructor(
        private typeORMService: TypeORMService
    ) {
    }

    $afterRoutesInit() {
        this.connection = this.typeORMService.get();
        $log.warn('Connected to the Users');
    }
}

In this case the $ afterRoutesInit hook will not be executed.
Can someone help me? Is there any solution that can be circumvented by this error?
Acceptance criterias

[x] Example (log, usecase, etc...)
[x] Ts.ED version (bug/question)

Ok I'll check in the example here ASAP https://github.com/Romakita/example-ts-express-decorator/blob/4.0.0/example-typeorm

@Romakita in your example there's no information about componentsScan

@Romakita any results? I have to fix it asap

@OskarLebuda I'd just tested and push working example with componentScan.
I the $afterRoutesInit isn't triggered correctly it means the componentScan isn't correctly configured.
Are sure the rootDir is good ? Because for me rootDir should have __dirname as value.
    mount: {
        '/v1': `${rootDir}/controllers/**/**.controller.{ts,js}`
    },
    componentsScan: [
        `${rootDir}/middlewares/**/**.{ts,js}`,
        `${rootDir}/services/**/**.{ts,js}`,
    ],

Can you test the example on example-ts-decorator and compare your project with?
See you,
Romain

You can add debug: true on ServerSettings decorator to display Services/Controllers built by the Server.

@Romakita
Ok I see. The problem is in the middleware. I Inject UserService to my auth service which is inject to my auth middleware like:
auth.middleware.ts

@OverrideMiddleware(AuthenticatedMiddleware)
export class AuthMiddleware implements IMiddleware  {

    constructor(private authService: AuthService) {
    }

    async use(@EndpointInfo() endpoint: EndpointMetadata, @Req() request: any) {
        // auth logic
    }

}

auth.service.ts
@Service()
export class AuthService {

    constructor(
        @Inject() userService: UserService,
    ) {
    }

user.service.ts

@Service()
export class UserService implements OnInit, BeforeRoutesInit, OnRoutesInit, AfterRoutesInit, OnServerReady {

    private connection: Connection;

    constructor(
        @Inject() private typeORMService: TypeORMService
    ) {
    }

    $onInit(): Promise<any> | void {
        $log.warn('All services is ready');
    }

    $beforeRoutesInit(): Promise<any> | void {
        $log.warn('Controllers and routes isn\'t mounted');
    }

    $onRoutesInit(components: IComponentScanned[]): Promise<any> | void {
        $log.warn('Controllers and routes are being built');

    }

    $afterRoutesInit(): Promise<any> | void {
        $log.warn('Controllers and routes are built');
    }

    $onServerReady(): Promise<any> | void {
        $log.warn('Server is ready and listen the port');
    }

In this case hooks will not be executed.

@Romakita any results? I have to fix it asap

@OskarLebuda Are you sure the UserService is correctly loaded by the componentScan ?
Because Service Lifecycle works fine for me and the example with TypeORM works fine with the componentScan configuration (and by default, Ts.ED have a componentScan configured).
To know if your service is correctly loaded and built, add debug: true on your server, then send me de log displayed by the server.
@ServerSettings({
   debug: true, // Add this line
})

Note: @Inject is not necessary to Inject service in another service when the Symbol is a class.
See you,
Romain

I've added a new branch aligned with you convention naming and your configuration. This example works perfectly. No fix is required, but I'm sure your problem come from the configuration on ServerSettings ;)
https://github.com/Romakita/example-ts-express-decorator/tree/oskarlebuda/example-typeorm
See you
Romain

@Romakita  Actually it's not working.. Look at the log from console. The hooks are not executed. I just change the url to the database. It's 100% correct. Server starts, but after create come request I have an error.
{
    host: "<IP_ADDRESS>",
    port: 5432,
    username: "postgres",
    password: "postgres",
    database: "postgres",
}

[2018-09-25T19:30:00.869] [ERROR] [TSED] - {
  "error": {
    "status": 500,
    "message": "Cannot read property 'manager' of undefined",
    "stack": "TypeError: Cannot read property 'manager' of undefined\n    at UsersService.<anonymous> (/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/services/users.service.ts:47:35)\n    at Generator.next (<anonymous>)\n    at /Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/services/users.service.ts:16:71\n    at new Promise (<anonymous>)\nat __awaiter (/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/services/users.service.ts:12:12)\n    at UsersService.create (/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/services/users.service.ts:46:16)\n    at UsersCtrl.create (/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/controllers/users/users.ctrl.ts:31:34)\n    at HandlerBuilder.<anonymous> (/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/node_modules/@tsed/common/src/mvc/class/HandlerBuilder.ts:117:67)\n    at Generator.next (<anonymous>)\n    at /Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/node_modules/tslib/tslib.js:107:75"
  },
  "reqId": "1",
  "method": "POST",
  "url": "/v1/users",
  "duration": 36,
  "headers": {
    "content-type": "application/json",
    "authorization": "Bearer {{authToken}}",
    "content-length": "69",
    "host": "<IP_ADDRESS>:8080",
    "connection": "Keep-Alive",
    "user-agent": "Apache-HttpClient/4.5.5 (Java/1.8.0_152-release)",
    "accept-encoding": "gzip,deflate"
  },
  "body": {
    "firstName": "Oskar",
    "lastName": "Lebuda",
    "age": 10
  },
  "query": {},
  "params": {}
}

Server start info
><EMAIL_ADDRESS>start /Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm
> ts-node src/index.ts

[2018-09-25T19:25:23.665] [DEBUG] [TSED] - Initialize settings
[2018-09-25T19:25:23.668] [INFO ] [TSED] - settings.rootDir => /Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src
[2018-09-25T19:25:23.668] [INFO ] [TSED] - settings.env => development
[2018-09-25T19:25:23.668] [INFO ] [TSED] - settings.httpPort => 8080
[2018-09-25T19:25:23.668] [INFO ] [TSED] - settings.httpsPort => 8000
[2018-09-25T19:25:23.668] [INFO ] [TSED] - settings.version => 1.0.0
[2018-09-25T19:25:23.668] [INFO ] [TSED] - settings.uploadDir => /Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/uploads
[2018-09-25T19:25:23.668] [INFO ] [TSED] - settings.logger => { requestFields: undefined,
  debug: true,
  logRequest: true,
  jsonIndentation: 2 }
[2018-09-25T19:25:23.670] [INFO ] [TSED] - settings.scope => singleton
[2018-09-25T19:25:23.670] [INFO ] [TSED] - settings.errors => { headerName: 'errors' }
[2018-09-25T19:25:23.674] [INFO ] [TSED] - settings.mount => { '/v1': '/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/controllers/**/*.ctrl.{ts,js}' }
[2018-09-25T19:25:23.674] [INFO ] [TSED] - settings.exclude => [ '**/*.spec.ts', '**/*.spec.js' ]
[2018-09-25T19:25:23.675] [INFO ] [TSED] - settings.componentsScan => [ '/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/middlewares/**/*.middleware.{ts,js}',
  '/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/services/**/*.service.{ts,js}' ]
[2018-09-25T19:25:23.675] [INFO ] [TSED] - settings.acceptMimes => [ 'application/json' ]
[2018-09-25T19:25:23.675] [INFO ] [TSED] - settings.passport => {}
[2018-09-25T19:25:23.675] [INFO ] [TSED] - settings.typeorm => [ { name: 'default',
    type: 'postgres',
    host: '<IP_ADDRESS>',
    port: 5432,
    username: 'postgres',
    password: 'postgres',
    database: 'postgres',
    synchronize: true,
    logging: false,
    entities:
     [ '/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/entity/*{.ts,.js}' ],
    migrations:
     [ '/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/migrations/*{.ts,.js}' ],
    subscribers:
     [ '/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/subscriber/*{.ts,.js}' ] } ]
[2018-09-25T19:25:23.675] [INFO ] [TSED] - settings.swagger => { path: '/api-docs' }
[2018-09-25T19:25:23.675] [INFO ] [TSED] - Build services
[2018-09-25T19:25:23.677] [DEBUG] [TSED] - InjectorService loaded
[2018-09-25T19:25:23.677] [DEBUG] [TSED] - ServerSettingsService loaded
[2018-09-25T19:25:23.677] [DEBUG] [TSED] - ExpressApplication loaded
[2018-09-25T19:25:23.677] [DEBUG] [TSED] - HttpServer loaded
[2018-09-25T19:25:23.677] [DEBUG] [TSED] - HttpsServer loaded
[2018-09-25T19:25:23.677] [DEBUG] [TSED] - JsonSchemesService built
[2018-09-25T19:25:23.677] [DEBUG] [TSED] - ConverterService built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - ArrayConverter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - DateConverter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - MapConverter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - PrimitiveConverter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - SetConverter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - SymbolConverter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - ParseService built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - BodyParamsFilter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - CookiesFilter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - HeaderParamsFilter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - LocalsFilter built
[2018-09-25T19:25:23.678] [DEBUG] [TSED] - PathParamsFilter built
[2018-09-25T19:25:23.679] [DEBUG] [TSED] - QueryParamsFilter built
[2018-09-25T19:25:23.679] [DEBUG] [TSED] - SessionFilter built
[2018-09-25T19:25:23.679] [DEBUG] [TSED] - FilterService built
[2018-09-25T19:25:23.679] [DEBUG] [TSED] - ValidationService built
[2018-09-25T19:25:23.679] [DEBUG] [TSED] - SendResponseMiddleware built
[2018-09-25T19:25:23.679] [DEBUG] [TSED] - GlobalAcceptMimesMiddleware built
[2018-09-25T19:25:23.679] [DEBUG] [TSED] - GlobalErrorHandlerMiddleware built
[2018-09-25T19:25:23.680] [WARN ] [TSED] - ===> UsersService { typeORMService: TypeORMService { _instances: Map {} } }
[2018-09-25T19:25:23.680] [WARN ] [TSED] - AuthMiddleware built
[2018-09-25T19:25:23.680] [DEBUG] [TSED] - AuthenticatedMiddleware built from class AuthMiddleware
[2018-09-25T19:25:23.680] [DEBUG] [TSED] - ResponseViewMiddleware built
[2018-09-25T19:25:23.680] [DEBUG] [TSED] - RouteService built
[2018-09-25T19:25:23.683] [DEBUG] [TSED] - ControllerService built
[2018-09-25T19:25:23.683] [DEBUG] [TSED] - MiddlewareService built
[2018-09-25T19:25:23.684] [DEBUG] [TSED] - LogIncomingRequestMiddleware built
[2018-09-25T19:25:23.684] [DEBUG] [TSED] - SwaggerService built
[2018-09-25T19:25:23.684] [DEBUG] [TSED] - SwaggerModule built
[2018-09-25T19:25:23.684] [DEBUG] [TSED] - TypeORMModule built
[2018-09-25T19:25:23.684] [DEBUG] [TSED] - RestCtrl built
[2018-09-25T19:25:23.684] [DEBUG] [TSED] - UsersCtrl built
[2018-09-25T19:25:23.685] [DEBUG] [TSED] - Call hook $onInit
[2018-09-25T19:25:23.685] [INFO ] [TSED] - Create connection with typeorm to database: default
[2018-09-25T19:25:23.685] [DEBUG] [TSED] - options: {"name":"default","type":"postgres","host":"<IP_ADDRESS>","port":5432,"username":"postgres","password":"postgres","database":"postgres","synchronize":true,"logging":false,"entities":["/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/entity/*{.ts,.js}"],"migrations":["/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/migrations/*{.ts,.js}"],"subscribers":["/Users/oskarlebuda/Downloads/example-ts-express-decorator-oskarlebuda/example-typeorm/src/subscriber/*{.ts,.js}"]}
[2018-09-25T19:25:23.710] [DEBUG] [TSED] - Call hook $onInit  promises built
[2018-09-25T19:25:23.766] [INFO ] [TSED] - Connected with typeorm to database: default
[2018-09-25T19:25:23.767] [DEBUG] [TSED] - Settings and injector loaded
[2018-09-25T19:25:23.767] [DEBUG] [TSED] - Mount middlewares
[2018-09-25T19:25:23.768] [DEBUG] [TSED] - Call hook $onMountingMiddlewares
[2018-09-25T19:25:23.814] [DEBUG] [TSED] - Call hook $beforeRoutesInit
[2018-09-25T19:25:23.815] [DEBUG] [TSED] - Call hook $onRoutesInit
[2018-09-25T19:25:23.815] [INFO ] [TSED] - Map controllers
[2018-09-25T19:25:23.815] [DEBUG] [TSED] - Call hook $afterRoutesInit
[2018-09-25T19:25:23.815] [INFO ] [TSED] - Routes mounted :
[2018-09-25T19:25:23.816] [INFO ] [TSED] -
┌────────────────┬────────────┬──────────────────────┐
│ Method         │ Endpoint   │ Class method         │
│────────────────│────────────│──────────────────────│
│ GET            │ /v1/rest/  │ RestCtrl.getRoutes() │
│────────────────│────────────│──────────────────────│
│ POST           │ /v1/users/ │ UsersCtrl.create()   │
│────────────────│────────────│──────────────────────│
│ GET            │ /v1/users/ │ UsersCtrl.getList()  │
└────────────────┴────────────┴──────────────────────┘
[2018-09-25T19:25:23.859] [DEBUG] [TSED] - Start server on http://<IP_ADDRESS>:8080
[2018-09-25T19:25:23.859] [DEBUG] [TSED] - Start server on https://<IP_ADDRESS>:8000
[2018-09-25T19:25:23.861] [INFO ] [TSED] - HTTP Server listen on http://<IP_ADDRESS>:8080
[2018-09-25T19:25:23.861] [INFO ] [TSED] - HTTP Server listen on https://<IP_ADDRESS>:8000
[2018-09-25T19:25:23.862] [DEBUG] [TSED] - Call hook $onServerReady
[2018-09-25T19:25:23.862] [INFO ] [TSED] - [default] Swagger JSON is available on http://<IP_ADDRESS>:8080/api-docs/swagger.json
[2018-09-25T19:25:23.862] [INFO ] [TSED] - [default] Swagger UI is available on http://<IP_ADDRESS>:8080/api-docs/
[2018-09-25T19:25:23.862] [INFO ] [TSED] - Started in 239 ms

I just add that everything is on the Mac OS X environment (mojave)

Here my logs:

[2018-09-26T07:47:46.521] [DEBUG] [TSED] - Initialize settings
[2018-09-26T07:47:46.524] [INFO ] [TSED] - settings.rootDir => /Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src
[2018-09-26T07:47:46.525] [INFO ] [TSED] - settings.env => development
[2018-09-26T07:47:46.525] [INFO ] [TSED] - settings.httpPort => 8080
[2018-09-26T07:47:46.525] [INFO ] [TSED] - settings.httpsPort => 8000
[2018-09-26T07:47:46.525] [INFO ] [TSED] - settings.version => 1.0.0
[2018-09-26T07:47:46.525] [INFO ] [TSED] - settings.uploadDir => /Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/uploads
[2018-09-26T07:47:46.525] [INFO ] [TSED] - settings.logger => { requestFields: undefined,
  debug: true,
  logRequest: true,
  jsonIndentation: 2 }
[2018-09-26T07:47:46.526] [INFO ] [TSED] - settings.scope => singleton
[2018-09-26T07:47:46.526] [INFO ] [TSED] - settings.errors => { headerName: 'errors' }
[2018-09-26T07:47:46.527] [INFO ] [TSED] - settings.mount => { '/v1': '/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/controllers/**/*.ctrl.{ts,js}' }
[2018-09-26T07:47:46.527] [INFO ] [TSED] - settings.exclude => [ '**/*.spec.ts', '**/*.spec.js' ]
[2018-09-26T07:47:46.527] [INFO ] [TSED] - settings.componentsScan => [ '/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/middlewares/**/*.middleware.{ts,js}',
  '/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/services/**/*.service.{ts,js}' ]
[2018-09-26T07:47:46.527] [INFO ] [TSED] - settings.acceptMimes => [ 'application/json' ]
[2018-09-26T07:47:46.527] [INFO ] [TSED] - settings.passport => {}
[2018-09-26T07:47:46.528] [INFO ] [TSED] - settings.typeorm => [ { name: 'default',
    type: 'postgres',
    host: 'localhost',
    port: 5432,
    username: 'postgres',
    password: '',
    database: 'testdb',
    synchronize: true,
    logging: false,
    entities:
     [ '/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/entity/*{.ts,.js}' ],
    migrations:
     [ '/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/migrations/*{.ts,.js}' ],
    subscribers:
     [ '/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/subscriber/*{.ts,.js}' ] } ]
[2018-09-26T07:47:46.528] [INFO ] [TSED] - settings.swagger => { path: '/api-docs' }
[2018-09-26T07:47:46.528] [INFO ] [TSED] - Build services
[2018-09-26T07:47:46.530] [DEBUG] [TSED] - InjectorService loaded
[2018-09-26T07:47:46.530] [DEBUG] [TSED] - ServerSettingsService loaded
[2018-09-26T07:47:46.530] [DEBUG] [TSED] - ExpressApplication loaded
[2018-09-26T07:47:46.530] [DEBUG] [TSED] - HttpServer loaded
[2018-09-26T07:47:46.530] [DEBUG] [TSED] - HttpsServer loaded
[2018-09-26T07:47:46.531] [DEBUG] [TSED] - JsonSchemesService built
[2018-09-26T07:47:46.531] [DEBUG] [TSED] - ConverterService built
[2018-09-26T07:47:46.532] [DEBUG] [TSED] - ArrayConverter built
[2018-09-26T07:47:46.532] [DEBUG] [TSED] - DateConverter built
[2018-09-26T07:47:46.532] [DEBUG] [TSED] - MapConverter built
[2018-09-26T07:47:46.532] [DEBUG] [TSED] - PrimitiveConverter built
[2018-09-26T07:47:46.533] [DEBUG] [TSED] - SetConverter built
[2018-09-26T07:47:46.534] [DEBUG] [TSED] - SymbolConverter built
[2018-09-26T07:47:46.534] [DEBUG] [TSED] - ParseService built
[2018-09-26T07:47:46.534] [DEBUG] [TSED] - BodyParamsFilter built
[2018-09-26T07:47:46.534] [DEBUG] [TSED] - CookiesFilter built
[2018-09-26T07:47:46.534] [DEBUG] [TSED] - HeaderParamsFilter built
[2018-09-26T07:47:46.535] [DEBUG] [TSED] - LocalsFilter built
[2018-09-26T07:47:46.535] [DEBUG] [TSED] - PathParamsFilter built
[2018-09-26T07:47:46.535] [DEBUG] [TSED] - QueryParamsFilter built
[2018-09-26T07:47:46.535] [DEBUG] [TSED] - SessionFilter built
[2018-09-26T07:47:46.536] [DEBUG] [TSED] - FilterService built
[2018-09-26T07:47:46.536] [DEBUG] [TSED] - ValidationService built
[2018-09-26T07:47:46.536] [DEBUG] [TSED] - SendResponseMiddleware built
[2018-09-26T07:47:46.536] [DEBUG] [TSED] - GlobalAcceptMimesMiddleware built
[2018-09-26T07:47:46.536] [DEBUG] [TSED] - GlobalErrorHandlerMiddleware built
[2018-09-26T07:47:46.537] [WARN ] [TSED] - ===> UsersService { typeORMService: TypeORMService { _instances: Map {} } }
[2018-09-26T07:47:46.537] [WARN ] [TSED] - AuthMiddleware built
[2018-09-26T07:47:46.537] [DEBUG] [TSED] - AuthenticatedMiddleware built from class AuthMiddleware
[2018-09-26T07:47:46.537] [DEBUG] [TSED] - ResponseViewMiddleware built
[2018-09-26T07:47:46.538] [DEBUG] [TSED] - RouteService built
[2018-09-26T07:47:46.542] [DEBUG] [TSED] - ControllerService built
[2018-09-26T07:47:46.542] [DEBUG] [TSED] - MiddlewareService built
[2018-09-26T07:47:46.542] [DEBUG] [TSED] - LogIncomingRequestMiddleware built
[2018-09-26T07:47:46.542] [DEBUG] [TSED] - SwaggerService built
[2018-09-26T07:47:46.543] [DEBUG] [TSED] - SwaggerModule built
[2018-09-26T07:47:46.543] [DEBUG] [TSED] - TypeORMModule built
[2018-09-26T07:47:46.543] [DEBUG] [TSED] - RestCtrl built
[2018-09-26T07:47:46.543] [DEBUG] [TSED] - RestCtrl built
[2018-09-26T07:47:46.544] [DEBUG] [TSED] - UsersCtrl built
[2018-09-26T07:47:46.544] [DEBUG] [TSED] - UsersCtrl built
[2018-09-26T07:47:46.544] [WARN ] [TSED] - ===> UsersService { typeORMService: TypeORMService { _instances: Map {} } }
[2018-09-26T07:47:46.544] [DEBUG] [TSED] - AuthService built
[2018-09-26T07:47:46.544] [DEBUG] [TSED] - UsersService built
[2018-09-26T07:47:46.545] [DEBUG] [TSED] - Call hook $onInit
[2018-09-26T07:47:46.545] [INFO ] [TSED] - Create connection with typeorm to database: default
[2018-09-26T07:47:46.545] [DEBUG] [TSED] - options: {"name":"default","type":"postgres","host":"localhost","port":5432,"username":"postgres","password":"","database":"testdb","synchronize":true,"logging":false,"entities":["/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/entity/*{.ts,.js}"],"migrations":["/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/migrations/*{.ts,.js}"],"subscribers":["/Users/romain.lenzotti/workspace/ts-projects/example-ts-express-decorator/example-typeorm/src/subscriber/*{.ts,.js}"]}
[2018-09-26T07:47:46.580] [WARN ] [TSED] - All services is ready
[2018-09-26T07:47:46.581] [DEBUG] [TSED] - Call hook $onInit  promises built
[2018-09-26T07:47:46.886] [INFO ] [TSED] - Connected with typeorm to database: default
[2018-09-26T07:47:46.886] [DEBUG] [TSED] - Settings and injector loaded
[2018-09-26T07:47:46.887] [DEBUG] [TSED] - Mount middlewares
[2018-09-26T07:47:46.887] [DEBUG] [TSED] - Call hook $onMountingMiddlewares
[2018-09-26T07:47:46.932] [DEBUG] [TSED] - Call hook $beforeRoutesInit
[2018-09-26T07:47:46.932] [WARN ] [TSED] - Controllers and routes isn't mounted
[2018-09-26T07:47:46.933] [DEBUG] [TSED] - Call hook $onRoutesInit
[2018-09-26T07:47:46.933] [INFO ] [TSED] - Map controllers
[2018-09-26T07:47:46.934] [WARN ] [TSED] - Controllers and routes are being built
[2018-09-26T07:47:46.934] [DEBUG] [TSED] - Call hook $afterRoutesInit
[2018-09-26T07:47:46.934] [INFO ] [TSED] - Routes mounted :
[2018-09-26T07:47:46.936] [INFO ] [TSED] -
┌────────────────┬────────────┬──────────────────────┐
│ Method         │ Endpoint   │ Class method         │
│────────────────│────────────│──────────────────────│
│ GET            │ /v1/rest/  │ RestCtrl.getRoutes() │
│────────────────│────────────│──────────────────────│
│ GET            │ /v1/rest/  │ RestCtrl.getRoutes() │
│────────────────│────────────│──────────────────────│
│ POST           │ /v1/users/ │ UsersCtrl.create()   │
│────────────────│────────────│──────────────────────│
│ GET            │ /v1/users/ │ UsersCtrl.getList()  │
│────────────────│────────────│──────────────────────│
│ POST           │ /v1/users/ │ UsersCtrl.create()   │
│────────────────│────────────│──────────────────────│
│ GET            │ /v1/users/ │ UsersCtrl.getList()  │
└────────────────┴────────────┴──────────────────────┘
[2018-09-26T07:47:46.942] [WARN ] [TSED] - Controllers and routes are built
this.connection => true
[2018-09-26T07:47:46.943] [DEBUG] [TSED] - Start server on http://<IP_ADDRESS>:8080
[2018-09-26T07:47:46.944] [DEBUG] [TSED] - Start server on https://<IP_ADDRESS>:8000
[2018-09-26T07:47:46.947] [INFO ] [TSED] - HTTP Server listen on http://<IP_ADDRESS>:8080
[2018-09-26T07:47:46.947] [INFO ] [TSED] - HTTP Server listen on https://<IP_ADDRESS>:8000
[2018-09-26T07:47:46.948] [DEBUG] [TSED] - Call hook $onServerReady
[2018-09-26T07:47:46.948] [INFO ] [TSED] - [default] Swagger JSON is available on http://<IP_ADDRESS>:8080/api-docs/swagger.json
[2018-09-26T07:47:46.948] [INFO ] [TSED] - [default] Swagger UI is available on http://<IP_ADDRESS>:8080/api-docs/
[2018-09-26T07:47:46.948] [WARN ] [TSED] - Server is ready and listen the port
[2018-09-26T07:47:46.949] [INFO ] [TSED] - Started in 446 ms

Ok I found a problem. It come from ts-node and tsc in the example. let me check the problem.

Great. Let me know if you find something 👍 🎉

@OskarLebuda Great news it's fixed with v4.30.9.
I've updated the example ;)
See you
Romain

It works! Thanks for your help! We can close this task


Incremental inserts

I would like to use river in feeder mode (pushing the changes to elasticsearch) by observing inserts into a database table.
Here is the relevant configuration section from my feeder script:
"jdbc" : {
...
"url" : "jdbc:hsqldb:hsql://localhost:9001/xdb",
"sql" : [
{
"statement" : "select * from public.observed_table where timestamp > ?",
"parameter" : [ "$river.state.last_active_begin" ]
}
],
...
}
I see multiple issues with this approach:

select statement has to rely on a timestamp column that may not be there always or may not be storing the timestamp information in UTC format. How do we handle that?
If the feeder has to be restarted again, what would be the value for last_active_begin it uses?
I wish there is a different strategy it uses to detect inserts: for example a custom strategy that checks table id column last synced or may be a timestamp in non UTC format, and stores that information in elasticsearch so that when we restart feeder, we know exactly where it would read from.

Any insights/suggestions are greatly appreciated!
-Madhav

You could've done something like this:
{
"statement" : "select * from public.observed_table where (? = 1) or (? > 1 and  timestamp > ?)",
"parameter" : [  "$river.state.counter ","$river.state.counter ","$river.state.last_active_begin" ]
}


Clean Docker images after ceph and minio tests.

Hoping this fixes https://github.com/rusoto/rusoto/issues/968 . 👍

Little more to go on the shell scripts: since the docker command is the last command run, the tests can fail but the build still passes.

I'd make cleaning another step in Semaphore CI. This way, the image isn't removed when the tests are executed locally.

👍 Added a note to the scripts and moved cleanup to the Semaphore job.

This is fixed in SemaphoreCI. Nothing in this PR is required. 👍


Switch to building the repo with VSSDK NuGet packages.

Instead of using the VSSDK on the machine, use the NuGet packages.
Building using RC packages fails on a Preview5 machine because it's still not cleanly separated out. Use a hack (that Roslyn put in today as well) to identify RC and switch packages based on that.
Add Prerequisites section so that it builds with the RC package.
Remove an errant BuildNumber property that broke local build versioning of <IP_ADDRESS>

@dotnet/project-system @mavasani @333fred @RaulPerez1

Tagging @MattGertz - this adds the prerequisites and make the repo buildable with RC bits so that we can dogfood.


Spacy upgrade to 3.0+

Hi there,
Just wondering - is there any reason spacy is locked with the old version spacy==2.2.4 in the main requirements.txt?
Spacy 3.0 was quite a big upgrade from 2.2.4, and 3.1.0 was just released today so it might make sense to look forward and make that a requirement instead.
I don't think any current implementations would break by this upgrade but I'm happy to make a PR for it and fix things if needed.

Tagging @samsontmr here!

#117  Upgraded spacy to 3.0.0


Discussion about CI and CD

With the PR #11, we will reach successful builds on CC7, Debian and Archlinux. We don't want to loose the compatibility with new commits, but we also want to avoid manually testing every release on every distribution manually. This is why we need CI. CI would also help us testing the compatibility between versions so that the superbuild is always in a working state.
There a multiple alternatives for CI :

GitLab CI : The CERN provides an instance where we can run tests. However, it is not possible to run GitHub integrated CI/CD for imported project, an Enterprise license a recent Premium version is required. The solution would be to move to GitLab and keep a public mirror on GitHub although the solution is not used for the other GEM projects. Using GitLab CI would also let us run private workers where tests involving hardware could be run.
Drone.io : It integrates well with GitHub and is mainly self-hosted. We'd only new one build server, ideally connected to GEM hardware for future tests. Can be a good solution if we do not reach 10.000 per year.
Travis-CI : While it is technically possible to run a self-hosted instance, it requires a lot of work and maintenance. This solution, which is integrable into GitHub, would then not allow tests involving hardware.

Regarding the CD, this is not high priority. In the short term, the superbuild will not be used to create packages, but will used to provide a development environment. Packaging the GEM software could definitively be done by calling each project packaging target, but that would require a environment where dependencies are installed.

As discussed in private, the GitLab-CI seems the most advantageous solution. We can rely on the CERN infrastructure with a very complete tool.
Did we converge on this ? @lmoureaux ?

Gitlab CI FTW.

The repository has been moved to the CERN GitLab in order to get the CI, so this issue is solved.


More string functionality

[ ] Regexes
[x] map
[ ] format
[x] intercalate
[x] words/unwords
[x] lines/unlines
[x] foldl
[ ] foldr

Hey @jkarni , is there a reason that foldr-string hasn't been implemented? It seem straightforward, which makes me wonder if I'm overlooking something. I'd be happy to add this.

@maxhallinan I think there is no reason, please go ahead and add this if you want!

Sounds good!


Release 0.12.1

There's been a couple of bugfixes to 0.12 which I think warrant a patch release before we start merging any new / breaking changes on the way to 0.13.
I'd like to publish this tomorrow evening (16th Sep). We can always make further patch releases if necessary (hopefully not).
I think it would be nice if a bugfix for #1185 was also included in this release. @kngwyu, I'm not sure if you're already working on one, I'll have a go at writing a fix tomorrow evening if I don't hear a reason not to.

As #1185 has been closed with conclusion that no fix needed, I'm going to proceed to release in a moment...

Thanks!


Use cached information to populate the Resources view Confluent Cloud single environment children

Summary of Changes

Use cached information to populate the Resources view Confluent Cloud single environment children,
issue #254. Makes the expansion of a single CCloud environment resource take 0 time, since no (redundant) deep read is performed. Makes use of what we learned right at login.
Follows similar pattern to the making use of cached knowledge of topics within a single cluster, but for the 'Confluent Cloud' children of the Resources panel (the ccloud environments, their kafka and schema registry clusters). We have all of this information in the resource manager cache, so use it.
Altered the 'refresh' icon / command for the resources panel to force a deep read (and re-cache).
Refreshing a single environment (as opposed to all CCloud environments) is near future work, https://github.com/confluentinc/vscode/issues/266

Any additional details or context that should be provided?

Pull request checklist
Please check if your PR fulfills the following (if applicable):
Tests

[x] Added new
[ ] Updated existing
[ ] Deleted existing

Other

[x] Does anything in this PR need to be mentioned in the user-facing CHANGELOG or README?
[x] Have you validated this change locally by packaging and installing the extension .vsix file?gulp clicktest

/sem-approve


Improvements to robustness of syncing

[x] create local genesis block that matches the rest of the protocol
[x] basic transaction encoding / decoding
[ ] validate chain structure in addition to individual blocks (note: partially done, prev hashes line up)
[x] local database validation optimizes local database (no longer needed, block ingestion creates optimized db by default)
[ ] local database validation reports non-contiguous parts of local chain and gets missing blocks
[x] create proper tiered structure of database "immature" -> "serialize buffer" (flushed every 1k) -> ancient (on disk)
[ ] sync from beginning (initial block download) detects incomplete block requests and resubmits
[x] ensure that database written from initial block download is optimized

Going to checkpoint this into main and release a new version that has the baseline syncing in place.
Deal with the remaining points here in a continuation PR.


chore: added codescan results to frontend optionally

Only appears if data found. This is mainly untracked but a possible option for users:

Looks great! My only suggestion is pulling the show/hide toggle and code snippet outside of the banner like this:
• Banner • Toggle Control Label (Hide Dockerfile | Show Dockerfile) • Code Snippet • Divider line

Going to do these suggestions in part of a common superset review later today most likely


cherri picked: move debug files to internal storage

Conflicts:
src/src/com/tns/JsDebugger.java

This is obsolete by #273


Response for unfound resource not JSON

When getting a single contact, if the resource can't be found, the API doesn't return usable JSON data. Instead it returns a string.
For example:
https://api.xero.com/api.xro/2.0/Contacts/some-imaginary-uuid
Response:
The resource you're looking for cannot be found
Please update this so the response is in a format consistent with the rest of the API, such as:
{     "Status": "Not Found",     "Errors": [       {           "Message": "The resource you're looking for cannot be found"       }     ]   }
...or something 🙂

@MatrixStefan
Thanks for the feedback - I will pass this along to our API team.  This issue is not something we can address in our OpenAPI specs or SDKs.  Our APIs have been around much longer than these specifications and therefore we do not create or update our APIs from these specs (it's the other way around).  We've written these OAS in order to generate SDKs to interact with existing APIs.
You can voice your feedback in a different forum so other developers can vote as well at https://xero.uservoice.com/forums/5528-xero-accounting-api
I'll be closing this issue now.


[Fix] #907 #908

Fix bug #907 #908

Could you please make sure the formatting checks pass? You can check them with:
poetry run black . --line-length=140 && poetry run pytype . --config .pytype.cfg -j 16

Hi Sarah, I add try and catch to the receiver and the sender. However, I can't test.

Could you please make sure the formatting checks pass? You can check them with:


Incompatible with Realistic Sleep

Realistic Sleep: https://modrinth.com/mod/realisticsleep
With this mod, waking up from a sleep won't reset Tiredness.
Also players won't be able to sleep at daytime, for this I'm not sure if the problem's actually on SleepRework's side

I am also getting this issue. I was wondering if there are any fixes planed for this or no?


Incorrect Docker name used for the vuestorefrontapi yarn restore step

Current behavior
If you follow the manual installation steps as outlined in https://docs.vuestorefront.io/guide/installation/linux-mac.html#install-the-vue-storefront-api
you will get the following error displayed when running the command:
docker exec -it vuestorefrontapi_app_1 yarn restore
Error: No such container: vuestorefrontapi_app_1
Expected behavior
No error, and the elastic.js restore works correctly.
Steps to reproduce the issue
docker exec -it vuestorefrontapi_app_1 yarn restore
Error: No such container: vuestorefrontapi_app_1
Repository
/vue-storefront/blob/master/docs/guide/installation/linux-mac.md
Can you handle fixing this bug by yourself?

[x] YES
[ ] NO

Which Release Cycle state this refers to? Info for developer.
Stable

[ ] This is a bug report for test version on https://test.storefrontcloud.io - In this case Developer should create branch from develop branch and create Pull Request 2. Feature / Improvement back to develop.
[ ] This is a bug report for current Release Candidate version on https://next.storefrontcloud.io - In this case Developer should create branch from release branch and create Pull Request 3. Stabilisation fix back to release.
[x] This is a bug report for current Stable version on https://demo.storefrontcloud.io and should be placed in next stable version hotfix - In this case Developer should create branch from hotfix or master branch and create Pull Request 4. Hotfix back to hotfix.

Environment details

Browser:
OS: MacOS 10.14.5
Node:
Code Version:

Additional information

Thanks @globaltopmedia ! :)


[cppgraphqlgen] update to v4.1.0

Library name: cppgraphqlgen
New version number: 4.1.0
The new version of cppgraphqlgen includes the latest GraphQL spec. Therefore, it will be nice if we can update the library.

Fix by #22411


Not everybody has logstash installed

We've got logstash installed on a production rails environemnt and I'd really like to use it to do more structured logging. The problem is, not everybody in dev has logstash installed, and so methods like reset and additional arguments to logging methods would completely fail.
Is there a straightforward way to resolve this? I tried building a logger that responds to those situations silently and delegated only if logstash was available, but that didn't work very well.

For anybody else running into this, I ended up adding an empty reset method to the base Logger class and passing a dict into the first argument (which seems to get converted to a string if it's not a logstash logger).


Cannot set geo restrictions on CLIENT_STEERING delivery services

Currently, you cannot set geo restrictions on CLIENT_STEERING DSs in the TO UI or the Traffic Portal.  Geo restrictions should be able to be set.

@dneuman64 - same with just STEERING DSs?

Yeah, I think so.  I can fix this.


Refactor Wiegand reading function

I'm trying to add some functionality to the Wiegand reader and I've noticed
that the code is duplicated between the official board and the others.
In extracting the code in the function I've minimized the changes.
While doing so I've also found a bug when using the keypad as a return statement
was commented out, for every keypress the function was going ahead and testing
if the user was in the memory, even if the uid wasn't filled in.
I don't have yet an official board to test (soon though!), but the code is
building properly.

sorry, not sure why this has been opened against the stable branc


Tabs widget causes decreasing font-size

Hi,
In a view generated by GII the fontsize turns out to be 16px, which is fine.  The view is generated by a Detailview.
When I add a Tab-widget to the view, the fontsize decreases into 10px. Not only for the view but for the entire page.
Questions: do you recognise the problem and is there a simple solution other than not using the tab-widget. If needed I can provide the source code of the view and screenshots of the application.

what causes

Hi, sorry to have bothered you... By fiddling around a bit I now use the yii\bootstrap4\Tabs and this works fine.
Issue can be closed


refactor(examples): divide components into packages and examples

Regarding Issue #327
Fix PR #281
Signed-off-by: Takuma TAKEUCHI<EMAIL_ADDRESS>

In this PR, I solved the problem Peter issued that package.json should reflects ones on other directories.

The automatic bot (The Travis CI) checked and said that there is some error.  But the log says that the error is regarding to packages/cactus-common/src/test/typescript/unit/key-converter.test.ts, which is not unrelated with my PR at all.

Error log: https://travis-ci.org/github/hyperledger/cactus/jobs/739153871
Then, please check my PR ignoring the above error? @petermetz @jonathan-m-hamilton @sfuji822

The automatic bot (The Travis CI) checked and said that there is some error. But the log says that the error is regarding to packages/cactus-common/src/test/typescript/unit/key-converter.test.ts, which is not related with my PR at all.

Error log: https://travis-ci.org/github/hyperledger/cactus/jobs/739153871

Then, please check my PR ignoring the above error? @petermetz @jonathan-m-hamilton @sfuji822

@takeutak Yeah sorry about that, a couple of our tests have been flaky recently, most of time it works if you just re-trigger the CI from the UI of TravisCI. Gonna do it right now, fingers crossed.

The automatic bot (The Travis CI) checked and said that there is some error. But the log says that the error is regarding to packages/cactus-common/src/test/typescript/unit/key-converter.test.ts, which is not related with my PR at all.

Error log: https://travis-ci.org/github/hyperledger/cactus/jobs/739153871

Then, please check my PR ignoring the above error? @petermetz @jonathan-m-hamilton @sfuji822

@takeutak Yeah sorry about that, a couple of our tests have been flaky recently, most of time it works if you just re-trigger the CI from the UI of TravisCI. Gonna do it right now, fingers crossed.

Re-ran the CI manually, now it passes. Hopefully we can fix this issue soon and get back to having a consistent CI experience.

I replied Peter's advice on PR #333 as the same pull-request.  Then I closed this PR.


x/build: add dev.boringcrypto automerger tool

We don't get to know when dev.boringcrypto stops merging cleanly with master, or when tests break.
On suggestion by @bradfitz, make a tool that periodically merges master into dev.boringcrypto.
If the merge fails, email a list; if the merge succeeds, the buildbots will run.

Seems no longer relevant due to #51940


Support non-utf8 file names

If given a non-utf8 filename, such as
$ touch `echo -e "notutf8\xbf.dat"`

SftpSession::read_dir(path_to_directory) will fail to list the directory, as filename in DirEntry is of type string and try_get_string throws an error when encountering a non-utf8 string.
    fn try_get_string(&mut self) -> Result<String, Error> {
        let bytes = self.try_get_bytes()?;
        String::from_utf8(bytes).map_err(|_| Error::BadMessage("unable to parse str".to_owned()))
    }

A solution could be to use OsString or PathBuf, I'll try to write a PR, but I've barely started climbing the Rust learning curve :)

I encountered the same problem

Here it is described why we cannot use PathBuf. I can't say how appropriate it would be to use OsString. As a temporary solution we can use #39 because it allows to use .as_bytes() and leave further work to the end user but still this is not a panacea.

Would just using a Vec<u8> instead of String be a way to go? I suppose it's not too different from #39, if as_bytes() returns the original array, except it would make it very explicit that filenames are just a vector of bytes, but with the trade-off that the API becomes less ergonomic 🤷 .

Yes

Would just using a Vec<u8> instead of String be a way to go?

but

API becomes less ergonomic

So I would like to find a more universal solution because I don't even like #39 that much

Yes

Would just using a Vec<u8> instead of String be a way to go?

but

API becomes less ergonomic

So I would like to find a more universal solution because I don't even like #39 that much

OsString would be an alternative then?

I could make a PR replacing with OsString @AspectUnk, don't see the disadvantage

@AspectUnk Made a most definitely working implementation with OsString in #44

After conducting research, I can say that UTF-16 support is not provided for in the specification because there is inconsistency regarding the size of a string character.
OsString like PathBuf is somewhat system-dependent, because its serialization will produce different results on different systems. The inconsistency in the size of the string character leads to a violation to packet integrity and byte order violations.
Linux:

Windows:

@gulrotkake Replacing String with Vec also won't yield results because the packet defines the number of characters, not bytes.
@kanpov The reason why your tests passed is because of the absence of tests on different systems.
Some leave the choice of encoding strings in UTF-16 to the end user, but warn that the remote server must also support it. It is possible to add this as a feature, but it is an extremely specific solution, so leaving #39 would be more appropriate. I may take care of it, but not very soon.
If I'm wrong about anything let me know.

So it's pretty much a choice between not having utf-8 or not supporting windows @AspectUnk ? I'd rather have the latter, but maybe there's another way

So it's pretty much a choice between not having utf-8 or not supporting windows @AspectUnk ? I'd rather have the latter, but maybe there's another way

Perhaps I misunderstood you, but it already supports utf-8 by default and at the same time windows, and #39 solves all problems with characters unsuitable for parsing.

Some leave the choice of encoding strings in UTF-16 to the end user, but warn that the remote server must also support it.

Btw, along with this you can add support for the unfinished version specification v6 (page 16) since it has a solution for sending charset.

When will the version be released?😂


Add header item "Content-Length"

Add item "Content-Length" to response header when shareFilesFromDirectory is being used to share files.
This PR is related to this issue #380

@nichbar Thanks for your contribution!


Convert to English

Request type :question:

[ ] :bug: Bug ( fixed or found bug )
[x] :new: New feather ( make new feather or add some things new in project )
[ ] :star2: Config ( change config files or install )
[ ] :rocket: performance ( when you do something that can improve performance )
[ ] :books: Document ( when code has documentation or document change )

Description :books:
Numbers and some Arabic and Persian characters have different ASCI codes than English numbers. This causes that if these characters are stored incorrectly in the database, it will not be possible to retrieve the correct data in the future when receiving information.
For this reason, it is necessary to apply the same format to the characters so as not to face problems
For example
Someone signup in our website with his PC. His pc's keyboard type is Arabic. Then after while he want to login with his phone,
But his phone keyboard type is English. if he use Persian or Arabic number for his password he can't login.
What stored in database with Arabic keyboard by PC=> 'username-۱۲۳۴'
What send with phone to login => 'uesrname-1234'
and 'username-1234' != 'username-۱۲۳۴'
How to solve this kind of issue
At store
public function create_user(Request $request) {
    User::query()->firstOrCreate(['username' => Str::toEnglish($request->username)], [
        'password' => Hash::make( Str::toEnglish($request->password) )
    ]);
    return 'success';
}

At fetch
public function login (Request $request) {
    $login = false;
    if ($user = User::query()->where('username', '=', Str::toEnglish($request->username'))->first()) {
        $login = Hash::check($user->password, Str::toEnglish($request->password));
    }
    return $login ? 'success' : 'failed' ;
}

Suggest
Use toEnglish method in perpareForValidation method. Doing this will make your code cleaner
protected function prepareForValidation() {
     return $this->merge([
         'mobile' => Str::toEnglish($this->input('mobile')),
         'email' => Str::toEnglish($this->input('email')),
         'password' => Str::toEnglish($this->input('password')),
     ]);
}

This feature doesn't make sense. We already have ascii conversion, which allows specification of the languagae.

And the other issue you are facing is the incorrect column collation on the database.

And the other issue you are facing is the incorrect column collation on the database

No that was just an example I will fixed it.

username-۱۲۳۴

This is the difference between toEnglish result and sample
code

result

This is a feature that Persian or Arabic speakers usually need to use.

I don't think you understand either of my points here.

Laravel already has a way to normalize input into ascii (Str::ascii), and the language paramter (param 2) allows you to tell it the source language, so you can have the conversions that make the most sense for you - I don't know any arrabic, but I do know that many European languages have different conversions, so à in one language may go to a, but in another, to two characters.
Databases already have a way to specify how you want to identify characters together - it is called collation. For example, a, A, à` are all treated as the same letter by default.

You should also be super careful allowing non-ascii characters in user names and mushing them together, as this can result in bad security where two people have different user names that end up being stored as the same username in the database. Best practice is to have strict validation and change the collation, not to transform the input, causing collisions.

OK I got it
Thank you for your guidance 💯

Graham, you are a wizard.😅
I learnt many thing by just reading your comments on GitHub.

Glad I could get us pointed in the right direction. 🚀


[Refactor] Improve the way Meshery Server handles Meshsync data

Description
This PR introduces some changes in the startup behaviour of Meshery Server. From now on, whenever Meshery Server is started, it would

Dump the stale MeshSync data in the database
Ask Meshsync to send the data in its informer store and persist them in the database
Subscribe to the events sent by MeshSync and configure what should happen when an event is received, which at the moment is to update the database using the information from the event.

Notes for Reviewers
I haven't pushed the changes to the GraphQL subscriptions because, the subscriptions that we have now are very rigid, that is, it is difficult to change something in one subscription without affecting the other, so we have to change this in a separate PR.
In this PR, the case when the operator is not deployed or not functional when Meshery Server starts is not being handled, it is to be done in a separate PR.
Signed commits

[x] Yes, I signed my commits.

Codecov Report

Merging #4972 (d8a7dad) into master (f34d398) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4972   +/-   ##
=======================================
  Coverage   32.47%   32.47%
=======================================
  Files          61       61
  Lines        5028     5028
=======================================
  Hits         1633     1633
  Misses       2991     2991
  Partials      404      404

Flag
Coverage Δ

gointegrationtests
9.36% <ø> (ø)

integrationtests
∅ <ø> (?)

unittests
25.87% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f34d398...d8a7dad. Read the comment docs.

Yes @bariqhibat

@sudo-NithishKarthik there are a couple merge conflicts to look at.

Syncing with master

Pending an update to support multiple Kubernetes contexts.

Refreshing with master again.


[REFACT] - lambda service invoke function

This PR just refacts the Lambda Invoke function, to use just one.
We divided the invokeLambda function into two, to use one with a custom secret and another (default) passing no secret value.
I refact the two functions into one

Guys! I requested your reviews so we can discuss this again. We, at BombSquad, were not happy with the two functions approach.

PS: We need to improve the version. Is this a Major or a Minor? Or just a Patch?

PS: We need to improve the version. Is this a Major or a Minor? Or just a Patch?

Major; this is a breaking change

PS: We need to improve the version. Is this a Major or a Minor? Or just a Patch?

Major; this is a breaking change

I agree

I think this function is an invoke function, this code to get accessKey from secrets manager is another function, and if needed to use with another accessKey, you need to use this by params directly to invoke
obs: this is not the normal flow from an invoke function...