Optimize spsc queue : optimizations without Obj.magic

This PR regroups the "safe" optimization implemented for the spsc queue  in #121 by @polytypic. Safe here means that they do not introduce new calls to  'Obj.magic'  or related unsafe OCaml features (like external ... "%identity").
This PR includes the following optimizations:

caches of the other side index
avoid wide pattern in record, assuming CSE instead
drop mask field
micro-optimize empty/full checks
avoid code duplication with GADTs
use of Atomic.make_contended if OCaml version >= 5.2.0

This results in a performance increase of a factor 4 to 5 for a large queue.

For reference, here is a C++ implementation of a SPSC queue:
#include <atomic>
#include <cassert>
#include <iostream>
#include <thread>

struct end {
  std::atomic<size_t> m_index;
  size_t m_other_index;

  end() : m_index(0), m_other_index(0) {}
};

template <class T> struct spsc {
  size_t m_mask;
  T *m_array;
  alignas(64) end m_lo;
  alignas(64) end m_hi;

  spsc(size_t capacity) {
    assert((capacity & (capacity - 1)) == 0);

    m_mask = capacity - 1;
    m_array = new T[capacity];
  }

  ~spsc() { delete[] m_array; }

  bool try_push(const T &value) {
    size_t hi = m_hi.m_index.load(std::memory_order_relaxed);
    if (hi - m_hi.m_other_index <= m_mask) {
      m_array[hi & m_mask] = value;
      m_hi.m_index.store(hi + 1, std::memory_order_release);
      return true;
    }
    m_hi.m_other_index = m_lo.m_index.load(std::memory_order_acquire);
    if (hi - m_hi.m_other_index <= m_mask) {
      m_array[hi & m_mask] = value;
      m_hi.m_index.store(hi + 1, std::memory_order_release);
      return true;
    }
    return false;
  }

  bool try_pop(T &out) {
    size_t lo = m_lo.m_index.load(std::memory_order_relaxed);
    if (0 < (m_lo.m_other_index - lo)) {
      out = m_array[lo & m_mask];
      m_lo.m_index.store(lo + 1, std::memory_order_release);
      return true;
    }
    m_lo.m_other_index = m_hi.m_index.load(std::memory_order_acquire);
    if (0 < (m_lo.m_other_index - lo)) {
      out = m_array[lo & m_mask];
      m_lo.m_index.store(lo + 1, std::memory_order_release);
      return true;
    }
    return false;
  }

  spsc(const spsc &) = delete;
  spsc &operator=(const spsc &) = delete;
};

int main(int argc, char *argv[]) {
  spsc<size_t> queue(512);

  const size_t n_messages = 1000 * 1000 * 500;

  std::thread producer([&]() {
    size_t n = n_messages;
    while (0 < n) {
      if (queue.try_push(n))
        --n;
    }
  });

  size_t n = n_messages;
  while (0 < n) {
    size_t message;
    if (queue.try_pop(message))
      --n;
  }

  producer.join();

  return 0;
}

On my laptop this seems to run in about a second, which translates to 500 million messages per second.
➜  clang++ -O2 -std=c++17 spsc.cpp && hyperfine -N --warmup 1 ./a.out
Benchmark 1: ./a.out
  Time (mean ± σ):     990.8 ms ±   6.0 ms    [User: 1976.3 ms, System: 1.2 ms]
  Range (min … max):   981.2 ms … 999.5 ms    10 runs

This is not a production quality implementation (I didn't test this extensively — there might be bugs; this doesn't guard against some obvious programming errors) and I didn't spend a lot of time tweaking this —  I would have actually assumed to reach closer to 1 billion message per second on my laptop.  I did look at the generated code on godbolt and didn't spot any obvious significant issues with the generated code.

So, I wanted to show the C++ SPSC implementation for a specific reason.  When optimizing code, it is important to understand the context.  For example, if you are optimizing an application then you might have a goal such as "make a specific operation in the application run in some specific time period".  For example, it might take 0.5 seconds for an operation in the application.  That is long enough that a person will notice that something is not instantenous.  A goal could then be to optimize that operation to run in less than 0.25 seconds, which is typically fast enough that a person does not notice that something is taking a lot of time.  If and once you make the operation run in 0.25 you are (or may be) done.  You don't have to think about whether things could still be faster.
Unfortunately, when implementing a library, we do not know the requirements of any particular application.  What should we compare against?  The way I think about optimizing libraries like this is that I want to understand what is the (estimated) theoretical peak performance that could be achieved.  There is no single way to figure out what is the theoretical peak.  You could, for example, consider things like maximum memory bandwidth and cost of atomic (fenced) instructions, a single threaded reference implementation (and multiply by two for 2 threads working mostly without interference), a reference implementation in C++.  The way I then judge my optimizations is with respect to the theoretical peak performance I came up with.  I want to get as close as possible to the theoretical peak performance — and if I don't get close enough, I spend time to figure out why (is my estimated theoretical peak too optimistic? is something in my implementation working poorly?).  Depending on what exactly one is working on, getting to within some percentage of theoretical peak could then be acceptable — for something like this, I'd want to get within maybe 90%-95% of what is possible with OCaml and perhaps within 2x of what is possible with a language like C++.  I basically want things to so fast that almost nobody should feel the urge to write their own implementation to optimize their application.

Optimizing these kinds of data structures is all about optimizing for the cache architecture and memory bandwidth.
For example, this implementation uses 3x more words per memory due to the _ option array than the unsafe implementation using a non-option array.  This basically means is that if you estimate the theoretical peak performance based on memory bandwidth, then you would expect this to perform up to 3x slower in cases when you are bandwidth bound.  Try to, for example, make the queue very large (large size exponent) and pass a lot of messages through the queue. Quite possibly at some point you will be bound by memory bandwidth and this queue will then likely be about factor of three slower than the unsafely optimized queue.  This is basically a physical law of nature.  It simply takes 3x more time to write/read 3x more memory.  There is no way around that.
When you are not bound by memory bandwidth, you are likely bound by the cache architecture.  On a modern CPU every write to memory by CPU core effectively "takes ownership" of the cache line aligned memory region affected by the write.  This happens via the cache coherency protocol, i.e. after a write the L1 cache of the core that performed the write "owns" the region of memory.  Another CPU core accessing the same region will then take a performance hit when trying to access (read or write) that same cache line aligned region of memory.  This is the reason to avoid false sharing where a single cache line aligned region of memory is being written to by at least one CPU core and also accessed by other cores (but not accessing the exact same words).  The cached indices and the queue record in this version of the queue are not protected against false sharing.  Sometimes you get lucky and things run fast.  Other times there is false sharing/contention and things run much slower.  For an application that has any sort of real-time requirements can be very undesirable.

Personally I think that we should "get a blessing" to at least use the equivalent of Multicore_magic.copy_as_padded.  I hope there is a way to write such a function in a way that it is guaranteed to work even with a compiler Flambda.  Otherwise we should actually warn ⚠  that the safe implementation are unsafe against false sharing — that is, their performance can sometimes be dramatically worse due to false sharing.
The factor of 3.5 slowdown is actually a lucky case.  I would assume that I could actually make that way worse by writing a program that has a falsely shared mutable integer field with the queue that the program would be writing to from a thread that gets allocated to a different core cluster.  Then the cache line transfer would need to go between core clusters, which is likely much slower than within a core cluster.  It might also be that if the other core cluster is an "efficiency" cluster (i.e. the cores are slower in such a cluster) the transfer cost would be even higher.

In summary:

I think we should use the _exn naming pattern and raise_notrace for operations that raise exceptions mostly as a control flow mechanism to avoid memory allocations.  (I actually benchmarked that with raise_notrace using exceptions for control flow was slightly faster on my laptop than returning an option.)
I'd strongly recommend picking some way to explicitly guard even safe implementations agaist false sharing.  I think the simplest way to do that would be to get a "blessed" implementation of copy_as_padded.  Othewise I would recommend labelling safe data structure implementations as ⚠  unsafe against false sharing.  Doing this will also not only make performance more stable, but will likely also improve thruput in most cases.

Otherwise this PR looks good to me. 👍

This method can be used by at most one domain at a time.

I noticed this wording in the documentation.  I've actually been asked about these in a private communication.  I think we should clarify our definition of "single consumer" or "single producer".
The comment implies a too restrective meaning.  It is perfectly fine for a single consumer/producer to move from one domain or (sys)thread to another.  What we want, informally, is that actions on the queue from a single consumer/producer are ordered as if they would come from a single thread of execution.  It is not even required that a single producer/consumer would be strictly speaking a single thread of execution.
Perhaps we should say something like that producer (and consumer) actions must be totally ordered according to the happens-before relation defined by the OCaml memory model.

With the padding of the _cache indices, performance in the benchmark is now more stable and improved on my laptop:
➜  saturn git:(optimize-spsc-queue-safe) ✗ dune exec --release -- bench/main.exe -budget 1 -brief Single
Saturn_lockfree Single_prod_single_cons_queue:
  time per message/2 workers, capacity 1:
    345.68 ns
  messages over time/2 workers, capacity 1:
    5.79 M/s
  time per message/2 workers, capacity 8:
    134.05 ns
  messages over time/2 workers, capacity 8:
    14.92 M/s
  time per message/2 workers, capacity 64:
    66.87 ns
  messages over time/2 workers, capacity 64:
    29.91 M/s
  time per message/2 workers, capacity 512:
    55.23 ns
  messages over time/2 workers, capacity 512:
    36.22 M/s
  time per message/2 workers, capacity 4096:
    40.16 ns
  messages over time/2 workers, capacity 4096:
    49.80 M/s
  time per message/2 workers, capacity 32768:
    21.65 ns
  messages over time/2 workers, capacity 32768:
    92.38 M/s

Because the main record of the queue is not padded, it is still possible for the queue to suffer from false sharing, but this does not show in our benchmark, because the benchmark is written so that no false sharing could really happen.  A real program could still easily accidentally get false sharing on the main queue record.


control maximum segment length

@cuddandr 's plot of segment lengths shows we have some pretty long hit segments inside the 2x2.

We should put in something into https://github.com/DUNE/2x2_sim/blob/main/run-edep-sim/macros/2x2_beam.mac like:
/edep/hitLength volTPCActive 5 mm
/process/eLoss/StepFunction 0.0001 5 mm
/process/eLoss/StepFunctionMuHad 0.0001 5 mm

@peter-madigan has studied different settings in the past and should be able to advise us on what was used to get good data/mc agreement out of module0,1.

You can see here on page 63 / figure 4.20 how this modifies the geant step size
For module 0, I used 0.001 1mm to make sure that the steps are always about 1mm for muons with an energy less than ~300MeV
For the hit length, I did 1mm as well.
I found that this produced segments that were actually 2mm for whatever reason - I'm thinking it probably has to do with how edep-sim merges geant4 steps.
I looked at 5mm, 1mm, 0.1mm, and 0.01mm, and honestly didn't see much of a difference between 1mm and 0.1mm. At 5mm and 0.01mm, the dQ/dx width visibly decreases, so I would say that you need to set it somewhere between these two bounds.
That said, we do get about a 10% smaller width of the dQ/dx peak in larndsim than what we see in module 0 data at the 1mm step size.

Copying these plots here for reference

Forgot the dQ/dx plot

I wanted to add to this discussion an alternative option to control the step size. It allows for a firm upper limit on the segment size in edep-sim, which can eliminate, if we want, the O(cm) segment sizes that we've been seeing.
There is a place in the detector GDMLs where auxiliary aux types are specified, i.e.
<volume name="volTPCActive">
      <materialref ref="LAr"/>
      <solidref ref="TPCActive_shape"/>
      <auxiliary auxtype="SensDet" auxvalue="TPCActive_shape"/>
      <auxiliary auxtype="EField" auxvalue="(500.0 V/cm, 0.0 V/cm, 0.0 V/cm)"/>

These are parsed from the GDML in this script src/EDepSimUserDetectorConstruction.cc  and are used to set user defined limits for the simulation. We can add an auxiliary auxtype for StepLimit in the GDML e.g. :
<auxiliary auxtype="StepLimit" auxvalue="0.1 mm"/>
It will act as a step size limiter (this is mentioned in the edep-sim README). But edep-sim does combine segments to some extent, so to minimize that, I set in the macro:
/edep/hitLength TPCActive_shape 0.00001 mm
/edep/update

Making hitlength very small to ensure no combining. I first tested this configuration with low energy electrons in module0 (because it was the easiest and quickest for me to do) where I tried 0.1 mm and 0.01 mm for step limit (note that the electron track lengths are O(1 mm) ) and with the hitLength parameter above. I get a strict upper bound now on the segment lengths of 0.1 mm and 0.01 mm respectively. With the hitLength set to default (2 mm) and no step limit, I get segment lengths up to 1 mm.

Here’s an example with some small 2x2 spill samples I made. I plotted the edep-sim segments within the active LAr only, not including the other segment detectors since I only applied StepLimit and hitLength to the active LAr and not the other segment detectors.

Avg segments per event in active LAr with No StepLimit / Default hitlength: ~160
Avg segments per event in active LAr with StepLimit / hitlength: ~3500

So it looks like this is a suitable option for limiting the step size. We should consider then how small of step sizes we really need. An important consideration is how it affects file sizes and requirements on computing resources. Additionally we should consider what benefits we would gain, if any, w.r.t data-MC comparisons by reducing the step size.


api_server.py: error: unrecognized arguments: --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/

run codes below
python -m vllm.entrypoints.api_server \     --model meta-llama/Llama-2-7b-hf \     --enable-lora \     --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
raise an error:
api_server.py: error: unrecognized arguments: --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/

Same error when trying to launch docker from vllm/vllm-openai:latest

Ok, I still get the error with the new docker. It isn't an available argument.
I recommend checking out this example for how to load the lora after launching the engine:
https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py

yes same error as well. Is there a solution for this? ... to serve lora trained models using vlllm? Thanks!

build env from source can fix it.

Upgrading to vllm==0.3.3 solved this for me.

Instead of filing a new issue, tagging on to this open issue. I am having a similar experience with vllm-openai:0.5.0 where it is giving me the same message. It seems that the --lora-modules argument is unrecognized. I'll try building from source, but is this a bug?


Consider integrating with OSS-Fuzz?

OSS-Fuzz is a Google-provided service that can automatically and continuously fuzz OSS libraries and report bugs for discovered issues. It supports fuzzing Rust.
The requirement for being accepted in OSS-Fuzz is "project must have a significant user base and/or be critical to the global IT infrastructure" which I personally believe definitely includes the Rust regex library, since it's at the core of so much stuff. List of projects that are currently in OSS-Fuzz.
Can't hurt to file a form and ask.
(Disclaimer: I work for Google, but not on anything fuzzing or security-related. I do not speak for my employer. This suggestion comes from "personal Val", not "business Val". :smiley:)

Oh, forgot to mention: OSS-Fuzz offers monetary rewards for integration, up to $20k. If this isn't personally interesting, it can be donated to a charity of your choice and Google will double the reward.
$40k to a charity for a few days of work doesn't sound like a bad deal...

Sounds like a good idea to me. Not sure if regex would qualify though or when I'll get to this. It's likely we could crib ideas from RE2: https://github.com/google/re2/tree/master/re2/fuzzing
If someone else wanted to work on this, I'd be happy to give guidance and mentor it.

@Valloric What kind of bugs do you hope to catch by fuzzing? It is my understanding that - since this library written without the use of unsafe - most bugs caught by fuzzing can not occur anyway.
For reference: https://opensource.googleblog.com/2017/05/oss-fuzz-five-months-later-and.html

The fuzzer is built by way of cargo-fuzz and the default of Cargo-fuzz is to build by way of release mode, i.e. the --release flag is used when calling cargo build: https://github.com/rust-fuzz/cargo-fuzz/blob/799bf534b889859b090b76f2ca1cffdca4ee7455/src/project.rs#L138
I did minor changes in a pull here: https://github.com/rust-lang/regex/pull/720 but once that has been merged we should be good to go with the new set up. OSS-Fuzz automatically pulls the latest, but it may take a few days.


Received response for /v2/getcodeactions but could not find request. Using namespace autocomplete not working.

Issue Description
using namespace autocomplete not working, vscode intellisense shows No quick fixes available.
It seems like a server error as the log shows could not find request.

Steps to Reproduce
Doesn't work on two of my machine, never worked
Expected Behavior
shows namespace autocomplete option.
Actual Behavior
intellisense shows No quick fixes available
Logs
OmniSharp log
Starting OmniSharp server at 2021/5/19下午4:03:03
    Target: c:\Users\cl930\Desktop\LeetCode
OmniSharp server started.
Path: c:\Users\cl930.vscode\extensions\ms-dotnettools.csharp-1.23.11.omnisharp\1.37.8\OmniSharp.exe
PID: 4948
    Starting OmniSharp on Windows 6.2.9200.0 (x64)

    DotNetPath set to dotnet

    Located 1 MSBuild instance(s)
        1: StandAlone 16.9.0 - "c:\Users\cl930\.vscode\extensions\ms-dotnettools.csharp-1.23.11\.omnisharp\1.37.8\.msbuild\Current\Bin"

    MSBUILD_EXE_PATH environment variable set to 'c:\Users\cl930\.vscode\extensions\ms-dotnettools.csharp-1.23.11\.omnisharp\1.37.8\.msbuild\Current\Bin\MSBuild.exe'

    Registered MSBuild instance: StandAlone 16.9.0 - "c:\Users\cl930\.vscode\extensions\ms-dotnettools.csharp-1.23.11\.omnisharp\1.37.8\.msbuild\Current\Bin"
        CscToolExe = csc.exe
        CscToolPath = c:\Users\cl930\.vscode\extensions\ms-dotnettools.csharp-1.23.11\.omnisharp\1.37.8\.msbuild\Current\Bin\Roslyn
        MSBuildExtensionsPath = c:\Users\cl930\.vscode\extensions\ms-dotnettools.csharp-1.23.11\.omnisharp\1.37.8\.msbuild
        MSBuildToolsPath = c:\Users\cl930\.vscode\extensions\ms-dotnettools.csharp-1.23.11\.omnisharp\1.37.8\.msbuild\Current\Bin

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpFormattingWorkspaceOptionsProvider, Order: 0

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.Completion.CompletionOptionsProvider, Order: 0

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.RenameWorkspaceOptionsProvider, Order: 100

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.ImplementTypeWorkspaceOptionsProvider, Order: 110

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.BlockStructureWorkspaceOptionsProvider, Order: 140

    Detecting Cake files in 'c:\Users\cl930\Desktop\LeetCode'.

    Could not find any Cake files

    No solution files found in 'c:\Users\cl930\Desktop\LeetCode'

    Queue project update for 'c:\Users\cl930\Desktop\LeetCode\LeetCode.csproj'

    Detecting CSX files in 'c:\Users\cl930\Desktop\LeetCode'.

    Could not find any CSX files

    Configuration finished.

    Omnisharp server running using Stdio at location 'c:\Users\cl930\Desktop\LeetCode' on host 2872.

    Loading project: c:\Users\cl930\Desktop\LeetCode\LeetCode.csproj

    Successfully loaded project file 'c:\Users\cl930\Desktop\LeetCode\LeetCode.csproj'.

    Adding project 'c:\Users\cl930\Desktop\LeetCode\LeetCode.csproj'

    Update project: LeetCode

Received response for /v2/getcodeactions but could not find request.
Received response for /quickinfo but could not find request.
[warn]: OmniSharp.Roslyn.CSharp.Services.Navigation.FindUsagesService
No symbol found. File: c:\Users\cl930\Desktop\Leetcode\637.average-of-levels-in-binary-tree.cs, Line: 22, Column: 12.
Received response for /v2/getcodeactions but could not find request.
Received response for /v2/getcodeactions but could not find request.
[warn]: OmniSharp.Roslyn.CSharp.Services.Navigation.FindUsagesService
No symbol found. File: c:\Users\cl930\Desktop\Leetcode\637.average-of-levels-in-binary-tree.cs, Line: 23, Column: 23.
[warn]: OmniSharp.Roslyn.CSharp.Services.Navigation.FindUsagesService
No symbol found. File: c:\Users\cl930\Desktop\Leetcode\637.average-of-levels-in-binary-tree.cs, Line: 23, Column: 37.
[warn]: OmniSharp.Roslyn.CSharp.Services.Navigation.FindUsagesService
No symbol found. File: c:\Users\cl930\Desktop\Leetcode\637.average-of-levels-in-binary-tree.cs, Line: 22, Column: 37.

C# log
No C# log
Environment information
VSCode version: 1.56.2
C# Extension: 1.23.11
Dotnet Information
.NET Core SDK (reflecting any global.json):
 Version:   3.1.409
 Commit:    217439b5ae
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.19042
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\3.1.409\
Host (useful for support):
Version: 3.1.15
Commit:  dc339bd0fb
.NET Core SDKs installed:
3.1.409 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

Extension
Author
Version

csharp
ms-dotnettools
1.23.11

vscode-leetcode
LeetCode
0.17.0

Consider reading this: https://github.com/OmniSharp/omnisharp-vscode/issues/5029#issuecomment-1023766141. It fixed the problem on my machine.

A lot has changed with O# code analysis since this issue was opened. If you are still having problems, please open a new issue at https://github.com/omnisharp/omnisharp-roslyn/issues


Signed transaction generator

Description:
Related issue(s):
Fixes #
https://github.com/hashgraph/hedera-json-rpc-relay/issues/164
Notes for reviewer:
Checklist

[ ] Documented (Code comments, README, etc.)
[ ] Tested (unit, integration, etc.)

@Nana-EC I am merging this PR in order to continue with the next acceptance tests within the 125-acceptance-test.. branch. If you have any other comments, please leave them in #139


uri module lowercases drive letters on Windows

Hello, we are using vscode-uri (https://github.com/Microsoft/vscode/blob/master/src/vs/base/common/uri.ts in this repo) to deal with URIs in https://github.com/flowtype/flow-language-server (a Language Server Protocol implementation for flow) and noticed an issue when converting fs paths to uri then back to fs path we lose drive letter casing, which then causes diagnostic not to show properly in atom.
Here's what happens:
C:\my\path -> file:///c/my/path -> c:\my\path
Looking at the code, this lowercasing is definitely intentional and was wondering why it was done, and if it would be possible not to do it since it would fix my use case. If there is a valid reason why it is done and could cause issues I can just fork it but wanted to ask here first :).
PS: The irony of posting a flow/atom issue in a ts/vscode repo 😆

If there is a valid reason why it is done and could cause issues

We wanted consistency and picked on of two options ;-). I wonder why that's an issue for you?

@jrieken  My guess is that atom uses strict comparison for file paths so C:\my\path !== c:\my\path when displaying diagnostics. I just debugged the inputs / outputs and noticed the paths were different when converting a fs path to uri then back to fs path, I don't really know more than that.
Technically it should work on Windows since paths are not case sensitive by in practice this is not always the case. I feel like fixing it here would probably be easier than fixing all paths comparisons and make sure they are not case sensitive on Windows (Are you doing this in vscode?).

I feel like fixing it here would probably be easier than fixing all paths comparisons and make sure they are not case sensitive on Windows

Yeah, we kind of do that. When creating paths/uris we know how we create them, when accepting uris from 3rd parties, e.g. extensions & user content, we normalise them.


Skip JSON content type check

Seeing as we're attempting to parse JSON anyway, I don't see the point of checking the JSON content type. I don't really like how we're using the exception here but it makes sense from a practical point of view.

Makes sense to me.

Looks good, but there are test failures, so please check on that.

I missed that an UnexpectedValueException should be thrown if a JSON content type was set but the JSON parse failed, but not be thrown when a JSON content type was not set.
It's very similar to master, the only difference is that the JSON content type check is only done if the parse actually failed. So the gain isn't significant, but still of value.
Also refactored the tests for parseContent a bit. :hammer:


Pipeline QC: Automated QC after each step of the pipeline

Summary
The plan is generate QC metrics while the pipeline is running to ensure the pipeline is running as intended. We started monitoring certain metrics for each release in early 2018 on this GitHub data_release repository wiki page. To move away from manually updating mark-up tables on the wiki page, we started a google sheet and explored more metrics that cold be monitored. These metrics have been generated for the 18.08 and 18.10 data releases. There was also a data QC meeting in August 2018 and the notes can be found here. These documents are very much a work in progress and will be the basis for the more detailed specification of what the automated QC should look like.
Work in progress

[ ] Document current status of QC here (see summary above, add details about --qc and --metrics steps)
[ ] Define the workflow for QC of evidence strings
[ ] Define the workflow for QC of base data (look-up tables)
[ ] Specify implementation details and steps

QC-related tickets on GitHub

Pipeline QC: Re-enable automated QC post 18.10 - need to define implementation details first.
Pipeline QC: Automated QC distribution of scores - this might be good in the context of comparing data between releases (score distribution, new and 'lost' associations, targets and diseases)
Pipeline QC: Document --qc and --metrics steps

Initial use case: @gkos-bio commented on [Tue Aug 07 2018]
The description of this issue is based on my experience running the pipeline.
Overall, the stability has increased over time.
On the last run, we observed that the Reactome step didn't fail but also didn't have the correct information. As a consequence, the facet served on the association page didn't contain the right data and were classified as 'Others'.
This ticket is to propose to develop a set of QC after each step to ensure that the lookup tables contain the right information.

Something to include is gene plugin results - to check if they injected anything or not.

These gene plugins can already be checked and are recorded on the first sheet of the magical metrics document:

chemicalprobes.probeminer
chemicalprobes.portalprobes
hallmarks
cancerbiomarkers
mouse_phenotypes.phenotypes
tractability

I think we can add reactome to this list. The tricky bit will be to define how to check that a plugin worked.


Enable long file paths usage on Windows & patch issues with using . shortcut for current working directory

Manually tested for all upload & download scenarios. Mixes with wildcard characters.
Fixes issues #104 #376 and #398.

IMPORTANT: If we do end up changing the slash format that is persistent into plan files, we will have to change the version number for plan files.

Closing PR to redo implementation in just-in-time conversion style


Fix for spawning chrome on non-win environment

Fix Chrome not spawning in non-windows environment

@r0ssing reminder about code review


add length limits for MPD module tags

adds the following configuration options to the MPD module:

artist-len
album-len
album-artist-len
title-len

to control the maximum length of the tags when they are displayed on the bar.

alright, i've fixed it up :smile:

LGTM, thx

Could you also update the github wiki?


Markdown cells don't use workspace settings for styles

Describe the bug
When adding a markdown cell to a polyglot notebook, the rendered markdown does not apply styles that I have set in a "styles.css" file, which my VS Code Workspace Settings includes. I have a settings.json file in the .vscode folder at the root of my workspace, which contains the following (note that I don't normally use consolas as the font - that is included here just to illustrate the point):
{
    "markdown.preview.fontFamily": "consolas, -apple-system, BlinkMacSystemFont, 'Segoe WPC', 'Segoe UI', system-ui, 'Ubuntu'",
    "markdown-mermaid.darkModeTheme": "dark",
    "markdown.styles": ["styles.css"]
}

In the root of my workspace I have a file named styles.css which contains some of my preferred styling for various markdown elements.
I have a mermaid renderer extension installed (https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid) which uses one of the other settings in the settings.json file, with no issues.
If I write a plain markdown file in the same workspace, the styles defined in styles.css are applied. In addition, the font family setting and the mermaid dark mode theme setting are also applied as expected in the preview window.
However, when I add a polyglot notebook file to the same workspace, and add a markdown cell to it, the styles are not applied. Additionally, the font family setting is also ignored. The mermaid dark mode theme IS applied as expected, however. Presumably this is because it's a third party extension which is able to use its settings correctly.
I have tried this both with .ipnyb and .dib file types and the result is the same. I know the styles I have defined can be used by the polyglot notebooks extension when rendering markdown because if I add them to a markdown cell, they are applied correctly. This problem appears be related only to the use of styles defined and linked from a VS Code workspace settings file.
Please complete the following:
Which version of .NET Interactive are you using? (In a notebook, run the #!about magic command. ):

OS

[x] Windows 11
[ ] Windows 10
[ ] macOS
[ ] Linux (Please specify distro)
[ ] iOS
[ ] Android

Browser

[x] Chrome
[ ] Edge
[ ] Firefox
[ ] Safari

Frontend

[ ] Jupyter Notebook
[ ] Jupyter Lab
[ ] nteract
[x] Visual Studio Code
[ ] Visual Studio Code Insiders
[ ] Visual Studio
[ ] Other (please specify)

Screenshots
Font rendered in consolas and styles applied correctly to a code block in the preview of a .md markdown file:

Same markdown rendered in a markdown cell in a polyglot notebook - default font and styles:

Any thoughts on this? Is there something I'm missing with this - or does the "External" label that's been added indicate that this is caused by an external dependency of this library?


Vendor all dependencies.

@mlerner

CircleCI weirdness caused by old hook talked about in person. Otherwise vendor-ing those libs LGTM.


Make tool more user friendly

@KidDaedalus please let me know if you have time for a quick review.


Docformatter conflict with isort

Hi!
Description
I recently update docformatter v1.6.0  -> v1.6.4 and actions on my project was failed(
After spending time searching, reason turned out to be following - isort and docmatter make opposite changes to the file:

docmatter - remove empty line between import and constant comment
isort - add this line)

Also was checked v1.6.1, v1.6.2 and v1.6.3 - they have same problem.
As I see it, PEP8 says:

You should put a blank line between each group of imports.

I think this applies to constants as well, so docmatter shouldn't remove that line.
Reproduction
You can reproduce my problem with next data:
Python code:
import os
from typing import Iterator

"""Don't remove this comment, it's cool."""
IMPORTANT_CONSTANT = "potato"

.pre-commit-config.yaml:
repos:
  - repo: https://github.com/pycqa/docformatter
    rev: v1.6.4
    hooks:
      - id: docformatter
        args: [
          --wrap-summaries=120,
          --wrap-descriptions=0,
          --in-place,
          --blank
        ]

  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort

After run pre-commit run --all-files you will see that file changed twice and actions was failed:
import os
from typing import Iterator
-
"""Don't remove this comment, it's cool."""
IMPORTANT_CONSTANT = "potato"

import os
from typing import Iterator
+
"""Don't remove this comment, it's cool."""
IMPORTANT_CONSTANT = "potato"

@M1troll should be fixed in v1.6.5.

@weibullguy , Thank you! Now everything works without problems)


fb.RunMainMenuCommand("File Operations/Move to/Anything") doesn't work

I got a few "Move to" configurations set up, but i can't call them using fb.RunMainMenuCommand. Other calls, like "File/Preferences" work though.
I'd like to run this from code so that i can run some related code afterwards.

File operations are accessed via the CONTEXT menu, not the MAIN menu.

@marc2k3 that was my issue! thanks a lot! the context/main concept still catches me every now and then xD

No. Ultimately the component/script is letting you execute C++ methods in the SDK via javascript. There is no magic dialog manipulation.


SSL connection not maintained

SSL connection with OKHTTP is not maintained getting following error
java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
Version used : implementation 'com.squareup.okhttp3:okhttp:3.10.0'
my code
Function used to create client as follows
`public static OkHttpClient getOkHttpClient() {
if(okHttpClient==null){
        try{
            CertificateFactory cf ;
            InputStream cert ;
            Certificate ca ;
            SSLContext sslContext ;

            cf = CertificateFactory.getInstance("X.509");

            cert = AppController.getInstance().getResources().openRawResource(R.raw.my_cert); // Place your 'my_cert.crt' file in `res/raw`

            try {
                ca = cf.generateCertificate(cert);
                Log.v(TAG,"ca=" + ((X509Certificate) ca).getSubjectDN());
            } finally {
                cert.close();
            }

            String keyStoreType = KeyStore.getDefaultType();
            KeyStore keyStore = null;
            try {
                keyStore = KeyStore.getInstance(keyStoreType);
            } catch (KeyStoreException e) {
                e.printStackTrace();
            }
            keyStore.load(null, null);
            keyStore.setCertificateEntry("ca", ca);

            String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
            TrustManagerFactory tmf = null;
            tmf = TrustManagerFactory.getInstance(tmfAlgorithm);

            tmf.init(keyStore);

            sslContext = SSLContext.getInstance("TLS");
            sslContext.init(null, tmf.getTrustManagers(), null);

            okHttpClient = new OkHttpClient.Builder()
                    .sslSocketFactory(sslContext.getSocketFactory())
                    .connectTimeout(CONNECTION_TIME_OUT, TimeUnit.SECONDS)
                    .readTimeout(CONNECTION_TIME_OUT, TimeUnit.SECONDS)
                    .writeTimeout(CONNECTION_TIME_OUT, TimeUnit.SECONDS)
                    .retryOnConnectionFailure(true)
                    .build();

        } catch (KeyManagementException | NoSuchAlgorithmException | java.security.cert.CertificateException | IOException | KeyStoreException e) {
            e.printStackTrace();
        }

    }

    return okHttpClient;
}`

Your trust managers don't trust the server you're connecting to.

@swankjesse Thats what I need to understand what Im doing wrong here, I tried a similar thing using Volley and it worked completely fine.

I'm not sure. It looks reasonable. Any way you can express this problem as a standalone test case?

This example works. Maybe see where your code and the example disagree?
https://github.com/square/okhttp/blob/master/samples/guide/src/main/java/okhttp3/recipes/CustomTrust.java

@swankjesse Did exactly what mentioned above, getting same error.

Could you provide an executable test case? Presumably this is a bug in OkHttp but I don’t have enough information to reproduce it.

Having same issue in my Tablet (Micromax P480) Android version 4.4.2, but in other devices it works fine. I don't have an idea of how to solve this.

please reopen with more information e.g. a fully reproducible test case or URL to test against.

I guess the above code is all the details needed.
URL?? we are using app side for this.
share little brief what exactly required for this.

@Hemant0601 I can't paste that code into Intellij and run it and reproduce the failure.  The results could vary depending on Android version, server and URL, certificates used.  Similar examples are known to work, so there isn't much we can do.
If you are able to debug the issue and make it a reproducible bug report we can help.  But as is, it isn't something we can help with.  Try stackoverflow as you will get a wider response.

@yschimke sure, will add if i get any running code or could not find any solution soon.
Thanks


Update for Apache Arrow >= 0.14 with arrow::Column removed

The arrow::Column C++ class was removed from the project. See also ARROW-6428

Codecov Report

Merging #226 into master will decrease coverage by 1.73%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #226      +/-   ##
==========================================
- Coverage    98.6%   96.86%   -1.74%
==========================================
  Files         136      141       +5
  Lines        3003     2268     -735
==========================================
- Hits         2961     2197     -764
- Misses         42       71      +29

Impacted Files
Coverage Δ

...odbc_numpy/Library/turbodbc_numpy/unicode_column.h
0% <0%> (-100%)
:arrow_down:

cpp/turbodbc_numpy/Library/src/unicode_column.cpp
0% <0%> (-86.67%)
:arrow_down:

...bc_python/Library/src/determine_parameter_type.cpp
83.05% <0%> (-13.62%)
:arrow_down:

...p/cpp_odbc/Library/src/level1/unixodbc_backend.cpp
71.42% <0%> (-7.15%)
:arrow_down:

cpp/turbodbc/Library/src/cursor.cpp
90% <0%> (-7.06%)
:arrow_down:

...pp/turbodbc_numpy/Library/src/numpy_result_set.cpp
96.42% <0%> (-3.58%)
:arrow_down:

.../turbodbc_python/Library/src/python_result_set.cpp
91.42% <0%> (-2.45%)
:arrow_down:

...urbodbc_numpy/Library/src/set_numpy_parameters.cpp
99.35% <0%> (-0.12%)
:arrow_down:

...pp/turbodbc/Library/src/result_sets/result_set.cpp
100% <0%> (ø)
:arrow_up:

...brary/src/python_bindings/python_parameter_set.cpp
100% <0%> (ø)
:arrow_up:

... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 089ebe7...e7a6dfd. Read the comment docs.

@wesm Could you try to use #if macros to detect the Arrow version? Then we could merge this to master and avoid https://github.com/apache/arrow/pull/5354

Done


Update to Helidon 4.1.3.

Also add configuration/validated tracing support for all integration modules.


Does not connect with Facebook token

checkpoint_url=null)InstagramLoginResult(super=StatusResult(super=dev.niekirk.com.instagram4android.requests.payload.InstagramLoginResult@35d4228a, status=fail, message=You can log into Instagram with your username or create a new account.), logged_in_user=null, error_type=fb_no_contact_point_found, checkpoint_url=null)
I do something wrong?

Yep that issue means you have an invalid FB access token.  To get the correct access token please use webview FB login, along with Instagrams App ID (do not create a Facebook app in their portal, use Instagram details instead)
Their app ID is:<PHONE_NUMBER>87414


Using date_bin with a time zone in a time range that contains daylight savings does not work

Describe the bug
When using the date_bin function, while providing a time zone to the column used in date_bin , and using a range that contains daylight savings (i.e. part of the range would be for example with an offset of +01:00, and another part with an offset of +02:00), the querying fails with the following error:
rpc error: code = InvalidArgument desc = External error: Arrow error: Cast error: Cannot cast timezone to different timezone

The query that I used is:
select
    date_bin(interval '1 hour', time at time zone 'Europe/Brussels')
from
    raw_data
where
    time >= '2021-03-27T22:00:00.000Z'
and
    time <= '2021-03-29T00:00:00.000Z'
group by date_bin(interval '1 hour', time at time zone 'Europe/Brussels')

Notice that the interval used [2021-03-27T22:00:00.000Z, 2021-03-29T00:00:00.000Z] contains daylight savings within the range, meaning that in this case part of the range will be with an offset of +01:00 (before the daylight savings), and another part will be with an offset of +02:00 (after the daylight savings)
I suspect the issue is that Arrow is attempting to case the times with a 2 hours offset to the times of an 1 hour offset
Update:
The issue is the daylight savings hour. This query works fine:
select
    date_bin(interval '1 hour', time at time zone 'Europe/Brussels')
from
    raw_data
where
    time >= '2021-03-27T22:00:00.000Z'
and
    time <= '2021-03-29T00:00:00.000Z'
-- exclude the daylight savings hour
and time not between '2021-03-28T02:00:00Z' and '2021-03-28T03:00:00Z'
group by date_bin(interval '1 hour', time at time zone 'Europe/Brussels')

In the query above, we exclude the daylight savings hour of that year for that timezone.
To Reproduce
Execute the following query
select
    date_bin(interval '1 hour', time at time zone 'Europe/Brussels')
from
    raw_data
where
    time >= '2021-03-27T22:00:00.000Z'
and
    time <= '2021-03-29T00:00:00.000Z'
group by date_bin(interval '1 hour', time at time zone 'Europe/Brussels')

Expected behavior
I should get the data aggregated by time. Part of the interval will be with a 2 hours offset, another part will be with a 1 hour offset
Additional context
raw_data is a table that contains values for each 15 minutes. The table with its data is stored in influxDB 3.0 that uses the FDAP stack
No response

I think the wider image of this issue is the fact that the hour at which the daylight savings occur will not be able to get transformed to the desired timezone. For example:
select '2019-03-31T02:00:00Z'::timestamp at time zone 'Europe/Brussels';

This would also fail for the same reason

I remember the daylight savings time is quite tricky because it has ambiguous possible time.
See https://github.com/apache/datafusion/issues/8899#issuecomment-1900598499

I am testing some things out with the at time zone operator to understand it better. I think the UTC timestamp casting should be explictly specified first. Some context:
Daylight savings in 2019 is in March 31 at 2AM in local time.
This is 2019-03-31T00:00:00 in UTC I think.
The query select '2019-03-31T02:00:00Z'::timestamp at time zone 'Europe/Brussels'; seems to be ignoring the Z at the end, and it considers the timestamp local time.
Some experiments:
select '2019-03-31T02:00:00Z'::timestamp at time zone 'Europe/Brussels';

result> Arrow error: Cast error: Cannot cast timezone to different timezone

select '2019-03-31T00:00:00'::timestamp at time zone 'UTC' at time zone 'Europe/Brussels'
result> 2019-03-31T01:00:00+01:00

select '2019-03-31T01:00:00'::timestamp at time zone 'UTC' at time zone 'Europe/Brussels'
result> 2019-03-31T03:00:00+02:00

It seems to me that, regardless of the offset provided in the timestamp of the query, the timestamp will considered as a naive date time, and the additional timezone information that is provided in the query will be considered the source of truth.


Help, example needed for ERC20 account balance enquiry

Hi, thank you so much for the great library. I'm kind of new in Crypto and Contract. I'm able to check for Ether account balance, but the Token balance of ERC20 seems different from NodeJS sample.
Sample in scOVu/web3.php:
$this->contract = new Contract($web3->provider, $this->testAbi);
Sample in NodeJS web3.js:
var tokenInst = new web3.eth.Contract(minABI,tokenAddress);
In your case now, how shall we check balance of a specify ERC20 account balance?

I end up manage to get ERC20 token balance from following code:
    var $testAbi = '[
              {
                "constant":true,
                "inputs":[{"name":"_owner","type":"address"}],
                "name":"balanceOf",
                "outputs":[{"name":"balance","type":"uint256"}],
                "type":"function"
              },
              {
                "constant":true,
                "inputs":[],
                "name":"decimals",
                "outputs":[{"name":"","type":"uint8"}],
                "type":"function"
              }
    ]';
    $web3 = new Web3(new HttpProvider(new HttpRequestManager($infuraURL, 10)));
    $tokenAddress = "0xdac17f958d2ee523a2206206994597c13d831ec7";
    $walletAddress = "0xbe0eb53f46cd790cd13851d5eff43d12404d33e8";     // Binance 7, Get from etherscan.io

    $this->contract = new Contract($web3->provider, $testAbi);
    $this->contract->at($tokenAddress)->call('balanceOf', $walletAddress, function ($err, $balance) {
        dd($balance['balance']);
    });

I end up manage to get ERC20 token balance from following code:
    var $testAbi = '[
              {
                "constant":true,
                "inputs":[{"name":"_owner","type":"address"}],
                "name":"balanceOf",
                "outputs":[{"name":"balance","type":"uint256"}],
                "type":"function"
              },
              {
                "constant":true,
                "inputs":[],
                "name":"decimals",
                "outputs":[{"name":"","type":"uint8"}],
                "type":"function"
              }
    ]';
    $web3 = new Web3(new HttpProvider(new HttpRequestManager($infuraURL, 10)));
    $tokenAddress = "0xdac17f958d2ee523a2206206994597c13d831ec7";
    $walletAddress = "0xbe0eb53f46cd790cd13851d5eff43d12404d33e8";     // Binance 7, Get from etherscan.io

    $this->contract = new Contract($web3->provider, $testAbi);
    $this->contract->at($tokenAddress)->call('balanceOf', $walletAddress, function ($err, $balance) {
        dd($balance['balance']);
    });

it's not working, I tried it and always return this:
  "balance": {
    "value": {

    },
    "is_negative": false,
    "precision": -1,
    "bitmask": false,
    "hex": null
  }
}```

have you check it again to see if it works?


Integrate jitsi-meet in OpenSlides (WIP)

Integrate Jitsi-meet in OpenSlides
Currently, we have a jitsi meet client on the start page.
Clicking a button openes the client and joins a room.

Conversations over OpenSlides are already possible.

I played a little more with jitsi's high level library. Although customization is limited, functionality can surely be wrapped inside an adapter. We should offer the full featured Jitsi-client nevertheless.
A jitsi window can now be shown and hidden anytime, can move around, stays in focus even on navigation and even works nicely in the background.
Opening OpenSlides automatically connects to a room defined by the "event name" on the meet.openslides.com server. Clients are usually muted and on audio only mode.
The connected user will have their short_name on jitsi. Hanging up will detach the jitsi client rather than jumping to the select-room screen.
Worked pretty fine for me so far. Please test around.

I can't test your branch because I get a 504 (Gateway timeout) with a fresh db after restart server:
e.g. Error occured while trying to proxy to: localhost:4200/apps/users/login/
Maybe some files are missing?

@emanuelschuetze cannot reproduce. The previous commit had some proxy files changed. I suppose you are on the previous commit. Compare the commit hash.

Works now correcly with new commit. Thx.

@emanuelschuetze updated:
reloading / reconnecting is more stable
Jitsi is only active in the first tab
link to external jitsi meet client
cleaned to toolbar in the jitsi client
server configs - default should work. Changes on the fly should also work.
mobile clients should work

new:

transpatent mute button
call button is now in accent
call button will be present as long as the user was not joining a room (this will prevent some errors)
current user is in list
projector is now compatible with rtc functions
focus current speaker with bold text
user list is scrollable (does neither shrink now grow)
firefox is allowed

new:

Rooms will be secured with passwords if "JITSI_ROOM_PASSWORD" was set in settings.py
rename events reflect in OpenSlides

new:

hide audio conference button if enable-jitsi was not set to true
show the current speaker above user list

Hi, I pushed the first version with jwt support.
Settings.py:
JITSI_DOMAIN = "localhost:8443"
JITSI_ROOM_NAME = "test"
# The issuer is also the app_id
JITSI_ISSUER = "os3_server"
JITSI_AUDIENCE = "os3_server"
JITSI_SECRET = "my_secret"

docker-jitsi-meet .env
# Enable authentication
ENABLE_AUTH=1

# Enable guest access
ENABLE_GUESTS=1

# Select authentication type: internal, jwt or ldap
AUTH_TYPE=jwt

# JWT authentication
#

# Application identifier
JWT_APP_ID=os3_server

# Application secret known only to your token
JWT_APP_SECRET=my_secret

# (Optional) Set asap_accepted_issuers as a comma separated list
JWT_ACCEPTED_ISSUERTS=os3_server

# (Optional) Set asap_accepted_audiences as a comma separated list
JWT_ACCEPTED_AUDIENCES=os3_server

After every change in .env you have to delete the ~/.jitsi-meet-cfg (or what ever you have set up for the CONFIG variable) and recreate the config folders as given in the readme.
I did not test ENABLE_GUESTS=0 yet. But if you are a guest and new in the room, there is a popup that you have to wait for a organisator to join (via jwt). I still haven't found out, if you can specify the role in the jwt. Also it is still possible to change one's name, if the iframe is visible, so the name is not locked...

I still haven't found out, if you can specify the role in the jwt

I found this repository.
https://github.com/nvonahsen/jitsi-token-moderation-plugin
In the Jitsi forum, I found the following by one of the developers ( https://community.jitsi.org/t/jwt-tokens-and-guest-access/18119/2 )

How we handle that is that we enable all_owners module so everyone in the room is moderator, enable jwt and enable enableUserRolesBasedOnToken. The later make sure that guests(those with no token) cannot do stuff like calling, recording etc.

That was closest thing I found to a clear statement to actual role based authentication.
So, if the user does have a JWT they will be moderators. Without a JWT users will be guests. I suppose we can either include the LUA module mentioned above or adjust the JWT path in OpenSlides accordingly
To sum it up:
If the user is in a certain OpenSlides group, send JWT, the user will be a Jitsi-moderator.
if not, do not send a JWT, the user will log in as a guest in Jtisi and thus not being a moderator.
A totally different approach:
We block direct Jitsi access, thus only allow OpenSlides-iFrame access, make everyone a Jitsi moderator and block moderating features in the Jitsi-iFrame / the OpenSlides UI depending on our own user roles.

Since this supposed to be merged without JWT, I am going to remove the last commit. Please create a branch of this branch to work on it.


Fallback on specific failures

Fallbacks should be more along the lines of RetryPolicies and CircuitBreaker, where they only get used for certain failures.

Added in 788e9361c456cf76cd8994ebb3e2d3290f824e78.


17

Changed ScalarTest code according to checkstyle.
Assert moved to MatcherAssert.

Coverage decreased (-7.6%) to 83.333% when pulling 55b46243a60c07fb41514fa95e836abe587d2712 on salikjan:17 into 1a9b77f7c067087648e7fdf5b9a9c6200b7538d2 on decorators-squad:master.

@salikjan Looks good, but just for the sake of the coverage, can you add 2 more asserts, for these cases?

By the way, you don't need to make another PR. Just make the changes on th ebranch, commit and push them and the current PR will update.
Now I closed the first PR, so we can go with this one. I.e. Add the 2 asserts, commit and push and this PR will be updated.

So now i can delete branch 11 ?

Forgot to check build with maven.
preparing new commit

@salikjan right, I made 2 comments. And yes, I'm pretty sure branch 11 can be removed. But let's merge this first, to be sure

Coverage decreased (-2.02%) to 88.889% when pulling 0f55f29b56f0f643cc4035c2a43fded6746f91b7 on salikjan:17 into 1a9b77f7c067087648e7fdf5b9a9c6200b7538d2 on decorators-squad:master.

@salikjan By the way, you know you can run the unit tests from the IDE, without running a full build, right?

@amihaiemil i always run tests from command line in the folder camel.
I do not know that i can do it in Intellij IDEA.
BTW where can i check the coveralls report?

@salikjan

I do not know that i can do it in Intellij IDEA.

I don't remember, I used Intellij a long time ago. But google it, it should be among the first results.

BTW where can i check the coveralls report?

Just click on the badge from the coveralls comment. It'll take you to coveralls.io (you might need to login?) and then look for the class that you are interested in.

Coverage increased (+3.5%) to 94.444% when pulling ab467cbba0137b0aa0e7b1c2b6e383158740c59a on salikjan:17 into 1a9b77f7c067087648e7fdf5b9a9c6200b7538d2 on decorators-squad:master.

Coverage increased (+3.5%) to 94.444% when pulling ab467cbba0137b0aa0e7b1c2b6e383158740c59a on salikjan:17 into 1a9b77f7c067087648e7fdf5b9a9c6200b7538d2 on decorators-squad:master.

@salikjan nice!

@rultor merge pls

@rultor merge pls

@amihaiemil OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge pls

@amihaiemil @salikjan Oops, I failed. You can see the full log here (spent 13s)
+ chmod 600 /home/rultor/.ssh/config
+ git clone git@github.com:decorators-squad/camel.git repo
Cloning into 'repo'...
+ cd repo
+ git config user.email<EMAIL_ADDRESS>+ git config user.name rultor
+ '[' -z 'mvn clean install -Pcheckstyle' ']'
+ cd ..
+ cat
+ '[' false = true ']'
+ cat
+ chmod a+x entry.sh
+ cat
+ echo 'mvn clean install -Pcheckstyle' ';'
+ gpgconf --reload gpg-agent
+ gpg --keyserver hkp://pool.sks-keyservers.net --verbose --recv-keys 9AF0FA4C
gpg: requesting key 9AF0FA4C from hkp server pool.sks-keyservers.net
Version: SKS 1.1.6
gpg: armor header:
Comment: Hostname: keyserver.vsund.de
gpg: armor header:
gpg: pub  2048R/9AF0FA4C 2014-08-17  Rultor.com<EMAIL_ADDRESS>gpg: key 9AF0FA4C: "Rultor.com<EMAIL_ADDRESS>not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
+ gpg --version
gpg (GnuPG) 2.0.22
libgcrypt 1.5.3
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2
++ pwd
++ pwd
+ gpg --verbose --keyring=/tmp/rultor-GqoJ/.gpg/pubring.gpg --secret-keyring=/tmp/rultor-GqoJ/.gpg/secring.gpg --decrypt repo/rcfg/settings.xml.asc
Version: GnuPG v2
gpg: armor header:
gpg: public key is 9A839C3A
gpg: using subkey 9A839C3A instead of primary key 9AF0FA4C
gpg: using subkey 9A839C3A instead of primary key 9AF0FA4C
gpg: encrypted with 2048-bit RSA key, ID 9A839C3A, created 2014-08-17
      "Rultor.com<EMAIL_ADDRESS>gpg: CAST5 encrypted data
gpg: original file name='rultor-hhD4'
+ gpg --no-tty --batch --verbose --decrypt --passphrase rultor-key:decorators-squad/camel settings.xml.enc
Version: GnuPG v2
gpg: armor header:
gpg: CAST5 encrypted data
gpg: encrypted with 1 passphrase
gpg: decryption failed: Bad session key
'cid' file is absent, container wasn't started correctly

@rultor merge pls

merged to master and pushed already

@rultor merge pls

@amihaiemil OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge pls

@amihaiemil @salikjan Oops, I failed. You can see the full log here (spent 15s)
+ chmod 600 /home/rultor/.ssh/config
+ git clone git@github.com:decorators-squad/camel.git repo
Cloning into 'repo'...
+ cd repo
+ git config user.email<EMAIL_ADDRESS>+ git config user.name rultor
+ '[' -z 'mvn clean install -Pcheckstyle' ']'
+ cd ..
+ cat
+ '[' false = true ']'
+ cat
+ chmod a+x entry.sh
+ cat
+ echo 'mvn clean install -Pcheckstyle' ';'
+ gpgconf --reload gpg-agent
+ gpg --keyserver hkp://pool.sks-keyservers.net --verbose --recv-keys 9AF0FA4C
gpg: requesting key 9AF0FA4C from hkp server pool.sks-keyservers.net
Version: SKS 1.1.5
gpg: armor header:
Comment: Hostname: key1.dock23.de
gpg: armor header:
gpg: pub  2048R/9AF0FA4C 2014-08-17  Rultor.com<EMAIL_ADDRESS>gpg: key 9AF0FA4C: "Rultor.com<EMAIL_ADDRESS>not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
+ gpg --version
gpg (GnuPG) 2.0.22
libgcrypt 1.5.3
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2
++ pwd
++ pwd
+ gpg --verbose --keyring=/tmp/rultor-FlQI/.gpg/pubring.gpg --secret-keyring=/tmp/rultor-FlQI/.gpg/secring.gpg --decrypt repo/rcfg/settings.xml.asc
Version: GnuPG v2
gpg: armor header:
gpg: public key is 9A839C3A
gpg: using subkey 9A839C3A instead of primary key 9AF0FA4C
gpg: using subkey 9A839C3A instead of primary key 9AF0FA4C
gpg: encrypted with 2048-bit RSA key, ID 9A839C3A, created 2014-08-17
      "Rultor.com<EMAIL_ADDRESS>gpg: CAST5 encrypted data
gpg: original file name='rultor-hhD4'
+ gpg --no-tty --batch --verbose --decrypt --passphrase rultor-key:decorators-squad/camel settings.xml.enc
Version: GnuPG v2
gpg: armor header:
gpg: CAST5 encrypted data
gpg: encrypted with 1 passphrase
gpg: decryption failed: Bad session key
'cid' file is absent, container wasn't started correctly

@salikjan I don't know what's wrong with rultor - I merged it myself for now. But you don't merge anything. Remember: first, the PR is merged in the upstream repo (where you don't have write access), and then you have to pull the changes from there to your master. ^^

So i do not have to merge from for example branch 17 to forked master?
Instead you do merge froim branch 17 to repo master and after that i pull from remote repo?

@salikjan exactly. You never merge anything. Wait for the branch to be merged to remote repo master and then you sincronize your local and fork with it (pull the changes locally and push them to your fork) :)

ok, catched )


Required Status Note Reminder Date in Mobile also.

as per below PC Screen View...

We can see Next Reminder Date.
In project list we can check status reminder date by hovering on status column but in mobile view hover is not possible so we can not get status note reminder dat
But as per below Mobile Screen view ...we can not able to show next reminder date......

e

over to @vinpahilwani


Firefox addon is not updated

Hello, I wish to try this on Firefox but when I access https://addons.mozilla.org/firefox/addon/overlay/ the available version is 0.0.1 (published 7 months ago). Could you update this please?

Thanks, just submitted for FireFox to review.
Going to integrate an automation soon

@valrieux Please let us know if you installed the newest version.


Convert Picture to BufferedImage

Create a stable Java2D specific method to convert a Picture to a BufferedImage. This will allow fun pixel art in conjunction with #85. This should be a new kind of effect type class, I think.
We already have a ToPicture type class. Perhaps FromPicture is the natural counterpart here? Not sure they would be symmetric though---rendering a picture is an effect but converting to a picture is not.

Hi Noel,
Is this what is needed here??
https://stackoverflow.com/questions/13605248/java-converting-image-to-bufferedimage

That's not really the issue. This is about having a way to convert the Picture type in Doodle into a BufferedImage. All the code already exists, it just needs to presented in a stable API.

I have one question about the end result. After the succesful implementation we want to be able to do something like:
val picture = circle(100)
val bi = picture.fromPicture

where bi is a BufferedImage or we want this function to evaluate the picture as well and get something like:
val picture = circle(100)
val (res, bi) = picture.fromPicture

Another possibility is to have a method .bufferedImage on picture that converts to the BufferedImage since there is such a method for the base64

The closest equivalent is the Base64 effect.
I think the following is probably what is needed:

An effect with a single method

def bufferedImage[A](picture: Picture[Alg, A]): IO[(A, BufferedImage)]

Syntax to make it easier to use.

An implementation for the Java2D backend. (Code already exists to do this conversion. It just needs to be exposed in this API.)

Yes, I think you're right. The parameters to renderBufferedImage are essentially the elements of Frame. So I think the correct signature is
def bufferedImage[A](frame: Frame, picture: Picture[Alg, A]): IO[(A, BufferedImage)]

The different order of types is just an accident. The external APIs should be as consistent as possible, and hence should follow the precedent in base64.

Does the binary compatibillity check failing mean that I cannot make the Java2dWriter extend the BufferedImageConverter and should create a separate class for that?

You can ignore binary compatibility issues. If you bump the base version defined in build.sbt the problem will go away.

I made everything specific for java2d backend but now I started thinking that maybe the approach of making the converter generic with respect to the bitmap is better as other backends might have their own bitmap classes and that could be helpful for example if we want draw pixels on the screen or do the convolutions (#94)


Parallelization with more than one port

Hey Jonathan,
hope your 2018 is going well.
I have a quick question about parallel streams. Say I want to have following configuration:
 m += a <= b ["processOne"]=> c
 m +=      b ["processTwo"]=> d

Then the port processOne is multiplied (e.g. processOne0, processOne1,, etc) as many times as threads in c and the same for processTwo and d.
As I understand the current framework does not support this. Is there any specific reason why it is not supported now? And what do you think would be the efforts for introducing something like this (i.e. do you think I can implement it myself) ?
Thanks a lot for your help..

Ok. So, the syntax
m += a <= b => c

would replicate b for as many times as there are output ports in _a, assuming that the output port count on _a and the input port count on _b would be equal, otherwise an exception would be thrown.
The syntax
 m += a <= b ["processOne"]=> c
 m +=  b ["processTwo"]=> d

Would in this case would create something that looks like this

The runtime would then have to check to ensure that the output port count on _a matched the input port count on both _c and _d. Then the one instantiated _b kernel would be replicated to match the number of ports as in the next figure.

Is that the intended behavior of what you would like to implement? If so, I can help you out if you're still interested in implementing. If not, let me know and lets think though it a bit. In either case, happy to help. Apologies for the delayed response. It's been a very busy few months.
-Jonathan

Hi Jonathan,
now I am the one apologizing for the late response, it's been crazy busy here too.
Indeed that was the intended behavior of what I wanted at the time. In between the designed has changed a little bit in my mind, I am thinking now more of a paralellization of every single kernel independently of the paralellization of the next one. So given:
 m += a <= b ["processOne"]=> c >> e
 m +=      b ["processTwo"]=> d

I would like to give b and c different number of threads. I think I can achieve this by adding before and after each of those kernels a split one, sort of what you do with raft::order::out, but keeping the order.
This feature has very low priority in my project right now so I will not need it for now. When I need it I think this can be achieved from outside the library (i.e. making my own split kernels), so we can consider this post closed for now..
Thanks again for a great library..


TexInput - textAlign={"start"} prop not working

I have been using TextInput .
As per the documentation textAlign enum('start', 'center', 'end')
Set the position of the cursor from where editing will begin.
I had made the following changes
<TextInput
ref="company"
autoCapitalize='none'
style={[styles.input,styles.formLabel,commonStyles.smallText]}
value={this.state.company}
underlineColorAndroid={'transparent'}
onChangeText={(text) => this.setState({company:text})}
textAlign={"start"}
/>
and getting this error
Error while updating property 'textAlign' in shadow node of type AndroidTextInput

Hey, thanks for reporting this issue!
It looks like your description is missing some necessary information, or the list of reproduction steps is not complete. Can you please add all the details specified in the template? This is necessary for people to be able to understand and reproduce the issue being reported.
I am going to close this, but feel free to open a new issue with the additional information provided. Thanks!


如何获取当前主题的主颜色？

Bug Report or Feature Request (mark with an x)

[ ] Bug report -> please search issues before submitting
[ ] Feature request
[ ] Documentation issue or request

Reproduction link

Steps to reproduce

What is expected?

What is actually happening?

Environment

ng-alain version: X.Y.Z
Angular version: X.Y.Z
Browser:
- [ ] Chrome (desktop) version XX
- [ ] Firefox version XX
- [ ] Safari (desktop) version XX
- [ ] IE version XX

ThemeService


Validation note and pointer to a related utility

Thanks to Zmajzlik for asking this question and for the pointer about the TypeHelper solution!
(This is a resubmit of https://github.com/MicrosoftDocs/dynamics365smb-devitpro-pb/pull/2282 from a forked branch to isolate the change)

Thanks for the contribution, much appreciated!


Is it possible to update data from main UI thread -> already running background task?

Hi,
Thanks a lot for creating this library first of all.
I have an app that records a user's heart rate and monitors our some statistics. I would like this monitoring to continue even when the user minimizes the application, which is why I have started looking into Headless JS and your library. Right now, I can start up a background task using your library and read HR data continuously whether the app is on the foreground or not, which is great. I can also update main thread UI state from my background task ( I pass in a function to options), to keep a log of these measurements which is also good. (communication from running background task -> main UI thread)
My main issue right now is that I would like to send some data to the active background task  ( communication from Main thread to running background task) and I am unable to do so.  I have tried doing this by passing a piece of useState from my main UI thread when I initialize the background job on startup via:
BackgroundJob.start(taskRandom, options)
The issue I'm experiencing is that if I update this state within my main UI thread, this does not get reflected on the current background task. Is this expected or am I doing something wrong?

@act-naturally, you could pass a reference to an already instantiated event emitter. For example:
const uiEventEmitter = new EventEmitter();

const task = async (taskDataArguments) => {
    await new Promise(() => {
           uiEventEmitter.addEventListener('event', () => {
              // Your UI wanted to change something
           };
           // Listen to hear rate
    });
};

// UI
BackgroundJob.start(task, options);
// Do stuff
// Emit the event with the info you want to communicate
uiEventEmitter.emit('event');

Great, this is perfect. I couldn't use EventEmitter but DeviceEventEmitter did work for me for anyone interested:
import { DeviceEventEmitter} from 'react-native';

const task = async (taskDataArguments) => {
    await new Promise(() => {
        // background thread
 DeviceEventEmitter.addListener('event.test', (event) => {
                // do something with event from UI
            });
    });
};

// UI
BackgroundJob.start(task, options);
// Do stuff
// Emit the event with the info you want to communicate
 DeviceEventEmitter.emit('event.test', { anyEventProps.. })

`


RecursionError calling get_converter() on Python 3.7.1

Hi folks. Thanks for this project! Excited to try out pygls 1.0 on top of lsprotocol.
I noticed the following RecursionError; not sure whether you want to try to work around it, or just bump the minimum supported version. (either is OK by me)
$ python
Python 3.7.1 (default, Dec 18 2022, 11:28:25)
[Clang 11.0.3 (clang-1<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from lsprotocol.converters import get_converter
>>> get_converter()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/site-packages/lsprotocol/converters.py", line 17, in get_converter
    return _hooks.register_hooks(converter)
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/site-packages/lsprotocol/_hooks.py", line 35, in register_hooks
    _resolve_forward_references()
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/site-packages/lsprotocol/_hooks.py", line 30, in _resolve_forward_references
    attrs.resolve_types(value, lsp_types.ALL_TYPES_MAP, {})
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/site-packages/attr/_funcs.py", line 410, in resolve_types
    hints = typing.get_type_hints(cls, globalns=globalns, localns=localns)
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/typing.py", line 973, in get_type_hints
    value = _eval_type(value, base_globals, localns)
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/typing.py", line 262, in _eval_type
    ev_args = tuple(_eval_type(a, globalns, localns) for a in t.__args__)
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/typing.py", line 262, in <genexpr>
    ev_args = tuple(_eval_type(a, globalns, localns) for a in t.__args__)
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/typing.py", line 263, in _eval_type
    if ev_args == t.__args__:
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/typing.py", line 658, in __eq__
    return frozenset(self.__args__) == frozenset(other.__args__)
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/typing.py", line 664, in __hash__
  [... repeats many times ...]
  File "/Users/pschanely/.pyenv/versions/3.7.1/lib/python3.7/typing.py", line 477, in __hash__
    return hash((self.__forward_arg__, self.__forward_value__))
RecursionError: maximum recursion depth exceeded

Does this happen with a newer version of Python? Just trying to narrow down if this is from our code, typing, or attrs. Since the __hash__ isn't by us and that's where the recursion is, my guess is Python 3.7 might be the problem.


Handle more than 7 apps

When there are more than 7 apps in the config folder, the list overflows and there does not seem to be a way to scroll it.

Firstly, congratulations on having at least 7 apps! That's a great sign
that the rM community is growing nicely.
Secondly, I agree, that is a problem. I'm afraid I don't have a fix at the
moment and I am knee deep in about 3 major projects.
If anybody would like to fork draft and add a scrollable zone, that would
be great! It should not be too challenging so I suggest it would be a good
way to get into hacking rM.
On Sun, 4 Oct 2020, 23:06 Mattéo Delabre<EMAIL_ADDRESS>wrote:

When there are more than 7 apps in the config folder, the list overflows
and there does not seem to be a way to scroll the list.
[image: screendump]
https://user-images.githubusercontent.com/1370040/95028166-920f2e80-069e-11eb-986c-e804131d772b.png
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dixonary/draft-reMarkable/issues/24, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4T3F5UXEISYZ7RPFSJ7N3SJDWWRANCNFSM4SD7DIOA
.

If anybody would like to fork draft and add a scrollable zone, that would be great! It should not be too challenging so I suggest it would be a good way to get into hacking rM.

coughs... Time to use Oxide?

I believe it might be! :-)
On Mon, 5 Oct 2020, 02:19 Nathaniel van Diepen<EMAIL_ADDRESS>wrote:

If anybody would like to fork draft and add a scrollable zone, that would
be great! It should not be too challenging so I suggest it would be a good
way to get into hacking rM.
coughs... Time to use Oxide?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dixonary/draft-reMarkable/issues/24#issuecomment-703348726,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4T3F4AOQFDKJZ7KKTQSMTSJENIZANCNFSM4SD7DIOA
.


Avoid rendering errors by passing in the webGLContext when creating a new CanvasGraphics in getColorN_Pattern (PR 9095 follow-up)

This was an oversight in PR #9095, which unfortunately breaks rendering in some PDF files (e.g. the one from issue #6737).
It thus appears that we don't have any test-coverage for this code-path, and given the relative complexity of the PDF files affected by this bug I wasn't able to easily create a reduced test-case.
Please note: The linked test-case included in this patch is currently not rendered correctly (that'd be the PR #6606), but it at least gives us some test-coverage here.

/botio test

/botio-linux preview

/botio makeref

Good fix, thank you!


Add a label in delete profile button.

@comzeradd r?
Regarding the font-weight in the label, there is a selector added in PR #1384 which will tackle this one as well.

r+


Error deleting all sponsors

Internal Server Error: /2016/admin/sponsorship/sponsor/
Traceback (most recent call last):
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/options.py", line 616, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/utils/decorators.py", line 110, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 233, in inner
return view(request, *args, **kwargs)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/utils/decorators.py", line 34, in _wrapper
return bound_func(*args, **kwargs)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/utils/decorators.py", line 110, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/utils/decorators.py", line 30, in bound_func
return func.get(self, type(self))(*args2, **kwargs2)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/options.py", line 1574, in changelist_view
response = self.response_action(request, queryset=cl.get_queryset(request))
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/options.py", line 1333, in response_action
response = func(self, request, queryset)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/actions.py", line 37, in delete_selected
queryset, opts, request.user, modeladmin.admin_site, using)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/utils.py", line 153, in get_deleted_objects
to_delete = collector.nested(format_callback)
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/utils.py", line 213, in nested
roots.extend(self._nested(root, seen, format_callback))
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/utils.py", line 196, in _nested
children.extend(self._nested(child, seen, format_callback))
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/utils.py", line 198, in _nested
ret = [format_callback(obj)]
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/contrib/admin/utils.py", line 126, in format_callback
force_text(obj))
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/utils/encoding.py", line 92, in force_text
s = six.text_type(s)
File "/srv/pycon/pycon/pycon/sponsorship/models.py", line 372, in unicode
return u"%s - %s (%s)" % (self.sponsor, self.benefit,
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 610, in get
rel_obj = qs.get()
File "/srv/pycon/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get
self.model._meta.object_name
DoesNotExist: Benefit matching query does not exist.

That turned out to be a problem in the data; somehow the benefit records we are not using anymore were deleted but not other records that still had foreign keys to them.
I've fixed it up and you should be able to delete the sponsors now.


Feature request: Add support for modifier keys like right ctrl and right alt

Issue Type
Feature Request
Description
Currently VS Code treats left and right modifier keys as the same one (ctrl, alt and shift).
I want to define different keyboard shortcuts for left and right modifier keys (e.g. left ctrl + s for saving files, and right ctrl + s for something else), but unfortunately this cannot be done.
VS Code Info
VS Code version: Code 1.20.0 (c63189deaa8e620f650cc28792b8f5f3363f2c5b, 2018-02-07T17:10:15.949Z)
OS version: Linux x64 4.14.16-300.fc27.x86_64

possible code pointer?
https://github.com/Microsoft/vscode/blob/master/src/vs/base/common/keyCodes.ts#L21

Came here because I'm running out of available key combinations to use as shortcuts :) This would allow me to pretty much double it.
It's definitely possible to differentiate between them as shown here: https://w3c.github.io/uievents/tools/key-event-viewer.html

Any update on this request?
It doesn't make any conflicts with current settings and also expands possible shortcuts.
I myself usually do commands with one of the sides, and the other can be used for another shortcut.

Is very interesting feature request.
As example, I want to use WASD as arrows when right cmd is pressed.

I think this feature is almost out of scope.
However, this feature could be implemented by a separate tool, independent of VS Code.
You could use for example AutoHotkey to override keybindings on windows, so that right-ctrl + W maps to F13. Afaik you can bind to F13 in VS Code.

Would it be better to start by just allowing users to specify keycodes in the shortcuts json? Might fix some issues for international keyboards as well?
Or a distinct handleKeycodes object with either keyEvent.key:
{
  "name": "Right Alt",
  "key": "Alt",
  "Location": "2"
}

or keyEvent.code:
{
  "name": "Right Alt",
  "code":"AltRight"
}

Then the shortcuts UI could check it an entry is displayed in handleKeycodes and display the specified name when setting new shortcuts.

Would it be better to start by just allowing users to specify keycodes in the shortcuts json? Might fix some issues for international keyboards as well?
Or a distinct handleKeycodes object with either keyEvent.key:
{
  "name": "Right Alt",
  "key": "Alt",
  "Location": "2"
}

or keyEvent.code:
{
  "name": "Right Alt",
  "code":"AltRight"
}

Then the shortcuts UI could check it an entry is displayed in handleKeycodes and display the specified name when setting new shortcuts.

I believe this is also an interesting approach for custom non-standard keycodes, although I do think this is a separate feature, since Right Alt is a fairly common modifier and doesn't deserve a custom configuration, IMO.

A thing to solve here is to help extension authors avoid some key bindings that would become available. For instance, I don't have a right ctrl key on my keyboard (a MacBook Pro). And since it is also a Swedish layout, I already have enough problems with some VS Code built-ins shortcut defaults.

I'm here because I'd love to get support for NumpadEnter as distinct from Enter. I blame TextMate for training me this way so many years ago by binding NumpadEnter — or Fn + Return on Mac laptops — to several useful sorts of commands.
It's such a convenient way of expressing “like Enter, only more so” in various contexts. Imagine using NumpadEnter

to continue your line- or block-comment on the next line with whatever delimiter is appropriate for the file you're in, instead of the simple line break that Enter would add;
to open the URL that your cursor is within in an HTML or Markdown file;
to continue a Markdown list;
and, well, these are the exact three things I use NumpadEnter for in Atom right now, but I'm sure others exist.

It's definitely possible to differentiate between them as shown here: https://w3c.github.io/uievents/tools/key-event-viewer.html

unfortunatley afaik that's another brokey key API, it doesn't differentiate between left/right key modifiers in a combo, only when you press them as non-modifier keys, you'd need some not-fully-reliable listener/tracker of previous events (which may happen outside of your app, so you'd miss them)

You could use F13-F24, including various combinations with ctrl and shift.

then you'd lose all the direct hints in the app and have to remember/lookup what is mapped to what.
lso external keybinding tools don't have knowledge of the app state so, e.g., if you have modal-vim-like editing plugin, they wouldn't know what mode you're in.
Also it's not x-platform unlike native support, so you'd have to duplicae work
So this approach is very unergonomic, although it beats having nothing

I think this feature is almost out of scope.

In what way is keyboard shortcut handling for an application that supports keyboard shortcuts remotely out of scope? Genuinely asking, because I'm not parsing the logic here. I would think this fits the stated goals of the project without need for further evaluation. The current behavior undesirably (for many) merges the assignment of multiple keys. There could very easily be a toggleable setting that determines whether these sorts of keys are merged.

You could use for example AutoHotkey to override keybindings on windows, so that right-ctrl + W maps to F13. Afaik you can bind to F13 in VS Code.

Wouldn't that also trigger the built-in Ctrl + W shortcut and close the window?

You could use F13-F24, including various combinations with ctrl and shift.

Does it seem reasonable for an application with keyboard shortcuts to support F13-F24 -- keys not on a typical keyboard -- but not support Right Ctrl, Right Shift, Right Alt, Numpad Enter, etc. -- keys generally on both full ANSI and ISO keyboards? Again, I am genuinely asking. This seems to fall firmly within the precise definition of a workaround.

Requesting this reminds me a lot of this xkcd comic, but hey... I'm trying to bind my right alt key (altgr+.) to a snippet, and my xmodmap config maps that combination to » (which is what I want) but inside vscode I can't catch it neither as alt+. or ». I know I can map things to e.g. F13, and I even have a Streamdeck to play around if I need extra keys... but I just expect vscode to handle this (simple?) case. Many applications support this, and considering developers are often power users, it'd make sense to see support for this out of the box.

You can see in keycodes.ts that the left/right modifier keys are mapped to the simple versions:
[1, ScanCode.ControlLeft, 'ControlLeft', KeyCode.Ctrl, empty, 0, 'VK_LCONTROL', empty, empty],
[1, ScanCode.ShiftLeft, 'ShiftLeft', KeyCode.Shift, empty, 0, 'VK_LSHIFT', empty, empty],
[1, ScanCode.AltLeft, 'AltLeft', KeyCode.Alt, empty, 0, 'VK_LMENU', empty, empty],
[1, ScanCode.MetaLeft, 'MetaLeft', KeyCode.Meta, empty, 0, 'VK_LWIN', empty, empty],
[1, ScanCode.ControlRight, 'ControlRight', KeyCode.Ctrl, empty, 0, 'VK_RCONTROL', empty, empty],
[1, ScanCode.ShiftRight, 'ShiftRight', KeyCode.Shift, empty, 0, 'VK_RSHIFT', empty, empty],
[1, ScanCode.AltRight, 'AltRight', KeyCode.Alt, empty, 0, 'VK_RMENU', empty, empty],
[1, ScanCode.MetaRight, 'MetaRight', KeyCode.Meta, empty, 0, 'VK_RWIN', empty, empty],

https://github.com/microsoft/vscode/blob/1cc266cc82838981034c2baa637f357a83711d5d/src/vs/base/common/keyCodes.ts#L657C1-L664C89
Why this s done I don't know.

I am suffering the same issue, but in another, a bit more complex scenario on Linux under Sway and some keys remapped with xremap; everything is explained here: https://github.com/xremap/xremap/issues/490
The remap works only on the terminal editor; even though the keycode seems to be sent correctly to the process, it didn't work.
Do you have any idea why?

I have an issue because I use LShift + RShift in Ubuntu to change the keyboard layout. In IntelliJ IDEA, it works fine, but in VSCode, every time I change the layout, it opens 'Go to file' in focus. The problem is that LShift + RShift triggers the Shift Shift binding, which is the 'Go to File' key binding from the IntelliJ IDEA keybindings plugin.

+1


[Usage] error about Chinese data finetuning

Describe the issue
Issue:
There is no problem fine-tuning using the provided dataset, but using local Chinese data will report this error, which is the same as the issue 134 issue, but it has been closed.
Command:
sh finetune_lora.sh[change data_path and image_folder]

finetune_lora.sh
Log:
Traceback (most recent call last):
  File "/workspace/LLaVA/llava/train/train_mem.py", line 13, in <module>
    train()
  File "/workspace/LLaVA/llava/train/train.py", line 932, in train
    trainer.train()
  File "/xxx/python3.10/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/xxx/python3.10/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/xxx/python3.10/site-packages/transformers/trainer.py", line 2654, in training_step
    loss = self.compute_loss(model, inputs)
  File "/xxx/python3.10/site-packages/transformers/trainer.py", line 2679, in compute_loss
    outputs = model(**inputs)
  File "/xxx/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/xxx/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/xxx/python3.10/site-packages/deepspeed/runtime/engine.py", line 1735, in forward
    loss = self.module(*inputs, **kwargs)
  File "/xxx/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/xxx/python3.10/site-packages/peft/peft_model.py", line 922, in forward
    return self.base_model(
  File "/xxx/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/workspace/LLaVA/llava/model/language_model/llava_llama.py", line 79, in forward
    ) = self.prepare_inputs_labels_for_multimodal(
  File "/workspace/LLaVA/llava/model/llava_arch.py", line 178, in prepare_inputs_labels_for_multimodal
    cur_image_features = image_features[cur_image_idx]
IndexError: index 16 is out of bounds for dimension 0 with size 16.

I've tried to delete all explicit '\n' in the json file.For example:
"value": "<image>\n图中的人是xxx,请以这个人为主体描述一下图中内容"-->"value": "<image>图中的人是xxx,请以这个人为主体描述一下图中内容"
but it doen't work.

Had the same issue today with english data, problems were: \n
and also "\ in some cases.
check those.

Had the same issue today with english data, problems were: \n and also "\ in some cases.
check those.

thank you.I checked before,but there is no '' except '\n'.
I print some vars
            for i in range(num_images + 1):
                cur_new_input_embeds.append(cur_input_embeds_no_im[i])
                cur_new_labels.append(cur_labels_noim[i])
                if i < num_images:
                    print(f"Batch Index: {batch_idx}\n, Current Image Index: {cur_image_idx}\n, Num Images: {num_images}")

in fileLLaVA/llava/model/llava_arch.py,line 178.
when it runs normally, Num Images: 1
when it runs with error, Num Images: 4

suggest to make sure one sample is right at https://github.com/haotian-liu/LLaVA/blob/main/llava/train/train.py#L402
if the input_ids and labels/targets are as expected, then there will be no error in the later processes. @CrazyBrick

Had the same issue today with english data, problems were: \n and also "\ in some cases.
check those.

thank you.I checked before,but there is no '' except '\n'. I print some vars
            for i in range(num_images + 1):
                cur_new_input_embeds.append(cur_input_embeds_no_im[i])
                cur_new_labels.append(cur_labels_noim[i])
                if i < num_images:
                    print(f"Batch Index: {batch_idx}\n, Current Image Index: {cur_image_idx}\n, Num Images: {num_images}")

in fileLLaVA/llava/model/llava_arch.py,line 178. when it runs normally, Num Images: 1 when it runs with error, Num Images: 4

Based on your message, I think you have similar problems like me, and it is because the data preprocessing is wrong. You should check that for every instance in your data list, mostly in multi turn conversations, the tag  should appear only once.

Had the same issue today with english data, problems were: \n and also "\ in some cases.
check those.

thank you.I checked before,but there is no '' except '\n'. I print some vars
            for i in range(num_images + 1):
                cur_new_input_embeds.append(cur_input_embeds_no_im[i])
                cur_new_labels.append(cur_labels_noim[i])
                if i < num_images:
                    print(f"Batch Index: {batch_idx}\n, Current Image Index: {cur_image_idx}\n, Num Images: {num_images}")

in fileLLaVA/llava/model/llava_arch.py,line 178. when it runs normally, Num Images: 1 when it runs with error, Num Images: 4

@CrazyBrick Based on your message, I think you have similar problems like me, and it is because the data preprocessing is wrong. You should check that for every instance in your data list, mostly in multi turn conversations, the tag  should appear only once.

thx, your guess is correct, I do have multi-turns of dialogue and multipletags. Yesterday, I regenerated a version that only had one dialogue. The Num_Images have become what I expected, but the index will still increase until IndexError: index 16 is out of bounds for dimension 0 with size 16.. I have no clue...

That is weird, maybe you should find a way to print out the actual instance that cause this error. Because it loads the images and texts seperately, which means the length of image_features is fixed (e.g. the length of  image_features is 16 in your case) before dealing with the texts. And in one batch, the cur_image_idx should only increase in two cases, first is that there is no image in this instace, second is that there is only one <image> in the instance which makes the cur_image_idx increases only once in one instance, so if there are two or more tags in the instance, the cur_image_idx would increase twice or more and then exceed the bound of already fixed image_features.

@CrazyBrick That is weird, maybe you should find a way to print out the actual instance that cause this error. Because it loads the images and texts seperately, which means the length of image_features is fixed (e.g. the length of image_features is 16 in your case) before dealing with the texts. And in one batch, the cur_image_idx should only increase in two cases, first is that there is no image in this instace, second is that there is only one  in the instance which makes the cur_image_idx increases only once in one instance, so if there are two or more tags in the instance, the cur_image_idx would increase twice or more and then exceed the bound of already fixed image_features.

thanks @henrycjh
quite weird, I debug for a while but I didn't find anything strange.But after regenerating the custom dataset for multi-tunes of dialogue with 1 tag, it seems like there are no errors. I don't understand, but in the end, it can run.
But I have to admit that my fine-tuning did not achieve the expected effect and seems to have no effect. I don't know if it's due to too little data or other parameter reasons.

@CrazyBrick  Glad that you make it work. I guess the main reason that fine-tuning has no effect is that the LLM is pretrained mainly on English data, so it may still have no effect even you have a lot of Chinese data during fine-tuing stage.

@CrazyBrick Glad that you make it work. I guess the main reason that fine-tuning has no effect is that the LLM is pretrained mainly on English data, so it may still have no effect even you have a lot of Chinese data during fine-tuing stage.

@henrycjh thank you for your help, I selected a portion from the dataset to generate English data, but the prediction is quite poor.
I am confused about how to achieve effective finetune, how many constraints are needed, and what kind of expectations can be achieved
I've proposed a new issue #884

@CrazyBrick Glad that you make it work. I guess the main reason that fine-tuning has no effect is that the LLM is pretrained mainly on English data, so it may still have no effect even you have a lot of Chinese data during fine-tuing stage.

@henrycjh thank you for your help, I selected a portion from the dataset to generate English data, but the prediction is quite poor. I am confused about how to achieve effective finetune, how many constraints are needed, and what kind of expectations can be achieved I've proposed a new issue #884

Did you do the 2 stage training from zero to one? In that case, intervening from stage 1 with Chinese data and mixing in some in stage 2 might work much better. That's what I did myself, and my Chinese skills still improved visibly

@CrazyBrick Glad that you make it work. I guess the main reason that fine-tuning has no effect is that the LLM is pretrained mainly on English data, so it may still have no effect even you have a lot of Chinese data during fine-tuing stage.

@henrycjh thank you for your help, I selected a portion from the dataset to generate English data, but the prediction is quite poor. I am confused about how to achieve effective finetune, how many constraints are needed, and what kind of expectations can be achieved I've proposed a new issue #884

I'm assuming you're translating the Chinese data from English? That's what I did, so I ran into a situation where multiple  appeared in a single session.


Update layout.hbs

downgrade JQuery to 2.2.4

Jquery is loaded via NPM and is already version 2.x in latest commit.

i don't think so,  npm load the latest version of JQuery
/*! jQuery v3.1.0 | (c) jQuery Foundation | jquery.org/license */

Shouldn't be. See PR: https://github.com/mrvautin/adminMongo/pull/91

i try to update the code to the worked version , i found those problems, so i think that maybe other need help ^^

We would prefer not to load dependencies from CDN in an application like this which is often used locally and without outside internet access.

okay so to prevent those problems, you can keep the latest version of JQuery and change "success" && "error" callbacks on "editor.js"

The latest version of Bootstrap now supports jQuery 3.x. After a little testing, we will be upgrading both jQuery and Bootstrap in package.json.

Changed in commit: 4641afa91c7f5690e6750fb979a9100f9d7af9cd.


Configuration#7: Step#1: alpha_S=0.2, matter: q_0=1 | matterlbt: q_0=2.5: matterlbt03

run the simulation with the above config by jetscape,
transfer the final state hadron to the related directory named as config-XX-final-state-hadrons
after doing step1 on all 12 simulations go to step 2 for the configurationX

port to Excel for overleaf