Save a trained model

Is there a way to save a trained model?

#14

hi again @alfredcs
At long last I have implemented this (NERDA==1.0.0). See functions:
python model.save_network
python model.load_network_from_file
Thank you so much for you feedback.


Add Music

Didn't really know where to add this, so I created a new category called Creating, right after Gaming.

Put it in the Misc section. I only create new categories when there's at least two items that could go into it.

Please read the contributing guidelines and the awesome manifest. The list is missing a license.

Also, I added a license to https://github.com/ciconia/awesome-music.

The list should have a header # Awesome Music.

Sorry for the messy commits, my poor git skills... :-)

Added


fix: stencil create

add arguments to get releasing working for native extensions

What this PR does / why we need it

Jira ID
[XX-XX]

Notes for your reviewers

Coverage decreased (-0.1%) to 40.768% when pulling f4b0c7b4e033326838d32fd0ac38fb2bc49e4fa5 on george-e-shaw-iv/fix/stencil-create into 2b3a527caffe08b4b0ce57b93ca5c88d82eb779a on main.


error writing parquet dataframe with only 1 row

Hello!  Thanks again for your quick fixes for my last couple of issues!
I'm having an odd problem where writing out a parquet dataframe fails if there's only 1 row.
In the below example, when one partition only has 1 row I get the error ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column col1 with type Int64').
If you uncomment the additional row in the example below so that each partition has > 1 row the dataframe writes out fine.
I believe the issue is with this line https://github.com/awslabs/aws-data-wrangler/blob/master/awswrangler/pandas.py#L896 .  For some reason slice notation with a single row isn't actually assigning the new float64 casted dataframe.  I'm not entirely sure why that would be.
There also seems to be an interaction with procs_cpu_bound, when it's > 1 the 4 row partitioned dataframe throws the arrow type error when it didn't with procs_cpu_bound=1.
Let me know if I can provide any more info!
Thanks!
import pandas as pd
import awswrangler

session = awswrangler.Session(procs_cpu_bound=1)

# create some dummy data
data = {
    'col1': [
        1,
        2,
        3,
        # 4,
    ],
    # date column to see valid dates
    'datecol': [
        '2019-11-09',
        '2019-11-09',
        '2019-11-08',
        # '2019-11-08',
    ],
    # dates in partition column to see invalid date strings
    'partcol': [
        '2019-11-09',
        '2019-11-09',
        '2019-11-08',
        # '2019-11-08',
    ]
}

df = pd.DataFrame(data)
# cast date columns to pandas datetime, similar to the result of read_csv() with parse_dates
df = df.astype({'datecol': 'datetime64', 'partcol': 'datetime64'})

# tell Athena what types you want
schema = {
    'col1': 'bigint',
    'datecol': 'date',
    'partcol': 'date',
}

session.pandas.to_parquet(
    dataframe=df,
    database="default",
    path="s3://...,
    partition_cols=['datecol'],
    table="r...",
    mode="overwrite_partitions",
    cast_columns=schema
)

Thanks again @chtorr !
Already fixed: https://github.com/awslabs/aws-data-wrangler/releases/tag/0.0.19


feat: AppRouter component

Adding a base AppRouter component to easily setup a Squide application React Router with Squide registration and routing primitives along with MSW.
Also update the define functions to automatically infer the publicPath by default. More info here:                         setIsPublicDataLoaded(true);
What changed
@squide/core

completeLocalModuleRegistrations data argument is now required.

@squide/webpack-module-federation

defineBuildHostConfig, defineDevHostConfig, defineBuildRemoteModuleConfig and defineDevRemoteModuleConfig publicPath prop is not mandatory anymore. It's default value is now auto.
defineBuildHostConfig and defineBuildRemoteModuleConfig set the HtmlWebpackPlugin publicPath prop to / by default.
completeRemoteModuleRegistrations data argument is now required.

@squide/firefly
This is a new package that offer a base AppRouter component for the "firefly" stack: React Router + MSW + Webpack
Basic usage of the new base AppRouter component:
import { AppRouter as FireflyAppRouter } from "@squide/firefly";

function Loader() {
    return (

Loading...

    );
}

export function AppRouter() {
    return (
        <FireflyAppRouter
            fallback={<Loader />}
            waitForMsw={false}
        />
    );
}

Advanced usage with public data, protected data, MSW and deferred registrations:
export function AppRouter({ waitForMsw, sessionManager, telemetryService }: AppRouterProps) {
    // Could be done with a ref (https://react.dev/reference/react/useRef) to save a re-render but for this sample
    // it seemed unnecessary. If your application loads a lot of data at bootstrapping, it should be considered.
    const [featureFlags, setFeatureFlags] = useState<FeatureFlags>();
    const [subscription, setSubscription] = useState<Subscription>();

    const logger = useLogger();
    const runtime = useRuntime();

    const handleLoadPublicData = useCallback((signal: AbortSignal) => {
        return fetchPublicData(setFeatureFlags, logger, signal);
    }, [logger]);

    const handleLoadProtectedData = useCallback((signal: AbortSignal) => {
        const setSession = (session: Session) => {
            sessionManager.setSession(session);
        };

        return fetchProtectedData(setSession, setSubscription, logger, signal);
    }, [logger, sessionManager]);

    const handleCompleteRegistration = useCallback(() => {
        return completeModuleRegistrations(runtime, {
            featureFlags
        });
    }, [runtime, featureFlags]);

    return (
        <FeatureFlagsContext.Provider value={featureFlags}>
            <SubscriptionContext.Provider value={subscription}>
                <TelemetryServiceContext.Provider value={telemetryService}>
                    <FireflyAppRouter
                        fallback={<Loader />}
                        waitForMsw={waitForMsw}
                        onLoadPublicData={handleLoadPublicData}
                        onLoadProtectedData={handleLoadProtectedData}
                        onCompleteRegistration={handleCompleteRegistration}
                    />
                </TelemetryServiceContext.Provider>
            </SubscriptionContext.Provider>
        </FeatureFlagsContext.Provider>
    );
}

I think there's already enough in this PR for a good review but I will need to update the code this week to handle errors while loading the data.
Something in missing.


Custom SSH key

I'm new to kamal and I can't find a way to specify the private ssh key to use for ssh. I can only modify the user or setup a proxy. Did I miss something? Otherwise I think this would make a great option.

You can add a config file into the .ssh folder and specify your key.
Here is an example:
Host MY.HOST.IP
  User ubuntu
  IdentityFile ~/.ssh/your-key.pem

@fabiendv
I have this setup for my ~/.ssh/config file but kamal setup command doesn't seem to respect the ssh config setting.
When I run kamal setup
$ kamal setup
  INFO [9b0f2031] Running /usr/bin/env mkdir -p .kamal on x.x.x.x
  INFO [98f565a2] Running /usr/bin/env mkdir -p .kamal on y.y.y.y
  Finished all in 0.8 seconds
  ERROR (Net::SSH::AuthenticationFailed): Exception while executing on host x.x.x.x: Authentication failed for user<EMAIL_ADDRESS>
~/.ssh/config file
Host x.x.x.x
  IdentityFile ./kamal-rails-pg-tutorial-app-1_key.pem
  User azureuser
Host y.y.y.y
  IdentityFile ./kamal-rails-pg-tutorial-app-2_key.pem
  User azureuser

Solved the problem.
I needed to update the permission of the pem keys.
chmod 400 kamal-rails-pg-tutorial-app-1_key.pem
And add the key to ssh-add
ssh-add kamal-rails-pg-tutorial-app-1_key.pem


develop#1

this branch/PR will be working on paths, food and detection scripts in general

develop stage achieved. proceed with graphics stage


Parallelism and concurrency overview

Hi
I'm further reading the source code, wanted to clarify my understanding of concurrency/parallelism implementation is correct.
Inference options:

FakeDualNet - returns fixed policy/value
LiteDualNet - TFLite integer inference, why? faster CPU inference? play against Minigo on mobile?
RandomDualNet - returns random policy/value
TFDualNet - standard TensorFlow CPU/GPU inference
TPUDualNet - TPU inference
WaitingModel - for testing
ModelBatcher - request asynchronous mini-batch inference from BufferedModel
BufferedModel - runs in it's own thread, does inference possibly combining mini-batches from multiple ModelBatcher

Concurrency/parallelism sources:

distributed execution - Kubernetes orchestrates multi-machine cluster execution
parallel self-play on single machine - Selfplayer manages multiple threads
within single thread:

concurrent execution of multiple games: select-leafs (across games), inference (batch), update-trees (across games)
virtual-losses within single tree, as managed by SelfplayGame
the number of game-states in single mini-batch is: virtual_losses * concurrent_games_per_thread
mini-batches can be evaluated synchronously or asynchronously (via ModelBatcher/BufferedModel)

Reasons for above architecture is as follows:

distributed - obviously for scalability
parallel self-play:

in principle, allow parallel execution on CPU/GPU?
while some CPU threads are blocked waiting for GPU, other can generate next mini-batch?

concurrent games in single thread:

so number of threads doesn't get out of hand when generating large mini-batches for GPU?

virtual-losses

required for ranked tournament games (only one game played, so all other methods are not applicable)?

Is above correct? Especially are there any other reasons for this setup, that I'm missing?
Also, I don't seem to see any way to execute single tree in parallel for tournament game (e.g. against human champion)?
Also 2, I will have possibly more questions, what would be a preferred communication channel for not-an-issues? Should I keep creating GitHub issues?
Thanks again for your time

re: the second part, happy to have these documented on github and available for posterity :)  Feel free to keep opening issues!

Your understanding is spot on.
ModelBatcher and BufferedModel are deprecated and will be deleted once I find the time to rewrite cc/eval.cc.
With regards to threading, inference is always performed synchronously from the point of view of the selfplay thread. Because we run Minigo selfplay at scale by playing multiple games concurrently, we don't need to asynchronous inference to achieve good GPU utilization. Even on very small models, the engine runs at >95% utilization on a v100, and close to 100% for a full sized model.
It has never been the goal of the project to write the fastest tournament engine (I doubt we'll ever enter one), so as you guessed this means the engine is somewhat slower when playing a single game than something like Leela.

What is the ratio of number of concurrent_selfplay executables to number of GPUs in the system? Is it 1:1 during normal execution?
Just to confirm my understanding (sorry for silly questions): assuming ratio is 1:1, and considering synchronous CPU/GPU execution, after GPU compute finishes there is a small gap in GPU utilisation while CPU does it thing to advance games and prepare next batch. But because in Go neural network is fairly large (GPU compute takes long) and game logic is comparatively quick, the GPU utilisation gap is small and thus non issue?
Thanks!

Gentlemen, could you confirm my further analysis of concurrent_selfplay are correct?

Selfplayer: always single instance, which:

in constructor creates ShardedExecutor with a thread pool, which will be used later
in Selfplayer::Run() (run from thread of the executable):

creates inference cache
creates abort file watcher
creates bunch of Models in InitializeModels, but does not spin new threads (even for BatchingModel/ModelBatcher/BufferdModel)
creates multiple SelfplayThread
creates single OutputThread

Then, in each SelfplayThread::Run:

execute multiple tree searches in parallel via ShardedExecutor created earlier
execute synchronously GPU inference via one of the model types (TF, TPU, etc.)

As for parameters in ml_perf/flags/19/selfplay.flags:

selfplay_threads=3 - number of game threads, a thread is blocked on GPU inference
parallel_search=4 - number of threads in ShardedExecutor to execute tree searches in parallel
parallel_inference=2 - number of models, maximum number of GPU inferences to run simultaneously, too high would cause OOM on GPU?
concurrent_games_per_thread=32 - number of separate games per thread
virtual_losses=4 - number of tree leafs to evaluate per iteration

Ahh, I think I see it now. As long as selfplay_threads > parallel_inference (and there is beefy enough CPU) there should be no "GPU gaps", because as soon as one inference finishes, another selfplay thread can "hop in" immediately as long as it has a batch ready. In the case above there is one extra selfplay_thread (3>2) which will run on CPU even if both GPU models are occupied.
Presumably parallel_search=4 is driven by number of CPU cores on the system, where 4-per-gpu seems about usual.
Does above seem right?

Your analysis of the threading parameters is correct.
Their values were all chosen to get >95% utilization on the VM that I'm using to test the MLperf benchmark: it has 48 physical cores (96 hyperthreads) running at 2GHz and 8 v100 GPUs. Optimal values will be different depending on the relative performance of your CPUs and GPUs.
parallel_inference=2 is used to double-buffer the inference requests so that one thread can prepare the GPU commands while the other is actually executing them. On some setups, parallel_inference=3 is also a viable choice because inference happens in three stages: transfer the feature tensor to GPU, evaluate the model, transfer the output tensor back to CPU.
parallel_search=4 is set so that the tree search doesn't take too long relative to inference: the MLperf model is about 50x smaller than the full size Minigo model. In a full run, we don't need to be so careful tuning these parameters.
If you're interested in how these parameters affect performance, I recommend reading the
profiling section of the MLperf docs, which describe how to get CPU traces of the selfplay code.

Hi
By "double-buffer the inference", do you mean simply running multiple independent models on same GPU (with obvious memory penalty)? Or is there something more going on? Assuming it's just independent models, is there any explicit mechanism to make sure models execute non-overlapping stages of GPU pipeline (transfer, evaluate, transfer back)? I kind of expect simply running 2-3 models per GPU would sort itself out on it's own in this use case, but just want to confirm.
Also, sorry for repeating, but could you explicitly confirm it's 1x concurrent_selfplay per GPU? While it seems pretty obvious by now (especially that I found hard coded gpu:0 somewhere), I'm still new to the code base and I don't want this important detail to be lost in translation.
I will definitely look into profiling after I manage to setup sacrificial CUDA dev. box for compilation purposes.
Thanks again for all the help, this was super useful! I think this wraps up my questions for now!

Yes, it's 1x concurrent_selfplay per GPU.
As for the double-buffering, well now we're getting to the interesting part. We currently create a new tensorflow::Session for every instance of the model. It's not sufficient to simply run multiple threads each performing instance: what tends to happen is that the TensorFlow framework ends up executing the Session::Run calls in lock-step, so if you're running N threads, every Session::Run call starts at the same time and they all take Nx longer to complete than a single call.
This is where the parallel_search flag comes into play. The fact that all threads share a global thread pool forces their calls to Session::Run to be staggered, which causes the TensorFlow framework to pipeline their execution correctly.
Here's old trace I found that illustrates this (it's from an experiment running on TPU with slightly different flags --selfplay_threads=4 --parallel_search=4 --parallel_inference=3 but you should get the idea):

Note that the SelectLeaf calls are scheduled more efficiently in the current master branch so if you generate a trace yourself it will look a bit different.
Now, it's possible that it would be more efficient to have all model instances loaded from the same file share the same TensorFlow session but that's just one more entry on my list of things I haven't had time to try out :)

Aha!
I had a sneaky suspicion that having ShardedExecutor shared between selfplay threads is not by accident, now we know why!
Off topic: what do you think about slightly alternative approach, having multiple game threads push eval requests to an async queue and then having neural-net thread(s) picking them up to form batches and execute on GPU? Basically implementing multiple-producer-multiple-consumer pattern. It seems to me MiniGo approach of having multiple concurrent games per thread is a strong benefit to keep total number of threads low. What's your opinion on other possible pros/cons of both approaches?

The threading model you describe is actually how Minigo selfplay used to be set up: we ran one selfplay thread for each game, and their inference requests were batched up and executed on separate inference threads. This was absolutely fine for the full sized Minigo run, we'd have maybe 8 games playing in parallel on a VM with 48 physical cores.
However, the model used for MLPerf is much smaller and we had to run significantly more selfplay threads than there were CPU cores to generate enough work for the GPU. This resulted in a large context switching overhead and reduced maximum GPU utilization.
The large number of threads and context switching overhead also made profiling the CPU code difficult. Once we switched to the current threading model, the simpler CPU traces showed there were some surprising hotspots in the code (e.g. calling argmax to select which node to visit during search). Here are some functions we found directly as a result of the simpler architecture that were optimized for up to 5x performance improvements:
https://github.com/tensorflow/minigo/blob/e44f4123f6611b4e6c6f01e66252625072a68198/cc/algorithm.cc#L28
https://github.com/tensorflow/minigo/blob/df51963aab4c54e73896045480aff79b8f9ded11/cc/mcts_tree.cc#L258
https://github.com/tensorflow/minigo/blob/64d5410724323dd371c46b0305a3aea7ef61fea6/cc/model/features.h#L114
We also found that it was measurably faster to have the tree search thread call Session::Run directly, rather than have inference and tree search run on separate threads. This was most likely because the cache of the CPU core running the inference thread wouldn't have the tree search data in it. I only ever profiled this on TPU so I don't know if this finding also applies to GPU TensorFlow.

We also found that it was measurably faster to have the tree search thread call Session::Run directly, rather than have inference and tree search run on separate threads.

But if I understand correctly, in your current model tree search thread offloads leaf selection to thread pool in ShardedExecutor anyway, so in a sense they do run on separate threads. Yeah, it's interesting what actually is going on and how it would work on GPUs.

Yep, optimizing a multithreaded system is hard :)
The original implementation of the SharedExecutor was careful to always schedule the same games to the same thread for exactly this reason. However that lead to an imbalance of work across the threads because there's a large variation in the number of nodes tree search visits during a game. You can see this in the trace above where different SelectLeaf blocks take different amounts of time. Also note that the SelectLeaf block that runs on the selfplay thread is normally the fastest because it gets better cache utilization.
It turned out to be a net win for the SelectLeaf threads to share an atomic counter into the game array and pop the next available game to run tree search on. Since tree search for a game runs on an arbitrary thread each time, the individual SelectLeaf calls are slower, but the work is better distributed across the threads and so ends up taking less time.

Quick note: if you found the hard-coded gpu:0 where i think you did, it's because we isolate the selfplay jobs on GPUs using the CUDA_VISIBLE_DEVICES environment variable.

@tommadams I think what you say makes sense, but I need to think more about the implications.
@amj Yeah, that's exactly what I thought. The gpu:0 was a major clue :)

I'd caution against reading too much into what I wrote, these are specific optimizations I made for our architecture and hardware setup. Bottlenecks will vary based on model size, CPU & GPU compute speed, board size, code architecture, etc.
The most important take away should be: make sure it's easy to profile your code :)


Should serialize terms:Title into terms?

b17164835
{
  "nameLocal": "African-American Life.",
  "nameFast": "African American life",
  "fast": 1775605,
  "type": "terms:Title"
}

Modify shadowcat ingest to ignore FAST titles


chore: Remove dependency on inflight package

Description of changes

Bumps the glob dependency to 9.3.5 that is determined to be stable without inflight dependency.
This comes with an API change that sync is now replaced with globSync method.
CDK / CloudFormation Parameters Changed

Issue #, if available

Description of how you validated changes
Currently failing build in amplify-graphql-schema-generator package and that package is not using glob. There is some other breaking change within the dependency chain that's unclear.
Checklist

[ ] PR description included
[ ] yarn test passes
[ ] Tests are changed or added
[ ] Relevant documentation is changed or added (and PR referenced)
[ ] New AWS SDK calls or CloudFormation actions have been added to relevant test and service IAM policies
[ ] Any CDK or CloudFormation parameter changes are called out explicitly

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

After change, inflight is no longer in the dependency graph for this repo:
% grep -i inflight yarn.lock
    promise-inflight "^1.0.1"
    promise-inflight "^1.0.1"
    promise-inflight "^1.0.1"
<EMAIL_ADDRESS>  resolved "https://registry.npmjs.org/promise-inflight/-/promise-inflight-1.0.1.tgz#98472870bf228132fcbdd868129bad12c3c029e3"


[for discussion only][vm] fully backword compatible bytecode format with function visibility info

Motivation
Here is a fully backward compatible bytecode format for the function visibility scheme I have in mind. I'll explain later what do I mean by fully-backward compatible.
But before that, this PR is based on the following assumptions:

We prefer callee-side friend list declaration (so we don’t need to look at multiple files to know who can call a particular function).
Module-level friend list seems to be a reasonable granularity to start with.

In other words, the friend list will be embedded in the function definition bytecode. This is similar to what is currently done (i.e., the is_public bit is stored in the function definition bytecode).
Now, here is the explanation on why this change is backward compatible:
The current layout for the bytecode of a function definition:
| ULEB128: function_index | U8: flags | Vec<ULEB128>: acquires_list | Optional<Vec<U8>>: code_unit |

Note that this flags is a bit mysterious here, as it is essentially capturing two information in the current scheme

Whether this function is_public
Whether this function is_native

Decomposing this flag, we get essentially the following:

For public functions:

| ULEB128: function_index | U8: (0x1 | is_native) | Vec<ULEB128>: acquires_list | Optional<Vec<U8>>: code_unit |

For private functions:

| ULEB128: function_index | U8: (0x0 | is_native) | Vec<ULEB128>: acquires_list | Optional<Vec<U8>>: code_unit |

Now here is the new layout in this PR:
| ULEB128: function_index | U8: flags | Optional<Vec<ULEB128>>: friends | Vec<ULEB128>: acquires_list | Optional<Vec<U8>>: code_unit |

Note that a new section, | Optional<Vec<ULEB128>>: friends | is added, which maps to a vector of ModuleHandleIndex. But this section is Optional, meaning that it only exists when flags & IS_PROTECTED_BIT != 0, and if we further unroll the layout casing on the visibility flag, it means:

For public functions:

| ULEB128: function_index | U8: (0x1 | is_native) | Vec<ULEB128>: acquires_list | Optional<Vec<U8>>: code_unit |

For private functions:

| ULEB128: function_index | U8: (0x0 | is_native) | Vec<ULEB128>: acquires_list | Optional<Vec<U8>>: code_unit |

For protected functions:

| ULEB128: function_index | U8: (0x4 | is_native) | Vec<ULEB128>: friends | Vec<ULEB128>: acquires_list | Optional<Vec<U8>>: code_unit |

Note that in the public and private case, the new and old layout are exactly the same! Hence, the scheme is fully backward compatible.

Why is it Optional<Vec<ULEB128>>: friends and not Vec<ULEB128> ?

Why is it Optional<Vec<ULEB128>>: friends and not Vec<ULEB128> ?

The reason is because Public and Private do not have a friend list, and hence, do not have the Vec<ULEB128>. Adding an empty Vec<ULEB128> will break compatibility because in the old format, that is the position for Vec<ULEB128>: acquires_list.

The reason is because Public and Private do not have a friend list, and hence, do not have the Vec<ULEB128>. Adding an empty Vec<ULEB128> will break compatibility because in the old format, that is the position for Vec<ULEB128>: acquires_list.

Ah, maybe I missed this, serializing an empty Vec<ULEB128> does not yield zero bytes. It actually yield one byte, which is the count (although it is 0) of the Vec.

Ah, maybe I missed this, serializing an empty Vec<ULEB128> does not yield zero bytes. It actually yield one byte, which is the count (although it is 0) of the Vec.

I think the Option the option thing is starting to make sense, thanks!
I'll comment on the code for some other things

I think the Option the option thing is starting to make sense, thanks!
I'll comment on the code for some other things


Invalid integer numeric value

HI, with latest snapshot <IP_ADDRESS>,when I try to edit smallint/bigint values using grid I got always the error: Invalid integer numeric value
in screenshot I put 1 instead of 0 for TIP column,for decimal works
thanks

Same there. It also happens when editing String fields when the value contains only digits.
The produced SQL is something like set STRING_FIELD = '518.000000'

Hi, is it possible to have a new snapshot compiled with at least the current fixed bugs? It's quite frustrating to use two versions of flamerobin, the last to have support for data type in FB3 and one older to being able to edit numeric and string fields.

Indeed very ugly bug,makes new version of Flame Robin unsless for FB>2.5

Please update to the latest snapshot (<IP_ADDRESS>) and try again, if working please close this issue.
Thanks

for me seems it is working


fix: determine if stderr is empty

Signed-off-by: zhuzhenfan<EMAIL_ADDRESS>
What type of PR is this?
/kind bug

What this PR does / why we need it:
issue #294  Open again
Which issue(s) this PR fixes:

Fixes #294
Special notes for reviewers:
@x893675 @lixd

Does this PR introduced a user-facing change?

None

Additional documentation, usage docs, etc.:

@x893675 @lixd

/cc @lixd

/approve


Connection pool in typhoeus

Is it possible to have connection pooling in typhoeus?? configuring like below:
Typhoeus.configure do |config|
config.size = 5
config.pool_timeout = 5
config.keep_alive_timeout = 30
end

Connection pooling is handled by libcurl. It does pool connections. I am not aware of any configuration option to tune it like you want. It is kinda automatic. I believe you can change the keepalive via headers.


Confirm Due Date for M-21-31 EL2 Requirements

In order to meet logging/observability requirements, we want to confirm the due date for M-21-31 EL2
Acceptance Criteria

[ ] Confirm due date of Feb 27, 2023

Security considerations
Need to find out due date so we can remain compliant
Implementation sketch

[ ] Discuss with Peter if the due date of Feb 27, 2023 is still relevant

Meeting scheduled with Peter on 2/7 to discuss.

Resolved by https://github.com/cloud-gov/private/issues/624.


Set SameSite=Lax when un-setting cookies

Summary of Changes
Set the cookie same site to Lax when un-setting cookies.
This avoids warning / invalid cookies https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite#Fixing_common_warnings
Related Issues
Closes #1721
Pull Request Checklist
This is just a reminder about the most common mistakes.  Please make sure that you tick all appropriate boxes.  But please read our contribution guide at least once; it will save you a few review cycles!
If an item doesn't apply to your pull request, check it anyway to make it apparent that there's nothing to do.

[x] Applied changes to both WSGI and ASGI code paths and interfaces (where applicable).
[x] Added tests for changed code.
[x] Prefixed code comments with GitHub nick and an appropriate prefix.
[x] Coding style is consistent with the rest of the framework.
[x] Updated documentation for changed code.

[x] Added docstrings for any new classes, functions, or modules.
[x] Updated docstrings for any modifications to existing code.
[x] Updated both WSGI and ASGI docs (where applicable).
[x] Added references to new classes, functions, or modules to the relevant RST file under docs/.
[x] Updated all relevant supporting documentation files under docs/.
[x] A copyright notice is included at the top of any new modules (using your own name or the name of your organization).
[x] Changed/added classes/methods/functions have appropriate versionadded, versionchanged, or deprecated directives.

[x] Changes (and possible deprecations) have towncrier news fragments under docs/_newsfragments/, with the file name format {issue_number}.{fragment_type}.rst. (Run towncrier --draft to ensure it renders correctly.)

If you have any questions to any of the points above, just submit and ask! This checklist is here to help you, not to deter you from contributing!
PR template inspired by the attrs project.

Updated

There may be some errors in the inspect module
Didn't find RouteMethodInfo.suffix in falcon.inspect
Didn't find MiddlewareInfo.independent_text in falcon.inspect

Smth is FUBAR with Sphinx

Seems to be an issue with the -j flag, so I removed it for now: https://github.com/falconry/falcon/pull/1729


Counting nodes

node_count=5 implies 4 nodes, the first of which is listed twice, following the wide-spread practice (e.g. http://schema.org/polygon ). I find the terminology a bit mis-leading. schema.org use refer to "points" rather than nodes, which may be better. It may be useful to refer to https://www.w3.org/wiki/WebSchemas/GeoShapeExamples .. your proposal contains much detail which is missing from the w3c approach. It might be useful to summarise what information needs to be added to go from a w3c polygon to the proposed CF polygon.

Good points. The usage of the word "node" has been inherited from the UGRID conventions where "node" seemed to be more appropriate than "point". You indicate that the netCDF-CF-simple-geometry proposal contains much more detail which is missing from the w3c approach, but I don't see that immediately. The biggest difference seems the counting of nodes which is implicit in the w3c approach which only stores series of coordinates only for individual rings. while the proposed approach stores one long list of coordinates for all rings. This is similar to some conventions in Shape file storage (and maybe other geometry files).

Ok, two things here.
Point vs. Node: The canonical reference for what we are doing here is the OGC/ISO Simple Features model. -- glancing through that, the word "Point" is used rather than "Node". Curious what @twhiteaker thinks about changing the language to "point" rather than "node". I personally like "node" better to make the distinction between actual point geometries and "linestrings" or linear rings of what we are now calling "nodes". But I don't feel very strongly about it.
The GeoShapeExamples is an archived W3C document. I don't see it as a particularly important external reference to build on/relate to. A more recent document that is relevant is the Spatial Data on the Web best practice.  To be honest, I also don't think this is all that relevant to us  since this is intended for a binary archival format -- not publication of data on the web. As long as this proposal is in line with OGC/ISO simple features, I feel that we are in a good place for interoperability with the preponderance of geometry implementations in the wild.

Re: points vs nodes:
I’m fairly ambivalent on this.  I do have my preferences, but at this point I’m happy to let the CF community mull on this and accept a solution. To that end, may I suggest making this same comment in the cf-conventions thread discussing the pull request for this proposalhttps://github.com/cf-convention/cf-conventions/pull/115?  Since @dblodgett-usgs submitted the pull request, that has been the thread for active discussion on this work and would be a great place to expose your concern before the pull request is accepted. I’ve been waiting for that discussion to come to a conclusion before updating the documentation and reference implementation in this repository, but I can see that may have caused some confusion so I apologize for that.  Since this repository is distinct from that pull request, I may work on updating it to reflect the latest discussion on the pull request when I’m back in the office next week.
Re: this proposal having detail missing from the w3c approach:
Conceptually we’ve just tried to support OGC simple features geometry primitives (point, linestring, polygon) and their multipart equivalents, so I’d say our target is more OGC than W3C, though really we’re supporting the same geometry types as in GeoJSON (except for geometry collections).  Can @martinjuckes clarify the extra detail you’re referring to?  I suspect it may be the contiguous ragged arrays, or separate variables dealing with holes, which are a product of trying to describe geometries within the syntax of netCDF and CF. Though the syntax may be different, the information content is the same as those basic GeoJSON types and WKT types, so I don’t think any extra information is needed to go from a “W3C” polygon to its representation in netCDF unless you want to attach a coordinate system to the coordinates.
Tim Whiteaker
Research Scientist
The University of Texas at Austin
From: David Blodgett<EMAIL_ADDRESS>Sent: Monday, July 10, 2017 11:49 AM
To: twhiteaker/netCDF-CF-simple-geometry
Cc: Whiteaker, Timothy L; Mention
Subject: Re: [twhiteaker/netCDF-CF-simple-geometry] Counting nodes (#75)
Ok, two things here.
Point vs. Node: The canonical reference for what we are doing here is the OGC/ISO Simple Features modelhttp://www.opengeospatial.org/standards/sfa. -- glancing through that, the word "Point" is used rather than "Node". Curious what @twhiteakerhttps://github.com/twhiteaker thinks about changing the language to "point" rather than "node". I personally like "node" better to make the distinction between actual point geometries and "linestrings" or linear rings of what we are now calling "nodes". But I don't feel very strongly about it.
The GeoShapeExamples is an archived W3C document. I don't see it as a particularly important external reference to build on/relate to. A more recent document that is relevant is the Spatial Data on the Web best practice. https://www.w3.org/TR/sdw-bp/ To be honest, I also don't think this is all that relevant to us since this is intended for a binary archival format -- not publication of data on the web. As long as this proposal is in line with OGC/ISO simple features, I feel that we are in a good place for interoperability with the preponderance of geometry implementations in the wild.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/twhiteaker/netCDF-CF-simple-geometry/issues/75#issuecomment-314165902, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAqsaBVnQfec8XYTA9PuHQjFl6lqz3_Zks5sMlYHgaJpZM4OSn44.


Add a GUI to Scripture Phaser

Description
Currently, the only way to interface with scripture_phaser is through the CLI tool (found in cli.py). Most users (and certainly almost all users on Windows) will want to use scripture_phaser through a GUI.
Technical Requirements

The GUI library/implementation needs to be platform independent, working on Windows, Linux and MacOS.
Using the GUI should be default.
All GUI functionality should interface directly with the API and bypass the CLI.
The GUI should not call an external text editor to allow the user to input the passage text.

Tasks

[ ] Add a useGUI or similarly named option to the config file (default True)
[ ] Choose a GUI library
[ ] Implement List Function (CLI "L")
[ ] Implement Toggle Mode Function (CLI "M")
[ ] Implement Set Reference Function (CLI "R")
[ ] Implement View Passage Function (CLI "V")
[ ] Implement Set Translation Function (CLI "T")
[ ] Implement View Translations Function (CLI "I")
[ ] Implement Practice Passage Function (CLI "P")
[ ] Implement Stats Viewer Function (CLI "S")
[ ] Implement Reset Stats Function (CLI "Z")
[ ] Include Version and License Info Somewhere

I've removed the ability to manually get/write the config from the CLI version of scripture_phaser, so I've also removed the necessity to include these features in the GUI. They were unnecessary features since the config is loaded and saved at the appropriate times by the backend anyway.


Make "sign in" text consistent

Variations of "sign in" are used across the portal and could use alignment.

Hamburger menu (mobile): "Sign In"
Homepage button: "Login"
Sign in page: "Log In"

Seems like services tend to use "sign in" these days. GitHub uses "sign up", "sign in", "signed in". But as long as its consistent...

Text is all synced up now!
We'll be going with "Log in" because it's the term used by Orchard, so I don't want the titles to say "Sign in" and the URLs say "/login".
I don't mind "Sign in", but both terms are pretty widely used. We considered changing the URL to "/signin/ but compatibility is unknown, so we're going to play it safe.

Text is all synced up now!
We'll be going with "Log in" because it's the term used by Orchard, so I don't want the titles to say "Sign in" and the URLs say "/login".
I don't mind "Sign in", but both terms are pretty widely used. We considered changing the URL to "/signin/ but compatibility is unknown, so we're going to play it safe.

Done!


"/usr/bin/ld: cannot find -ltinfo" on first run instead of helpful message

Setting up my project on a previously unused computer running Ubuntu 15.10 I get this:
Intero is not installed in the Stack environment.

Installing intero-0.1.18 automatically ...

intero-0.1.18: configure
intero-0.1.18: build

--  While building package intero-0.1.18 using:
      /home/unhammer/.stack/setup-exe-cache/x86_64-linux/setup-Simple-Cabal-<IP_ADDRESS>-ghc-7.8.4 --builddir=.stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS> build --ghc-options " -ddump-h\
i -ddump-to-file"
    Process exited with code: ExitFailure 1
    Logs have been written to: /home/unhammer/src/foo/.stack-work/logs/intero-0.1.18.log

    Configuring intero-0.1.18...
    Building intero-0.1.18...
    Preprocessing executable 'intero' for intero-0.1.18...
    [1 of 8] Compiling GhciTypes        ( src/GhciTypes.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero-tmp/GhciTypes.o )
    [2 of 8] Compiling GhciMonad        ( src/GhciMonad.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero-tmp/GhciMonad.o )
    [3 of 8] Compiling GhciTags         ( src/GhciTags.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero-tmp/GhciTags.o )
    [4 of 8] Compiling GhciInfo         ( src/GhciInfo.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero-tmp/GhciInfo.o )
    [5 of 8] Compiling GhciFind         ( src/GhciFind.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero-tmp/GhciFind.o )
    [6 of 8] Compiling Paths_intero     ( .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/autogen/Paths_intero.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/inter\
o-tmp/Paths_intero.o )
    [7 of 8] Compiling InteractiveUI    ( src/InteractiveUI.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero-tmp/InteractiveUI.o )
    [8 of 8] Compiling Main             ( src/Main.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero-tmp/Main.o )
    Linking .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/intero/intero ...
    /usr/bin/ld: cannot find -ltinfo
    collect2: error: ld returned 1 exit status
Could not install Intero!

Apparantly I had to sudo apt-get install libtinfo-dev. Could this solution maybe go in the error message?

It could be in the error message if it detects cannot find -ltinfo, sure.

This could also be useful in stack BTW.


DracoEncode.cs not visible

The static class public static class DracoEncode or its methods like EncodeMesh are not accessible.
This is on Unity 202.3.22f1 with Draco 4.0.1

Hi @MoritzSkowronski ,
That seems wrong indeed, I'll have a look hopefully next week.
What do you want to do with it? Fyi: If it's runtime compression, I've not tested if it works yet.

My goal, although this is not possible yet, I presume, is to compress and decompress a LIDAR scanned point cloud to sent over WIFI.

@MoritzSkowronski Makes sense. Runtime compression (once it's implemented) will probably take some time (depending on data size, ofc).

I'm currently facing the same issue. Are there any updates ?

@AR-Vsx No updates so far. It's not a priority on my side, but I'm happy to review contributions if someone wants to tackle it sooner.

I was able to use the namespace by creating a Assembly Definiton File in my Scripts folder and referencing both Draco and DracoEncoder.

Revising this after some time:
Draco.Encoder.DracoEncoder is a public class, so that's not the problem. The assembly definition it is part of (DracoEncoder) is not auto-referenced though. That means if you want to use it, create your own assembly definition and reference it explicitly (more on asmdefs).
That should fix the original problem, thus I'm closing this issue.
Now regarding the greater picture feature - Runtime Export:
There's now a dedicated issue in #56 .
Have you been able to get it running? Could you report your experience? What platforms are you running it on?
Thanks


module smart-table error

Uncaught Error: [$injector:modulerr] Failed to instantiate module BlurAdmin due to:
Error: [$injector:modulerr] Failed to instantiate module smart-table due to:
Error: [$injector:nomod] Module 'smart-table' is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument.

Hi @bravility, this issue is not connected to this project. Please use blur-admin repo instead. Thanks.


Error when trying this blog

I'm facing this error, can you please help?
git submodule add <url> dotfiles/bash_itValue Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist.python3.6 at 'runtime' failed to satisfy constraint: Member must satisfy enum value set: [java8, java11, nodejs10.x, nodejs12.x, python2.7, python3.6, python3.7, python3.8, dotnetcore2.1, go1.x, ruby2.5] or be a valid ARN (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 05ec750a-4458-4536-bb9c-c528d6829312; Proxy: null)

YAML code expects ddb-to-firehose.py in a zipped format.


Change Package Structure of the Output Library according to the Eclipse Project Pattern

old: io.adminshell.aas
new: org.eclipse.aas4j...

Please change path to:
groupID: org.eclipse.digitaltwin.aas4j
as agreed within Eclipse DT TPL PMC.

PR for this issue: https://github.com/eclipse-aas4j/aas4j/pull/42

PR for this issue: eclipse-aas4j/aas4j#42

Looks fine to me! We can close the issue once we merge your PR?

Closing as PR has been approved and merged.


DuckDB can't read from TimescaleDB hypertables

What happens?
I am using the postgres extension TimescaleDB to handle an events table. Under the hood this is a timescaledb hypertable. When attaching with DuckDB's postgres scanner the scanner fails to read from the hypertable. I believe this has something to do with the compressed chunks on the hypertable. I am aware that this isn't vanilla postgres however, tools like connectorx and polars from the python/rust world is able to successfully read from the same table without issue.
D CALL postgres_attach('dbname=billing_platform user=postgres password=password host=<IP_ADDRESS>', source_schema='usage');

Error: IO Error: Unable to query Postgres: ERROR:  transparent decompression only supports tableoid system column
ERROR:  transparent decompression only supports tableoid system column

To Reproduce
To start you can launch a TimescaleDB instance with the following compose file.
# docker-compose.yaml
version: '3.7'
services:
  db:
    image: timescale/timescaledb-ha:pg15-ts2.12-all
    restart: always
    platform: linux/amd64
    environment:
      TZ: UTC
      POSTGRES_PASSWORD: password
      POSTGRES_DB: sandbox
    ports:
      - 5432:5432

Create a hypertable in timescale. You can connect with any postgres client such as psql
PGPASSWORD=password psql -h localhost -p 5432 -U postgres -d sandbox

-- TimescaleDB hypertable creation
CREATE TABLE IF NOT EXISTS events (
                timestamp TIMESTAMPTZ NOT NULL,
                eventname TEXT NOT NULL,
                account UUID NOT NULL,
                counter DOUBLE PRECISION
            );

            CREATE UNIQUE INDEX events_pk ON events (account, eventname, timestamp ASC);

            CREATE INDEX events_eventnametimestamp ON events (eventname, timestamp ASC);

            SELECT CREATE_HYPERTABLE('events', 'timestamp', chunk_time_interval => INTERVAL '15 minutes');

            ALTER TABLE events SET (
                timescaledb.compress,
                timescaledb.compress_orderby = 'timestamp ASC',
                timescaledb.compress_segmentby = 'account,eventname'
            );

            SELECT ADD_COMPRESSION_POLICY('events', INTERVAL '15 minuntes');

Insert some data
INSERT INTO events (timestamp, eventname, account, counter)
VALUES
  (NOW(), 'some-event', '69735c6b-f764-4620-b476-c668f7c0ae48', 1.2);

Now in DuckDB run the following.
INSTALL postgres_scanner;
LOAD postgres_scanner;
CALL postgres_attach('dbname=billing_platform user=postgres password=password host=<IP_ADDRESS>', source_schema='public');
PRAGMA show_tables;
SELECT COUNT(*) FROM events;

If you fire all these at it quickly the problem may not occur since the table partitions won't have been compressed yet. They should compress after about 15 minutes given the policy applied in the database setup.
OS:
MacOS/docker
PostgreSQL Version:
pg15.4/ts2.12
DuckDB Version:
v0.9.1 401c8061c6
DuckDB Client:
brew
Full Name:
Taylor Beever
Affiliation:
QuickNode
Have you tried this on the latest main branch?

[X] I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

[X] I agree

Thanks for the report! Could you try this with v0.9.2? We reworked the extension quite significantly since 0.9.1.

I've managed to reproduce the issue - note that it is limited to COUNT(*) in the latest version. SELECT * FROM events works as expected. I've pushed a fix for this in #132.

@Mytherin I guess I should have tested a few other queries... Thanks for the quick turnaround on this!


Update sentence about Office 365 suite

[Hello, would you mind updating this sentence under 'Office 365' headline: "Administrators can exclude specific apps from policy if they wish, including the Office 365 suite and excluding the specific apps in policy". The wording doesn't make sense here. Perhaps as I'm not native English speaking, but I don't get the phrase (specifically the ending), thanks.]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e5b78894-32c2-6712-6699-f3c87d40b1d7
Version Independent ID: dd791c15-d43b-b203-a93f-a8ec33c519b0
Content: Cloud apps, actions, and authentication context in Conditional Access policy - Azure Active Directory
Content Source: articles/active-directory/conditional-access/concept-conditional-access-cloud-apps.md
Service: active-directory
Sub-service: conditional-access
GitHub Login: @MicrosoftGuyJFlo
Microsoft Alias: joflore

@ChristianJBergstrom Thanks for the feedback! We are currently investigating and will update you shortly.

@ChristianJBergstrom
Thank you for pointing this out! I've created a PR to update the documentation and it's currently waiting for the author to review and sign off.

I will now close out this issue. Please allow some time for the author to review and the changes to reflect.
Thank you for your time and patience!


added docstring to logging.py and added pydocstyle to github workflow

Hi, @refeed I have added a docstring to logging.py and also added pydocstyle to github workflow. Do checkout and let me know am i going right?

Hi @refeed made a commit, please check.

Merged this via https://github.com/StackGuardian/tirith/pull/188 . Thanks for the contribution!


ImportError: pafy: youtube-dl not found

when I run spotdl this is happening:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/__init__.py", line 4, in <module>
    from .lazy_extractors import *
ModuleNotFoundError: No module named 'youtube_dl.extractor.lazy_extractors'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pafy/pafy.py", line 48, in <module>
    import youtube_dl
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/__init__.py", line 43, in <module>
    from .extractor import gen_extractors, list_extractors
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/__init__.py", line 9, in <module>
    from .extractors import *
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/extractors.py", line 55, in <module>
    from .ard import (
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/ard.py", line 8, in <module>
    from .generic import GenericIE
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/generic.py", line 79, in <module>
    from .instagram import InstagramIE
ModuleNotFoundError: No module named 'youtube_dl.extractor.instagram'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/spotdl", line 5, in <module>
    from spotdl.spotdl import main
  File "/usr/local/lib/python3.7/site-packages/spotdl/spotdl.py", line 14, in <module>
    from spotdl import youtube_tools
  File "/usr/local/lib/python3.7/site-packages/spotdl/youtube_tools.py", line 3, in <module>
    import pafy
  File "/usr/local/lib/python3.7/site-packages/pafy/__init__.py", line 7, in <module>
    from .pafy import new
  File "/usr/local/lib/python3.7/site-packages/pafy/pafy.py", line 52, in <module>
    "pafy: youtube-dl not found; you can use the internal backend by "
ImportError: pafy: youtube-dl not found; you can use the internal backend by setting the environmental variable PAFY_BACKEND to "internal". It is not enabled by default because it is not as well maintained as the youtube-dl backend.

but youtube-dl is installed I can use the command.
Any ideas?

Hi, can you try again after updating spotdl with:
$ pip3 install spotdl -U

spotdl no longer uses pafy as of now.

worked thanks a lot,
saved my day


Changing cursor to grab/grabbing with dragHandle

If anyone could provide an example with how I can use a grab and grabbing cursor with a drag handle, that would be great. Right now, no matter what CSS I try to use, the cursor seems to always go to default when dragging is active.
Edit: Closing this, I ended up using a workaround where I just applied a class to the whole document body

for other people who bump in here: https://github.com/clauderic/react-sortable-hoc/issues/253
this issue (and all the links there) can provide you with tips on ways to do this.

FYI heres's how i did it:
onSortStart={() => (document.body.style.cursor = 'grabbing')}
onSortEnd={({ newIndex, oldIndex }: any) => {
    document.body.style.cursor = 'default';
...

then on your drag handler make sure to put cursor: grab on it

Thank you for this @zackify —I would never have figured this out :( This should be documented somewhere :/

Completely agree. I searched forever then figured this out 😂 I don’t like having to call document methods inside react. But oh well!

Thanks @neckro it works the way I need it. Could you tell me please what the void operator does there?

Sorry for not checking my notifications.  That's Typescript -- you can remove the : void and the : React.ReactElement type hints for plain JS.

FYI heres's how i did it:
onSortStart={() => (document.body.style.cursor = 'grabbing')}
onSortEnd={({ newIndex, oldIndex }: any) => {
    document.body.style.cursor = 'default';
...

then on your drag handler make sure to put cursor: grab on it

Just to add to @zackify 's approach, to make sure that the cursor will not change when you drag over, say a button (it may change to a pointer), you can do this:
onSortStart={() => (document.body.className = 'grabbing')}
onSortEnd={({ newIndex, oldIndex }: any) => {
  document.body.className = '';
...

where grabbing is
grabbing {
  cursor: grabbing;
}
grabbing * {
  cursor: grabbing;
}

@arosisi Your solution works like a charm :+1: It should be in the doc. Team Work power, thx guys ;p

FYI heres's how i did it:
onSortStart={() => (document.body.style.cursor = 'grabbing')}
onSortEnd={({ newIndex, oldIndex }: any) => {
    document.body.style.cursor = 'default';
...

Just to add to @zackify 's approach, to make sure that the cursor will not change when you drag over, say a button (it may change to a pointer), you can do this:
onSortStart={() => (document.body.className = 'grabbing')}
onSortEnd={({ newIndex, oldIndex }: any) => {
  document.body.className = '';
...

where grabbing is
grabbing {
  cursor: grabbing;
}
grabbing * {
  cursor: grabbing;
}

I would like to add another improvement. This would erase all other classes on the body. Assuming that other features on the webpage might also want to add classes to the body, I recommend using:
onSortStart={() => (document.body.classList.add('grabbing'))}
onSortEnd={({ newIndex, oldIndex }: any) => {
  document.body.classList.remove('grabbing');
})}

Full browser support


Code Release

When is the code release planned for?

Hello, reynoldscem.
Thanks for your interest in our work, and we may release the code before Christmas.


Modified injured_accidents_with_pedestrians widget api

resolves #2044.

Changed to stacked bar graph api.
Ordering by injury severity.
Converted label key year from int to str.

@shaniwein looks great, can you make sure pylint test is passing as well?
See https://github.com/hasadna/anyway/blob/dev/docs/TESTS.md


Proposal: add a built-in reduce function on the type level

This is my first proposal here, so feedback welcome.
TL;DR I noticed arrays in the standard lib are assumed homogeneous e.g. Array<T>, I'd like to see better support for heterogeneous arrays (i.e. tuples), and in looking for solutions there concluded being able to use reduce in the type language might help a lot to facilitate hard typings, that is, cut down on walls of generics to get them into one line.
To prevent confusion, I started looking for a solution to typing reduce-like functions, but ended up finding a way using a reduce-like function (in the type language) so as to solve something more general.
Array.prototype.reduce definitions currently naively assume the initial value, output, and all intermediate values share one and the same type, all following the 'homogeneous array of unspecified length' Array<T> presumption:
reduce(callbackfn: (previousValue: T, currentValue: T, currentIndex: number, array: T[]) => T, initialValue?: T): T;
reduce<U>(callbackfn: (previousValue: U, currentValue: T, currentIndex: number, array: T[]) => U, initialValue: U): U;

This assumption may hold in quite some cases, though not in all. The reality for those cases may be harder to catch in the current typing system, as each successive result is really based on what came before it. I believe to properly capture this the typing system would need to be able to iterate just as the original function would (though dealing with types rather than values).
Code illustrating the issue:
const pathFn = <T, K extends string|number>(obj: T, path: K[]) => path.reduce(<U>(o: U, k: K extends keyof U) => o[k], obj);
// ^ mirrors Ramda's `R.path`, or Lodash's `_.get` with array

interface Cat {
  age: number;
  children: Cat[];
}
let bigglesworth: Cat = { age: 5, children: [ { age: 1 } ] }
let pets = { bigglesworth };
let firstChildAge = pathFn(pets, ['bigglesworth', 'children', 0, 'age']); // 1

Expected behavior:
Expected inferred type: number
Actual behavior:
Reduce concludes object in, object out.
Relevance:

Isn't this some weird edge case?

I hear you asking.
Well, out of the functions contained in Ramda, I'm under the impression the following functions utilize this reduce type of logic, and whose typings I think would benefit from solving this problem:

reduce, reduceBy, scan
path, pathEq, pathOr, pathSatisfies, assocPath, dissocPath, lensPath **
invertObj **, invert
mergeAll, mergeWithKey,  mergeWith (though not merge)
pickBy (infer which to pick, types if heterogeneous)
fromPairs, toPairs, toPairsIn **
where, whereEq (relevant if calculating the result on the type-level, which requires all values at compile-time, for where functions returning actual boolean values e.g. type guards, for whereEq a value equality check)
flatten *
indexBy
sequence,  transduce, traverse
applySpec
filter (for objects this allows typing properly rather than as a Partial given known boolean outcomes, as is possible using type guards)
xprod *
zipObj *, **
reverse *, **
pipe, pipeP, pipeK **
compose, composeP, composeK **
curry? **

*: added value presuming heterogeneous content (tuple input).
**: currently the case for pipe/compose/curry, and in retrospect for the others marked this way (including ironically the path I'd taken for my example), these can be worked around using lists covering the different possible sizes (squared in case of xprod), which works, though it makes definition files harder to read/write/maintain (esp. when all functions are curried as well). This would for those cases turn this proposal into no more than syntactic sugar to turn all those predictable variants into one line.
I ignored silly ones like range / times that'd technically get more type info (-> array size known) but wouldn't plausibly benefit from it.
I might be off by a bit, but this would seem to be in the order of 37 functions (/234), including ones I've yet to use, but some others I'd deem indispensable.
Summarizing:
Now, I'd wondered if we needed some generic construct for iteration on the type level, but that might raise a whole slew of new questions (what would this look like syntax-wise? where else might it be used? wouldn't the type language become a copy of the actual language?). That's not the way I'd like to go here.
Information we have:

The type language supports function application.
The type language has certain values built-in, e.g. the standard JavaScript types.
In this case, all the above-mentioned functions could get typed better because they specifically (directly or by dependency) follow this reduce pattern (the non-trivial version, i.e. cases where the current typing system fails).

Proposal:
I believe functional programming in TypeScript could be significantly improved by exposing a reduce function on the type level.
Note:

This would be a function rather than method like Array.prototype.reduce, I suppose ideally with the array as the third function argument, e.g. function reduce(callback /* pars: (accumulator, currentValue, currentIndex) */, initialValue, values /* Array, don't even bother running if info like length is unknown */).
Unlike JS functions like Array.prototype.reduce, this would instead deal with types rather than values.
I'd be cool with any recognizable name/capitalization, from reduce (good/bad because it seems like a regular function) to _RDC (looks more like a type-level name, unlikely to clash, still kinda recognizable, but ugly).

Examples:
Using _RDC as a name for example purposes, we can retry the example from above:
reduce<T, U, V>(callbackfn: T, initialValue?: U, values: V): _RDC(T, U, V);
const typedPathFn = <T, K extends string|number>(obj: T, path: K[]) => reduce(<U>(o: U, k: K extends keyof U) => o[k], obj, path);
let typedFirstChildAge = typedPathFn(pets, ['bigglesworth', 'children', 0, 'age']);
// number \o/

But we can do more than that.
The bigger picture here would be to also fix all of the Array methods from the standard library, so let's start looking at a simple example from there, reversing a tuple.
A function version might look like this:
reverse(tpl: []): [];
reverse<A>(tpl: [A]): [A];
reverse<A,B>(tpl: [A,B]): [B,A];
reverse<A,B,C>(tpl: [A,B,C]): [C,B,A];
...

Can we reduce that?
reverse<Ts>(tpl: [...Ts]): _RDC((acc, v) => [v, ...acc], [], Ts);

Let's look at a definition of compose, here with 1 param on the outer function:
compose<V0, T1>(fn0: (x0: V0) => T1): (x0: V0) => T1;
compose<V0, T1, T2>(fn1: (x: T1) => T2, fn0: (x0: V0) => T1): (x0: V0) => T2;
compose<V0, T1, T2, T3>(fn2: (x: T2) => T3, fn1: (x: T1) => T2, fn0: (x: V0) => T1): (x: V0) => T3;
...

Can we generalize this? I think so.
compose<V0, T0, V, T>(...fns: ...(x: V) => T, f0: (x0: V0) => T0): (x: V0) => ..._RDC((acc, f) => [f(...acc)], [V0], reverse([...fns, f0]));

We mentioned these all use 1 input param. What does varying that part look like?
compose<V0, T1>(fn0: (x0: V0) => T1): (x0: V0) => T1;
compose<V0, V1, T1>(fn0: (x0: V0, x1: V1) => T1): (x0: V0, x1: V1) => T1;
compose<V0, V1, V2, T1>(fn0: (x0: V0, x1: V1, x2: V2) => T1): (x0: V0, x1: V1, x2: V2) => T1;

Can we incorporate that?
compose<Vs, T0, V, T>(...fns: ...(x: V) => T, f0: (...xs: ...Vs) => T0): (...xs: Vs) => ..._RDC((acc, f) => [f(...acc)], [...Vs], reverse([...fns, f0]));

The opposite (R.pipe) is easier, no flipping:
pipe<Vs, T0, V, T>(f0: (...xs: ...Vs) => T0, ...fns: ...(x: V) => T): (...xs: Vs) => ..._RDC((acc, f) => [f(...acc)], [...Vs], [f0, ...fns]);

How about curry, as given by sandersn in his Variadic Kinds proposal? In there, he gaven an example for the case of dividing the parameters over 2 function applications: curry<...T,...U,V>(f: (...args:[...T,...U]) => V, ...ts:...T): (...us: ...U) => V. Let's try to make that more general:
curry<V,...A>(f: (...args:[...A]) => V, ...as:...A): V
curry<V,...A,...B>(f: (...args:[...A,...B]) => V, ...as:...A): (...bs: ...B) => V
curry<V,...A,...B,...C>(f: (...args:[...A,...B,...C]) => V, ...as:...A): (...bs: ...B) => (...cs: ...C) => V
curry<V,...A,...B,...C,...D>(f: (...args:[...A,...B,...C,...D]) => V, ...as:...A): (...bs: ...B) => (...cs: ...C) => (...ds: ...D) => V
...

Let's try to reduce.
curry<V,...T>(f: (...args:[...A, ...T]) => V, ...as?:...A): _RDC((acc, par) => (par) => acc, V, reverse(T))

This appears to be covering the case where arguments not passed immediately would need to be passed in one by one (so using a full wall as above is still more useful), but it's a start.
To revisit those reduce definitions from the standard library, these could then be fixed up with a list of generic tuple definition for various lengths:
interface Tuple2<T,U> extends [T,U] {
  reduce<X,Y>(callbackfn: X, initialValue?: Y): _RDC(X, Y, this /* is this right? */);
}

Note that these would also need to capture the case of homogeneous arrays, since info on array size (and separate keys) is relevant. Note that the maximum tuple length may this way become a bottleneck for the definition of the method version.
So, would you have these definitions generate the original definition walls?
I think if from TS -> .d.ts we'd generate the 'resulting' possible types (i.e. the list options [A], [A,B], [A,B,C], ...), we'd have to choose a balance between (1) limiting to a fixed number vs. (2) keeping our files clean. This sounds like a lose-lose trade-off, so I'd either generate options in-memory or have TS calculate them as needed.
Language Feature Checklist:
SYNTAX:
Since from a definition writer's perspective this feels intuitively similar as using 'regular' functions (function application: F(T)) on the type level, I'd be inclined to have this use the same syntax.
Since this is on the type level and using an existing syntax, I expect no clashes with ES.
SEMANTICS:

What is an error under the proposed feature? Show many examples of both errors and non-errors

I believe this should function largely the same as applying any function on the type level, although under the hood it would be different, in the sense that TS would need to trigger this 'special' function rather than doing the type-level function application directly. So if one iteration of applying the callback function were to result in a TS error, this should throw to bubble up rather than continue to iterate.

const prop = <U,T>(o: U, k: T extends keyof U) => o[k];
_RDC(prop, pets, ['bigglesworth', 'children', 0, 'age']) // ok, number
let runtime_path: string[];
_RDC(prop, pets, runtime_path) // not good, any
let runtime_pets: { [k: string]: Cat };
_RDC(prop, runtime_pets, ['bigglesworth', 'children']) // Cat[]? <-- this should probably work? assumes existance of Cat 'bigglesworth'. Otherwise make it `Cat[] | null` / `undefined`?
_RDC(prop, runtime_pets, ['bigglesworth', 'children', 0, 'age']) // number? <- also a 'should probably work I guess?', on top of the above does also assume existance of child 0.
// not shown: bad callback functions (parameter 1). This would just be any type-level function, and be applied as such in the reduce function with the accumulator and next value, so wherever that fails, you have a bad... something.

How does the feature impact subtype, supertype, identity, and assignability relationships?

I don't foresee special interaction.

How does the feature interact with generics?

I don't foresee special interaction. That said, arrays typed such that info on their distinct values/types has gone lost would mean no iteration could be made (-> any).

EMIT:

What are the effects of this feature on JavaScript emit? Be specific; show examples

N/A

Does this emit correctly in the presence of variables of type ‘any’? Features cannot rely on runtime type information

It'd need to return any as well in this case -- all required info must be inferable for this to add value.

What are the impacts to declaration file (.d.ts) emit?

N/A

Does this feature play well with external modules?

N/A

COMPATIBILITY:
No breaking changes.
OTHER:

Can the feature be implemented without negatively affecting compiler performance?

looping through would cost more compute than ignoring these cases. But then the relevant type info would go lost, and presumably most of the usage cases would be somewhat limited in scope -- when accurate info on the array (e.g. length) is unknown, iterating can be skipped as there would be no point (not enough info to find out anything more specific than using the existing methods relying on Array<T>).

What impact does it have on tooling scenarios, such as member completion and signature help in editors?

positive -- inference should greatly improve for code making use of this.

P.S.: I'm also hoping for improvement for map. I think these two form the basis for a lot of the basic FP operations, and with these out of the way FP in TS should become a lot more pleasant.

Edit: I now realized more of these than I thought can currently be covered using those walls of increasingly long definition lines with increasing amounts of generics. For those this proposal would just offer syntactic sugar for definition writers.

Note that one way to achieve this would be to enable type-level array iteration. I mentioned some issues in achieving that currently here. That'd probably an elegant approach, as it'd be more general than just this (though reduce may cover the majority of the use-cases).

Update: I've managed to emulate reduce-like logic, see e.g. #12290.
Missing pieces:

failing inference when used through a function
inability to get return types of functions (#6606)

Presuming we can tackle those, we should no longer need a hard-coded reduce, making this proposal obsolete.


refactor: improve get_clients_from_cmd_args

improve the get_clients_from_cmd_args for better readability and consistency.

Replaced dictionary-based storage with a table for clients in the get_clients_from_cmd_args function.
Utilized the '#' operator to accurately count clients and enhance code readability.
Updated the condition for checking if the result table is empty using the '#' operator.

Probably works, but not thoroughly tested yet

Manually, I confirmed that LspRestart and LspStop work.


Scipy riccati solvers are too naive

It basically implements the textbook definition. Implement the Newton + line search for both care and dare (with proper function names if possible). Slyvester and Lyapunov solvers use LAPACK versions hence they are OK on paper.
Reminder : For control purposes, speed is irrelevant! So being implemented in Fortran doesn't mean much. The residuals and conditioning of the variables decide whether certain synthesis algorithms work or not. This needs to be state-of-art without relying on anything off-the-shelf.

This has led to a turtle-all-the-way-down situation. Hence, I'm trying to replicate the Sylvester solver which is needed for Lyapunov solvers which is needed for Riccati solvers. Current implementation is related to SLICOT 2014 report by Köhler and
Saak and also Kågström and Westin and then Penzl. Need to figure out a way to make incremental advances. Full blown implementation is too much work and might resort back to LAPACK and implement an error bound warning.

Lyapunov solvers are in place but slow (~10x) but more precise. For problems up to 300x300 the difference is bearable.

Commits 96814038b2f68ba379e133e24f2a27cfddeab9be and a39813b4d80f70a9341a00ff3c4d91a520aacf1d for the regular Lyapunov equations brought down the speed difference to x3. The precision for not-ill-conditioned problems are at least equivalent.  For discrete Lyapunov equations the precision gets better with problem size (with respect to max residual) if compared to scipy and matlab.

Lyapunov solvers are working pretty OK. For Riccati solvers I'm stuck with the balancing algorithm of LAPACK's GEBAL not performing as good as matlab's balance.m. I surely appreciate any pointers if anybody wants to chime in.

With, commit ba4e9731d1214e0e394d7293af45128e3fda0e16 scipy solvers are dropped and custom ones are in place. There is still this difference between GEBAL and balance.m which is beyond me.

Changed my mind an started to commit riccati solvers to official scipy repo directly.

SciPy 0.19 will have the new Riccati solvers via https://github.com/scipy/scipy/pull/6616. Lyapunov solvers are on the way either to that or to the next release of SciPy. It actually makes more sense to commit to numpy and scipy as much as possible since it is maintained and reviewed more meticulously.


fix(test): fix fields derive unit test

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Seems these three tests fail in my local env but pass on CI, I don't know why. cc @BugenZhao
Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[x] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Now it fails on CI. 😄

Is it possible to check equality of generated rust code ignoring spaces?

I think it fails because your editor recognized the expected files.

I think it fails because your editor recognized the expected files, and auto fmts them.

Yep it's possible.

It seems not because of my editor reformatting, since the expected string is never written into a file...


set & get block cache capacity via cf_options

enable us change block cache  capacity via tikv-ctl

Please discard the unrelated changes.

LGTM


How to debug the LDAP parameterization

Hi all,
firt of all I think that is a very clean and written extension to paperless-ngx.
I used it in a vanilla test installation and the docker compose brings it up smoothly.
But still the authorization is driven by internal users' database.
I enter the django container and look at the environment:
PAPERLESS_AD_DOMAIN=colombini.loc
PAPERLESS_AD_AUTH_PASSWORD=*********
PAPERLESS_AD_AUTH_URL=ldap://<IP_ADDRESS>:389
PAPERLESS_AD_AUTH_SEARCH_BASE=DC=colombini,DC=loc
PAPERLESS_AD_ADMIN_GROUP_CN=CN=Ced,OU=ICT,OU=Gruppi,OU=Colombini,DC=colombini,DC=loc
PAPERLESS_AD_USER_GROUP_CN=CN=Domain Users,OU=Utenti_Servizio,DC=colombini,DC=loc
PAPERLESS_AD_AUTH_USER=ldap-wa
PAPERLESS_AD_UPN_DOMAIN=colombini.loc

and it seems OK.
My question is: is there any command line django tool to investigate if it is working ?
Otherwise is there any specific log to look at in order to diagnose the problem ?
Thank you for any advice you can give me.
I work for an italian company, but in Italy seems there is little expertise on Django and nothin at all on Paperless-ngx.

You can add your own debugging to settings_ldap.py. Don’t forget my image uses UPN by default.

OK, after a lot of modification, I made it works.
I had to resume to change the settings_ldap.py to:
#############################################################################
# Fare riferimento alla seguente pagina del perchè ho docuto cambiare il file
# https://github.com/etianen/django-python3-ldap
#############################################################################
import os
import django_python3_ldap

# Initiate TLS on connection.
LDAP_AUTH_USE_TLS = None
LDAP_AUTH_CONNECT_USE_SSL = False
LDAP_AUTH_USER_LOOKUP_FIELDS = ("username",)

LDAP_AUTH_URL = [os.getenv('PAPERLESS_AD_AUTH_URL')]
LDAP_AUTH_SEARCH_BASE = os.getenv('PAPERLESS_AD_AUTH_SEARCH_BASE')
LDAP_AUTH_CONNECTION_USERNAME = os.getenv('PAPERLESS_AD_AUTH_USER')
LDAP_AUTH_CONNECTION_PASSWORD = os.getenv('PAPERLESS_AD_AUTH_PASSWORD')
LDAP_AUTH_USER_FIELDS = {
    "username": "sAMAccountName",
    "first_name": "givenName",
    "last_name": "sn",
    "email": "mail",
}
LDAP_AUTH_OBJECT_CLASS = "user"
PAPERLESS_AD_USER_GROUP = os.getenv('PAPERLESS_AD_USER_GROUP_CN')
PAPERLESS_AD_ADMIN_GROUP = os.getenv('PAPERLESS_AD_ADMIN_GROUP_CN')

from .settings import *
INSTALLED_APPS.append("django_python3_ldap")
AUTHENTICATION_BACKENDS.insert(2, "django_python3_ldap.auth.LDAPBackend")

LDAP_AUTH_FORMAT_USERNAME = "django_python3_ldap.utils.format_username_active_directory"
LDAP_AUTH_ACTIVE_DIRECTORY_DOMAIN = "COLNET"

For sure, now it is less elegant.... :-(
Thanks.

Using sAMAccountName solved your issue I guess.  It’s best practice to use the UPN for everything, since this is what people will use anyway for everything Azure and Co related 😊. Glad it works now.


Update helm install steps

Signed-off-by: Paul McDonald<EMAIL_ADDRESS>Just a quick docs update to the helm install commands
Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO)

hey!
thanks for the contribution! What would you like to achieve with this change? I mean, the organization is kedacore.

I think both options are OK, but think kedacore is better to show that it's more than keda itself

Yes, I think KEDA is also correct, but as you say, the repo has more charts than KEDA, thanks for the clarification

I just found it a bit confusing, as when I go to Releases, the commands are different
It's up to you both really

I'll review the releases then, maybe that's wrong. Thanks for pointing it

you are totally right. I'm going to solve it right now.
Thanks again ❤

I have updated the already existing releases to show the correct commands and I have created another PR fixing the release template. I'll close this PR because it's not needed, but thanks again for pointing the issue ❤❤


RT-1.31: BGP 3 levels of nested import/export policy with match-set-options

RT-1.31: BGP 3 levels of nested import/export policy with match-set-options
Summary

AS-path prepending by more the 10 repetitions
Recursive policy subroutines (multi-level nesting). At least 3 levels
match-set-options of ANY, INVERT for match-prefix-set conditions
Applicable to both IPv4 and IPv6 BGP neighbors

https://github.com/openconfig/featureprofiles/issues/2612

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 52.141%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1668

Relevant Lines:
3199

💛  - Coveralls

A) do we have separate independent test against AS path prepend? I asume we do not.
It is also covered here: http://shortn/_7KIZD3ClNY
But i believe we will still need atomic test to only validate as-path prepend. I will write a test only to check AS path prepend
B) IMO as-path-prepend and nested policies shall be separate tests. This are unrelated features and I do not understand why they are tested together. Note that if this test Fail due to nesting, we have no coverage for prepend. and vice-a-versa.
Will write separate test for AS-PATH prepend, but the 3 levels of nested policies have to take some action, it cant be blank policies else how will i validate that all policy actions are carried out. Every policy should do something that can be verified
C) Setting origin is path we never requested and we do not use. Better use MED or LP or communities
Will remove and replace with community
D) I will replace ORIGIN w/ MED in 1.31.1 nad 1.31.2; Then remove 1.31.3 and 1.31.4 completely
Ok replacing ORIGIN with MED but why remove 1.31.3 and 1.31.4 do we not need to check for IPv6 also to make sure (I know it will work for IPv6) but lets have the coverage, what do you think?
E) make separate test (separate README) fopr prepend.
On it


removes semi colon after export default

before
function thing(arg) {
  return arg;
}

export default thing;

after
function thing(arg) {
  return arg;
}

export default thing // <-- missing semi colon

Duplicate of #22

oops didn't see it on the next page. closing.


Question about the code of CHAIR

Hello, thanks for your contribution to the VLM evaluation! Besides the metric Pope, will you also release the code of the metric CHAIR you use in your paper? Thanks!

Thanks for your attention to our work. We adopt the code in the original paper about CHAIR. You can refer to LisaAnne/Hallucination for more details.

Thanks!


Implement Prometheus statser

This would allow for internal metrics to be monitored from prom.
This could be useful when there's a large number of ephemeral hosts, and other backends may not scale as well with the high cardinality of internal metrics across many hosts.
This is mostly stand-alone and doesn't interact with the more complicated aspects of how gostatsd handles things, as it's mostly just adding an http server + prom, so it's probably a good issue for someone looking to get across the code base.
This is explicitly not a prom backend, this is only for internal metrics, see #285 for that.

This would allow for internal metrics to be monitored from prom.
This could be useful when there's a large number of ephemeral hosts, and other backends may not scale as well with the high cardinality of internal metrics across many hosts.
This is mostly stand-alone and doesn't interact with the more complicated aspects of how gostatsd handles things, as it's mostly just adding an http server + prom, so it's probably a good issue for someone looking to get across the code base.
This is explicitly not a prom backend, this is only for internal metrics, see #285 for that.

Hey, I am keen to give this issue a go. All good if I take it from here?

That'd be great!

Would pkg/web be the right place to make the changes at,, or is there a more suitable directory where I can add the HTTP server with Prometheus to?

I'm not familiar with the Prom SDK in go, but my approach would be to create an appropriate PromStatser (which also includes the necessary Handler) in createStatser, then propagate that through to NewHttpServer, then if it's enabled (for that particular http server), attempt a type assertion to convert the interface to a PromStatser, and then attach the Handler if it's the right type.
If it's the wrong type, return an error - fast fail on startup is encouraged.

What do you mean by propagating the PromStatser to NewHttpServer? I am not able to find the related code.
And when you said attaching the Handler at the end, that is the Handler for the NewHttpServer, right? And specifically, what would this handler do?

You don't need to worry about Metric/EventHandler, that would be for making a prom backend (#285), this is purely for internal metrics.
I would suggest approaching this in two steps.  First, implement the stats.Statser interface, exposing prom metrics without regard for pkg/web.  This is just so that you can get the prom stuff working.  Ideally this would be done without using the DefaultRegistry/DefaultGatherer, so as to avoid singletons.
Once you've got the metrics exposed and working that way, you need to get the http.Handler through to pkg/web.  As mentioned, I recommend just passing the stats.Statser through and doing a type check on it.


Enable field/index/invocation access on grouped varRef

Purpose
$title.
Fixes (Partially) #19020. However, still cannot use expressions within the group (i.e ("A" + x).length(), ((dataMap)["a"]).getName()) due to current grammar limitations (tracked in #20238).
Remarks

#20238
#18298
#19020

Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #20234 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #20234   +/-   ##
=======================================
  Coverage    19.9%    19.9%
=======================================
  Files          55       55
  Lines        1437     1437
  Branches      218      218
=======================================
  Hits          286      286
  Misses       1138     1138
  Partials       13       13

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0649a94...39d8258. Read the comment docs.


Can't get ip address on a sails io socket request.

I got those error when use sails socket.io js.
code:
error: Sending 500 ("Server Error") response:
TypeError: Cannot read property 'remoteAddress' of undefined
at module.exports (/Users/chipgata/Projects/Backend-Tracking-Marketing/node_modules/sails-permissions/api/policies/AuditPolicy.js:6:67)
error: Sending 500 ("Server Error") response:
TypeError: undefined is not a function
at sanitizeRequestUrl (/Users/chipgata/Projects/Backend-Tracking-Marketing/node_modules/sails-permissions/api/policies/AuditPolicy.js:29:15)
Problem is this connection really don't have ip attribute in req object. But ip come alone with socket object that included in req object.
Hope you have a fix for this soon.
Thank you.

A PR would be great. I'm not going to have time to look at this until at least next week.

This fixes the IP issue, at least it's in the right direction.
var ipAddress = req.isSocket ? req.socket.handshake.address.address + ":" + req.socket.handshake.address.port : req.headers['x-forwarded-for'] || req.connection.remoteAddress;

Okay, I will update and check it out.  Thank you guy.

Is this resolved by #47?

Hey team, after update current Sails-Permission version,  I no-longer get Ipaddress error. But get this error when try to get collection by sails's io.socket GET method.
error: Sending 500 ("Server Error") response:
 TypeError: Object #<MockIncomingMessage> has no method 'get'
    at sanitizeRequestUrl (/home/hank/git/node/*****/node_modules/sails-permissions/api/policies/AuditPolicy.js:26:15)

This is line 24 to 28 of AuditPolicy.
 var requestUrl = url.format({
    protocol: req.protocol,
    host: req.get('host'),
    pathname: req.originalUrl,
    query: req.query
  });


Improve the return type of the refreshToken mutation

Currently the return type is named Refresh and contains two fields:

token which I assume is the new token
payload which has no description and a type of GenericScalar

The return type of refreshToken contains a description and is now returning a User type 👍


Add colcon_cd bash completer

Address #12
I had some time to work on this and managed to add in the auto-completion feature for colcon_cd. The solution is a bit hackish, but I hope it will be helpful.
How to use
. <path-to-workspace>/install/colcon-cd/share/colcon_cd/function/colcon_cd.sh
. <path-to-workspace>/install/colcon-cd/share/colcon_cd/function/colcon_cd-argcomplete.bash
# . <path-to-workspace>/install/colcon-cd/share/colcon_cd/function/colcon_cd-argcomplete.zsh

My original intention is to make use of argcomplete's --external-argcomplete-script to create a fake console command (entrypoint name: _colcon_cd_completions) and use that console command for bash completion. This is implemented in completer.py.
However, due to the argcomplete version differences between bootstrap setup pip (1.12.3) and Ubuntu 20.04 native python3-argcomplete (1.8.1), I have to include the entire bash script from argcomplete 1.12.3, and also not being able to remove the auto-completion for unused variable -- base_path (ref).
I have verified that it works with bash on Ubuntu 20.04 native and bootstrap setup, I am not familiar with zsh testing, it will be helpful if I could get help from the maintainers on this.
Tests pass with colcon test locally, CI pending https://github.com/colcon/colcon-cd/pull/13, passed after rebase
It might be easier way to do this completely in shell, but I just cannot figure it out and this was what I currently decide to go with.

Rebased to latest, ready for review!

Pinging here again. Please help review this if anybody have time..

Pinging here again. Please help review this if anybody have time..

@Briancbn is this PR still up-to-date to be merged?
@cottsay Please merge this one, when all issues are addressed.

@cottsay friendly ping to review (and merge) this PR.

CI is broken for now, I'll get that cleaned up this week so we can get a green build here.

Do you intend to distribute this extra functionality on an opt-in basis

Yes, only when sourcing the bash since the colcon_cd command doesn't work on Windows at the current point.

2. Is it possible to use register-python-argcomplete instead of creating the completion script manually like colcon-argcomplete does?

Looks promising. Let me verify.
Is there any reason that colcon-argcomplete didn't choose to use register-python-argcomplete? I would assume it's best to keep the autocompletion setup consistent within colcon.

@cottsay friendly ping ;)

CI is broken for now

CI is now fixed.

Is there any reason that colcon-argcomplete didn't choose to use register-python-argcomplete?

It does: https://github.com/colcon/colcon-argcomplete/blob/master/completion/colcon-argcomplete.bash

Lets merge this thing and release it please.

@cottsay Sorry, I thought the only blocking thing was the broken CI.

I work on this next weekend. Sorry been busy with other things lately. @cottsay thanks for the suggestion!

@cottsay Thanks for the guidance. Works like charm! This is really brilliant.

To elaborate on why I suggested that might be a good idea, here's a simple change that could eliminate the copy/paste from shell_integration.py.
diff --git a/function/colcon_cd.sh b/function/colcon_cd.sh
index d85751b1aea4..3d1f961a952c 100644
--- a/function/colcon_cd.sh
+++ b/function/colcon_cd.sh
@@ -1,7 +1,9 @@
 # copied from colcon-cd/function/colcon_cd.sh

 colcon_cd() {
-  if [ $# = 0 ]; then
+  if [ "$_ARGCOMPLETE" == "1" ]; then
+    python3 -m colcon_cd.completer
+  elif [ $# = 0 ]; then
     # change the working directory to the previously saved path
     if [ "$_colcon_cd_root" = "" ]; then
       echo "No previous path saved. Either call 'colcon_cd <pkgname>' from a" \

I am still not entirely sure what is happening here. I think the above changes including the python3 -m colcon_cd.completer part gets executed within here (https://github.com/kislyuk/argcomplete/blob/v1.8.1/argcomplete/bash_completion.d/python-argcomplete.sh), Please correct me if I am wrong.

@cottsay pinging again. Ready to review if you have time.


ci: Repo stats workflow

Resolves #5667

Where does the data get stored?

They get stored in the repo-stats branch.
Example: https://github.com/luarss/awesome-conference-dates/tree/repo-stats/luarss/awesome-conference-dates
We can also specify the folder to store this in the target branch, but since I did not they just used author/repo

There is no such branch today - does the action create one?  Can it be configured to store the data in a different repo?

https://github.com/jgehrcke/github-repo-stats?tab=readme-ov-file#setup
Yes you don’t have to manually create the branch. It seems also that directing to a data repo is possible, but the token needs to be able to given sufficient permissions


C++ stacktrace enable

This pull request adds a switch for enabling extended C++ exception stacktraces.   It is a private internal module function.  It will be enabled during testing (as it makes tracking problems in the CI much easier).
This is built on numbers2.  (It is only a 5 line change)
Fixed #623

I will have to deal with this one once the master catches up with this point.


Issue with k8s.io/docs/reference/kubectl/kubectl/

This is a Bug Report

Problem:
Proposed Solution:
Page to Update:
https://kubernetes.io/...

@knbz Please provide more information
/assign
/triage needs-more-information

pls feel free to open with actionable issue details or file a new one.


confirmation on completion of assessment

I suggest we change the button to say 'See my scores'.
And when someone clicks 'See my scores', there should be a warning message (like when the user clicks 'exit' button) to warn the user that 'Your answers have been set; you will not be able to edit after this point' or something like that?

What do you think of the text/changes I've made so far? Currently it has "Complete assessment" on the button, plus some additional text underneath


Stop old CCGs appearing as outliers on Analyse results

Since April old CCGs (Newcastle, Manchester) are appearing as outliers on some searches on the Analyse page.

https://openprescribing.net/analyse/#org=CCG&numIds=0703021Q0BB&denomIds=0703021Q0&selectedTab=summary
This appears to be caused by the closed practices still in the CCG:

https://openprescribing.net/analyse/#org=practice&orgIds=00F&numIds=0703021Q0BB&denomIds=0703021Q0&selectedTab=chart
The above search mentions Manchester CCGs in the screenshot as outliers even though it is a search for practices in Gateshead. If you click the link above it doesn't have this message at the bottom. This only appears if you first run this search, and then without refreshing change the search to practices:Gateshead CCG.
This message wasn't appearing on analyse before the April upload (I run the Cerazette/Desogestrel search in demos).
Other examples:
https://openprescribing.net/analyse/#org=CCG&numIds=0601060D0BHABA0,0601060D0CUABA0&denomIds=6.1.6&selectedTab=summary
https://openprescribing.net/analyse/#org=CCG&orgIds=01Y&numIds=090402000BBQLA0,090402000BBRBA0,090402000BBRAA0,090402000BBQKA0,090402000BBQMA0&denomIds=9.4.2&selectedTab=chart

Relates to #408

Wrapped up into #891


Support Send and serialization/deserialization of Gc<T> types

Would it be possible to implement Send for Gc<T> to allow migration of a Gc<T> pointer from one mutator to another?
In addition, would it be possible to implement serialize/deserialize for Gc<T> types? I mean, instead of converting Gc<T> -> Box<T> -> serialize -> deserialize -> Box<T> -> Gc<T>, I'd like to go Gc<T> -> serialize -> deserialize -> Gc<T>

It is possible to implement Send. Serialize and Deserialize is a harder question to solve, but I know for sure you do not need to box types for serialization and deserialization. In the future Serialize can be implemented, but Deserialize is impossible because we cannot pass mutator to deserialize to allocate memory when needed.

I think it's possible to use serde_state. Probably it can be handled by the user

Do you think it's possible to have zero-copy serialization/deserialization using for example rkyv? Maybe it's a long shot.


Cat-select doesn't update properly when using search and clicking outside after selection

This bug only seems to happen with single select, if you wait a few seconds after you select a value it will work

#315


Avoid lock when token is still valid

For performances reasons, it's better to avoid locks when possible.

When the token is still valid, there is no need for a lock
When the token need to be refreshed, the time spent to lock is negligible versus the http request.

So in Service_.prototype.hasAccess, before using this.lockable_ I think it should check if there is an unexpired token, and is so, returns it without lock.

So my original thought was that I wanted to prevent duplicate refreshes of the access token, and in order to do that you need to lock before you check to see if it's expired. If we only lock once we determine that it is expired, we end up with the following:

Execution #1
Execution #2

Is token expired? Yes
...

Acquire lock
...

Start refreshing token
...

...
Is token expired? Yes

...
Wait for lock

Refresh complete
...

Release lock
...

...
Acquire lock

...
Start refreshing token

...
Refresh complete

...
Release lock

To prevent the token from being refreshed twice, we need to lock before we check if it's expired, so that the second execution see's a non-expired token when it checks.
Now the original problem was due to the tokens getting saved out of order, such that the token from the first execution would be the one persisted in the properties, but it was invalidated by the token requested in the second execution. Locking after we check for expiration would result in two tokens, but I think it would guarantee that they are saved in the correct order.
Thoughts?

It will works but I think a better design would be to only request the token once. So if two concurrent thread request the token, one refresh it and both gains access to it.
I feel it's a better design because it avoid duplicate tokens and is faster because if one precess is already in the middle of the renewal, the second one will get the renewed token as soon as the first one gets it.

Execution 1
Execution 2

Is token N expired? Yes

Acquire lock

is current token N expired? Still Yes

Is token expired? Yes

Wait for lock

Refresh complete : current token=N+1

Release lock

is current token N+1 expired? No

Release lock

(of course, if there is no token, no lock should be done)

I hadn't thought of that, but makes sense. I sent out PR #147 to implement this. Please take a look.


Build target marked as build, but not required will fail builds if the build support is not installed

Affects: Release v0.3.8
Internal Ticket: UTY-2668

Description
If you have a build target which has a platform marked as 'Build', but not 'Required' and the build support for that platform is not installed, attempting to build this target will fail rather than skip the platform without build support.
This will be fixed in #1435 and will be available in the next release.
Workaround
Untick the platform marked as 'Build'.

Released in v0.3.9


use panel path from the config

this pr will fix the route been static, and get the path from the config rest-presenter.panel.path

:rocket: This pull request is included in v1.1.1. See Release 1.1.1 for release notes.


Improve tf/custom directory handling.

Describe the feature you're requesting.
In scenarios where TF2 is installed on a separate drive from Steam, the editor is supposed to be able to find the tf/custom directory using the registry or else ask the user to specify where that folder is located.

[x] Update the Troubleshooting page to include the steps and video on how to manually resolve this issue. https://www.youtube.com/watch?v=NqSqLyROBwk
[x] Make the settings dropdown button of the editor a little more prevalent.
[x] Improve the tf/custom path finding to account for TF2 being installed on a separate drive. Avoid immediately asking the user to set the path if TF2 is not installed on the same drive as Steam.

Additional context
This was brought up by @Hakej on Discord, who also created the linked video.

All three changes have been implemented in https://github.com/CriticalFlaw/TF2HUD.Editor/pull/136 and will be released in version 2.6