cluster multiple thresholds

Working example of how we can cluster efficiently at multiple thresholds.
We start with the lowest/loosest threshold.
At each iteration, we use the clusters from the previous iteration to find 'stable clusters' - ones which we know in advance won't change.
The algorithm for identifying stable clusters is as follows:

Suppose initial clustering was at 50%
Next threshold is 60%
For each 50% cluster, identify the edges, then find the minimum match_probability amongst the dges
If the minimum match_probability is >60%, we know the cluster is unaffected by the new cluster
Prior to running the clustering at 60%, remove all nodes and edges associated with stable clusters

This method is independent of the clustering algorithm so we should be able to use it even if we find a faster way of clustering at a single threshold

Click to expand
import pandas as pd

import splink.comparison_library as cl
from splink import DuckDBAPI, Linker, SettingsCreator, block_on, splink_datasets
from splink.internals.pipeline import CTEPipeline
from splink.internals.vertically_concatenate import (
    compute_df_concat_with_tf,
)

db_api = DuckDBAPI()

df = splink_datasets.fake_1000

settings = SettingsCreator(
    link_type="dedupe_only",
    comparisons=[
        cl.ExactMatch("first_name"),
        cl.ExactMatch("surname"),
        cl.ExactMatch("dob"),
        cl.ExactMatch("city").configure(term_frequency_adjustments=True),
        cl.ExactMatch("email"),
    ],
    blocking_rules_to_generate_predictions=[
        block_on("first_name"),
        block_on("surname"),
    ],
    max_iterations=4,
)

linker = Linker(df, settings, db_api)

linker.training.estimate_probability_two_random_records_match(
    [block_on("first_name", "surname")], recall=0.7
)

linker.training.estimate_u_using_random_sampling(max_pairs=1e6)

linker.training.estimate_parameters_using_expectation_maximisation(block_on("dob"))

pairwise_predictions = linker.inference.predict(threshold_match_weight=-10)

clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(
    pairwise_predictions, 0.5
)

clusters.templated_name
# __splink__df_predict
pipeline = CTEPipeline()
nodes_with_tf = compute_df_concat_with_tf(linker, pipeline)

pipeline = CTEPipeline([pairwise_predictions, clusters, nodes_with_tf])

OLD_THRESHOLD = clusters.metadata["threshold_match_probability"]
NEW_THRESHOLD = 0.51

sql = f"""
select unique_id_l, unique_id_r, match_probability
 from __splink__df_predict
union all
select unique_id as unique_id_l, unique_id as unique_id_r, 1.0 as match_probability
from __splink__df_concat_with_tf

"""
pipeline.enqueue_sql(sql, "edges")

sql = """
SELECT e.*,
        cl.cluster_id AS cluster_id_l,
        cr.cluster_id AS cluster_id_r
FROM edges e
LEFT JOIN __splink__df_representatives cl ON e.unique_id_l = cl.unique_id
LEFT JOIN __splink__df_representatives cr ON e.unique_id_r = cr.unique_id
"""
pipeline.enqueue_sql(sql, "edges_with_clusters")

sql = f"""
SELECT
   cluster_id_l as cluster_id, min(match_probability) as min_match_probability
FROM edges_with_clusters
WHERE match_probability >= {OLD_THRESHOLD}
GROUP BY cluster_id_l
HAVING MIN(match_probability) >= {NEW_THRESHOLD}
ORDER BY cluster_id_l
"""

pipeline.enqueue_sql(sql, "__splink__stable_clusters")

stable_clusters = linker._db_api.sql_pipeline_to_splink_dataframe(pipeline)
stable_clusters.as_duckdbpyrelation()
pipeline = CTEPipeline([stable_clusters, clusters])

sql = """
SELECT cluster_id, unique_id
FROM __splink__df_representatives ic
WHERE ic.cluster_id IN (
    SELECT cluster_id FROM __splink__stable_clusters
)
"""
pipeline.enqueue_sql(sql, "__splink__stable_nodes")

stable_nodes = linker._db_api.sql_pipeline_to_splink_dataframe(pipeline)
stable_nodes.as_duckdbpyrelation()

pipeline = CTEPipeline([stable_nodes, pairwise_predictions, nodes_with_tf])

sql = f"""
select unique_id_l, unique_id_r, match_probability
 from __splink__df_predict
union all
select unique_id as unique_id_l, unique_id as unique_id_r, 1.0 as match_probability
from __splink__df_concat_with_tf

"""
pipeline.enqueue_sql(sql, "edges")

sql = """
SELECT e.unique_id_l, e.unique_id_r, e.match_probability
FROM edges e
WHERE e.unique_id_l NOT IN (SELECT unique_id FROM __splink__stable_nodes)
  AND e.unique_id_r NOT IN (SELECT unique_id FROM __splink__stable_nodes)
"""
pipeline.enqueue_sql(sql, "__splink__edges_in_play")

edges_in_play = linker._db_api.sql_pipeline_to_splink_dataframe(pipeline)
edges_in_play.as_duckdbpyrelation()

pipeline = CTEPipeline([nodes_with_tf, stable_nodes])
sql = """
SELECT unique_id
FROM __splink__df_concat_with_tf
WHERE unique_id NOT IN (SELECT unique_id FROM __splink__stable_nodes)
"""
pipeline.enqueue_sql(sql, "__splink__nodes_in_play")

nodes_in_play = linker._db_api.sql_pipeline_to_splink_dataframe(pipeline)
nodes_in_play.as_duckdbpyrelation()

import logging

logger = logging.getLogger("splink")
logger.info("done one ")

linker2 = Linker(df, SettingsCreator(link_type="dedupe_only"), db_api=db_api)
linker2.table_management.register_table_input_nodes_concat_with_tf(
    nodes_in_play.physical_name
)
clusters_2 = linker2.clustering.cluster_pairwise_predictions_at_threshold(
    edges_in_play, 0.98
)

df_clusters_1 = stable_nodes.as_pandas_dataframe()

df_clusters_2 = clusters_2.as_pandas_dataframe()

combined_clusters = pd.concat([df_clusters_1, df_clusters_2], ignore_index=True)
combined_clusters

Once nice feature of this is we'll be able to produce a chart that shows, for each threshold probability/match weight, how much the end results change.  Could this chart be used to identify sensible thresholds?  e.g. we expect it to be steepest at 50%.  If it's not does that indicate an issue with modell?
At each iteration, we use the clusters from the previous iteration to find which clusters do not change.
For each iteration, when we compute the clusters, should we add on the min probability in the cluster so it can be used in the next iteration?
On the first run, we can actually get out a table of cluster vs. at what probability we need to re-analyse the cluster., that's what min_match_probability does.  So on the next run, before we do the expensive join of clusters to edges, we can prior to that filter out all the nodes/clusters/edges  we know in advance are going to be unaffected
TODO:

[ ] Clean up tables create by Splink
[ ] Try it on more complex graphs

Here's the current terrible testing script:
import logging
import random
import string
import time

import duckdb
import pandas as pd

import splink.comparison_library as cl
from splink import DuckDBAPI, Linker, SettingsCreator, block_on, splink_datasets
from splink.blocking_analysis import (
    cumulative_comparisons_to_be_scored_from_blocking_rules_chart,
)

original_df = splink_datasets.historical_50k
from splink.internals.clustering import (
    cluster_pairwise_predictions_at_multiple_thresholds,
    cluster_pairwise_predictions_at_threshold,
)

# # stack 10 copies of the dataframe vertically
# # Function to generate a random 4-character prefix
# def random_prefix():
#     return "".join(random.choices(string.ascii_lowercase, k=4))

# # Create 4 copies of the dataframe with unique prefixes
# dfs = []
# for i in range(2):
#     temp_df = original_df.copy()
#     prefix = random_prefix()

#     # Add prefix to relevant columns
#     for col in ["full_name", "first_and_surname", "first_name", "surname"]:
#         temp_df[col] = temp_df[col] + prefix

#     dfs.append(temp_df)

# # Concatenate the modified dataframes
# df = pd.concat(dfs, ignore_index=True)
# # Reset the index to ensure it starts from 0 and is continuous

# df = splink_datasets.historical_50k.head(100)
# df = df.reset_index(drop=True)
# df.drop(columns=["unique_id"], inplace=True)
# # Rename the index to 'unique_id'
# df = df.rename_axis("unique_id").reset_index()

# print(f"Length of modified df: {len(df)}")
# df.head()
# # Reset the index to ensure it starts from 0 and is continuous
# df = df.reset_index(drop=True)
# # drop unique_id
# df = df.drop(columns=["unique_id"])
# # Rename the index to 'unique_id'
# df = df.rename_axis("unique_id").reset_index()
# len(df)

df = splink_datasets.historical_50k
db_api = DuckDBAPI()
linker = Linker(df, "50k_model.json", db_api=db_api, set_up_basic_logging=True)

logging.getLogger("splink").setLevel(logging.WARNING)

# blocking_rules = [
#     block_on("substr(first_name,1,3)", "substr(surname,1,4)"),
#     block_on("surname", "dob"),
#     block_on("first_name", "dob"),
#     block_on("postcode_fake", "first_name"),
#     block_on("postcode_fake", "surname"),
#     block_on("dob", "birth_place"),
#     block_on("substr(postcode_fake,1,3)", "dob"),
#     block_on("substr(postcode_fake,1,3)", "first_name"),
#     block_on("substr(postcode_fake,1,3)", "surname"),
#     block_on("substr(first_name,1,2)", "substr(surname,1,2)", "substr(dob,1,4)"),
# ]

# cumulative_comparisons_to_be_scored_from_blocking_rules_chart(
#     table_or_tables=df,
#     blocking_rules=blocking_rules,
#     db_api=db_api,
#     link_type="dedupe_only",
# )

# settings = SettingsCreator(
#     link_type="dedupe_only",
#     blocking_rules_to_generate_predictions=blocking_rules,
#     comparisons=[
#         cl.ForenameSurnameComparison(
#             "first_name",
#             "surname",
#             forename_surname_concat_col_name="first_name_surname_concat",
#         ),
#         cl.DateOfBirthComparison("dob", input_is_string=True),
#         cl.PostcodeComparison("postcode_fake"),
#         cl.ExactMatch("birth_place").configure(term_frequency_adjustments=True),
#         cl.ExactMatch("occupation").configure(term_frequency_adjustments=True),
#     ],
#     retain_intermediate_calculation_columns=True,
#     max_iterations=2,
# )

# df["first_name_surname_concat"] = df["first_name"] + " " + df["surname"]

# linker = Linker(df, settings, db_api=db_api, set_up_basic_logging=True)

# linker.training.estimate_probability_two_random_records_match(
#     [
#         "l.first_name = r.first_name and l.surname = r.surname and l.dob = r.dob",
#         "substr(l.first_name,1,2) = substr(r.first_name,1,2) and l.surname = r.surname and substr(l.postcode_fake,1,2) = substr(r.postcode_fake,1,2)",
#         "l.dob = r.dob and l.postcode_fake = r.postcode_fake",
#     ],
#     recall=0.6,
# )

# linker.training.estimate_u_using_random_sampling(max_pairs=1e6)

# training_blocking_rule = block_on("first_name", "surname")
# training_session_names = (
#     linker.training.estimate_parameters_using_expectation_maximisation(
#         training_blocking_rule, estimate_without_term_frequencies=True
#     )
# )

# training_blocking_rule = block_on("dob")
# training_session_dob = (
#     linker.training.estimate_parameters_using_expectation_maximisation(
#         training_blocking_rule, estimate_without_term_frequencies=True
#     )
# )

# linker.misc.save_splink_model("50k_model.json")

df_predict = linker.inference.predict(threshold_match_probability=0.5)
df_predict.as_duckdbpyrelation().count("*")
linker.visualisations.match_weights_histogram(df_predict)

logging.getLogger("splink").setLevel(logging.WARNING)

thresholds = [0.5, 0.501, 0.502, 0.503, 0.504, 0.505, 0.506, 0.507]

time_start = time.time()

quick_results = cluster_pairwise_predictions_at_multiple_thresholds(
    df,
    df_predict,
    node_id_column_name="unique_id",
    db_api=db_api,
    match_probability_thresholds=thresholds,
)
time_end = time.time()
print(f"Quick clustering time: {time_end - time_start:.2f} seconds")

time_start = time.time()
slow_results = {}
for threshold in thresholds:
    start_iteration_time = time.time()
    clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(
        df_predict, threshold
    )
    slow_results[threshold] = clusters
    end_iteration_time = time.time()
    print(
        f"Slow Iteration Time taken: {end_iteration_time - start_iteration_time:.2f} seconds"
    )

time_end = time.time()
print(f"Slow clustering time: {time_end - time_start:.2f} seconds")

for threshold in thresholds:
    print(threshold)
    quick_result = quick_results[threshold].as_duckdbpyrelation()
    slow_result = slow_results[threshold].as_duckdbpyrelation()

    sql = """
    SELECT count(distinct cluster_id)
    from quick_result
    """
    quick_count = db_api._con.sql(sql).fetchone()[0]

    sql = """
    SELECT count(distinct cluster_id)
    from slow_result
    """
    slow_count = db_api._con.sql(sql).fetchone()[0]

    print(f"Quick count: {quick_count}")
    print(f"Slow count: {slow_count}")
    assert quick_count == slow_count

Sort of helpful:
import random
import time

import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
import pandas as pd

from splink import DuckDBAPI
from splink.internals.clustering import (
    cluster_pairwise_predictions_at_multiple_thresholds,
    cluster_pairwise_predictions_at_threshold,
)

db_api = DuckDBAPI()

def generate_uniform_probability_graph(graph_size=1000, num_edges=2000, master_seed=42):
    random.seed(master_seed)

    def generate_random_nodes(num_rows):
        return pd.DataFrame({"unique_id": range(num_rows)})

    def generate_random_edges(num_rows, num_edges):
        edges = []
        for _ in range(num_edges):
            unique_id_l = random.randint(0, num_rows - 1)
            unique_id_r = random.randint(0, num_rows - 1)
            if unique_id_l != unique_id_r:
                match_probability = random.uniform(0, 1)
                edges.append(
                    {
                        "unique_id_l": unique_id_l,
                        "unique_id_r": unique_id_r,
                        "match_probability": match_probability,
                    }
                )
        return pd.DataFrame(edges)

    nodes = generate_random_nodes(graph_size)
    edges = generate_random_edges(graph_size, num_edges)

    return nodes, edges

combined_nodes, combined_edges = generate_uniform_probability_graph(100_000, 100_000)

# combined_nodes.to_parquet("combined_nodes.parquet")
# combined_edges.to_parquet("combined_edges.parquet")

# combined_nodes = pd.read_parquet("combined_nodes.parquet").head(1_000_00)
# combined_edges = pd.read_parquet("combined_edges.parquet").head(2_000_00)

import logging

logging.basicConfig(format="%(message)s")
logging.getLogger("splink").setLevel(15)

start_time = time.time()
res = cluster_pairwise_predictions_at_multiple_thresholds(
    combined_nodes,
    combined_edges,
    node_id_column_name="unique_id",
    db_api=db_api,
    match_probability_thresholds=[0.5, 0.501, 0.5010001],
)
end_time = time.time()
print(f"Time taken: {end_time - start_time:.2f} seconds")
end_time = time.time()

start_time = time.time()
res[0.5].as_duckdbpyrelation().count("distinct cluster_id")

Useful graph creation script:
import random
import time

import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
import pandas as pd

from splink import DuckDBAPI
from splink.internals.clustering import (
    cluster_pairwise_predictions_at_multiple_thresholds,
    cluster_pairwise_predictions_at_threshold,
)

db_api = DuckDBAPI()

def generate_random_graph(graph_size, seed=None):
    if not seed:
        seed = random.randint(5, 1000000)

    density = 1 / graph_size
    print(f"Graph size: {graph_size}, Density: {density}")

    graph = nx.fast_gnp_random_graph(graph_size, density, seed=seed, directed=False)
    return graph

def nodes_and_edges_from_graph(G):
    edges = nx.to_pandas_edgelist(G)
    edges.columns = ["unique_id_l", "unique_id_r"]

    nodes = pd.DataFrame({"unique_id": list(G.nodes)})

    return nodes, edges

G = generate_random_graph(100_000)
combined_nodes, combined_edges = nodes_and_edges_from_graph(G)
combined_edges["match_probability"] = np.random.uniform(0, 1, len(combined_edges))

clusters = cluster_pairwise_predictions_at_threshold(
    combined_nodes,
    combined_edges,
    node_id_column_name="unique_id",
    db_api=db_api,
    threshold_match_probability=0.5,
)

clusters_ddb = clusters.as_duckdbpyrelation()
clusters_ddb.count("distinct cluster_id").show()

db_api._con.sql("""
    SELECT cluster_size, COUNT(*) as cluster_count
    FROM (
      SELECT cluster_id, COUNT(*) as cluster_size
      FROM clusters_ddb
      GROUP BY cluster_id
    )
    GROUP BY cluster_size
    ORDER BY cluster_size
""").show()

The length of the neighbours table is the length of:

nodes + edges*2  (one row for each self link, two rows for each edge)

The following script demostrates that joining 100 to 1000 vs 100 to 10_000_000 takes basically the same amount of time:

import duckdb
import pandas as pd
import time

# Create in-memory DuckDB connection
con = duckdb.connect()

# Generate DataFrames
df_small = pd.DataFrame({
    'id': range(1, 101),  # 100 rows
    'value': ['small_value_' + str(i) for i in range(1, 101)]
})

df_medium = pd.DataFrame({
    'id': range(1, 1001),  # 1000 rows
    'value': ['medium_value_' + str(i) for i in range(1, 1001)]
})

df_large = pd.DataFrame({
    'id': range(1, 10000001),  # 10,000,000 rows
    'value': ['large_value_' + str(i) for i in range(1, 10000001)]
})

# Load the DataFrames into DuckDB
con.execute("CREATE TABLE small AS SELECT * FROM df_small")
con.execute("CREATE TABLE medium AS SELECT * FROM df_medium")
con.execute("CREATE TABLE large AS SELECT * FROM df_large")

# Function to measure join time
def measure_join_time(table1, table2):
    start_time = time.time()
    con.sql(f"SELECT * FROM {table1} JOIN {table2} ON {table1}.id = {table2}.id").show()
    end_time = time.time()
    return end_time - start_time

# Measure performance for small vs medium join
time_small_medium = measure_join_time('small', 'medium')
print(f"Join time for 100 rows to 1000 rows: {time_small_medium} seconds")

# Measure performance for small vs large join
time_small_large = measure_join_time('small', 'large')
print(f"Join time for 100 rows to 10,000,000 rows: {time_small_large} seconds")

You can't remove represenataives that are out of play because the in_play column does not identify converged clusters
# This sql code for solving connected components takes inspiration
# from the following paper: https://arxiv.org/pdf/1802.09478.pdf

# While we haven't been able to implement the solution presented
# by the paper - due to SQL backend restrictions with UDFs, -
# we have been able to use the paper to further our understanding
# of the problem and come to a working solution.

# See also https://github.com/RobinL/clustering_in_sql

from __future__ import annotations

import logging
import time
from typing import Optional

from splink.internals.database_api import DatabaseAPISubClass
from splink.internals.pipeline import CTEPipeline
from splink.internals.splink_dataframe import SplinkDataFrame

logger = logging.getLogger(__name__)

def _cc_create_unique_id_cols():
    pass

def _cc_generate_neighbours_representation() -> str:
    """SQL to generate all the 'neighbours' of each input node.

    The 'neighbour' of a node is any other node that is connected to the original node
    within the graph.  'Connected' means that at the threshold match probability,
    the nodes are considered to be a match (i.e. the nodes are estimated to
    be same entity)

    This table differs to the edges table in two respects:
    1.  Unlike the edges table, it's not guaranteed that the node (left hand side)
        has a lower id than the neighbour.  That is, the left column (node_id) contains
        all the nodes, not just the ones with a higher id on the right hand side
    2. The table contains all nodes, even those with no edges (these are represented)
        as a 'self link' i.e. ID1 -> ID1, ensuring they are present in the final
        clusters table.
    """

    sql = """
    select n.node_id,
        e_l.node_id_r as neighbour
    from nodes as n

    left join __splink__edges as e_l
        on n.node_id = e_l.node_id_l

    UNION

    select n.node_id,
        coalesce(e_r.node_id_l, n.node_id) as neighbour
    from nodes as n

    left join __splink__edges as e_r
        on n.node_id = e_r.node_id_r
    """

    return sql

def _cc_generate_initial_representatives_table() -> str:
    """SQL to generate our initial "representatives" table.

    The 'representative' column will eventually become the cluster ID.

    As outlined in the paper quoted at the top:

    '...begin by choosing for each vertex (node) a representatative by picking the
    vertex (node) with the minimum id amongst itself and its neighbours'.

    e.g. node ids 1, 2 and 3 may all have representative 2, indicating
    they are a cluster.

    This is done initially by grouping on our neighbours table
    and finding the minimum neighbour for each node.
    """

    sql = """
    select
        neighbours.node_id,
        min(neighbour) as representative

    from __splink__df_neighbours as neighbours
    group by node_id
    order by node_id
    """

    return sql

def _cc_update_neighbours_first_iter() -> str:
    """SQL to update our neighbours table - first iteration only.

    Takes our initial neighbours table, join on the representatives table
    and recalculates the mimumum representative for each node.

    This works by joining on the current known representative for each node's
    neighbours.

    i.e. rather than looking at a node's minimum representative, we look at the node's
    neighbour's minimum representative.

    So, if we know that B is represented by A (B -> A) and C is represented by B
    (C -> B), then we can join on B to conclude that (C -> A).
    """

    sql = """
    select
        neighbours.node_id,
        min(representatives.representative) as representative

    from __splink__df_neighbours as neighbours
    left join representatives
    on neighbours.neighbour = representatives.node_id
        group by neighbours.node_id
        order by neighbours.node_id
    """

    return sql

def _cc_update_representatives_first_iter() -> str:
    """SQL to update our representatives table - first iteration only.

    From here, standardised code can be used inside a while loop,
    as the representatives table no longer needs generating.

    This is only used for the first iteration as we

    In this SQL, we also generate "in_play", which is a boolean
    that indicates whether the current representative differs
    from the previous representative.

    This value is used extensively to speed up our while loop and as
    an exit condition.
    """

    sql = """
    select
        n.node_id,
        n.representative,
        n.representative <> repr.representative as in_play
    from neighbours_first_iter as n
    left join representatives as repr
    on n.node_id = repr.node_id
    """

    return sql

def _cc_generate_representatives_loop_cond(
    prev_representatives: str,
) -> str:
    """SQL for Connected components main loop.

    Takes our core neighbours table (this is constant), and
    joins on the current representatives table from the
    previous iteration by joining on information about each node's
    neighbours representatives.

    So, reusing the same summary logic mentioned above, if we know that B
    is represented by A (B -> A) and C is represented by B (C -> B),
    then we can join (B -> A) onto (C -> B) to conclude that (C -> A).

    Doing this iteratively eventually allows us to climb up the ladder through
    all of our neighbours' representatives to a solution.

    The key difference between this function and 'cc_update_neighbours_first_iter',
    is the usage of 'in_play'.

    The logic behind 'in_play' is summarised in 'cc_update_representatives_first_iter'
    and it can be used here to reduce our neighbours table to only those nodes that need
    updating.
    """

    sql = f"""
    select

    source.node_id,
    min(source.representative) as representative

    from
    (

        select

            neighbours.node_id,
            repr_neighbour.representative as representative

        from __splink__df_neighbours as neighbours

        inner join {prev_representatives} as repr_neighbour
        on neighbours.neighbour = repr_neighbour.node_id

        where repr_neighbour.in_play = True

    ) AS source
    group by source.node_id
        """

    return sql

def _cc_update_representatives_loop_cond(
    prev_representatives: str,
) -> str:
    """SQL to update our representatives table - while loop condition.

    Reorganises our representatives output generated in
    cc_generate_representatives_loop_cond() and isolates 'in_play',
    which indicates whether all representatives have 'settled' (i.e.
    no change from previous iteration).
    """

    sql = f"""
    select

        r.node_id,
        r.representative,
        r.representative <> repr.representative as in_play

    from r

    inner join {prev_representatives} as repr
    on r.node_id = repr.node_id
        """

    return sql

def _cc_assess_exit_condition(representatives_name: str) -> str:
    """SQL exit condition for our Connected Components algorithm.

    Where 'in_play' (summarised in 'cc_update_representatives_first_iter')
    it indicates that some nodes still require updating and have not yet
    settled.
    """

    sql = f"""
            select count(*) as count
            from {representatives_name}
            where in_play
        """

    return sql

def solve_connected_components(
    nodes_table: SplinkDataFrame,
    edges_table: SplinkDataFrame,
    node_id_column_name: str,
    edge_id_column_name_left: str,
    edge_id_column_name_right: str,
    db_api: DatabaseAPISubClass,
    threshold_match_probability: Optional[float],
) -> SplinkDataFrame:
    """Connected Components main algorithm.

    This function helps cluster your linked (or deduped) records
    into single groups, which can then be more easily visualised.

    Args:
        linker:
            Splink linker object. For more, see splink.linker.

        edges_table (SplinkDataFrame):
            Splink dataframe containing our edges dataframe to be connected.

    Returns:
        SplinkDataFrame: A dataframe containing the connected components list
        for your link or dedupe job.

    """

    pipeline = CTEPipeline([edges_table, nodes_table])

    match_prob_expr = f"where match_probability >= {threshold_match_probability}"
    if threshold_match_probability is None:
        match_prob_expr = ""

    sql = f"""
    select
        {edge_id_column_name_left} as node_id_l,
        {edge_id_column_name_right} as node_id_r
    from {edges_table.physical_name}
    {match_prob_expr}

    UNION

    select
    {node_id_column_name} as node_id_l,
    {node_id_column_name} as node_id_r
    from {nodes_table.physical_name}
    """
    pipeline.enqueue_sql(sql, "__splink__edges")
    edges = db_api.sql_pipeline_to_splink_dataframe(pipeline)

    pipeline = CTEPipeline([edges])

    sql = f"select {node_id_column_name} as node_id from {nodes_table.physical_name}"

    pipeline.enqueue_sql(sql, "nodes")

    sql = _cc_generate_neighbours_representation()
    pipeline.enqueue_sql(sql, "__splink__df_neighbours")
    neighbours = db_api.sql_pipeline_to_splink_dataframe(pipeline)

    # Create our initial representatives table
    pipeline = CTEPipeline([neighbours])
    sql = _cc_generate_initial_representatives_table()
    pipeline.enqueue_sql(sql, "representatives")
    sql = _cc_update_neighbours_first_iter()
    pipeline.enqueue_sql(sql, "neighbours_first_iter")
    sql = _cc_update_representatives_first_iter()
    # Execute if we have no batching, otherwise add it to our batched process
    pipeline.enqueue_sql(sql, "__splink__df_representatives")

    representatives = db_api.sql_pipeline_to_splink_dataframe(pipeline)

    prev_representatives_table = representatives
    repr_name = "__splink__df_representatives"

    print("Initial representatives:")
    print(representatives.as_duckdbpyrelation().show())

    converged_repr_tables = []

    sql = f"""
    SELECT *
    FROM {representatives.physical_name} r
    WHERE r.representative IN (
        SELECT representative
        FROM {representatives.physical_name}
        GROUP BY representative
        HAVING bool_and(NOT in_play)
    )
    """

    pipeline = CTEPipeline([representatives])
    pipeline.enqueue_sql(sql, "__splink__df_representatives_marginal")

    marginal_converged_repr = db_api.sql_pipeline_to_splink_dataframe(pipeline)
    converged_repr_tables.append(marginal_converged_repr)
    print("Marginal converged representatives:")
    print(marginal_converged_repr.as_duckdbpyrelation().show())

    # Loop while our representative table still has unsettled nodes

    iteration, root_rows_count = 0, 1
    while root_rows_count > 0:
        start_time = time.time()
        iteration += 1

        # Loop summary:

        # 1. Update our neighbours table.
        # 2. Join on the representatives table from the previous iteration
        #    to create the "in_play" column.
        # 3. Assess if our exit condition has been met.

        # Generates our representatives table for the next iteration
        # by joining our previous tables onto our neighbours table.
        pipeline = CTEPipeline([neighbours])
        sql = _cc_generate_representatives_loop_cond(
            prev_representatives_table.physical_name,
        )
        pipeline.enqueue_sql(sql, "r")
        # Update our in_play column in the representatives table.
        sql = _cc_update_representatives_loop_cond(
            prev_representatives_table.physical_name
        )

        repr_name = f"__splink__df_representatives_{iteration}"

        pipeline.enqueue_sql(
            sql,
            repr_name,
        )

        representatives = db_api.sql_pipeline_to_splink_dataframe(pipeline)

        print(f"Representatives after iteration {iteration}:")
        print(representatives.as_duckdbpyrelation().show())

        pipeline = CTEPipeline()
        # Update table reference
        prev_representatives_table.drop_table_from_database_and_remove_from_cache()
        prev_representatives_table = representatives

        # Remove any converged representatives to make table smaller
        sql = f"""
        SELECT *
        FROM {representatives.physical_name} r
        WHERE r.representative IN (
            SELECT representative
            FROM {representatives.physical_name}
            GROUP BY representative
            HAVING bool_and(NOT in_play)
        )
        """
        pipeline = CTEPipeline([representatives])
        pipeline.enqueue_sql(sql, "__splink__df_representatives_marginal")

        marginal_converged_repr = db_api.sql_pipeline_to_splink_dataframe(pipeline)
        converged_repr_tables.append(marginal_converged_repr)
        # print("Marginal converged representatives:")
        # print(marginal_converged_repr.as_duckdbpyrelation().show())

        pipeline = CTEPipeline()
        # Check if our exit condition has been met...
        sql = _cc_assess_exit_condition(representatives.physical_name)

        pipeline.enqueue_sql(sql, "__splink__df_root_rows")

        root_rows_df = db_api.sql_pipeline_to_splink_dataframe(
            pipeline, use_cache=False
        )

        root_rows = root_rows_df.as_record_dict()
        root_rows_df.drop_table_from_database_and_remove_from_cache()
        root_rows_count = root_rows[0]["count"]
        logger.info(
            f"Completed iteration {iteration}, root rows count {root_rows_count}"
        )
        end_time = time.time()
        logger.log(15, f"    Iteration time: {end_time - start_time} seconds")

    pipeline = CTEPipeline()

    sql = " UNION ".join(
        [f"select * from {t.physical_name}" for t in converged_repr_tables]
    )

    pipeline.enqueue_sql(sql, "__splink__clustering_output")

    sql = f"""
    select representative as cluster_id,
    node_id as {node_id_column_name}
    from __splink__clustering_output
    order by cluster_id, node_id
    """

    pipeline.enqueue_sql(sql, "__splink__clustering_output_final")

    final = db_api.sql_pipeline_to_splink_dataframe(pipeline)

    return final

To identify stable clusters, the condition could be that the size of the cluster does not change between iterations

Good testing script:
import random
import time

import duckdb
import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
import pandas as pd

from splink import DuckDBAPI
from splink.internals.clustering import (
    # cluster_pairwise_predictions_at_multiple_thresholds,
    cluster_pairwise_predictions_at_threshold,
)

def generate_random_graph(graph_size, seed=47):
    if graph_size < 10:
        density = 1 / graph_size
    else:
        density = 2 / graph_size
    # print(f"Graph size: {graph_size}, Density: {density}")

    graph = nx.fast_gnp_random_graph(graph_size, density, seed=seed, directed=False)
    return graph

def nodes_and_edges_from_graph(G):
    edges = nx.to_pandas_edgelist(G)
    edges.columns = ["unique_id_l", "unique_id_r"]

    nodes = pd.DataFrame({"unique_id": list(G.nodes)})

    return nodes, edges

# Create 10 copies of nodes and edges with unique IDs
all_base_nodes = []
all_base_edges = []
unique_id_counter = 0
for size in range(1, 100):
    num_graphs = max(1, 20 - size // 5)
    for i in range(num_graphs):
        G = generate_random_graph(size)
        nodes, edges = nodes_and_edges_from_graph(G)

        # Update unique IDs
        nodes["unique_id"] += unique_id_counter
        edges["unique_id_l"] += unique_id_counter
        edges["unique_id_r"] += unique_id_counter

        unique_id_counter += len(nodes)

        all_base_nodes.append(nodes)
        all_base_edges.append(edges)
        # plot graph
        # nx.draw(G, with_labels=True)
        # plt.show()

base_nodes = pd.concat(all_base_nodes, ignore_index=True)
base_edges = pd.concat(all_base_edges, ignore_index=True)

all_nodes = []
all_edges = []

for _ in range(1, 20):
    offset = _ * unique_id_counter

    new_nodes = base_nodes.copy()
    new_nodes["unique_id"] += offset
    all_nodes.append(new_nodes)

    new_edges = base_edges.copy()
    new_edges["unique_id_l"] += offset
    new_edges["unique_id_r"] += offset
    all_edges.append(new_edges)

# Combine all copies
nodes = pd.concat(all_nodes, ignore_index=True)
edges = pd.concat(all_edges, ignore_index=True)

print(f"Final graph size: Nodes: {len(nodes):,.0f}, Edges: {len(edges):,.0f}")

db_api = DuckDBAPI(connection=":default:")
import logging

logging.basicConfig(format="%(message)s")
logging.getLogger("splink").setLevel(15)

clusters = cluster_pairwise_predictions_at_threshold(
    nodes,
    edges,
    node_id_column_name="unique_id",
    db_api=db_api,
)
# clusters.as_duckdbpyrelation().show()
clusters_ddb = clusters.as_duckdbpyrelation()
sql = """
with cluster_sizes as (
select
cluster_id, count(*) as cluster_size
from clusters_ddb
group by cluster_id)
select
cluster_size, count(*) as count
from cluster_sizes
group by cluster_size
order by cluster_size
"""
duckdb.sql(sql).show()

Superceded by https://github.com/moj-analytical-services/splink/pull/2437


Added xml tag at the beginning of xml files

I think its a good idea to add it, even though it is not mandatory.

Sure, why not.


Cannot hide the loading indicator

I'm trying to build this app that uses a third party library to access an API and thus does not use the Ajax component of atvjs.
When I use ATV.start() and pass to it the loadingTpl, it will show the loading indicator when I change page from the menubar, however even though by ready() method does call resolve(), the loading indicator is still being displayed.
How can I hide it?

Looks like it's the way I use JS to generate the template.
I've now used the boilerplate and modified my code to use .hbs templates and all works


fix: re-import duplicate, request name, and localhost with port #5

Fix localhost. #2
Cache user input URL
Rename request from operation_{mutation,query} -> operation
Fix duplicated after re-import

Thanks 🙏 this looks amazing!
Didn't have time to test it yet but caching and duplicate handling looks great! 👍

@gatzjames
Is there any update on this?
I can't use this awesome plugin.


load more than one module?

:load seems not to work with more than one module,
:load a.hs
:load b.hs

gives on the second load the cryptic message:
Failed to load module B: attempting to use module ‘main:A’ (a.hs) which is not loaded

(a and b doesn't reference each other at all)
:load a.hs b.hs as one can do in ghci doesn't work either, it seems to interpret this as the single filename "a.bs b.hs"

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/5861236-load-more-than-one-module?utm_campaign=plugin&utm_content=tracker%2F332399&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F332399&utm_medium=issues&utm_source=github).

This issue should be handled with #449.

Nice, thanks!


Create who-myth-buster.md

do I name this .md correct? I noticed that I could not see my previous pull request in the deployment.

Please use this form to add new resources. They will be automatically integrated within ~1 day after submission.
https://docs.google.com/forms/d/e/1FAIpQLSeq8S04bKTHPQ_SjUoagZGWGx4ymHIGlkXuKAxVAeZDZcPQWg/viewform

I have added the resource via form so this PR is no longer needed.


Rockets fetch from API

In this milestone, I have completed the following changes:

[ ] Created rockets basic structure issue #45.
[ ] Fetch Rocket from API issue #41.

Hello @lengushuru,
Thanks for the approval and investing time to review my PR.🎉🎉🎉


Cluster trust domain is hardcoded to cluster.local

See: https://github.com/istio/istio/issues/43673
cc @SirNexus

also @kdorosh


Characters sometimes (rarely) printed when using getchar

Please consider the following example:
from click import getchar
ESC = '\x1b'
i = 0
while True:
    c = getchar()
    i += 1
    if c == ESC:
        print("\n total characters typed:", i)
        exit()

(Press ESC to exit when running it)
When I run this in gnome-terminal under Linux User 3.13.0-37-generic #64-Ubuntu x86_64 x86_64 x86_64 GNU/Linux and randomly smash on the keyboard, the output is similar to the following:
~/Programming/click $ python click-test.py
dfsjakfh
total characters typed: 205
~/Programming/click-test $

I am actually trying to solve [this bug] (https://github.com/sloria/doitlive/issues/6) in case anyone is wondering why I am smashing on my keyboard ;)
So the problem is that a handful of characters gets printed instead of being catched by getchar(). I also suspect this is because of the nearly simultaneous key presses.
Does anyone know a solution to this problem?

Please also note that this does not happen with this code:
import os
import sys
import tty
f = open('/dev/tty')
fd = f.fileno()
tty.setraw(fd)
ESC = '\x1b'
i = 0
while True:
    c = os.read(fd, 32)
    #print(c.decode("utf-8"))
    i += 1
    if c.decode("utf-8") == ESC:
        print("total characters typed:", i)
        exit()

Please forgive the crappy example. You will have to reset your terminal after running it and you will not see that you're typing reset while doing so...
Could it be the case that all the [stuff in the getchar function] (https://github.com/mitsuhiko/click/blob/master/click/_termui_impl.py#L512) (especially the terminal configuration for every char) makes it so slow that it can't cope with all the characters coming in?

It seems it's impossible to fix this race condition without breaking the API.

Thanks for the quick reply. Do you know what exactly is causing this behaviour? Am I correct with the assumption that [tty.setraw(fd)] (https://github.com/mitsuhiko/click/blob/master/click/_termui_impl.py#L522) isn't set fast enough again between key presses?

Yes exactly, but "making it fast enough" is not a good fix. As said in the other issue, setraw should be done and undone only once each to avoid that race condition.

So introducing something like a function to enable/disable a "raw state" for the length of the desired getchar capture would fix this problem? Do you think that could work without breaking the API?

Well, if you want to use getchar without experiencing this bug, you'll have to change your code.
Right now you'd write:
while True:
    click.getchar()

To avoid this bug, you'd write:
# hypothetical API
with click.raw_terminal() as x:
    while True:
        x.getchar()

I don't think a bugfix that only changes code in Click is possible.

Allright, I'll use the solution you suggested over at the other issue for now and maybe look into the details here later if I have some time. Should we close this issue as not fixable in the meantime?

Let's keep this open.

@untitaker One more question: Is it intended behaviour that the echo function does only produce a newline and no carriage return? Because this leads to [this problem] (http://stackoverflow.com/questions/12231794/python-in-raw-mode-stdin-print-adds-spaces) when echoing in raw mode, so it's not possible to stay in raw mode while using echo. Would it break anything to add \r to the echo function?

That's definetly intentional. Sending \r looks like a workaround to me.
Please open a new issue for that if you think that's an issue Click should be
concerned with.
On Sat, Oct 10, 2015 at 02:53:12PM -0700, Stefan wrote:

@untitaker One more question: Is it intended behaviour that the echo function does only produce a newline and no carriage return? Because this leads to [this problem] (http://stackoverflow.com/questions/12231794/python-in-raw-mode-stdin-print-adds-spaces) when echoing in raw mode, so it's not possible to stay in raw mode while using echo. Would it break anything to add \r to the echo function?

Reply to this email directly or view it on GitHub:
https://github.com/mitsuhiko/click/issues/435#issuecomment-147128664

Fixed this in #1007


Enhance Nexus .type JSDoc with scalar info as found in GraphQL doc

Perceived Problem

Current JSDoc example: https://github.com/prisma/nexus-prisma/blob/c8b5f6cd644ae3fa077d5a1b1678e637fee4ae07/tests/unit/typescriptDeclarationFile/snapshots/modelScalarFields.test.ts.snap#L1204-L1207

Ideas / Proposed Solution(s)

Current GraphQL Doc example: https://github.com/prisma/nexus-prisma/blob/c8b5f6cd644ae3fa077d5a1b1678e637fee4ae07/tests/unit/graphqlSchema/snapshots/json.test.ts.snap#L5-L7
Improve JSDoc with some of the info we already are putting into the GraphQL doc.

For BigInt lets link to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt.


❓ [Question] Is there support for optional arguments in model's forward()?

❓ Question
Is there support for optional arguments in model's forward()? For example, I have the following: def forward(self, x, y: Optional[Tensor] = None): where y is an optional tensor. The return result is x + y if y is provided, otherwise just x.
What you have already tried
I added a second torch_tensorrt.Input() in the input spec, then at inference time got the error:
Expected dimension specifications for all input tensors, but found 1 input tensors and 2 dimension specs
I then removed the Optional annotation and just pass in None or the actual tensor for y. When None is passed in, I got the error: RuntimeError: forward() Expected a value of type 'Tensor' for argument 'input_1' but instead found type 'NoneType'.
I also tried passing in just 1 argument for x, and got:
RuntimeError: forward() is missing value for argument 'input_1'
Environment

Build information about Torch-TensorRT can be found by turning on debug messages

PyTorch Version (e.g., 1.0): 1.10.0+cu113
CPU Architecture:
OS (e.g., Linux): Ubuntu 18.04
How you installed PyTorch (conda, pip, libtorch, source): pip
Build command you used (if compiling from source):
Are you using local sources or building from archives:
Python version: 3.7.11
CUDA version: 11.1
GPU models and configuration: Tesla V100 with 32GB memory
Any other relevant information:

Additional context

@lhai37  I don't think we support optional tensors at the moment. @narendasan  to confirm. We expect inputs and outputs of a module to be torch::Tensors. Can you share how your torchscript model looks like ? I tried to convert the following to TS but torch.jit.script fails on this
class Optional(torch.nn.Module):

    def __init__(self):
        super(Optional, self).__init__()

    def forward(self, x, y: Optional[torch.Tensor] = None):
        return x + y

model = Optional()
scripted_model = torch.jit.script(model)

@peri044 Your code fails because you are doing x + None. If you add an if statement to prevent that, then it will work in TorchScript.

This is not currently supported and requires the next phase collections feature (#629). The issue is we need to be able to generate the torchscript code to manage mapping from function input to tensorrt input when potentially any arbitrary input could be None.


it keeps giving me an error "please upload either a minecraft's skin or refresh the page"

it keeps giving me an error "please upload either a minecraft's skin or refresh the page"

Same

SameSame

Now MineSkin has many request for sign a value and signature, it cause a problem on SkinSyste, SkinFile Generator and SkinsRestorer for /skin  command.

https://github.com/riflowth/SkinFile-Generator/pull/10 could possibly help increase the success rate of uploads


Compute AotProxyHints for Spring HATEOAS.

This PR computes required ClassProxyHints for Spring HATEOAS based on user code and spring jars.
It contains a dedicated Spring Boot sample application using HATEOAS and the required changes to NativeContext that allow dynamic registration of ClassProxyDescriptor and includes fixes for potential NPEs in Method.isAtController() and proxySuperclass lookup.
Related to: #356

Sorry for being late to merge this, but could you please rebase it to use the new names JDK/AOT proxy names?

Merged via 1a2059fbf295bedf72e10996c4cd2065f47e5013.


Update spring-cloud-quickstart-launch-app-maven to use the official release

@yucwan : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@jpconnock similar to https://github.com/MicrosoftDocs/azure-docs-pr/pull/94334, this pr is updating maven plugin version to the official release. Please help merge this pr as well.

#sign-off

@yucwan: I'm sorry - only the author of this article, @v-vasuke, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

I'll fold this change into my PR.  Thanks @yucwan

I've pushed this into a PR that is to be published on Monday.  https://github.com/MicrosoftDocs/azure-docs-pr/pull/94207
Feel free to close this one.  Thanks again!


Use enablement for commands in the extension

https://code.visualstudio.com/updates/v1_35#_command-enablement

Run Test File should be disabled when not in a test file.

Run Cells commands could be disabled when no cells are defined.

cc @rchiodo @IanMatthewHuff

Actually I just noticed that cells commands only appear if a cell is defined, so no need to consider that here

"Run Selection/Line in Terminal" and "Run Selection/Line in Terminal" commands could be disabled when empty lines are selected (since we already don't send anything for empty selections)

@luabud I guess it would make sense to not run on an empty file, but I don't think it's worth the complexity. As for the "rename symbol", I think that one also comes down to complexity as that would be dependent on parsing the selection to figure out if more than one symbol was chosen and we don't have a way to do that from TS right now without asking the language server to do it for us (we would have to write a Python parser in TS otherwise)

Agreed, not worth the complexity. Also, does the extension check if the selection is empty after the "run selection/line in terminal" command is selected? if so, we may also be adding unnecessary overhead if we implement the first two bullet points from my comment above

@luabud don't know about the empty selection case.

"Rename Symbol" could be disabled if cursor is on an empty line

This is a VS Code command. We don't create this, we just handle it indirectly.

Should we turn off "Sort imports" if you have Python 2.7 selected due to #2022 ?

sounds good, as long as we document that sort imports are not supported with Python 2.7

@luabud I think there's a comment about the bug in the docs, but I didn't check.

@brettcannon we should add it here: https://code.visualstudio.com/docs/python/editing#_sort-imports

Requirements

Disable Run Selection/Line in Terminal when line is empty
Disable Run Test File when not a test file

Prescribed Solution

Add necessary when clauses to check if line is empty
Add necessary context value when a test file has been activated in the editor.

We have since then used when clauses to hide commands when not relevant, testing related commands though are now managed by VSCode, and other commands mentioned here are mostly removed.


DSI Laptop Clean

Please check up the documents on the OneDrive for the DSI laptops. I have sent the credentials   on  for access to the laptops

Done


Delete orphaned configuration file

The config file target/dovecot/dovecot.conf is not used to build the
docker-mailserver image. All tests are working without this file. The
additional config file config/dovecot.cf won’t be loaded if
target/dovecot/dovecot.conf is used.

https://github.com/tomav/docker-mailserver/issues/212


Missing hid_error() return type declaration

hid_error() return type is not specified.
The following line has to be added (at line 60) :
hidapi.hid_error.restype = ctypes.c_wchar_p

@Nicolasp38 I don't currently have anything to test this fix with, so if you could please give the latest master a try and confirm it is working properly I will increment the version number and publish it to pypi.
Thanks for reporting this issue.
Austin

This is ok for me.


Versioned Clientlibs Transformer require  tag to include type attribute, which is not recommended

Required Information

[x] AEM Version, including Service Packs, Cumulative Fix Packs, etc: 6.5.9
[x] ACS AEM Commons Version: latest
[x] Reproducible on Latest? yes

The current recommendation from the w3c regarding script and link tags is to NOT include a type attribute in them.
From link:

The common use of this attribute is to define the type of stylesheet being referenced (such as text/css), but given that CSS is the only stylesheet language used on the web, not only is it possible to omit the type attribute, but is actually now recommended practice.

Still, in the current version of com.adobe.acs.commons.rewriter.impl.SaxElementsUtils does not match up this recommendation for <link> tags. The behavior for the script tags seems correct since https://github.com/Adobe-Consulting-Services/acs-aem-commons/commit/d71a9a9d538854bc9f34efc776b34bd299728336 but it was not changed for the <link> tag
Expected Behavior
The Versioned Clientlib transformer should pick up <link> tags without the need of the type attribute if rel="stylesheet" is provided.
Actual Behavior
The <link> tag is only picked up when type="text/css" is present.
Steps to Reproduce
have a <link> tag in the form <link rel="stylesheet" href="/etc.clientlibs/path/to/clientlib.css">

pushed a PR regarding this https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/2868

thanks for the quick merge


docs: update functions page

PR Checklist

[x] A description of the changes is added to the description of this PR.
[x] If there is a related issue, make sure it is linked to this PR.
[x] If you've fixed a bug or added code that should be tested, add tests!
[x] If you've added or modified a feature, documentation in docs is updated

Description of changes
This PR updates the Functions docs page with more explanations and additional examples. This will close #140

@MrPowers - this is ready for review. looks like there's a test failing related to auth permissions (?) -- is that something I should fix on my side?

you're good @avriiil, this is broken workflow that we'll fix

This is fun:
bin/uc function create --full_name unity.default.myPythonFunction --data_type INT --input_params "n_sides int, n_dice int" --language "python" --def "import numpy as np\nrolls=np.random.randint(1, n_sides + 1, size=n_dice)\nreturn print(rolls)"

But is this considered good or bad?  I actually don't know.


Multi User Chest has an issue recovering tomb stones

In multi player I often encountered that getting your equipment from a tombstone is buggy when using MultiUserChest (together with Epic Loot?). I also posted a but to EL discord, but I guess Randy does not work on it anymore again...
I really had to literally remove multi user chest mod, then the bug was gone. Otherwise the other player in the area of the tomb stone had to leave the area, so the owner of the tomb stone could open it again. The behavior was that when pressing E key to open, nothing happened. But an exception was thrown:
[Error  : Unity Log] InvalidOperationException: Collection was modified; enumeration operation may not execute.
Stack trace:
System.Collections.Generic.Dictionary`2+KeyCollection+Enumerator[TKey,TValue].MoveNext () (at <695d1cc93cca45069c528c15c9fdd749>:0)
EpicLoot.Data.ItemInfo.LoadAll () (at <3445a66d20694f088e28b9e0c6e99344>:0)
EpicLoot.Data.ItemInfo.IsStackableWithOtherInfo (EpicLoot.Data.ItemInfo info) (at <3445a66d20694f088e28b9e0c6e99344>:0)
EpicLoot.Data.ItemInfo.IsStackable (ItemDrop+ItemData a, ItemDrop+ItemData b) (at <3445a66d20694f088e28b9e0c6e99344>:0)
EpicLoot.Data.ItemInfo.CheckItemDataStackableAddItem (Inventory __instance, ItemDrop+ItemData item, System.Int32 x, System.Int32 y, System.Collections.Generic.Dictionary`2[System.String,System.String]& __state, System.Boolean& __result) (at <3445a66d20694f088e28b9e0c6e99344>:0)
(wrapper dynamic-method) Inventory.DMD<Inventory::AddItem>(Inventory,ItemDrop/ItemData,int,int,int)
Inventory.MoveAll (Inventory fromInventory) (at <0a1a436be26b421bb46752e573a7aa2f>:0)
MultiUserChest.InventoryHelper.MoveAll (Inventory from, Inventory to) (at D:/Projects/No-Chest-Block/MultiUserChest/Helper/InventoryHelper.cs:145)
MultiUserChest.InventoryHelper.GetAllMoveableItems (Inventory from, Inventory to, MultiUserChest.InventoryHelper+MoveAction moveAction) (at D:/Projects/No-Chest-Block/MultiUserChest/Helper/InventoryHelper.cs:135)
MultiUserChest.InventoryHelper.GetAllMoveableItems (Inventory from, Inventory to) (at D:/Projects/No-Chest-Block/MultiUserChest/Helper/InventoryHelper.cs:141)
MultiUserChest.ContainerHandler.TakeAll (Container container, Inventory playerInventory) (at D:/Projects/No-Chest-Block/MultiUserChest/ContainerHandler.cs:10)
MultiUserChest.Patches.ContainerPatch.TakeAllPatch (System.Boolean& __result, Container __instance, Humanoid character) (at D:/Projects/No-Chest-Block/MultiUserChest/Patches/ContainerPatch.cs:103)
(wrapper dynamic-method) Container.DMD<Container::TakeAll>(Container,Humanoid)
(wrapper dynamic-method) TombStone.DMD<TombStone::Interact>(TombStone,Humanoid,bool,bool)
(wrapper dynamic-method) Player.DMD<Player::Interact>(Player,UnityEngine.GameObject,bool,bool)
(wrapper dynamic-method) Player.DMD<Player::Update>(Player)

Maybe you can have a look if there is something you can do on this side? I would really love to make use of the mod again.
Maybe tomb stones could be excluded from any logic that MUC does add to the game?

This is a bug in EpicLoot duo to an outdated DataManager implementation. I will try to contact Randy and Vapok again about this, it's a known issue for quite some time now.
Otherwise I don't want to disable features of my mod because another mod throws errors, at most the tomb stones could be ignored if EpicLoot is installed.

Seems to be fixed in EpicLoot 0.9.9


added basic disambiguation of same name files

This PR introduces some visual syntactic sugar around same name files from different folders.

Every path created is truncated to 20 characters, which is completely arbitrary and can be discussed/changed*

When two same-name files are open, a path hint will appear for as long as multiple same name files are opened
When two files from the same workspace are open, the names of the folders (full path minus workspace root) are used
When a file from outside of the workspace is opened, the full path is used
When a file at the workspace root is opened, ./ is used

The implementation is not as efficient as it could be (some clones and a lot of conversions), but I am no Rust expert, so feel free to show me how to improve :)
closes #551

The file-counting logic should maybe go in update instead of layout, and it should be a field on data somewhere so that it can be reused. However, I think we can avoid having the hashmap and string allocations at all.
Most uses of the editor would have significantly less than even 100 editors open and so just iterating over each of them to check if they have a duplicate filename seems like it would be better (compared to allocating a bunch of strings).
Ex:
for (i, child) in editor_tab.children.iter().enumerate() {
    // the variables defined at the start
    match child {
        EditorTabChild::Editor(view_id, _, _) => {
            let editor = data.main_split.editors.get(view_id).unwrap();
            if let Some(file_name) = path.file_name() {
                if has_other_editors_with_same_name(data, editor_tab, i, file_name) {
                    // though, file_path could perhaps be an Option<String>?
                    file_path = format!(" {}", self.get_effective_path(data.workspace.path.clone()));
                }
            }
        }
    }
}

// define some separate helper function if you want
/// Checks if there is another editor with the same file name
fn has_other_editors_with_same_name(data: &LapceTabData, editor_tab: &LapceEditorTabData, index: usize, file_name: &OsStr) -> bool {
    for (i, child) in editor_tab.children.iter().enumerate() {
        // Skip past itself
        if i == index {
            continue;
        }

        match child {
            EditorTabChild::Editor(view_id, _, _) => {
                let editor = data.main_split.editors.get(view_id).unwrap();
                if let BufferContent::File(path) = &editor.content {
                    if let Some(child_filename) = path.file_name() {
                        if child_filename == file_name {
                            // There is a copy
                            return true;
                        }
                    }
                }
            }
            EditorTabChild::Settings(_, _) => {}
        }
    }

    // There was no copy
    false
}

(Note: The above is untested, I wrote it out from looking at the existing code)

The file-counting logic should maybe go in update instead of layout

I should have read the comments before adding my own. Sorry about that :')

Wait, I have reviewed this PR already? Can we move forward on this?

@bugadani there was some comments about the way to store and handle / compute the data that I never got around to resolve in the end. If someone wants to take care of it then by all means. Otherwise, I'll revisit and complete this PR in a few weeks, as I want to finish PSP first.

Let me give it a shot.

Let me give it a shot.

Gladly! thanks a lot @hbina

Do i need to create a new PR or can I reuse this somehow?

Do i need to create a new PR or can I reuse this somehow?

create new PR

@hbina Create a new PR but feel free to copy whatever you need from this PR :)

closing as #1521 got merged


[FEATURE]: Add support for Offline Install of CSM Operator in non OLM environment

Describe the solution you'd like
Add support for Offline Install of CSM Operator in non OLM environment.

Completed.


WIP: Downstream

Includes (most) of API and some registry changes.

/test all

/retest

/retest


Asim core

Coverage increased (+0.1%) to 96.608% when pulling f2c2cd78af14d7140af462f9e3112fed071e44eb on RSGInc:asim-core into dce856381bc4f4bed720200da62b808d86a5b53e on UDST:master.

we're not quite ready to accept this pull request since we need to update the docs and do a little more housekeeping, but I want to keep it open so we can discuss the improvements

Coverage increased (+0.1%) to 96.608% when pulling f2c2cd78af14d7140af462f9e3112fed071e44eb on RSGInc:asim-core into dce856381bc4f4bed720200da62b808d86a5b53e on UDST:master.

Re merging or not the pull request, it seems like it might be time to implement git-flow whereby there is a develop branch so that this can sit there while documentation is updated.  Leaving pull requests open for a long time is less desirable from my perspective.


[pulsar-client]Add a optional params scope for pulsar oauth2 client

Motivation
In some scenarios (e.g. azure cloud), when the client exchanges tokens with the server, an optional scope parameter is required, this pr fixes this issue, to ensure compatibility, when the user does not fill in this parameter, all behavior is the same as before.
Modifications

Add an optional parameter scope when exchanges token

Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] doc-required
https://github.com/apache/pulsar/issues/11932
(If you need help on updating docs, create a doc issue)

[ ] no-need-doc
(Please explain why)

[ ] doc
(If this PR contains doc changes)

/pulsarbot run-failure-checks

@tuteng Thanks for your contribution. Please do not forget to add docs accordingly to allow users to know your great code changes. And you can ping me to review the docs, thanks.

/pulsarbot run-failure-checks

@eolivelli please review @tuteng 's latest change.

/pulsarbot run-failure-checks


 doesn't set input-has-value class

Ionic version:
[x] 2.x
I'm submitting a ...
[x] bug report
Current behavior:
When an <ion-select> has a selected value, it doesn't set the input-has-value class on the surrounding <ion-item>. Other inputs such as <ion-input> and <ion-datetime> set that class when they have a value.
Expected behavior:
<ion-select>  should set the input-has-value class on the surrounding <ion-item> when it has a selected value.
Steps to reproduce:
See this plunker:
http://plnkr.co/edit/YM9m1E8F1AonrQhfYx44?p=preview
You'll notice that the .input-has-value { color: #FF0000;} style does not apply to the <ion-select> even though it has a value.
Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
 ordova CLI: 6.5.0
Ionic Framework Version: 2.2.0
Ionic CLI Version: 2.2.1
Ionic App Lib Version: 2.2.0
Ionic App Scripts Version: 1.1.4
ios-deploy version: Not installed
ios-sim version: Not installed
OS: Windows 10
Node Version: v6.9.5
Xcode version: Not installed

Thanks for opening an issue with us, we will look into this.


Assigned roles of the group need to show role's audience

Describe the issue:

Create roles with the same name in different audiences (eg: role1 in organization audience, role1 in different application audience)
Create a group
Assign the group to roles with the same name in different audiences.
Group-to-role assignments will be successful
Go to the Group's assigned roles view
Only the assigned roles names are shown
But the role audience is also required to distinguish the assigned roles. (API returns the audience data)

https://github.com/wso2/product-is/assets/25483865/29dcf047-af25-40d1-8f6d-7a474224c6f6

There is also a backend issue. In the new authorization runtime, when retrieving a group's roles, roles with similar names are assigned the same audience (You can see this in the screen shot).
Fix for backend issue : https://github.com/wso2-extensions/identity-inbound-provisioning-scim2/pull/509


Roll engine f0371e2f8f97..7ef587220aca (2 commits)

git@github.com:flutter/engine.git/compare/f0371e2f8f97...7ef587220aca
git log f0371e2f8f97..7ef587220aca --no-merges --oneline
2019-11-16<EMAIL_ADDRESS>Roll fuchsia/sdk/core/mac-amd64 from s9Vl7... to rX-Bh... (flutter/engine#13887)
2019-11-16<EMAIL_ADDRESS>Roll src/third_party/dart d6b0a27665..8617da2bf0 (1 commits)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

Auto-roller completed checks. About to merge.


Box2DWorld is a Node2D but has undefined behavior when its transform changes

This can be resolved in a few ways:
Solution 1.
Make Box2DWorld inherit Node. This fixes undefined behavior but isn't fun.
Solution 2.
Moving the node utilizes b2World.ShiftOrigin and all bodies stay in the same screen-space coordinates.
Perhaps there could be an option/property to either A. lock bodies to screen-space, or B. inherit Box2DWorld's translation changes. This might be stupid, I don't know.

Whats are the cons to 1?

Does solving in the Box2dWorld node solve it everywhere?  Does any transform in the scene tree (above or below the Box2DWorld node) have undefined behavoir?

What are the cons to 1?

If there's any opportunity to incorporate the functionality of b2World.ShiftOrigin in an intuitive way, I think this is it.
The first solution fixes the bug in the short term, but I'm not sure it'll be the final design.

what about scale and rotation?

Node2d at the root has a scale X of 2.

Screenshot editor

Screenshot play
Sprite under box2d doesn't scale X

Godot doesn't support scale for collision shapes, so maybe it's not unreasonable to leave out support for those types of transformations and just document it.

Sorry, my screenshots may be confusing.  i'm not scaling anything of the collision shapes.  I'm scaling the root node in the scene tree (Node2D).

forgive me I'm still getting up to speed with box2d_physics_body.cpp
I'm just looking through code there now and seeing how global transforms are done.  I may try and tweak a little.
Do we want to be doing anything with global_transforms?  Why not base everything local?  Or at lease relative to the Box2dWorld node?

Sorry, I didn't see the screenshots when I replied from my phone. I think I understand now.
At the moment, everything is using global transforms because local transforms make the following hierarchy impossible:
Box2DWorld
└─ (Node2D) PileOfBricks
└─ (several Box2DPhysicsBody nodes)
I currently aim to use neither of these transforms in the future and instead use "relative to Box2DWorld transform", though I don't know how that may interact with scaling and rotation. Unless there's a good use-case for rotating and scaling the world, I think it might be a bit complex to figure out.

Great.  That's what I was hoping for, as I think the easiest would be relative to box2dworld transform.
I mean, i still want to be able to 'zoom' in on my world from above the Box2DWorld node.  I definitley do this all the time.  The World node itself isn't scaled, but say a node above it.

SomeViewNode2D
└─ Box2DWorld
└─ (several Box2DPhysicsBody nodes)

It would be great to support local transforms in the tree under Box2DWorld, although I'm thinking how it would even work.  It would need to adjust shapes, and I'm considering the ramifications of that, wrecking contacts, etc.  Even too many b2Fixture::Synchronize will call performance problems.  Still would be cool though if there's an elegant solution.   I can't think of one right now though, that doesn't involve changing fixtures though.

Yeah, changing transforms would have potentially drastic implications. If I scale my PileOfBricks example mid-simulation, what happens? I'd have to see what Godot supports, and likely follow that. Though, I don't expect this is something it allows.

Exactly.  Ya i mean, godot doesn't support it.  It's just that it's apparently been a source of confusion for new developers in godot.
I mean the only saving grace, is that any modification would be outside a world->step().  I've thought about this a few times, and it continually boils down to moving shape vertices (or parameters for shapes like circle).  (or removing and re-adding fixtures with new adjusted vertices or parameters).  But at that point contacts are going to be blown out and collisions are going to be not great weird.

So Node2D is a necessity so that you can create instanced that can be placed throughout the world, but scaling should definitely not be supported.

Is that a real workflow people have?  I actually have no clue.  So you can 'group' Box2DPhysicsBody's together under a Node2D and move them all around in the editor?  Then you still have the issue of potentially moving that Node2D as well mid-simulation no?

Maybe there's a possibility to detect Node2D's between the Box2DWorld and Box2DPhysicsBody's and disable scale even in the editor, or even show a warning on that property or something.

I've modified the rigid bodies to use 'relative to world' transforms, instead of global.  This project just has a ball bouncing on a box.  In this video, I've scaled the world node, and I'm rotating it every frame in a script.  Transforms from the WorldNode and up the scene tree not longer have an impact on the transforms of the physics nodes.  And because the transform is relative to world, it still supports putting 'Node2D's in-between the Box2dWorldNode and Box2dPhysicsBody's.  Everything kinda works as expected.
See here:
https://youtu.be/PcussH-2q3c
Here is what it looks like un-scaled, and not rotating:

Thoughts @briansemrau ?

Here is again, the same project, but I am scaling the world node as well:
https://youtu.be/jKf4eTU42Xs

Am I correct in understanding that these different transforms are leaving the simulation untouched, but is still applying the scale/rotation/offset to the visual elements?
Also, does this lead to any visual bugs with the fixture shape editor handles?

It's a wip, and I'll clean it up and I'll verify the editor tools are working correctly next, but they will just work if I swap any set/get global xform funcs, to the world xform func.  But basically think of it this way.  Everytime you move from b2d to godot, instead of going to the global space, we go to the world space.  And vice versa.

So take any transformation from the Box2dWorld node and above in the scene tree (inclusive), and it won't change any actual placement of anything in the box2d simulation.

The editor tools should 'just work'.  Even if you are working in a node, and a root node higher in the tree is 'scaled' or anything, you will actually used the tools in the editor under a 'scaled' view.  But when we actually set the properties of the b2Body (pos, rotation), or shape vertices, they are always in coordinates in the world space.  So the skewed view at the editor level doesn't really change any values.
Will clean up, and submit something tomorrow 👍

Alright, sounds good. Thanks for figuring this one out!

Can probably close with with https://github.com/briansemrau/godot_box2d/pull/20


feat: list datasources

What type of PR is this?
/kind feature

What this PR does / why we need it
list datasources
Which issue(s) this PR fixes

Fixes #
Special notes for your reviewer

Please rebase the code, there may be some conflict with the latest code.


[O]llama support?

What do you think about adding support for locally running [o]llama? I'd be happy to see JSON prompt used with it. :)

Of course, it is a good idea to support Llama. I think it is a good idea to make a separate package, named something like "WWW::Llama". Then the LLM functions of this package can be just use Llama access configurations.

After reading the Llama's documentation it looks like Llama uses an Web API that is OpenAI compatible. Hence, just proper extensions of "WWW::OpenAI" have to be made to support Llama.

People might get confused with naming.
Anyway, off-topic, but don't know where it's better to ask. What is the right way to attach a file to chat request? I mean, for llm-vision I found that image-import function and the parameter is there. But for textual synthesis?

People might get confused with naming.

Yes. After reading the Llama documentation some more I think there should be a  "WWW::Llama" package.
BTW, "WWW::OpenAI" can work with Llama already, and this package has/supports a 'llama' configuration.

What is the right way to attach a file to chat request? I mean, for llm-vision I found that image-import function and the parameter is there. But for textual synthesis?

I assume you can just slurp and concatenate file's content. The function llm-synthesize also takes arrays of strings. (And concatenates them into one prompt.)

I assume you can just slurp and concatenate file's content. The function llm-synthesize also takes arrays of strings. (And concatenates them into one prompt.)

Works for small files. But I would like to analyze PDFs, for example. I'm getting promising results when submitting via their Web UI, but can't figure out how to do via API.

There are some discrepancies between what:

OpenAI's documentation says
API can do
Web Interface can do

I think in the use case you outlined -- feeding a large PDF file to an LLM -- you have to do one of the following:

Convert the PDF file to plain text and use a model that allows large number of tokens of the input (e.g. 128K)
Convert the PDF file pages into images, and feed the images to one of OpenAI's vision models
Use the texts of the PDF file pages with a chat object, progressively feeding them to the LLM

I think 1) is the easiest option. 3) requires to learn how to use llm-chat -- should be fairly easy, but not as straightforward as llm-synthesize.

The original request is implemented -- there is a dedicated "llama" configuration.
Here is example usage:
use LLM::Functions;
use LLM::Prompts;

my $prompt = llm-prompt("Yoda");
say $prompt;

say "-" x 120;
my $conf = llm-configuration('Llama', model => 'gpt-3.5-turbo');

my $chatObj = llm-chat($prompt, :$conf);

say $chatObj.eval("Who are you?" ~ llm-prompt('Translate')('German'));

LLMs of "llamafile" can be used via "WWW:OpenAI" by providing suitable base URL. This what the current implementation uses. Configuration-wise and function-wise nothing will change when the dedicated access package "WWW::LlamaFile" is implemented.

So, for now it's chat only? No way of attaching a file to message yet?

Anyway, thank you!

So, for now it's chat only? No way of attaching a file to message yet?

I am not sure should file-attachment be implemented for the functions of this package. It might be a good idea, but ultimately the file content is serialized at some point, so I would rather send those serializations with llm-synthesize and co.

As for the original request -- using llama file models -- I am working on it. And if file attachment is part of any of API's endpoints, then I will include it.

What are the files sizes you are dealing with? (Say, of a PDF file you would attach to the message to the LLM.)

It's doesn't really matter which way I do it. I'm not well familiar with the structure of the related packages, so might be asking dumb questions. :)
There is no particular limits on the kinds of files. So far, PDFs are likely. Might submit emails for estimating scam content probabilities. A side project is to wait until they get llava model support fixed and try attaching keywords/descriptions to images for better OS-level search.


json Decode Error

The Error is:
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\tkinter_init_.py", line 1892, in call
return self.func(*args)
File "C:\Users\Desktop\coding!Coding\Tkinter-Designer-master\tkinter_designer.py", line 29, in btn_clicked
backend.generate_code(token,URL, output_path)
File "C:\Users\Desktop\coding!Coding\Tkinter-Designer-master\backend.py", line 91, in generate_code
data = response.json()
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Link to Figma FIle:
https://www.figma.com/file/SmMH601IW47DIQ8RsyiMZQ/Untitled?node-id=0%3A1
Btw before this i got this error:
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\tkinter_init_.py", line 1892, in call
return self.func(*args)
File "C:\Users\Desktop\coding!Coding\Tkinter-Designer-master\tkinter_designer.py", line 29, in btn_clicked
backend.generate_code(token,URL, output_path)
File "C:\Users\Desktop\coding!Coding\Tkinter-Designer-master\backend.py", line 147, in generate_code
window_elements = fig_window["children"]
KeyError: 'children

Thanks for creating an issue, we are eager to solve it.

Hey @Vinyzuz, The JSON decode error is unexpectedly occurring today. And it goes away after a second time.
And let me see about the second.

Hey @Vinyzuz, The keyError children is due to there are no valid elements inside the frame. Add some text and it should work.
Also use the latest version.

Also I recorded the response and the JSON decoder error is caused because of The Figma server.
When this error occurs the status code is - 504 Gateway Timeout.
It means that there's no problem with Tkinter Designer. It would go away soon. Until then we just have option of retrying.

ok thanks

Hey @Vinyzuz , the json decode error
Must have solved now.

Same error now, even after updating tkd @ParthJadhav


[Fix #13284] Add new target_gem_version API

This allows cops that require a gem to handle multiple different versions of a gem at the same time.

For example in rack 3.1, new status code aliases were added that should not be handled on earlier versions.
Since this only looks at the lockfile, default gems are not considered.

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Isn't this quite similar to the existing require_gem API?

Is is as similar to requires_gem as target_ruby_version is to minimum_target_ruby_version/maximum_target_ruby_version. It allows you do handle multiple gem versions at the same time, in the same cop.
For example, in Rack 3.1, some status codes were renamed/added. I can write a cop that correctly handles either Rack < 3.1 or Rack >= 3.1, but not both. This allows you to check at runtime and decide what to do. Otherwise I'd have to write two cops since all I know with requires_gem is that some constraint is satisfied, but not with what version.

Ah, got it. Potentially this version checks could be part of the requires_gem API, but I can see some merit into having a separate method for this as well.

Can you elaborate on that? How would that work, or are you just saying that the method should have the same name?

I was thinking of something along the lines of (requires_gem gem_name, version_specifier), but I'm also open to having a dedicated method for this.

Fair point, I didn't think about that.


Command Palette background color doesn't update correctly when toggling with keyboard

Windows Terminal version
1.13.11432.0
Windows build number
10.0.25136.0
Other Software
No response
Steps to reproduce

Ctrl+Shift+P to open command palette
type color to select color scheme
toggle color schemes using up/down keys
the command palette background color doesn't update accordingly (scroll on the mouse scrollwheel seems to force it to update and you can see the difference)

Expected Behavior
No response
Actual Behavior

That's wild. I can't get anything like that to happen locally. Do you have animations enabled in the OS/? I honestly don't know what could possibly cause that...

@zadjii-msft

I can consistently reproduce this if I close all other windows and keep my desktop still.
That's definitely not an us issue, but maybe there's something we can do to manually trigger the XAML repaint here.


0.2.0 - better connection management, API improvements

Replace sliver with pluggable idStrategy approach
Update mutate to always return document (changed or not)
Make _indexes available during mutate
Add default timeout to ensure promises reject on connection failure
Use whistlepunk for logging
Add ability to specific # of connections per node to start with in the pool
Bug fix - connection pool doesn't cycle connections per node correctly
Bug fix - correct how defaults are applied to node settings
Simplify and clean-up log entries
Update/Cleanup README

Tests passing :+1: :+1: :+1: :fireworks:


Integrate build and dev process after hasura integration

this is to Integrate build and dev process after hasura integration

I setup change using generate as not git ignored file.
But it is not effective...
so I try to integrate hasura migration in setup too.


fixed #309 (Extra space around multiple scatter plot panels)

Thank you!
One further thing to consider: with all these frequent image updates the repository will at some point explode in size. In all the image-based tests, we should use the smallest sizes possible. Images are already relatively small, but we can further reduce the size in the future. No necessary to remake all of them now, but something to keep in mind for future PRs.
What do you think?


[stdlib] Constrain MutableCollection's subscript default implementation

This only provides the subscript(Range) -> Slice implementation when the collection's subsequence is actually a Slice.
(The diff is funny—I moved the subscript down, but it shows the change as moving the swapAt method up.)

@swift-ci Please Test Source Compatibility

Thanks, good spot!

@airspeedswift It occurred to me that this could cause source breakage, if some has passed myArray[2...] to a function taking a Slice, or has used a type annotation to explicitly choose this subscript. Should I do a deprecated/obsoleted/introduced dance so this can't break source in the Swift 4 timeline?

What if you also added a RangeExpression variant, wouldn't that resolve the issue?

Sorry, the issue is that if someone has this (weird) code, this change will break it:
func sumArraySlice(_ x: ArraySlice<Int>) -> Int { return x.reduce(0, +) }
func sumSlice(_ x: Slice<Array<Int>>) -> Int { return x.reduce(0, +) }

let numbers = [5, 10, 15, 20]
sumArraySlice(numbers[1..<4])
sumSlice(numbers[1..<4])       // <-- this stops working

@swift-ci Please smoke test


feat(scheduler): loadaware support nodemetric

close #22

Split relevant code changes into two commits (vendor + feature), PTAL @NickrenREN


Custom keyboards: Make input toolbar's height configurable.

Pull request checklist

[x] All tests pass. Demo project builds and runs.
[x] I have resolved any merge conflicts.
[x] I have followed the coding style, and reviewed the contributing guidelines. Confirmation: ✅

What's in this pull request?
This commit enables one to configure key aspects of the input toolbar and the keyboard controller from the outside, in order to be able to use a custom keyboard: see attached screencast of what this allows me to build.
Thanks for your great work @jessesquires! 🤗🤗

@jessesquires What's your Git workflow for this kind of PR? Should I duplicate a merge request on develop as well as on the release branch? Thx 😄

hmm, should It be against the develop branch?

Current coverage is 62.79% (diff: 40.00%)

Merging #1796 into release_7.3 will increase coverage by 0.13%

@@           release_7.3      #1796   diff @@
=============================================
  Files               63         63
  Lines             2378       2360    -18
  Methods            643        625    -18
  Messages             0          0
  Branches           153        153
=============================================
- Hits              1490       1482     -8
+ Misses             812        802    -10
  Partials            76         76

Powered by Codecov. Last update 929f1d8...2c52f16

Thanks @julien-c !

Should I duplicate a merge request on develop as well as on the release branch?

Actually, I don't think this is necessary on develop. Do you mind testing to see? If it's not needed, then I think I'd prefer to pass on this patch since the issue will be resolved in 8.0.

Sure, I will check. As 8.0 might still be a little ways into the future, I was hoping not to have to maintain my fork of the library, going forward. It is not that big of a deal though. Thanks!

Thanks @julien-c ! 👍

@julien-c How much effort do you need to build it by your own? I feel this is not too necessary to implement into library level as developers can handle it from the app level.

@julien-c @jessesquires sorry, accidentally press the close button.

Haven't heard from @julien-c for awhile, so going to close this.
I'm pretty sure we don't need this on develop since the KeyboardController is now dead. 😄

Sure @jessesquires @Lucashuang0802. You might want to merge 29ae478d2fea3b0dd6f077304c7b13d052757b2c though (which at least gives a little guarantee that the shipped Swift demo builds with the corresponding library)

Thanks. @julien-c, that's not a bad idea. Do you want to make a PR?

Yep -- I'm planning on redo-ing all the demo project stuff soon


Zig 0.14-dev: no std.rand.DefaultPrng

just tried to run the lib tests with the latest Zig master; that failed since this version of Zig now doesn't have a std.rand.DefaultPrng anymore. To fix this issue, I think you can just replace it with std.Random.DefaultPrng.
Otherwise: awesome library, thanks a lot for putting this out there!

since we're at it, you might want to add the new cache dir to .gitignore (.zig-cache) ;-)

Thanks :)


⚠ Turnos 24x7 has degraded performance

In e05be99, Turnos 24x7 (https://reservas-dev.alternativasinteligentes.com/) experienced degraded performance:

HTTP code: 200
Response time: 1119 ms

Resolved: Turnos 24x7 performance has improved in ced73d6.


how to generate file using fs

fs.writeFile('t1.pdf', pdf.stream)
I tried this but it seems not a right way to generate pdf file.

pdf.stream.pipe(fs.createWriteStream('path/to/my/file.pdf'));


Long Press | Button | Move key to Lock ctl key from being active

[ Before posting bugs, please check
https://code.google.com/p/hackerskeyboard/wiki/FrequentlyAskedQuestions and
the existing bugs for known issues - my responses may be delayed since I
have very little time to work on this project. ]

What steps will reproduce the problem?
1.type really fast
2.accidentally hit ctl
3.delete everything by hitting another key

What is the expected behavior? Data typed doesn't get lost.
What do you see instead? Data gets highlighted then deleted once the 'any key'
is pressed :(

What version of Hacker's Keyboard are you using, for example "1.37"? 1.37.1419..

On what phone or tablet? asus padfone X

If applicable, does this affect the 4-row or 5-row layout, or both? Which
language(s)? Both / All with ctl key.

Please provide any additional information below.

Please fix, its exaspersting!

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 13 Dec 2014 at 1:26

Duplicate of issue #487, this is fixed in v1.39.1 alpha.


How to replace nvim with nix-config built one?

I really like this project mainly due to how it doesn't try to implement every plugin as a module I feel I have more control that way.
I would like to try replace nvim with mine from nvim.buildNeovim. I have tried using overlays but without much luck mainly due to me not knowing nix well enough.
Both HM module nixpkgs.overlays and overlays.default
            ({ pkgs, ...}: {
              nixpkgs.overlays = [
                (final: prev: {
                  nvim = args.nix-neovim.buildNeovim {
                    configuration = ./nix/home-manager/nvim/configuration.nix;
                  };
                })
              ];
            })

doesn't seem to do anything can you please help me?

Wrong project sorry:)


Get a Free Trial Account on SAP Cloud Platform

Tutorial URL: https://developers.sap.com/tutorials/hcp-create-trial-account.html
Please specify the step you are referring to
buen dia. porque no se actualizan los porcentajes de los tutoriales si ya jan sido realizados?
gracias

What do you mean percentages? Where do you see them not updating?

Thank you for your feedback. We haven't heard from you in a while so we are closing the issue.
If you still have questions, feel free to reopen the issue.


Add dispcount:transaction/2 for running funs in watcher context

To avoid unnecessary calls and waiting for responses allow for a fun to
be passed as a transaction that can be run by the watcher. As soon as
the fun completes, the watcher is marked as free again without
intervention by the calling process. This should improve throughput for
use cases where the caller having direct access to the resource is
not necessary.
This still needs tests, but I have it working in the field.

Here's how I'm using it, with epgsql's async mode:
  prepared_query(Pool, Name, Params) ->
      Ref = make_ref(),
      Fun = fun(From, {Stmts, Conn}) ->
                    Statement = maps:get(Name, Stmts),
                    #statement{types = Types} = Statement,
                    TypedParameters = lists:zip(Types, Params),
                    %% construct the same kind of cast the epgsqla:prepared_statement does, but redirect
                    %% the output to the elli process directly
                    gen_server:cast(Conn, {{cast, From, Ref}, epgsql_cmd_prepared_query, {Statement, TypedParameters}})
            end,
      case dispcount:transaction(Pool, Fun) of
          ok ->
              receive
                  {_Conn, Ref, Res} ->
                      Res
              after
                  500 ->
                      throw(?RESPONSE_503)
              end;
          {error, busy} ->
              throw(?RESPONSE_503)
      end.

Sounds workable; might make sense to identify which request returns which results with a Ref, allowing people to parallelize calls.


Add equality autocorrection option to PredicateMatcher

I think it would be a good idea to use equality matcher when autocorrecting a PreficateMatcher to be explicit.
In our codebase we don't want to allow be matchers in order to keep things explicit and consistent, so we would like to use this cop to automatically correct any use of be_something or be true to eq true.
I was not sure if we should drop the previous version (be boolean), so I added this as an config option.

Before submitting the PR make sure the following are checked:

[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Added an entry to the changelog if the new code introduces user-observable changes.
[x] The build (bundle exec rake) passes (be sure to run this locally, since it may produce updated documentation that you will need to commit).

Thank you for the PR @tiagotex, but I’m afraid I disagree with the proposed change.
The documentation says

[rspec-expectations] also ships with two matchers that have more of a DSL feel to them:
expect(a).to be(b) # passes if a.equal?(b)
expect(a).to eq(b) # passes if a == b

I would recommend always using the be matcher when comparing booleans (and nils, and Numerics)


SA1003 fix not working for conditional directive

class Program
{
    static int Main(string[] args)
    {
#if! NOT
        return 1;
#endif
    }
}

When applying the fix at line 5 it adds a space right before 'return'.

Nice catch. :smile:


Feature: RRTA Integration

RRTA integration with things like BusFinder and maybe real-time route planning if the official app can be reverse-engineered.
python-rrta currently supports various myStop endpoints.

Closing this due to transit data finally being readily available via Google.


Collect pystats in subprocesses

There are two ways in which "real work" that we'd want to measure ends up in a subprocess.  Neither of these cases presently collects pystats in the subprocess.

Using the pyperf.Runner.bench_command API (eg. python_startup).  These should be handled automatically by this update to pyperf, we just need a new pyperf release and then update pyperformance and bench_runner to use it.

  These benchmarks will need to be individually updated to enable pystats in the subprocess, under the right circumstances.  It should basically amount to checking for --hook pystats on the commandline and then setting the envvar PYTHONSTATS=1 in the subprocess.  I can't think of a way to do that automatically for all of these, but there aren't that many, we should just update them individually.  (This also requires the pyperf update above, first).

Cc: @brandtbucher

The tricky thing for case two is that we don't really have a way of knowing if we've "caught" all of the Python subprocesses (especially the ones that use library code). We just need to either notice that the benchmark stats seem off, or know exactly what the benchmark is doing.
I think the current default for stats builds is to not collect stats by default, and require an environment variable to enable them at startup.
Could we flip that? So collect stats by default, but turn them off with an env var? That way all new subprocesses will collect stats by default, without us needing to do anything. And pyperf/pyperformance can explicitly disable stats collection at startup for its subprocesses and only turn them on around the invocation of the benchmark itself.
The one catch is that the main pyperformance invocation needs to be called with the env var set (to avoid collecting stats from pyperformance itself). But that seems straightforward enough to deal with - our benchmarking infrastructure can be updated to set everything correctly, and we could update pyperformance/pyperf to exit with a helpful error message if it's called on a stats build without the env var.

we could update pyperformance/pyperf to exit with a helpful error message if it's called on a stats build without the env var

I'm a little worried this isn't great UX, but it's a balance, right?
Just to throw an alternative out there, importing pyperf could monkeypatch subprocess.Popen and inject the environment variable turn stats on (if --hook pystats is passed in).  That's maybe a bit too magical.

We’re probably some of the only people who actually use the stats builds, so I doubt many people would be frustrated by the error message (or be surprised by the subprocess magic).
I still prefer the error message, I think. Not only a bit simpler and less error-prone, but I can’t think of a way to do the subprocess solution without “leaking” an additional stats file for the parent process.
I actually just thought of a third option... could the initial process call sys._stats_off(); sys._stats_clear()? As long as it’s not reenabled at some point, that should just write an empty file at process exit, I think.

I actually just thought of a third option... could the initial process call sys._stats_off(); sys._stats_clear()? As long as it’s not reenabled at some point, that should just write an empty file at process exit, I think.

Yeah, that's a good solution.


Verifying github

Unable to verify GitHub on account various tries no result

Someone fixed this on the back end thanks


Cold Structure placement

is there a way you could fix this?

Thank you for creating an issue.
This is unfortunately another bug in world generation. luckily we are currently working on a fix! (135.8.6 bug fix update)
This will be published soon.