Elasticsearch Ddb To ES Lambda - Not updating removed or edited items.

Describe the bug
I noticed when I remove items manually from DynamoDB the items on the ES index don't update at all and are left there. However adding items seems to work just fine.
To Reproduce
I'm not sure how one can reproduce it, I was using the latest amplify released a month ago if that helps but i'm unsure of what version I used.
If you would like logs or anything just let me know what logs from where and I will get them. As far as I can tell I looked around and couldn't find any errors regarding the deletion of an item.
Expected behavior
When removing items from Dynamo they should automatically update the ES index and update or remove the item.

From the people I have spoken with, so far no one knows why this is the case.

I've attached below the generated python script.
import base64
import datetime
import json
import logging
import os
import time
import traceback
from urllib.parse import urlparse, quote

from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
from botocore.credentials import get_credentials
from botocore.endpoint import BotocoreHTTPSession
from botocore.session import Session
from boto3.dynamodb.types import TypeDeserializer

# The following parameters are required to configure the ES cluster
ES_ENDPOINT = os.environ['ES_ENDPOINT']
ES_REGION = os.environ['ES_REGION']
DEBUG = True if os.environ['DEBUG'] == "1" else False
ES_USE_EXTERNAL_VERSIONING = True if os.environ['ES_USE_EXTERNAL_VERSIONING'] == "true" else False

# ElasticSearch 6 deprecated having multiple mapping types in an index. Default to doc.
DOC_TYPE = 'doc'
ES_MAX_RETRIES = 3              # Max number of retries for exponential backoff

logger = logging.getLogger()
logger.setLevel(logging.DEBUG if DEBUG else logging.INFO)
logger.info("Streaming to ElasticSearch")

# custom encoder changes
# - sets to lists
class DDBTypesEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, set):
            return list(obj)
        return json.JSONEncoder.default(self, obj)

# Subclass of boto's TypeDeserializer for DynamoDB to adjust for DynamoDB Stream format.
class StreamTypeDeserializer(TypeDeserializer):
    def _deserialize_n(self, value):
        return float(value)

    def _deserialize_b(self, value):
        return value  # Already in Base64

class ES_Exception(Exception):
    '''Capture status_code from request'''
    status_code = 0
    payload = ''

    def __init__(self, status_code, payload):
        self.status_code = status_code
        self.payload = payload
        Exception.__init__(
            self, 'ES_Exception: status_code={}, payload={}'.format(status_code, payload))

# Low-level POST data to Amazon Elasticsearch Service generating a Sigv4 signed request
def post_data_to_es(payload, region, creds, host, path, method='POST', proto='https://'):
    '''Post data to ES endpoint with SigV4 signed http headers'''
    req = AWSRequest(method=method, url=proto + host +
                    quote(path), data=payload, headers={'Host': host, 'Content-Type': 'application/json'})
    SigV4Auth(creds, 'es', region).add_auth(req)
    http_session = BotocoreHTTPSession()
    res = http_session.send(req.prepare())
    if res.status_code >= 200 and res.status_code <= 299:
        return res._content
    else:
        raise ES_Exception(res.status_code, res._content)

# High-level POST data to Amazon Elasticsearch Service with exponential backoff
# according to suggested algorithm: http://docs.aws.amazon.com/general/latest/gr/api-retries.html
def post_to_es(payload):
    '''Post data to ES cluster with exponential backoff'''

    # Get aws_region and credentials to post signed URL to ES
    es_region = ES_REGION or os.environ['AWS_REGION']
    session = Session({'region': es_region})
    creds = get_credentials(session)
    es_url = urlparse(ES_ENDPOINT)
    # Extract the domain name in ES_ENDPOINT
    es_endpoint = es_url.netloc or es_url.path

    # Post data with exponential backoff
    retries = 0
    while retries < ES_MAX_RETRIES:
        if retries > 0:
            seconds = (2 ** retries) * .1
            logger.debug('Waiting for %.1f seconds', seconds)
            time.sleep(seconds)

        try:
            es_ret_str = post_data_to_es(
                payload, es_region, creds, es_endpoint, '/_bulk')
            logger.debug('Return from ES: %s', es_ret_str)
            es_ret = json.loads(es_ret_str)

            if es_ret['errors']:
                logger.error(
                    'ES post unsuccessful, errors present, took=%sms', es_ret['took'])
                # Filter errors
                es_errors = [item for item in es_ret['items']
                            if item.get('index').get('error')]
                logger.error('List of items with errors: %s',
                            json.dumps(es_errors))
            else:
                logger.info('ES post successful, took=%sms', es_ret['took'])
            break  # Sending to ES was ok, break retry loop
        except ES_Exception as e:
            if (e.status_code >= 500) and (e.status_code <= 599):
                retries += 1  # Candidate for retry
            else:
                raise  # Stop retrying, re-raise exception

# Extracts the DynamoDB table from an ARN
# ex: arn:aws:dynamodb:eu-west-1:123456789012:table/table-name/stream/2015-11-13T09:23:17.104 should return 'table-name'
def get_table_name_from_arn(arn):
    return arn.split(':')[5].split('/')[1]

# Compute a compound doc index from the key(s) of the object in lexicographic order: "k1=key_val1|k2=key_val2"
def compute_doc_index(keys_raw, deserializer, formatIndex=False):
    index = []
    for key in sorted(keys_raw):
        if formatIndex:
            index.append('{}={}'.format(
                key, deserializer.deserialize(keys_raw[key])))
        else:
            index.append(deserializer.deserialize(keys_raw[key]))
    return '|'.join(index)

def _lambda_handler(event, context):
    logger.debug('Event: %s', event)
    records = event['Records']
    now = datetime.datetime.utcnow()

    ddb_deserializer = StreamTypeDeserializer()
    es_actions = []  # Items to be added/updated/removed from ES - for bulk API
    cnt_insert = cnt_modify = cnt_remove = 0

    for record in records:
        # Handle both native DynamoDB Streams or Streams data from Kinesis (for manual replay)
        logger.debug('Record: %s', record)
        if record.get('eventSource') == 'aws:dynamodb':
            ddb = record['dynamodb']
            ddb_table_name = get_table_name_from_arn(record['eventSourceARN'])
            doc_seq = ddb['SequenceNumber']
        elif record.get('eventSource') == 'aws:kinesis':
            ddb = json.loads(base64.b64decode(record['kinesis']['data']))
            ddb_table_name = ddb['SourceTable']
            doc_seq = record['kinesis']['sequenceNumber']
        else:
            logger.error('Ignoring non-DynamoDB event sources: %s',
                        record.get('eventSource'))
            continue

        # Compute DynamoDB table, type and index for item
        doc_table = ddb_table_name.lower()
        doc_type = DOC_TYPE
        doc_table_parts = doc_table.split('-')
        doc_es_index_name = doc_table_parts[0] if len(doc_table_parts) > 0  else doc_table

        # Dispatch according to event TYPE
        event_name = record['eventName'].upper()  # INSERT, MODIFY, REMOVE
        logger.debug('doc_table=%s, event_name=%s, seq=%s',
                    doc_table, event_name, doc_seq)

        # Treat events from a Kinesis stream as INSERTs
        if event_name == 'AWS:KINESIS:RECORD':
            event_name = 'INSERT'

        is_ddb_insert_or_update = (event_name == 'INSERT') or (event_name == 'MODIFY')
        is_ddb_delete = event_name == 'REMOVE'
        image_name = 'NewImage' if is_ddb_insert_or_update else 'OldImage'

        if image_name not in ddb:
            logger.warning(
                'Cannot process stream if it does not contain ' + image_name)
            continue
        logger.debug(image_name + ': %s', ddb[image_name])
        # Deserialize DynamoDB type to Python types
        doc_fields = ddb_deserializer.deserialize({'M': ddb[image_name]})

        # Sync enabled APIs do soft delete. We need to delete the record in ES if _deleted field is set
        if ES_USE_EXTERNAL_VERSIONING and event_name == 'MODIFY' and '_deleted' in  doc_fields and doc_fields['_deleted']:
            is_ddb_insert_or_update = False
            is_ddb_delete = True

         # Update counters
        if event_name == 'INSERT':
            cnt_insert += 1
        elif event_name == 'MODIFY':
            cnt_modify += 1
        elif event_name == 'REMOVE':
            cnt_remove += 1
        else:
            logger.warning('Unsupported event_name: %s', event_name)

        logger.debug('Deserialized doc_fields: %s', doc_fields)

        if ('Keys' in ddb):
            doc_id = compute_doc_index(ddb['Keys'], ddb_deserializer)
        else:
            logger.error('Cannot find keys in ddb record')

        # If DynamoDB INSERT or MODIFY, send 'index' to ES
        if is_ddb_insert_or_update:
            # Generate ES payload for item
            action = {'index': {'_index': doc_es_index_name,
                                '_type': doc_type,
                                '_id': doc_id}}
            # Add external versioning if necessary
            if ES_USE_EXTERNAL_VERSIONING and '_version' in doc_fields:
                action['index'].update([
                    ('version_type', 'external'),
                    ('_version', doc_fields['_version'])
                ])
                doc_fields.pop('_ttl', None)
                doc_fields.pop('_version', None)
            # Append ES Action line with 'index' directive
            es_actions.append(json.dumps(action))
            # Append JSON payload
            es_actions.append(json.dumps(doc_fields, cls=DDBTypesEncoder))
            # migration step remove old key if it exists
            if ('id' in doc_fields) and (event_name == 'MODIFY') :
                action = {'delete': {'_index': doc_es_index_name, '_type': doc_type,
                    '_id': compute_doc_index(ddb['Keys'], ddb_deserializer, True)}}
                es_actions.append(json.dumps(action))
        # If DynamoDB REMOVE, send 'delete' to ES
        elif is_ddb_delete:
            action = {'delete': {'_index': doc_es_index_name,
                                '_type': doc_type, '_id': doc_id}}
            if ES_USE_EXTERNAL_VERSIONING and '_version' in doc_fields:
                action['delete'].update([
                    ('version_type', 'external'),
                    ('_version', doc_fields['_version'])
                ])
            # Action line with 'delete' directive
            es_actions.append(json.dumps(action))

    # Prepare bulk payload
    es_actions.append('')  # Add one empty line to force final \n
    es_payload = '\n'.join(es_actions)
    logger.info('Posting to ES: inserts=%s updates=%s deletes=%s, total_lines=%s, bytes_total=%s',
                cnt_insert, cnt_modify, cnt_remove, len(es_actions) - 1, len(es_payload))
    post_to_es(es_payload)  # Post to ES with exponential backoff

# Global lambda handler - catches all exceptions to avoid dead letter in the DynamoDB Stream
def lambda_handler(event, context):
    try:
        return _lambda_handler(event, context)
    except Exception:
        logger.error(traceback.format_exc())

Hello @Rafcin are you using @key in your schema?

I have my schema structured where the main object that holds everything does not use a key. It has a connection but only the smaller objects in it have keys. The main object in this case is a restaurant that stores menu items and then the menu items have reviews and those using @key. In the end its the main object that i'm trying to update.

I tried to repro with the following schema
type Todo
  @model
  @searchable
  @key(name: "nameCreatedAt", fields: ["name", "createdAt"], queryField: "byNameCreatedAt")
{
  id: ID!
  name: String!
  description: String
  createdAt: AWSDateTime
  updatedAt: AWSDateTime
}

Though it's deleting records for me. Did you use the ddb_to_es.py script to backfill your data?

I haven't modified ddb_to_es.py in any way. For me it hasn't removed any records it simply adds records and never updates or deletes them if a change occurs in DDB. The only difference with the above schema I see is you use queryField, something I didn't add.
Here is a chunk of my Schema:
type Restaurant
@model
@searchable
@auth(
  rules:
    [
      { allow: owner }
      { allow: public, provider: iam, operations: [read] }
      { allow: private, provider: iam }
      { allow: groups, groups: ["User"], operations: [read] }
      { allow: groups, groups: ["RestaurantUser"], operations: [read,create,update] }
      { allow: groups, groups: ["Admin"], operations: [read,create,update,delete] }
    ]
)
{
  id: ID!
  restaurantName: String!
  restaurantAbout: String!

  restaurantAddress: String!
  restaurantRoute: String!
  restaurantLocality: String
  restaurantZip: String!

  restaurantPhone: String!
  restaurantWebsite: String
  restaurantEmail: String!

  restaurantOwner: [RestaurantOwner!]!

  location: Location!

  restaurantOpeningHours: [String]

  restaurantAmenities: [RestaurantAmenities!]!
  restaurantPaymentOptions: [RestaurantPaymentOptions!]!

  restaurantRating: Float

  restaurantTuesdaySpecials: [RestaurantSpecial] @connection(keyName: "byRestaurant", fields: ["id"])
  restaurantTuesdaySpecialsBlurb: String

  restaurantImageHeaders: [String!]!

  restaurantPostmates: String
  restaurantDoorDash: String
  restaurantUberEats: String
  restaurantGrubhub: String

  restaurantPlaceID: String!
}

type Location @aws_iam @aws_cognito_user_pools{
 lat: Float
 lon: Float
}

input LocationInput {
  lat: Float
  lon: Float
}

type RestaurantAmenities {
  value: String!
  label: String!
}

type RestaurantPaymentOptions {
  value: String!
  label: String!
}

type RestaurantOwner {
  name: String!
  ownerid: String!
  group: String!
}

type RestaurantSpecial
@model
@key(name: "byRestaurant", fields: ["restaurantSpecialID"])
@auth(
  rules:
    [
      { allow: owner }
      { allow: public, provider: iam, operations: [read] }
      { allow: private, provider: iam }
      { allow: groups, groups: ["User"], operations: [read,create,update,delete] }
      { allow: groups, groups: ["RestaurantUser"], operations: [read,create,update,delete] }
      { allow: groups, groups: ["Admin"], operations: [read,create,update,delete] }
    ]
){
  id: ID!
  restaurantSpecialID: ID!
  restaurantName: String!
  title: String!
  description: String!
  price: String!
  picture: String!
  reviews: [SpecialReview] @connection(keyName: "byRestaurantSpecial", fields: ["id"])
  restaurant: Restaurant @connection(fields: ["restaurantSpecialID"])
}

Same for me - deleting an item directly from DynamoDB console does not affect the elasticsearch indice

We are having this issue as well.
@SwaySway help will be appreciated.

I tried to repro with version 4.27 given the schema provided by @racfin.
I did remove the reviews portion as the SpecialReview type was not included but the script was still able to delete an item for me.
@idanlo @yonatanganot
If the delete is failing could you provide the log for the lambda? The name of the lambda follows this format
DdbToEsFn-<YOUR_API_ID>-<ENV>

I'll send it over in a bit just need to finish up some auth stuff.

Sorry it's late... I broke a few auth things... anyways I forwarded the logs to the email above!

Hello @Rafcin
I noticed the logs did not have any errors. Are there any logs where the script failed to delete an item?
If you used this script before ddb_to_es.py we recently fixed a bug to support the use of secondary indexes.
re https://github.com/aws-amplify/amplify-cli/issues/3786

Is this the new python script? I can switch it and see if it works. As far as I could see I never saw an error but I can check again

I just triggered a change to an object in Dynamo and in cloudwatch and the only lambda error was
[ERROR]	2020-08-17T18:50:38.946Z	193e9b01-76af-4d6a-9dae-022966564add	Traceback (most recent call last):
  File "/var/task/python_streaming_function.py", line 257, in lambda_handler
    return _lambda_handler(event, context)
  File "/var/task/python_streaming_function.py", line 251, in _lambda_handler
    post_to_es(es_payload)  # Post to ES with exponential backoff
  File "/var/task/python_streaming_function.py", line 106, in post_to_es
    es_errors = [item for item in es_ret['items']
  File "/var/task/python_streaming_function.py", line 107, in <listcomp>
    if item.get('index').get('error')]
AttributeError: 'NoneType' object has no attribute 'get'

What should be done in place of the older handler? The function requires a handler to be set

Sorry it was late and I had put the ddb_to_es.py in the streaming function...
anyways I updated the streaming function and its throwing some new errors.
[DEBUG]	2020-08-19T20:08:52.635Z	6309177d-8d7e-4b3e-aca5-26abcfcbbc71	Return from ES: b'
{
    "took": 10,
    "errors": true,
    "items": [
        {
            "index": {
                "_index": "restaurant",
                "_type": "doc",
                "_id": "TacoTuesdayRestaurant_ZZQyFMn4SJlAIRhygkummKsQHbWfjFc3dPgLvkKg",
                "status": 409,
                "error": {
                    "type": "version_conflict_engine_exception",
                    "reason": "[doc][TacoTuesdayRestaurant_ZZQyFMn4SJlAIRhygkummKsQHbWfjFc3dPgLvkKg]: version conflict, current version [1] is higher or equal to the one provided [1]",
                    "index_uuid": "MjgE7XpzS-uIG8qJB3IalQ",
                    "shard": "0",
                    "index": "restaurant"
                }
            }
        },
        {
            "delete": {
                "_index": "restaurant",
                "_type": "doc",
                "_id": "id=TacoTuesdayRestaurant_ZZQyFMn4SJlAIRhygkummKsQHbWfjFc3dPgLvkKg",
                "_version": 1,
                "result": "not_found",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 8,
                "_primary_term": 3,
                "status": 404
            }
        }
    ]
}
'

[ERROR]	2020-08-19T20:08:52.635Z	6309177d-8d7e-4b3e-aca5-26abcfcbbc71	ES post unsuccessful, errors present, took=10ms

[ERROR]	2020-08-19T20:08:52.635Z	6309177d-8d7e-4b3e-aca5-26abcfcbbc71	Traceback (most recent call last):
  File "/var/task/python_streaming_function.py", line 258, in lambda_handler
    return _lambda_handler(event, context)
  File "/var/task/python_streaming_function.py", line 252, in _lambda_handler
    post_to_es(es_payload)  # Post to ES with exponential backoff
  File "/var/task/python_streaming_function.py", line 107, in post_to_es
    es_errors = [item for item in es_ret['items']
  File "/var/task/python_streaming_function.py", line 108, in <listcomp>
    if item.get('index').get('error')]
AttributeError: 'NoneType' object has no attribute 'get'

So far none of the changes have fixed it.

Hello @Rafcin
Are you still experiencing issues with this?
I noticed the error provided in the error logs was a due to a version mismatch.
version_conflict_engine_exception
Do you have datastore features enabled?

Is anyone still dealing with this issue?

Hi sorry, I commented on this issue somewhere else, I ended up going into one of the templates and disabling versioning and it seems work as it should

Actually sorry I just checked again, an array of strings I changed did not update at all.

It seems that my dev branch with ES updates fine, however the production branch fails to see the changes. I've taken a look at both lambda functions and they have the right connections, the py script is the same so i'm not sure why this is

I think the issue is resolved, I looked at both lambda functions in my dev branch and production and they are exactly the same. It seems to work after the version conflict detection was disabled. I did some tests and the reason some images don't update after the change is because of the extreme caching I have setup for the production branch that has now been fixed.
@r0zar I would suggest disabling the conflict detection, if I remember correctly I had to go into my templates as well to forcefully remove it because every time I ran amplify push it complained.

I might have an idea why it's happened to you, I had also the same issue.
what I did I have change the name of the table so the old name still indexed in Kianna so I was thinking that it didn't
update the table data. but what I forgot to do is to create a new index in Kianna. therefore I saw old data which I thought that the table didn't update but I was needed to create the new index.  : )
I hope this helps.
In this picture I am about to remove the old index, assetsearch, and keep the new one. the new one have updated data correctly

Closing this issue as this is happening due to conflict versioning enabled.


fix fix wrong result of bitmap intersect functions

Proposed changes
Issue Number: close #22476
Agg result of bitmap intersect_count, orthogonal_bitmap_intersect and orthogonal_bitmap_intersect_count should set first_init to false after merging.
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.38 seconds
stream load tsv:          516 seconds loaded<PHONE_NUMBER>9 Bytes, about 138 MB/s
stream load json:         30 seconds loaded<PHONE_NUMBER> Bytes, about 74 MB/s
stream load orc:          70 seconds loaded<PHONE_NUMBER> Bytes, about 15 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select:          29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size:<PHONE_NUMBER>1 Bytes

run buildall


Clean fails to DROP AGGREGATE in SQL Server

What version of Flyway are you using?
Maven plugin 4.0.3
What database are you using (type & version)?
SQL Server 2014
What operating system are you using?
Windows 7
What did you do?
(Please include the content causing the issue, any relevant configuration settings, and the command you ran)
Created repeatable script to drop and recreate CLR functions. 1 of the functions is an aggregate function.

drop aggregate dbo.HarmonicMeanWithWeight
go

CREATE AGGREGATE [dbo].[HarmonicMeanWithWeight](@value [float], @Weight [int])
RETURNS[float]
EXTERNAL NAME [MathUtils].[Intelematics.Data.HarmonicMeanWithWeight]
GO
<<
When running clean the aggregate function does not get dropped. From the error it looks like flyway is calling drop function instead of drop aggregate.
What did you expect to see?
no errors when cleaning the schema.
What did you see instead?
[ERROR] Failed to execute goal org.flywaydb:flyway-maven-plugin:4.0.3:clean (default-cli) on project mwra-test-db: org.flywaydb.core.api.FlywayException: Unable to clean schema [dbo]: Can
not use DROP FUNCTION with 'dbo.HarmonicMeanWithWeight' because 'dbo.HarmonicMeanWithWeight' is a aggregate function. Use DROP AGGREGATE. -> [Help 1]

This has been fixed as part of the Flyway 5.0.0 work.


int4 not faster than fp16 and fp8

System Info
x86_64, Debian 11, L4 GPU
Who can help?
No response
Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

Build and benchmark bf16

python3 TensorRT-LLM/examples/llama/convert_checkpoint.py --model_dir ./llama-3.1-8b --output_dir ./tllm_checkpoint_1gpu_bf16 --dtype bfloat16

trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_bf16 --output_dir ./tmp/llama/8B/trt_engines/bf16/1-gpu  --gpt_attention_plugin auto  --gemm_plugin auto  --max_num_tokens 16384 --max_batch_size 8 --logits_dtype=float32

python3 TensorRT-LLM/examples/run.py --engine_dir=./tmp/llama/8B/trt_engines/bf16/1-gpu --max_output_len 1 --run_profiling --tokenizer_dir ./llama-3.1-8b --max_input_length=100000 --input_file batched_input.txt

Build and benchmark fp8

python TensorRT-LLM/examples/quantization/quantize.py --model_dir ./llama-3.1-8b --dtype bfloat16 --qformat fp8 --kv_cache_dtype fp8 --output_dir ./tllm_checkpoint_1gpu_fp8 --calib_size 512

trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_fp8 --output_dir ./tmp/llama/8B/trt_engines/fp8/1-gpu  --gpt_attention_plugin auto  --gemm_plugin auto  --max_num_tokens 16384 --max_batch_size 8 --logits_dtype=float32

python3 TensorRT-LLM/examples/run.py --engine_dir=./tmp/llama/8B/trt_engines/fp8/1-gpu --max_output_len 1 --run_profiling --tokenizer_dir ./llama-3.1-8b --max_input_length=100000 --input_file batched_input.txt

Build and benchmark int4

python TensorRT-LLM/examples/quantization/quantize.py --model_dir ./llama-3.1-8b --dtype float16 --qformat int4_awq  --awq_block_size 128  --output_dir ./tllm_checkpoint_1gpu_int4_awq   --calib_size 32

trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_int4_awq --output_dir ./tmp/llama/8B/trt_engines/int4_awq/1-gpu  --gpt_attention_plugin auto  --gemm_plugin auto  --max_num_tokens 16384 --max_batch_size 8 --logits_dtype=float32

python3 TensorRT-LLM/examples/run.py --engine_dir=./tmp/llama/8B/trt_engines/int4_awq/1-gpu --max_output_len 1 --run_profiling --tokenizer_dir ./llama-3.1-8b --max_input_length=100000 --input_file batched_input.txt

Expected behavior
int4 faster than fp8 faster than bf16
actual behavior
bf16
batch_size: 2, avg latency of 10 iterations: : 1.1307756423950195 sec

fp8
batch_size: 2, avg latency of 10 iterations: : 1.2754581451416016 sec

int4
batch_size: 2, avg latency of 10 iterations: : 1.2559791564941407 sec

additional notes
The batched_input.txt has 2 inputs of 2k tokens

@ShuaiShao93 Which version of TrtLLM do you use? Could you use the latest version?

@ShuaiShao93 Which version of TrtLLM do you use? Could you use the latest version?

Yes it’s the latest 0.14.0

Why do you only generate a single output token? With that, performance will be dominated by prefill compute, which is roughly the same for each quant mode, no?

The int4 weights will first be dequantized to fp16 to run actual computations

The int4 weights will first be dequantized to fp16 to run actual computations.

Doesn't L4 GPU have int4 tensor cores? Why do we have to dequant first?

Only fp8 is capable of avoiding that step

fp8 is not faster than fp16 either

Why do we have to dequant first

Doing the math in int4 would not be accurate enough to produce useful results. They are trying something like it with fp4 on Blackwell but it seems to have questionable quality there also

Why do we have to dequant first

Doing the math in int4 would not be accurate enough to produce useful results. They are trying something like it with fp4 on Blackwell but it seems to have questionable quality there also

I see, thanks for the explanation! Does this mean fp8 is likely to be the fastest option for now?

@aikitoria is right. Weight-only quantization has benefit on generation stage. Prefill stage comparison is unfair. Please benchmark with gptManagerBenchmerk.

I just noticed if I change --dtype float16 to --dtype bfloat16 in int4 quantize
python TensorRT-LLM/examples/quantization/quantize.py --model_dir ./llama-3.1-8b --dtype float16 --qformat int4_awq  --awq_block_size 128  --output_dir ./tllm_checkpoint_1gpu_int4_awq   --calib_size 32

it becomes much faster. Is this expected? It's faster than bf16 now


Add files via upload

GSQ Feature Status for review please.
Paul, can you have a critical look at the various concepts please, especially in relation to the status drop-down for GA's ASUD. Please bear in mind that each status has to be mutually exclusive of the others. I found this to be a problem in dealing with them. And the vocab needs to apply to ALL geologic features, not just lithostratigraphic ones.
Cheers,
John
PS: Although the list seems simple, I found it to be a little complex and quite problematical. And, the vocab may still need to be kicked around a bit (or more)!!

Hi John,
I am out at Zillmere for the rest of this week.  I will have a look at it next week.
Regards
Paul
From: johnmckellar<EMAIL_ADDRESS>Sent: Wednesday, 10 February 2021 4:33 PM
To: geological-survey-of-queensland/vocabularies
Cc: BLAKE Paul; Review requested
Subject: [geological-survey-of-queensland/vocabularies] Add files via upload (#368)
GSQ Feature Status for review please.
Paul, can you have a critical look at the various concepts please, especially in relation to the status drop-down for GA's ASUD. Please bear in mind that each status has to be mutually exclusive of the others. I found this to be a problem in dealing with them. And the vocab needs to apply to ALL geologic features, not just lithostratigraphic ones.
Cheers,
John
PS: Although the list seems simple, I found it to be a little complex and quite problematical. And, the vocab may still need to be kicked around a bit (or more)!!

You can view, comment on, or merge this pull request online at:
https://github.com/geological-survey-of-queensland/vocabularies/pull/368https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=GFEEQ-X_mXgygcuI9W1Xs-N_hTIOGBSKnlGxqxuzgo4&e=
Commit Summary

Add files via upload

File Changes

A vocabularies/gsq-feature-status.ttlhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368_files-23diff-2D466fde294ca9d2f1e0690f7f7ea9bf39de496c1d690ef30808a78db6a8842ad0&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=aOliiLntl7jdgH9lphPS17oGkN-WPRIVj2L-_woEX1U&e= (152)

Patch Links:

https://github.com/geological-survey-of-queensland/vocabularies/pull/368.patchhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368.patch&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=sdPvjlCufbEBWVxg3rII9_40aVWL9VQkgsR0DZFnK9g&e=
https://github.com/geological-survey-of-queensland/vocabularies/pull/368.diffhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368.diff&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=bh3L-jpJdodotFxYsUsgGb1RrPgp0FAY5bri0G0HYQM&e=

—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=GFEEQ-X_mXgygcuI9W1Xs-N_hTIOGBSKnlGxqxuzgo4&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_APDQOLE2ZYAAZCH7LZ2ZSCDS6ISBDANCNFSM4XMLXKEQ&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=vustP-zG_ChTEXvxpA-Brc8P5cFbxyP1Kikwp5PvokU&e=.

The information in this email together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. There is no waiver of any confidentiality/privilege by your inadvertent receipt of this material.
Any form of review, disclosure, modification, distribution and/or publication of this email message is prohibited, unless as a necessary part of Departmental business.
If you have received this message in error, you are asked to inform the sender as quickly as possible and delete this message and any copies of this message from your computer and/or your computer system network.

Thanks Paul. Not a problem. 👍

From: BlakePaul<EMAIL_ADDRESS>Sent: Thursday, 11 February 2021 7:51 AM
To: geological-survey-of-queensland/vocabularies<EMAIL_ADDRESS>Cc: MCKELLAR John<EMAIL_ADDRESS>Assign<EMAIL_ADDRESS>Subject: Re: [geological-survey-of-queensland/vocabularies] Add files via upload (#368)
Hi John,
I am out at Zillmere for the rest of this week. I will have a look at it next week.
Regards
Paul
From: johnmckellar<EMAIL_ADDRESS>Sent: Wednesday, 10 February 2021 4:33 PM
To: geological-survey-of-queensland/vocabularies
Cc: BLAKE Paul; Review requested
Subject: [geological-survey-of-queensland/vocabularies] Add files via upload (#368)
GSQ Feature Status for review please.
Paul, can you have a critical look at the various concepts please, especially in relation to the status drop-down for GA's ASUD. Please bear in mind that each status has to be mutually exclusive of the others. I found this to be a problem in dealing with them. And the vocab needs to apply to ALL geologic features, not just lithostratigraphic ones.
Cheers,
John
PS: Although the list seems simple, I found it to be a little complex and quite problematical. And, the vocab may still need to be kicked around a bit (or more)!!

You can view, comment on, or merge this pull request online at:
https://github.com/geological-survey-of-queensland/vocabularies/pull/368https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=GFEEQ-X_mXgygcuI9W1Xs-N_hTIOGBSKnlGxqxuzgo4&e=
Commit Summary

Add files via upload

File Changes

A vocabularies/gsq-feature-status.ttlhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368_files-23diff-2D466fde294ca9d2f1e0690f7f7ea9bf39de496c1d690ef30808a78db6a8842ad0&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=aOliiLntl7jdgH9lphPS17oGkN-WPRIVj2L-_woEX1U&e= (152)

Patch Links:

https://github.com/geological-survey-of-queensland/vocabularies/pull/368.patchhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368.patch&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=sdPvjlCufbEBWVxg3rII9_40aVWL9VQkgsR0DZFnK9g&e=
https://github.com/geological-survey-of-queensland/vocabularies/pull/368.diffhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368.diff&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=bh3L-jpJdodotFxYsUsgGb1RrPgp0FAY5bri0G0HYQM&e=

—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=GFEEQ-X_mXgygcuI9W1Xs-N_hTIOGBSKnlGxqxuzgo4&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_APDQOLE2ZYAAZCH7LZ2ZSCDS6ISBDANCNFSM4XMLXKEQ&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=qRX8HqO1_UuI31pwgXAgdn_OgEMPETe1FODnBRsTde4&m=4RptsTI-vTk7JGxlTsTHYn53OkbC5u0WArkucJSUJUQ&s=vustP-zG_ChTEXvxpA-Brc8P5cFbxyP1Kikwp5PvokU&e=.

The information in this email together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. There is no waiver of any confidentiality/privilege by your inadvertent receipt of this material.
Any form of review, disclosure, modification, distribution and/or publication of this email message is prohibited, unless as a necessary part of Departmental business.
If you have received this message in error, you are asked to inform the sender as quickly as possible and delete this message and any copies of this message from your computer and/or your computer system network.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_368-23issuecomment-2D777055723&d=DwMFaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=7PGzJsKGfMQg6r_oMXGYtVoxRstXoRB0iYsyk0FLe8U&m=0Xe9aSYtrMv1OTYZkQyGYDqBcGnrj66pPEuvQw0RGb4&s=jC13A3XvgwK_2yJMJH1J4Pf75Sb5wltk1HAb4176VLg&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMXRW2VWQX62LQHZMLQFK2TS6L5VLANCNFSM4XMLXKEQ&d=DwMFaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=7PGzJsKGfMQg6r_oMXGYtVoxRstXoRB0iYsyk0FLe8U&m=0Xe9aSYtrMv1OTYZkQyGYDqBcGnrj66pPEuvQw0RGb4&s=y04P0GBgGQtNwd6bEsfGJAx0O4I8rC6KaLEryz-Nmfk&e=.

Hi @johnmckellar .
This is a tricky one. This to me seems like at least two vocabularies. As you've mentioned in other discussions, getting this to fit logically with the GA statuses requires composite terminology. I imagine that the GA list has grown organically over time to fit the needs of systems with significant human intervention, but without the kind of ontological rigour as we/they would apply if building it in the present. Let's discuss on teams how to best handle this, I have some ideas, but need to bounce them around to see if they make sense.

Sorry.  I forgot about this.  The definitions look OK to me and will have a lot of overlap with the Rock Unit Status from MERLIN that will be needed.

Request is now stale. To arrange file to be reuploaded for fresh review.


Efficiently split an SFrame into K SFrames

Received a feature request:
Efficiently split an SFrame into K SFrames based on some mask containing K unique values.
For example, SFrame.split_by_value(['a','b','a','c']) will return a dict where the keys are the unique values of the given mask, and the values are SFrames containing only rows of data that correspond to the matching key in the mask.
Currently this can be done in O(NK) by iterating over the unique values of the mask, and passing a corresponding binary mask to the SFrame.
@karlavega

@alonpal  @karlavega Can we please setup a phone call with Pontis. Let us follow up with them and get some details on the usage pattern. This will help us look at how we want to implement this feature request
Thanks


Setup Cloud Build for CI

Set up Cloud Build for the repo CI.
Fixes #16.
Overall comments: Cloud Build seems quite good. It seems to be a little faster than GHA, partly because I think we no longer need to download gcloud in Cloud Build as the Cloud Build agent has authenticaiton built into it.

Would external contributors be able to see the build log?
Where is the configuration for when this runs? On every pull request? On branches?
Otherwise, this looks good.

Currently the trigger is configured to run on commits to this branch cloud-build. However I can change it so it runs on every commit to all branches so our PR branches would be covered.
For now we would make it so it doesn't run on user fork PRs (pending resolution of our question with security team).

It looks like the GCB run can be configured for "approval" within the trigger. So we'd likely have two pairs of triggers, almost identical, but one is for push events and one for pull_request events with an owner/collaborator commenting /gcbrun to invoke the trigger.

It looks like the GCB run can be configured for "approval" within the trigger. So we'd likely have two pairs of triggers, almost identical, but one is for push events and one for pull_request events with an owner/collaborator commenting /gcbrun to invoke the trigger.

Oh nice, this /gcbrun command is related the the security question Mike had about his concerns on external users triggering our builds through forks. I think this resolves the issues we had.

I've just updated the builds as you suggested - basically 1 trigger for PRs and 1 trigger for commits to master:

@ttomsu - Hey can you PTAL to see if everything looks good to you?


97 hint box

Summary
Closes #97
Hint box is positioned to the right and expands to the left. If overflow text is put in the box, it is automatically scrollable so the user can read everything.

Screenshots
** disregard the scroll bar, this is here because of my own chrome settings

It looks good, but can you take a look at if you can make it so that the scroll bar doesn't randomly appear/disappear when it doesn't need to? It causes the screen to jitter.
For example, with the first text box, it looks like this during the transition:

If you can't prevent the scroll bar from showing, I think it would be better to probably just remove the transition.

@Dankco I found a fix for the jitter by removing the scrollbar during the transition. it still shows up once the transition is over, but this helps with the overall look of the component. let me know if theres anything else to change!

One last thing (its mostly relevant for just scrolling manually though)

The scroll bar overlays the text (assuming this is to fix the resizing of the component), but this ends up blocking the text while scrolling. If you can't figure out a fix, we can just leave as it's fairly minor.


CDI 3.0.0 on Windows compile issues

Something is still broken in either  cdi_proj.sln project files for CDI 3.0 or libfabric_new project files.
This part of README:
In the libfabric_new folder, rename the libfabric.vcxproj, libfabric_new.vcxproj.filters and libfabric_new.vcxproj.user files to use libfabric_new instead of libfabric. This is done for several reasons. The AWS CDI SDK solution file uses them as a project source and linking to the generated libraries. It also provides a clear identification of the libfabric versions while using the AWS CDI SDK solution in Visual Studio.
BOLD part is probably typo?
Renaming libfabric_-> libfabric_new breaks Release compilation for cdi_proj.sln.
We can make a copy of project file with  libfabric_new instead, but then ReleaseDLL fails ( for new api libs).
It also could be better to explicitly specify  git version for libfabric_new/libfabric for Windows build, like it was in 2.4 instead of referencing Linux, since it Linux has many other git requirements.

Couple breadcrumbs for windows issue:
in cdi_proj, project view
lib_fabric.new says loads failed, on reload , project does not contain requested configuration Release|x65
rdmawin - the project of web cannot be found

opening libfaric_new in VS, and replacing Replacing Release(v142) with Release

adding efawin after removing rdmawin fixes the issue, at last I get through compile and running cdi_min_tx

Not sure what happened, but the correct solution file did not get pushed to GitHub. I pushed some changes to the developer_preview branch that should fix the build issues. Still testing and I will update this issue with final findings.

I just pushed changes to the developer_preview branch with this commit. Libfabric v1.15.2 is required in order to use the EFA adapter. There are some issues still being resolved with later versions of libfabric and use of the EFA adapter.


Documatation

PR checklist

[ ] This comment contains a description of changes (with reason).
[ ] If you've fixed a bug or added code that should be tested, add tests!
[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] Make sure your code lints (nf-core lint).
[ ] Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
[ ] Usage Documentation in docs/usage.md is updated.
[ ] Output Documentation in docs/output.md is updated.
[ ] CHANGELOG.md is updated.
[ ] README.md is updated (including new tool citations and authors/contributors).

@muffato The pipeline is release ready once this PR is merged. Can you do the review? Thanks.

DeepVariant also creates tabix indexes for the (g)VCF files. Can they be outputted / published too ?
They are not merged yet like the vcf files themselves. We can add that as a future feature.


⚠ Site ANTS has degraded performance

In cfaf067, Site ANTS (https://ants.gouv.fr) experienced degraded performance:

HTTP code: 200
Response time: 19671 ms

Resolved: Site ANTS performance has improved in a93d242.


INT-11 [BREAKING CHANGES] Migrate to ESM

@vladholubiev Could you please setup the CircleCI project for this repo?

@vladholubiev Could you please set up the CircleCI project for this repo?

done


Tags are not shared

Description
Tags are not shared between users.
When the folder is accessed by the user (with access rights to that folders), Tags set on the shared documents/folders, are not shared with the target user.
Example
admin Taged folder_1 with "Receipt" Tag
admin has shared folder_1 to user_1 (Allow | Full Control)
user_1 can see the tag on the folder, but do not have access to this tag in the Tag menu (can't pin it/edit it). (No Tags from admin are visible at user context)
user_1 cannot add new Tag to the shared folder. (Tag appears for a moment but dissipaters after refresh).
Expected
User (preferably with proper role) can view, add, and edit tags.
Actual

user_1 cannot see any tags in Tags tab
user_1 can see the tag on the folder, but do not have access to this tag in the Tag menu
user_1 cannot add new Tag to the shared folder. (Tag appears for a moment but dissipaters after refresh).

Info:

OS: https://hub.docker.com/r/linuxserver/papermerge
Browser Firefox 94.0.2 (64 bity)
Database N/A
Papermerge Version 2.0

Good catch! Thanks for reporting this bug!

You are welcome let me know if you would like to to help somehow.

I just stumbled onto the same problem (using Version v2.1.5-ls61). The problem even exists, when no folders are shared between users. When user 1 creates a bunch of tags, user 2 is not allowed to create this tag, but is also does not show up in the tag list.
However, I found a workaround:
User 2 just needs to assign the respective tag(s) manually on a document/file once. Then the tag appears in the tag list.


Setup mergify

ref https://github.com/tkf/Kaleido.jl/pull/30

Codecov Report

Merging #9 into master will decrease coverage by 0.81%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master       #9      +/-   ##
==========================================
- Coverage   90.69%   89.88%   -0.82%
==========================================
  Files           4        4
  Lines          86       89       +3
==========================================
+ Hits           78       80       +2
- Misses          8        9       +1

Impacted Files
Coverage Δ

src/Aqua.jl
71.42% <0%> (-28.58%)
:arrow_down:

src/exports.jl
100% <0%> (ø)
:arrow_up:

src/unbound_args.jl
100% <0%> (ø)
:arrow_up:

src/ambiguities.jl
89.23% <0%> (+1.35%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 648792f...5cb4050. Read the comment docs.


Use PyTorch 2.0 in Marqo[ENHANCEMENT]

Describe the solution you'd like
Marqo should use PyTorch 2.0 rather than 1.x.x. PyTorch promises better performance, that Marqo will benefit from.
One of the problems is building a multiarchitecture image with PyTorch 2.

need to investigate all the models to implement torch.compile()

@wanliAlex we don't have to use .compile for everything if for some reason it doesn't work. upgrade could be done in two stages if there are issues (apparently there shouldn't be).


Histograms: Thoroughly review all the code changes in the sparsehistogram branch

With a special focus on the TSDB part, and generally a critical eye on test coverage.

Note that there will be more test coverage with #11170 (for PromQL processing).
Also, test tooling needs to be improved to increase test coverage, see #11171.

Closing as discussed with @codesome .


Attempt to build SDKs on default runners

Unfortunately, pulumi-aws test (go) is still too big to run on default runners.


Performance Overlay Screen Placement

I am running ChimeraOS on the OneXplayer 2 and when I enable the performance overlay it is displayed in the middle of the screen instead of the top left of the screen.  I think it may be related to the portrait panel that is used for the screen.  I am just not sure where I can change this.

Can you switch to the unstable build and verify it's already been resolved?
Press CTRL ALT F3 to enter the TTY and login using,
user: gamer
pass: gamer
and then run sudo frzr-deploy chimeraos/chimeraos:unstable

Yes that worked.


Template the tomcat/carbon/WEB-INF/web.xml.j2 file

Purpose
Template the tomcat/carbon/WEB-INF/web.xml.j2 file and add default values for the configurable properties.
Part of the fix for the issue https://github.com/wso2/product-is/issues/10380

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/1410814484

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/1410814484
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/1410814484

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/1410814484
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/1531824807

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/1531824807
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/1531824807

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/1531824807
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/1531824807

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/1531824807
Status: success


CRYPTO-141: Avoid calling operations on an invalid context.

There are two main changes here.
The first one wraps the OpenSSL cipher context with some extra state
needed by the JNI wrapper. This allows checking whether it's safe to
call operations such as EVP_CipherUpdate, which can cause problems if
the context is not properly initialized. This more closely follows the
semantics of the Java interface.
The second change avoids cleaning up the OpenSSL context when an error
happens. Doing that would invalidate the memory used by the context,
but the Java code would still have the address of the now invalid
context, and calling another JNI method would crash the JVM. This
ties the lifecycle of the OpenSSL context to the Java cipher instance.
As part of those changes, all uses of EVP_CIPHER_CTX_cleanup were
removed; the only remaining cleanup paths use EVP_CIPHER_CTX_free,
which is its replacement.
There are a few more changes that are not required for the above but
helped with testing and debugging:

some very minor changes in Java code, especially using final fields
where they make sense.
enable -Wall and -Werror when compiling on linux64.
fixed some minor issues with error handling in the native code.

Coverage decreased (-0.07%) to 74.008% when pulling 23d63fa30d677368010fad5c6084c7008971c144 on vanzin:CRYPTO-141 into c5c6220c2509c4d12de3fc08d591f82b89f882a0 on apache:master.

@winningsix are you still working with this code? Would be good to get a second set of eyes on this.

@squito just in case you don't have enough to do and want to review some more stuff.

sorry @vanzin I think this would be a really hard change for me to review.  hopefully you can find somebody more familiar with this


cas usage issue

Can you leave a email please?
I have a question , hope you can help.
Thanks a lot.

Just leave your question here.

OK, here is my question.
I use the django-cas-ng as the client .
When I logout the site, I also logout from the server, but  the other client sites didn't logout cause they have their own cookies.
Now I want to fix this issue. I tried to get a logout func for all the loggined sites in the logout clicked site.
Am I right?
what did you do with this?

Sounds single sign out.
Check out CAS protocol for detail. The implement ion should based on CAS protocol.
http://jasig.github.io/cas/development/protocol/CAS-Protocol.html

On Jan 27, 2015, at 00:57, mygithub<EMAIL_ADDRESS>wrote:
OK, here is my question.
I use the django-cas-ng as the client .
When I logout the site, I also logout from the server, but the other client sites didn't logout cause they have their own cookies.
Now I want to fix this issue. I tried to get a logout func for all the loggined sites in the logout clicked site.
Am I right?
what did you do with this?
—
Reply to this email directly or view it on GitHub.

It's helpful, thanks a lot.

You are welcome.


2024-Tooltip-allignment-in-project-listing-page-is-right-alligned

Targets #2024

Description
Right alignment of Tooltip-velocity-in-project-listing-page-is-done

Checklist:

[x] I have performed a self-review of my code.

@chetanbhatt10 This PR has some merge conflicts, please fix them and assign for review,

I think PR is not needed as we already discussed that this is fixed. Hence, closing this.


VS2017 RC - Get correct options for scripts

Prior to this, we never update the compiler options stored for scripts.  This means that as you add #r or #load the compilation options aren't updated.
This PR updates the options in the semantic analysis phase.  We can't update them on every request for options because it is too expensive.
This fixes some aspects of https://github.com/Microsoft/visualfsharp/issues/1808  but the underlying problem of not reacting to file update events in dependencies is not fixed.

This build on #1855 hence the large number of commits


[BUG] After clicking Signup, the screen shows like it's freezed

Description

Before this page , when frontend was trying to establish connection, it does not show loading and nothing.
Instead, I did not know anything about this so clicked multiple times and  faced with some duplicate database.key error.
Screenshots

another error also it throws like which moves very fast so could not take one screenshot

Now this error

If needed, I can work on this over the next few days

@CerealPlayer should I assign you?

Sure!

Shall I take this issue? I can fix it before next week.

Sure, anyone who gets to it first can take it, to be honest. Make sure to communicate your progress as you go, though.

Yeah @Akshay1018 you can take care if you want 👍

@CerealPlayer @ykdojo  Thanks. I will update the progress.

@ykdojo will take this issue

The issue is still there

@ykdojo may I help with this issue? ,then you  can assign me


[BUG]: Unhandled exception

Description
I'm playing with ComfyUI lately and now, when I want to open a PNG with PR, I get an error.
Can't tell what happened. 🤔
Reproduction steps
Open image with PR
Image file

Version
1.3.3
OS
Windows
Tool
ComfyUI

Have you tried 1.3.4beta1? If the same bug still appears, please post the image here or send to my email.
2023年10月10日 +0800 AM6:27 Woisek @.***>，写道：

Description
I'm playing with ComfyUI lately and now, when I want to open a PNG with PR, I get an error.
Can't tell what happened. 🤔
Reproduction steps
Open image with PR
Image file
Version
1.3.3
OS
Windows
Tool
ComfyUI
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


Public accounts for publishing the extensions

Should we create new accounts to publish the apps in the appstore of FF an Chrome?
Or does someone want to use his/her existing account?

Please create a new account so that all of us can access it :)


Compiling out of a string

Hey, so say for example I have my code to compile as a string in my code, and I want to compile it without referencing a file: let code = "console.log";. Basically what I want to do is compile my code that is stored as a string, not as an individual JS file. Is this possible? If so, how can it be achieved?

Hi, if I get you right: you want to be able to load javascript code from strings (or even database entries ;)).
This will work. index.js
/** Needed to load modules from string (or even a database) */
function requireFromString(src, filename) {
   var Module = module.constructor;
   var m = new Module();
   m._compile(src, filename);
   return m.exports;
}
/** Your "string" code **/
let code = `exports.hello = function(a){
   console.log("Hello ", a);
}`;

var helloWorld = requireFromString(code,"");
//console.log(helloWorld);
helloWorld.hello("World");

package.json (here for linux)
{
  "name": "test",
  "bin": "index.js",
  "pkg":{
   "targets": ["latest-linux-x64"]
  }
}

Run
pkg .

... and have fun with
./test
# output will be
Hello World


UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'tarball' of undefined after 'npm install @fortawesome/fontawesome-svg-core'

Describe the bug
When doing a npm install @fortawesome/fontawesome-svg-core, verdaccio throws an
 UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'tarball' of undefined
at convertDistRemoteToLocalTarballUrls (/opt/verdaccio/build/lib/utils.js:209:89).

To Reproduce
Steps to reproduce the behavior:

run latest verdaccio/verdaccio docker image
set local npm to use verdaccio
npm install @fortawesome/fontawesome-svg-core
verdaccio throws and crashes
local npm times out with error

Expected behavior

verdaccio should be able to proxy/cache the @fortawesome/fontawesome-svg-core package without crashing.
Docker || Kubernetes (please complete the following information):

Docker verdaccio tag: latest

Configuration File (cat ~/.config/verdaccio/config.yaml)
storage: /verdaccio/storage/packages
plugins: /verdaccio/plugins

auth:
  htpasswd:
    file: /verdaccio/storage/htpasswd

security:
  api:
    jwt:
      sign:
        expiresIn: 60d
        notBefore: 1
  web:
    sign:
      expiresIn: 7d
      notBefore: 1

uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    access: $all
    proxy: npmjs
    storage: cached

  '**':
    access: $all
    proxy: npmjs
    storage: cached

middlewares:
  audit:
    enabled: true

logs:
  - { type: stdout, format: pretty, level: http }

Environment information

Environment Info:

  System:
    OS: Linux 4.4 Alpine Linux undefined
    CPU: (4) x64 Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz
  Binaries:
    Node: 10.16.3 - /usr/local/bin/node
    Yarn: 1.17.3 - /usr/local/bin/yarn
    npm: 6.9.0 - /usr/local/bin/npm

Debugging output

2019-12-12T10:47:45.796372+01:00 | npm-registry_app_1 |  http <-- 200, user: XXX(XXX via XXXX), req: 'GET /@fortawesome%2ffontawesome-svg-core', bytes: 0/0
2019-12-12T10:48:36.147890+01:00 | npm-registry_app_1 | (node:7) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'tarball' of undefined
2019-12-12T10:48:36.147913+01:00 | npm-registry_app_1 |     at convertDistRemoteToLocalTarballUrls (/opt/verdaccio/build/lib/utils.js:209:89)
2019-12-12T10:48:36.147920+01:00 | npm-registry_app_1 |     at Object.getPackageMetaCallback [as callback] (/opt/verdaccio/build/api/endpoint/api/package.js:39:65)
2019-12-12T10:48:36.147924+01:00 | npm-registry_app_1 |     at getPackageSynUpLinksCallback (/opt/verdaccio/build/lib/storage.js:336:17)
2019-12-12T10:48:36.147927+01:00 | npm-registry_app_1 |     at self.localStorage.updateVersions (/opt/verdaccio/build/lib/storage.js:603:9)
2019-12-12T10:48:36.147930+01:00 | npm-registry_app_1 |     at _readCreatePackage (/opt/verdaccio/build/lib/local-storage.js:210:9)
2019-12-12T10:48:36.147935+01:00 | npm-registry_app_1 |     at storage.readPackage (/opt/verdaccio/build/lib/local-storage.js:712:7)
2019-12-12T10:48:36.147939+01:00 | npm-registry_app_1 |     at _readStorageFile.then.res (/opt/verdaccio/node_modules/@verdaccio/local-storage/lib/local-fs.js:199:9)
2019-12-12T10:48:36.148398+01:00 | npm-registry_app_1 | (node:7) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 5)

installing the package directly from registry.npmjs.com without using verdaccio succeeds without problems.
maybe https://github.com/verdaccio/verdaccio/blob/a7172dc9afd6fad91e582473a6301518aa7dffed/src/lib/utils.ts#L172 should rather be
if (distName && distName.tarball)

instead of strictly checking for !== null.
Seems distName can be undefined too for some packages/versions, but I'm not able to debug it further in the docker image, somehow my code changes are not applied / verdaccio doesn't respond anymore if I change some code within the container and restart it.

any update on this? is it verdaccio issue or font-awesome?

manually removing the downloaded package from the verdaccio cache folder and re-downloading it helped for the moment, however, I think the checks / error handling mentioned above should be improved anyway so verdaccio doesn't crash even if a package returns unexpected metadata. Maybe the metadata even got corrupted by verdaccio itself internally somehow... Currently, I cannot reproduce the error but will report back if it occurs again.

I can not reproduce it with the current master / 4.4.0:
yarn add @fortawesome/fontawesome-svg-core --registry=http://localhost:4873
yarn add v1.19.1
...
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
success Saved lockfile.
success Saved 2 new dependencies.
info Direct dependencies
└─<EMAIL_ADDRESS>info All dependencies
├─<EMAIL_ADDRESS>└─<EMAIL_ADDRESS>✨  Done in 0.77s.

Same for npm:
npm i @fortawesome/fontawesome-svg-core --registry=http://localhost:4873
...
+<EMAIL_ADDRESS>added 2 packages from 6 contributors and audited 2 packages in 1.679s
found 0 vulnerabilities

yarn run v1.19.1
$ cross-env BABEL_ENV=registry babel-node --extensions ".ts,.tsx" src/lib/cli
 warn --- config file  - /Users/danielruf/.config/verdaccio/config.yaml
 warn --- Verdaccio started
 warn --- Plugin successfully loaded: verdaccio-htpasswd
 warn --- Plugin successfully loaded: verdaccio-audit
 warn --- http address - http://localhost:4873/ - verdaccio/4.4.0
 http --> 200, req: 'GET https://registry.npmjs.org/@fortawesome%2Ffontawesome-svg-core' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@fortawesome%2Ffontawesome-svg-core', bytes: 0/176976
 http <-- 200, user: null(<IP_ADDRESS>), req: 'GET /@fortawesome%2ffontawesome-svg-core', bytes: 0/39289
 http --> 200, req: 'GET https://registry.npmjs.org/@fortawesome%2Ffontawesome-common-types' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@fortawesome%2Ffontawesome-common-types', bytes: 0/198951
 http <-- 200, user: null(<IP_ADDRESS>), req: 'GET /@fortawesome%2ffontawesome-common-types', bytes: 0/42643
 http <-- 200, user: null(<IP_ADDRESS>), req: 'GET<EMAIL_ADDRESS>bytes: 0/35419
 http <-- 200, user: null(<IP_ADDRESS>), req: 'GET<EMAIL_ADDRESS>bytes: 0/41665

I'm closing this, too old, please if the error persist don't hesitate to comment and provide more information 👍


New features needed for validation / synthesis

Prototypes on new -much needed- features, please optimize them when appropriate :)

Any idea why this is failing (since it has to do with the plots which were not modified to my knowledge). Also why does it repeat previous commits that I merged from the upstream?

Not sure, I will have to look into it.  Might be due to newer versions of dependent libraries.

Thanks! Let me know if I have to fix something....

Codecov Report

Merging #591 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #591   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          41     41
  Lines        2006   2018   +12
=====================================
+ Hits         2006   2018   +12

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d829ad...4cb4f4a. Read the comment docs.

@mgeplf thanks!

#595 is a copy of this, including a small change, so the changes are included.


Fix Integration Tests For ID Token Signing and then Encrypting

Fix Integration Tests For ID Token Signing and then Encrypting
Fixed ID Tokens to be signed and the singed payload to be be encrypted, creating a nested JWT. This PR contains the fixes for integration tests with regards to PR#1606
1] https://openid.net/specs/openid-connect-core-1_0.html#IDToken
[2] https://datatracker.ietf.org/doc/html/rfc7516#section-4.1.6
Issue : https://github.com/wso2/product-is/issues/11901

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/949282028

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/949282028
Status: success


Change no-op checks from SKIPPED to SUCCESS

Ref: STONEINTG-681

Still a draft because I need to test it and verify if e2e tests need any changes.

I don't see anything in e2e that checks for SKIPPED so likely no changes there.

Testing Notes
I've performed the following using a kind cluster, as these changes shouldn't be OpenShift specific. From what I can tell, it is working as expected. I'll go ahead and mark this PR out of draft.
deprecated-image-check
Create Task definition in cluster:
oc apply -f task/deprecated-image-check/0.3/deprecated-image-check.yaml

Start a TaskRun and inspect the results. Example:
$ tkn task start deprecated-image-check --param BASE_IMAGES_DIGESTS="" --use-param-defaults --showlog
TaskRun started: deprecated-image-check-run-d9p4m
Waiting for logs to be available...
[check-images] {"result":"SUCCESS","timestamp":"1699303314","note":"Task deprecated-image-check success: No base images to check.","namespace":"required_checks","successes":0,"failures":0,"warnings":0}

$ oc get tr deprecated-image-check-run-d9p4m -o json | jq '.status.results'
[
  {
    "name": "TEST_OUTPUT",
    "type": "string",
    "value": "{\"result\":\"SUCCESS\",\"timestamp\":\"1699303314\",\"note\":\"Task deprecated-image-check success: No base images to check.\",\"namespace\":\"required_checks\",\"successes\":0,\"failures\":0,\"warnings\":0}\n"
  }
]

sast-snyk-check
First, we need a volume that contains some source code. Do this by first creating a PVC:
cat << EOF | oc apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: source
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Second, create the git-clone Task in-cluster so we can populate the volume:
oc apply -f task/git-clone/0.1/git-clone.yaml

And, trigger a TaskRun for the git-clone Task using our volume:
tkn task start git-clone --use-param-defaults \
  --param url=https://github.com/MartinBasti/data-only-test-container.git \
  --param revision=main \
  -w name=output,claimName=source \
  --showlog

Next, login to create a snyk token, and create the snyk secret:
snyk auth
oc create secret generic snyk-secret --from-literal=snyk_token=`snyk config get api`

Create the sast-snyk-check Task in cluster:
oc apply -f task/sast-snyk-check/0.1/sast-snyk-check.yaml

Finally, start a TaskRun for the sast-snyk-check Task using the same volume as the git-clone TaskRun.
Example:
$ tkn task start sast-snyk-check --use-param-defaults -w name=workspace,claimName=source --showlog
TaskRun started: sast-snyk-check-run-82d2w
Waiting for logs to be available...
[sast-snyk-check] {"result":"SKIPPED","timestamp":"1699303729","note":"Task sast-snyk-check skipped: Snyk code test found zero supported files.","namespace":"default","successes":0,"failures":0,"warnings":0}

$ oc get tr sast-snyk-check-run-82d2w -o json | jq '.status.results'
[
  {
    "name": "TEST_OUTPUT",
    "type": "string",
    "value": "{\"result\":\"SUCCESS\",\"timestamp\":\"1699303975\",\"note\":\"Task sast-snyk-check success: Snyk code test found zero supported files.\",\"namespace\":\"default\",\"successes\":0,\"failures\":0,\"warnings\":0}\n"
  }
]

/retest

/retest

/retest


A complete node repository documentation is required

Describe the feature or improvement you're requesting
The current README of this repository does not fully illustrate all its usages, and the official documentation only provides API reference. I need a user manual for this Node.js project repository. When I try to call the v3+ version, an error occurs, indicating that my access path is incorrect. I cannot find a solution through the README, but I did find an answer in the issues section. However, I would prefer to directly locate the corresponding API call through documentation.
PLEASE!!!!!!
目前这个仓库的readme不能体现所有的用法，官方提供的文档只是接口文档。我需要一份这个nodejs项目仓库的使用说明文档。我现在调用v3+版本的时候，会报错，提示我的访问路径不对。我没办法通过readme来找到解决方法，不过从issue中找到了回答。然而我期望通过文档能直接找到对应的api调用。
Additional context
No response

Our upcoming v4 has a more thorough README: https://github.com/openai/openai-node/tree/v4#readme


Edit debug assert to match current behavior

I'm still trying to figure out when this changed, but it seems that the "timestamp object" either never existed or wasn't free'd immediately.
If this assert is still correct, then this pull request should be refused and a bug entered!

I believe that this was introduced with commit b1abeed5e88db5d874216d42d3216e00f73fe3f7 (issue #28) when the expire code was removed.

I've seen this assert, but I'm not quite sure yet whether the assert is wrong, or the +1 in SaveImageFile is wrong

@jgfoster, nope the assert is wrong. So this is correct, as suggested by the comment


[RFE] Create tests to validate gadget OCI image

Current situation
Create tests to validate that gadget OCI images generated with ig image build command are compliant with what is defined in the design document. For instance, the first manifest is an index, etc.
Impact
Issues like https://github.com/inspektor-gadget/inspektor-gadget/pull/2572 should be detected by tests.

Do you mean that some of our commands (ig run, ig image build or ig image push) should test that?
Or do you mean CI tests for the image-based gadgets that are implemented in the main inspektor-gadget git repository?

I mean CI tests. Sorry for the poor description.


Emit activeContentItemChanged event on layoutManager as well.

Emit activeContentItemChanged event on layoutManager as well.
This makes it easier to listen to these events without having to manage separate listeners for each component.
I use this in DomTerm. It uses goldenlayout to manage multiple terminals.  When a tab is made active, the corresponding terminal should get keyboard focus.  Using a single listener on the layoutManager made do so much easier.
I'm also unfamiliar with the world of grunt, gult, bower, or npm, so I haven't yet figured out how to rebuild things.

This seems OK to me; thanks.


Enrich datasets with context from knowledge base

Thanks, @schmika !
I can confirm that the sentence mapping is working, via:
import json

from flair.datasets import NER_HIPE_2022

for language in ["de", "en", "fr"]:
    ajmc = NER_HIPE_2022(dataset_name="ajmc", language=language, add_document_separator=True)

    for split in ["train", "dev"]:
        data = []
        with open(f"kb_data/ajmc/{language}/{language}_{split}.jsonl", "rt") as f_p:
            for line in f_p:
                data.append(line)

        corpus = ajmc.train if split == "train" else ajmc.dev
        for index, sent in enumerate(corpus):
            gold_sentence = sent.to_plain_string()

            jsonl = json.loads(data[index])

            actual_sentence = jsonl["original_sentence"]

            if gold_sentence.startswith("-DOCSTART-"):
                continue

            if gold_sentence != actual_sentence:
                print(f"Mismatch for {language}/{split} @ index:", index + 1)
                print("Gold sentence:", gold_sentence)
                print("Actual sentence:", actual_sentence)
                print("-" * 100)


Short project names cannot have dashes

This just became a problem again because Chu's sidewalk equity project had a Short Name of 'sidewalk-equity' and it brought down the entire server (temporarily until we changed sidewalk-equity -> sidewalkequity).
We should:

Add in advice about short names
Check for invalid short names
Try to investigate why we can't have dashes in short names

The original url routing code:
urlpatterns = [
    re_path(r"^admin/", admin.site.urls),
    re_path(r'^$', views.index, name='index'),
    re_path(r'^people/$', views.people, name='people'),
    re_path(r'^member/(?P<member_id>[0-9]+)/$', views.member, name='member'),
    re_path(r'^member/(?P<member_id>[-a-z]+)/$', views.member, name='member'),
    re_path(r'^publications/$', views.publications, name='publications'),
    re_path(r'^talks/$', views.talks, name='talks'),
    re_path(r'^videos/$', views.videos, name='videos'),
    re_path(r'^projects/$', views.project_listing, name='projects'),
    re_path(r'^projects/(?P<project_name>[a-zA-Z\- ]+)/$', views.project, name='project'),
    re_path(r'^project/(?P<project_name>[a-zA-Z\- ]+)/$', views.project, name='project'),
    re_path(r'^news/$', views.news_listing, name='news_listing'),

    # First try to match on the news id (for historical compatibility) then match on the slug
    path('news/<int:id>/', views.news_item, name='news_item_by_id'),
    path('news/<slug:slug>/', views.news_item, name='news_item_by_slug'),

    re_path(r'^faq/$', views.faq, name='faq'),

    # JEF (Oct 31, 2022): this makes it sound you can just type in a project name
    # and we'll try to go to that project without putting in 'projects' or 'project'
    # For example, http://makeabilitylab.cs.uw.edu/soundwatch will go to
    # http://makeabilitylab.cs.uw.edu/project/soundwatch
    re_path(r'(?P<project_name>[a-zA-Z\- ]+)/$', views.redirect_project, name='project'),
]

I asked ChatGPT to fix the regular expressions to support dashes and it responded:
To support project names with dashes, you can update the regular expression in your URL patterns to include the dash character -. You can do this by adding - to the character set in your regular expressions. Here's how you can do it:
urlpatterns = [
    # ... other url patterns ...
    re_path(r'^projects/(?P<project_name>[a-zA-Z\- ]+)/$', views.project, name='project'),
    re_path(r'^project/(?P<project_name>[a-zA-Z\- ]+)/$', views.project, name='project'),
    re_path(r'(?P<project_name>[a-zA-Z\- ]+)/$', views.redirect_project, name='project'),
]

In the above code, [a-zA-Z\- ]+ will match one or more occurrences of any uppercase letter, lowercase letter, dash, or space. This will allow project names with dashes to be correctly captured by your URL patterns. Please replace the existing lines in your code with these updated lines. Remember to test your changes to ensure everything works as expected. Let me know if you have any other questions! 😊


Feature Request: Omni-direction cropping.

Under the crop tool, make it possible to crop the Image in a freestyle manner (being able to drag the four edges of cropping not just in X and Y direction but also diagonally to any angle like in CamScanner app) and the image will auto-adjusts afterwards to maintain a standard aspect ratio. (I really anticipate this feature)

Not an exact solution, but a template that might help if you're willing to use opencv. This should take a polygon defined by any four points inside the image, and map/resize/transform the selection up to the original size of the image. You could also crop at this point, or resize to the longest edge of the selected polygon to avoid too much upscaling.

Define Source Points: Specify four points from the source image. These are the coordinates the user selects for the transformation.

pts_src = np.array([[x1, y1], [x2, y2], [x3, y3], [x4, y4]], dtype=float)

Define Destination Points: Set the points to the corners of the original image dimensions.

pts_dst = np.array([[0, 0], [width-1, 0], [width-1, height-1], [0, height-1]], dtype=float)

Calculate Transformation Matrix: Use the getPerspectiveTransform function with the source and destination points.

matrix = cv2.getPerspectiveTransform(pts_src, pts_dst)

Apply Transformation: Transform the image using the warpPerspective function.

transformed_image = cv2.warpPerspective(image, matrix, (width, height))


Add "Slovyansk" as incorrect spelling of "Sloviansk"

I assume this y spelling is deprecated.  I see it in many places, e.g., Google Maps.

You are right. Thanks!


Couchapp 'utf8' warnings

On Mac OS X Lion, Python 2.7, Couchapp v1.0.1.
2011-11-29 01:54:00 [WARNING] PATH_TO_DIR/_attachments/icon.png isn't encoded in utf8
2011-11-29 01:54:00 [WARNING] plan B didn't work, PATH_TO_DIR/_attachments/icon.png is a binary
2011-11-29 01:54:00 [WARNING] use plan C: encode to base64

Fixed with the proper install command ($ env ARCHFLAGS="-arch i386 -arch x86_64" pip install couchapp).

Regenerated application; no more UTF8 warnings.


Add Spout Instance and Bolt Instance for Python topologies

Instance part of #1196

👍

👍

Code looks good, but I will add unit tests for this PR.

@objmagic - is this ready to be merged?

Yes. But I'm sorry that unit tests are hard to write and I'm not in good health condition now. I may add some later. Integration test which Taishi now is writing could be helpful though.
Anyway, I'd day we can merge this PR.

On Aug 11, 2016, at 00:30, Karthik Ramasamy<EMAIL_ADDRESS>wrote:
@objmagic - is this ready to be merged?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Ok - sounds good. I will merge it tomorrow.


[KOGITO-3337] allow to use multisheets dtables

replaced by https://github.com/kiegroup/kogito-runtimes/pull/777


fix(tree): fix tree update error with animation and without animation

Brief Information
This pull request is in the type of:

[x] bug fixing
[ ] new feature
[ ] others

What does this PR do?

For polyline, with animation on, to filter out nodes that are about to exit, and clear sourceEdge if the number of remaining nodes is 0
For all tree, with animation off, pass the current node in removeNodeEdge since the current node is immediately deleted, making it impossible to get the current node by dataIndex.

Fixed issues

#18448
#18490

Details
Before: What was the problem?

For polyline, when a child node is removed by updating the node, the line is not removed
For all trees, when collapsing a node with animation off, lines will be left behind

After: How does it behave after the fixing?

for polyline, when a child node is removed by updating the node, the lines will be removed
For all trees, collapse the node with the animation off, no lines will be left

Document Info
One of the following should be checked.

[x] This PR doesn't relate to document changes
[ ] The document should be updated later
[ ] The document changes have been made in apache/echarts-doc#xxx

Misc
ZRender Changes

[ ] This PR depends on ZRender changes (ecomfe/zrender#xxx).

Related test cases or examples to use the new APIs
N.A.
Others
Merging options

[ ] Please squash the commits into a single one when merging.

Other information

@Ovilia When will this PR be reviewed? It's been almost a year now and the bug persists.


Update LICENSE to Nothing

@sudo-bot /agree

I disagree with the removal as opposed to the change to a no license license as depicted here.
https://choosealicense.com/no-permission/
I mean, c'mon! This is nocode. There should be nolicense.

+1


istioctl: improve error message when istio ConfigMap isn't found

Fixes #569

Jenkins job manager/presubmit passed

Jenkins job manager/e2e-smoketest passed

Jenkins job manager/presubmit passed

Jenkins job manager/e2e-smoketest passed

Jenkins job manager/presubmit passed

Jenkins job manager/e2e-smoketest passed


Feature request: Database item category

Hi @bsodmike!
This library looks very useful, thanks for creating it!
I'm looking to fetch database credentials. Would it be possible to implement the Database item category, or is there an existing way for me to retrieve credentials that I've missed?

Hi @sondrelg thanks for the kind comments. Been a while since I looked at this. Let me update you when I have time to extend the API.
Feel free to may be sponsor my opensource contributions if you have found them useful please. Every little bit of support helps ❤
Best M


s3_pagination test is flaky

Steps to reproduce
sometimes s3_pagination test fails
Expected result
it shouldn't fail
Actual result
it fails
Logs, links

https://github.com/neondatabase/neon/actions/runs/5941037366/job/16110958065

thread 's3_pagination_should_work' panicked at 'assertion failed: `(left == right)`
  left: `{RemotePath("folder0/"), RemotePath("folder2/"), RemotePath("folder3/"), RemotePath("test/"), RemotePath("folder1/")}`,
 right: `{RemotePath("test")}`: remote storage root prefixes list mismatches with the uploads. Returned prefixes: {RemotePath("folder0/"), RemotePath("folder2/"), RemotePath("folder3/"), RemotePath("test/"), RemotePath("folder1/")}', libs/remote_storage/tests/test_real_s3.rs:66:5

My guess why this happens

time stamps are used to generate the bucket names. That's a bad idea.

Another occurance: https://github.com/neondatabase/neon/actions/runs/6313466375/job/17141589142?pr=5384#step:18:938
---- s3_pagination_should_work stdout ----
2023-09-26T13:39:29.871902Z  INFO Using s3 bucket 'neon-github-public-dev' in region 'eu-central-1' as a remote storage, prefix in bucket: 'Some("pagination_should_work_test_1695735569871864268/")', bucket endpoint: 'None'

---- s3_list_files_works stdout ----
2023-09-26T13:39:29.871903Z  INFO Using s3 bucket 'neon-github-public-dev' in region 'eu-central-1' as a remote storage, prefix in bucket: 'Some("pagination_should_work_test_1695735569871864268/")', bucket endpoint: 'None'

Hmm, I wonder if this is a copy paste issue? :)


Integration is using deprecated DEVICE_CLASS_* constants

Please check Services, Known Bug / Issues and Troubleshooting over here first: https://github.com/dahlb/ha_kia_hyundai/blob/master/README.md
Region and Brand of car
n/a
Describe the bug
This custom integration uses deprecated DEVICE_CLASS_* constants in its codebase.
The DEVICE_CLASS_* constants have been deprecated and replaced in Home Assistant Core 2021.12 (over a year ago). I would highly suggest updating/migrating this integration to the new enums.
For example, for the device classes supported by the sensor platform, there is now a SensorDeviceClass enum. So if a sensor previously used the DEVICE_CLASS_ENERGY constant, it should now use SensorDeviceClass.ENERGY. Other platforms (like binary_sensor, and number) provide similar enumerations for their supported device classes.
The migration thus only consists of replacing constants with an enumeration member and is, therefore, very low impact and should be fairly straightforward.
If I can help resolve any questions regarding this change or migration, feel free to ask or respond to this issue. I'm happy to help!
Kindest regards,
../Frenck
Debug logs if an error occurred
n/a
To Reproduce
n/a
Expected behavior
n/a
Screenshots
n/a
Additional context
n/a

You are unstoppable! 💪
Thank you for the quick responses and effort, appreciated 👍
../Frenck


Remove solarish compat for openpty and forkpty

We currently polyfill these functions for solarish in https://github.com/rust-lang/libc/blob/3a0b0444d7206aa3feb30297c005597a960d0b33/src/unix/solarish/compat.rs, but it would be better to phase these out so we're not providing any implementations in libc.
It looks like Solaris supports openpty and forkpty https://docs.oracle.com/cd/E88353_01/html/E37843/openpty-3c.html, and Illumos has had them for at least a few years https://www.illumos.org/issues/5386, so at this point our implementation could be replaced with bindings.

Correction, those are not available on illumos, either never been accepted or reverted ; most likely the former.

Oh that's interesting, thanks for the context. Hm, I'm not sure what to do here in that case - this is about the only case of polyfill in this crate and ideally we wouldn't have any. I guess maybe we can remove these with 1.0.


i want the HTTPS schema for the site of “fauset.testnet.libra.org” .

🚀 Feature Request
i want the HTTPS schema for the site of  “fauset.testnet.libra.org” .
Motivation
i am making the site on the web, that is minting from “fauset.testnet.libra.org”
but the “fauset.testnet.libra.org” can not use https.
as a result, i made http page.
but, It is not secure without HTTPS.
Pitch
Describe the solution you'd like
i want to this.
https://fauset.testnet.libra.org
Additional context
it can mint 6. ok  (on HTTP)
http://kabuda.net/test/libra/libra-core/http-mint-fetch-nocors.html
it can not mint 7.  (on HTTPS)
https://kabuda.net/test/libra/libra-core/https-mint-fetch-nocors.html
see Libra-Vista
https://www.libravista.com/address/9291e14f5e7c1ce211ce9477154faddb0c308af2f6a10324f893e2e59a13ff80
console Error
Mixed Content: The page at 'https://kabuda.net/test/libra/libra-core/https-mint-fetch-nocors.html' was loaded over HTTPS, but requested an insecure resource 'http://faucet.testnet.libra.org/?amount=7&address=9291e14f5e7c1ce211ce9477154faddb0c308af2f6a10324f893e2e59a13ff80'. This request has been blocked; the content must be served over HTTPS.

Write again
With HTTP page (non ssl/tls)
    let url='http://faucet.testnet.libra.org/?amount=20000000&address=9291e14f5e7c1ce211ce9477154faddb0c308af2f6a10324f893e2e59a13ff80'
    fetch(url, {
        method: 'post',
        mode: 'no-cors'
    }).then(data  =>
        console.log(data)
    ).catch(error => console.error(error))

is work!
but With HTTPS page (ssl/tls)
    let url='http://faucet....

is  not work. because it is Mixed Content.
And https://faucet....  doesn't work
Because of that, I can't write mint code on the https page


An error:RuntimeError: Function 'SigmoidBackward0' returned nan values in its 0th output.

In  nnUNetTrainerV2_DDP.py，
with autocast(enabled=False):
output_act = output_ds[i].sigmoid() if is_sigmoid else softmax_helper(output_ds[i]) # bug occurs here..
There will raise an error:RuntimeError: Function 'SigmoidBackward0' returned nan values in its 0th output.
Could you tell me how to solve it,please?
Thank you.

I got the same issue.
For me it always happens at around epoch ~120.
Here is my stacktrace:
"/dkfz/cluster/gpu/data/OE0441/t006d/Code/transunet3d/nn_transunet/trainer/nnUNetTrainerV2_DDP.py", line 1039, in run_training
    l = self.run_iteration(self.tr_gen, True)
  File "/dkfz/cluster/gpu/data/OE0441/t006d/Code/transunet3d/nn_transunet/trainer/nnUNetTrainerV2_DDP.py", line 552, in run_iteration
    l = self.compute_loss(output, target, is_max, is_c2f, self.args.is_sigmoid, is_max_hungarian, is_max_ds, point_rend, num_point_rend, no_object_weight)
  File "/dkfz/cluster/gpu/data/OE0441/t006d/Code/transunet3d/nn_transunet/trainer/nnUNetTrainerV2_DDP.py", line 658, in compute_loss
    output_act = output_ds[i].sigmoid() if is_sigmoid else softmax_helper(output_ds[i]) # bug occurs here..
 (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:113.)


Redesign UAVCAN using Bootstrap framework

Resigned the UAVCAN website to use the Bootstrap framework.  The site features:

a responsive design layout
a new consortium page
a new carousel of images on the home page
new home page content
a fixed sidebar navigation menu

The design can be viewed on the UAT website.
This PR will merge code from UAT into production.

how do I review this? Is there a pre-prod URL I can view the rendered results or a procedure for previewing the change locally?

how do I review this? Is there a pre-prod URL I can view the rendered results or a procedure for previewing the change locally?

The design can be viewed on the UAT website (http://uat.uavcan.org).

I'm not a huge fan of the "above the fold" experience. For something that is supposed to be uncomplicated the first glance on the site is very busy with a lot of text. I'd like to see something that has more whitespace and a simpler structure on the homepage. That said, the content seems fine so I'm not going to block this release.


Simplify docker-gen and nginx setup

This fixes the broken proxy image. Currently a hack to generate the default.conf for nginx no longer works due to an image update. This should provide a long term solution which is also easier to work with.
To simplify the setup we separate docker-gen from the actual nginx instance. From the original nginx.tmpl we only take the necessary parts.
I've added nginx to the development setup and would like some input on that. My reasoning was, that it is closer to production and allows testing new nginx configs. But during operation requests are only handled through. An other thought was, that I'd like to replace nginx with the openresty distribution in order to use the nginx lua module. We could fix the current problems with the prometheus exporter with that by concatinating responses from all workers.

Henrik and I had a talk about this. There are some issues left, that prevent the exporter from working.

This is nearly done. So if you want to have a working setup, merge this now.
There is a minor problem left: /metrics shows garbage text in browsers but works with curl. Chrome lets you download a gzip somehow.

In my development setup, localhost refuses to connect when I try to log in:
HTTP Origin header (http://localhost:3000) didn't match request.base_url (http://workers)
Redirected to http://workers/

In my development setup, localhost refuses to connect when I try to log in:
HTTP Origin header (http://localhost:3000) didn't match request.base_url (http://workers) Redirected to http://workers/

Thanks for pointing this out. It is fixed now. I've tested this in experimental, so from a functionality point of view I consider this to be ready for review and merging.
I'm only unsure if it is easier to work with the codebase without nginx in development. So if you have any issues with code auto reloading or anything else, I can also remove it from the development docker-compose file.
Production deployment needs to be done manually once! This is because new containers are spawned and the config needs to be ready for this. So when this PR goes into production, let me know. mampf-next/mampf-dev is already configured.

If new containers are added more often, we could think about making the docker-compose files part of this repository. But then github workflows need to be setup to decline changes to them. If someone can make changes to docker-compose files in production, they are effectively root there.

Thanks, it is working nicely in development now as well. I think the addition/removal of containers will not happen so often, so I think the setup given in this PR should be fine.

If there are no more comments, I think this could be merged.


Use of cloudfoundry metric-store as a datasource?

We're experimenting with leveraging the metric-store-release as an additional datasource for grafana in order to pick up metrics emitted by apps and services on a CF.
Has anyone done something similar?

interesting. i would like to try the same. Please help with some document

interesting. i would like to try the same. Please help with some document


When logged in to non-enterprise account, cannot log in to enterprise account

Testing #3985
The extension wants me to sign in even though I'm already signed in (though on the non-enterprise account). The extension is on the trusted extension list in Manage Trusted Extensions.

@rzhao271 That sign in button should prompt you to sign in with GitHub Enterprise when you click it if you have a folder open to a Enterprise repo (you may need to wait a few seconds after opening the folder for this to happen though as we have to wait for information from the git extension).

Resolved offline

In case anyone else comes across this: I recommend cloning your Enterprise repo in an external terminal so that the git credential manager modal dialog shows.


test for max stack size

[x] issue #191
[x] follows contribution guide
[x] code change includes tests

@b-j-roberts This PR is Ready for review

this test function throw the following error
#[test]
fn test_max_stack_size_more() {
    let mut program: ByteArray = "";
    let op_1_string = "OP_1 ";
    let mut index: u64 = 0;
    while index < 1000 {
        program.append(@op_1_string);
        index += 1;
    };

    let mut engine = utils::test_compile_and_run_err(program, Error::SCRIPT_STACK_SIZE_EXCEEDED);

    utils::check_dstack_size(ref engine, 1000);
}

this errors

failures:
   shinigami::opcodes::tests::test_stack::test_max_stack_size_more - Panicked with "Execution of the program did not fail as expected".

which mean putting 1000 item in the stack does not break it  @b-j-roberts

which mean putting 1000 item in the stack does not break it @b-j-roberts

Yes, that is the expected behavior. 1000 items on the stack should work fine. So you can use utils::test_compile_and_run instead of the error one.


Anomalies during recording with H265

Hello,
I am in the process of collecting some data using a ZED Camera, now using H265 for recording.
About a third of the time, this works -- in 720P/60Hz mode, a 30s clip ends up ~60mb. However, sometimes, the same 30s recording is x10-15 times that size. When opening in ZED Explorer, the  larger SVO files are reported to be well below the 60FPS requested, and when opening using the Python API, it hangs indefinitely and I have to kill the Python process, making it impossible to pass through our processing pipeline.
An example of a failed recording is here: https://redfern.me/public/failed_file.svo, with a successful  example available here: https://redfern.me/public/ok_file.svo.
Do you have any idea what might be causing this issue, or have any ideas how it might be fixed?
The files were recorded under Ubuntu, on a machine with an i7 7700T, GTX 1070 and a 1tb Samsung 960 Evo SSD using SDK version 2.8.3.

Hi,
Thanks for reporting this issue.
There is indeed a mismatch between the FPS and the timestamps of each frame in the failed SVO. When reading the failed_svo with ZED Explorer, the FPS is about 5fps which obviously is wrong when you look at the images.
Do you use ZED Explorer or a recording sample (C++ or python) to save the SVO file ? Just to know where to look for the fix.
We will investigate on that issue.
Right now, to fix the failed SVO, there is a trick to recompress/restore the svo file. This idea is to use ZED SVOEditor with a correct svo (ok_file.svo) and a wrong svo (failed_file.svo) to "fake" the mismatch between fps and timestamps. Here is how to do :

Save the number of frames for the "ok" svo :
$ ./ZED\ SVOEditor -inf ok_file.svo
For example, on the "ok_file.svo" you shared, the number of frame is n=1741.

Create a single svo that merge ok and failed (in that order)
$ ./ZED\ SVOEditor --merge ok_fle.svo failed_file.svo temp_output.svo

Cut the merged svo to extract only the "failed" part that start at s = number of frame +1
$ ./ZED\ SVOEditor -cut temp_output.svo -s 1742 failed_file_restore.svo

failed_file_restore.svo should be now at the proper size (around 60Mb) + with correct FPS.
make sure that ok_file and failed_file are in the same resolution ,otherwise it won't work.

Thank you, I will try this out.
The recording was done using the Python SDK Bindings, with enable_recording().
Are you able to reproduce the crash/hang while trying to read in the failed SVO with Python?
If you need me to share any code then please let me know.
Thanks,
Joe

Your process seems to work as temporary workaround -- thank you. I wrapped it up into a bash script, in case anyone else has this issue:
#!/bin/bash
set -e

OK_FILE=svo_1574168934.svo

if [ "$#" -ne 1 ]; then
    echo "Illegal number of parameters"
    exit 1
fi

broken_file=$1
fixed_file="fixed_"$broken_file

ok_frame_count=$(ZED\ SVOEditor -inf $OK_FILE 2>&1 |grep 'Number of Frames' | awk '{print $5'})

echo "[i] $ok_frame_count frames in OK file"

merged_file=$(mktemp /tmp/zed_merged_tmp.XXXXXXXX)
broken_start_index=$(($ok_frame_count + 1))

echo "[i] Starting merge..."

echo ZED\ SVOEditor --merge $OK_FILE $broken_file $merged_file
ZED\ SVOEditor --merge $OK_FILE $broken_file $merged_file
echo "[+] Merge complete, cropping"

ZED\ SVOEditor -cut $merged_file -s $broken_start_index $fixed_file

echo "[+] Crop completed"
echo "[i] Deleting temporary file"

rm $merged_file

echo "Fixed file written to $fixed_file"

ok_file needs to be set to a known-good SVO, and ZED\ SVOEditor needs to be on your PATH.

@obraun-sl Any updates on this issue? I've seen it happening in SVO Explorer too, and it's quite inconvenient having to do the additional processing, and the corrupt files occupy a large amount of disk space.