[RFC] Low Level Design for Normalization and Score Combination Query Phase Searcher

Introduction
This document describes details of Low Level Query Phase Searcher Design in scope of Score Normalization and Combination Feature.  This is one of multiple LLDs in scope the Score Normalization and Combination Feature. Pre-read of following documents is highly recommended: high-level design [RFC] High Level Approach and Design For Normalization and Score Combination, antecedent LLD design [RFC] Low Level Design for Normalization and Score Combination Query.
Background
As per HLD and in scope of Normalization feature we need an ability to collect results of multiple sub-queries and send them unmerged for post-processing by coordinator node. New part that currently does not exist in OpenSearch is storing of such multiple results. Current logic assumes that search at shard level should return a single list of results (doc ids and scores), and merge should happen at shard level.
OpenSearch provides mechanism of extensions for this type of use cases. Such goal can be achieved through custom QueryPhaseSearcher, that can call custom DocCollector. Both are exwecuted at the shard level as part of the Query phase of search request (caller of the query phase searcher). Those abstractions along with new DTO that can hold results of multiple sub-queries will be a focus for this design.
New QueryPhaseSearcher and related classes will be added as part of the Neural Search plugin and code changes will be done in the plugin repo.
Requirements

Results with doc ids from multiple sub-queries need to be collected and passed to controller node as part of the query execution results. At later Fetch phase we need to be able to identify what result belongs to what sub-query.

New query should keep added latency (for functions like query parsing etc.) to minimum and not degrade performance in both latency and resource utilization comparing to a similar query that does combination at shard level. We will add exact latency numbers after the benchmark is done, initial expectations are: added latency within 15%.

Fetch phase of query execution (“reduce” in terms of OpenSearch, executed at coordinator node) should work without changes.

Scope
In this document we propose a solution for the questions below:

How do we collect scores for each sub-query and form final resulting DTO.

How do we use existing extension point of OpenSearch for implementation of custom QueryPhaseSearcher.

Solution Overview
New custom query phase searcher will be created as implementation of core OpenSearch interface QueryPhaseSearcher.  New custom DocCollector will be responsible for collecting search results from a single shard. We’ll be using new DTO object that holds collection of top scored docs for all sub-queries.
Risks / Known limitations

currently core OpenSearch supports only single custom QueryPhaseSearcher, and there is one under “Concurrent Search” feature that has been registered recently. Currently it’s implemented behind the feature flag (Moving concurrent-search out of the sandbox plugin to core behind the feature flag). There is an open issue in core to provide ability to register multiple phase searchers.

In first phase we’ll create a setting in the plugin that will allow to disable our query phase searcher in case user needs to use concurrent search feature. Using feature flag has a drawback of using command line arguments that are set during the distribution build. For setting user can change the value and restart the cluster. By default hybrid query searcher will be enabled, and setting will allow to disable it.

new DTO for top docs will be implementing two views for results: collection of doc ids for single query and collection of collections of matching doc ids for a new hybrid query. First collection is required to be complied with existing core API contracts and keep number of changes to minimum. Actual results per sub-query will be used in later implementations, in transformer that is part of the search pipeline; it will be processed on a Query phase and before the Fetch phase.

new doc collector will be implementing minimal set of features required to collect results of hybrid query. This is because collector will be used for only one query type, and some of the features supported by core doc collector (like pagination or max score threshold) are not supported in initial release (e.g. pagination, see [RFC] High Level Approach and Design For Normalization and Score Combination for details).

Future extensions

Resolve multiple query searchers limitation. This can be a feature in core, currently it’s under discussion (https://github.com/opensearch-project/OpenSearch/issues/7020). There are two possible ways here:

use a single query searcher, choose one of multiple implementations in runtime
create new query phase (or sub-phase), plugins can register multiple custom implementations

pagination for query results. Assumption is that this should work if we pass “to” and “from” to doc collector, but it needs testing (small POC using first implementation version or previous POCs).

Solution Details
We’re going to use existing plugin class NeuralSearch as an entry point to register new HybridQueryPhaseSearcher. Searcher will call new doc collector only in case query is of type HybridQuery, this check is required as query searcher is global at plugin level and will work for all queries (currently NeuralSearch query).

Figure 1: Class diagram for HybridQueryPhaseSearcher implementation
Below is the general data flow for collecting query results using custom doc collector for Hybrid query. QueryPhaseSearcher will be executing at shard level, after coordinator node sends the Query request to each shard. DocCollector will Get max scores for each sub-query using doc id iterator and priority queue, then it will form a collection of all results. This is set as a shard query result and sent back to coordinator node for fetch phase.

Figure 2: General sequence diagram for collecting query results from shards
Final query results will be set to the QueryResult object as a single instance of CompoundTopDocs object.
For example, we are sending Hybrid Query search request with 3 sub-queries:
POST <index-name>/_search
{
    "query": {
        "hybrid": {
            "queries": [
                { /* standard term query 1 */ },
                { /* standard term query 2 */ },
                { /* neural query */ }
            ]
        }
    }
}

Our DTO with query results will look something like this:
CompoundTopDocs: //new class
    docs:
        [0] TopDocs: //existing class
            totalHits
            scoreDocs:
                [0] ScoreDoc: //existing class
                    docId
                    score
                [1] ScoreDoc:
                    docId
                    score
        [1] TopDocs:
            totalHits
            scoreDocs:
                [0] ScoreDoc:
                    docId
                    score
        [2] TopDocs:
            totalHits
            scoreDocs:
                [0] ScoreDoc:
                    docId
                    score
                [1] ScoreDoc:
                    docId
                    score
                [2] ScoreDoc:
                    docId
                    score

Main difference between new TopDocs and existing core implementation is that new object has a collection of results. Standard core object has always single result, for instance
TopDocs:
    totalHits
    scoreDocs:
         [0] ScoreDoc:
               docId
               score
         [1] ScoreDoc:
               docId
               score

Testability
New query phase searcher is testable via existing /search REST API and lower level direct API calls. Main testing will be done via unit and integration tests. We don’t need backward compatibility tests as Neural-search is in experimental mode and there is no commitment for support of previous versions.
Tests will be focused on overall query stability and results that are collected. Actual explicit testing of result correctness is not possible at this stage, as score normalization and combination is done at later stage by future extension on text processor (or similar alternative implementation):

collect result doc ids for hybrid query that has no sub-queries, has one sub-query and has multiple sub-queries
test on cluster with multiple shards/nodes
check that query result object is set and accessible from SearchQueryThenFetchAsyncAction part of workflow

Mentioned tests are part of the plugin repo CI, main OpenSearch build CI, and also can be executed on demand from development environment.
Tests for metrics like normalized score correctness, performance etc. will be added in later implementations when end-to-end solution will be available.
Reference Links

Meta Issue for Feature: https://github.com/opensearch-project/neural-search/issues/123
[RFC] High Level Approach and Design For Normalization and Score Combination: https://github.com/opensearch-project/neural-search/issues/126
[RFC] Low Level Design for Normalization and Score Combination Query: https://github.com/opensearch-project/neural-search/issues/174

We have found during testing that for multiple node scenario, which is typical in production, custom implementation of TopDocs doesn't work well. We choose to adjust our format to existing logic and switch to existing TopDocs with a single list of scores. Details of our findings we explained in core OpenSearch issue, TL;DR: coordinator node will receive only single list of scores from data nodes:
Updated approach for DTO
We're using using TopDocs class as a DTO for sending results from data nodes to coordinator node. Scores of sub-queries will be in a single list of scores, each sub-query will be preceded with a special delimiter score. Special score will also be first and last element of the score list, this will mark such TopDocs as related to Hybrid Query and simplify parsing of that score list.
High level protocol
     *  doc_id | magic_number_1    //start/stop
     *  doc_id | magic_number_2   //delimiter for sub-query 1
     *  ...
     *  doc_id | magic_number_2   //delimiter for sub-query 2
     *  ...
     *  doc_id | magic_number_2   //delimiter for sub-query 3
     *  ...
     *  doc_id | magic_number_1  //start/stop

Example
TopDocs:
    totalHits
    scoreDocs:
         [0] ScoreDoc:
               0, start/stop_score
         [1] ScoreDoc:
               0, delimiter_score
         [2] ScoreDoc:
               0, 0.95
         [3] ScoreDoc:
               2, 0.9
         [4] ScoreDoc:
               3, 0.75
         [5] ScoreDoc:
               0, delimiter_score
         [6] ScoreDoc:
               0, 12.1
         [7] ScoreDoc:
               1, 8.9
         [8] ScoreDoc:
               0, start/stop_score

We have to utilize only score field of ScoreDocs, this is due to limitation in implementation of pipelines and processor execution. For case of only 1 shard it's not guaranteed that FETCH phase will be executed before the processor is called, even for case when processor is registered between QUERY and FETCH phases. In such case if docId is not valid then Fetch phase code fails.
Corresponding code in normalization processor needs to support this new format. Main logic of score processing will remain the same, changes will be only in logic of parsing TopDocs objects from shards/nodes.

Resolving this github issue as the changes for RC of 2.10 is finalized and merged. Please create a github issue if there are any further questions.


ForceNew if zone changes on CustomHostname resource

Custom Hostnames cannot be moved across zones. ForceNew if the zone changes.

Thanks for this one @cdloh! Can you please add a test exercising this new condition? https://github.com/cloudflare/terraform-provider-cloudflare/blob/f61f3ad5355dc8150d4e39589fba71ea6bd99b43/cloudflare/resource_cloudflare_custom_hostname_test.go#L188-L224 is an example of where we do this elsewhere and confirm the values are indeed changing and Terraform is doing the expected thing.

@jacobbednarz is there another zoneid within the environment to test with? If there isn't another zone I'm unsure how to test it moving across zones.

Yep, use CLOUDFLARE_ALT_ZONE_ID and confirm the plan isn't empty.


BadgerStore

On the Monet testnet - Camille-3 - we saw evidence that there is a bug in BadgerStore whereby some Events are not found when responding to a pull request.
We started a node without a database (so no bootstrap), forcing it to pull the entire hashgraph from other peers, and the peers would encounter and error like this:
[0124] ERROR babble: requestSync() error=EventCache, 0X0EF22788B30904B2B46488E4BFF41160A4A6C74D9607377E1FD4E1B1FE81AB93, Not Found [0124] ERROR babble: gossip pull error=EventCache, 0X0EF22788B30904B2B46488E4BFF41160A4A6C74D9607377E1FD4E1B1FE81AB93, Not Found [0125] DEBUG babble: requestSync() duration=1353432 [0125] ERROR babble: requestSync() error=EventCache, 0X0EF22788B30904B2B46488E4BFF41160A4A6C74D9607377E1FD4E1B1FE81AB93, Not Found [0125] ERROR babble: gossip pull error=EventCache, 0X0EF22788B30904B2B46488E4BFF41160A4A6C74D9607377E1FD4E1B1FE81AB93, Not Found
This suggests that BadgerStore is not finding some "old" events, which is potentially a bug with the caching system.
It could be useful to reproduce this issue with a very small cache-size setting in Babble.

Fixed in v0.5.9

We fixed the GetEvent method, but we should also fix the GetParticipantEvents method.
Just looking at the core.EventDiff method, which is used by nodes to respond to sync requests,
we can see that it uses GetParticipantEvents which will eventually start failing if we don't use
the persistant storage for it.
More generally, there is a case for cleanly separating store methods for pure hashgraph objects, from
consensus objects. The hashgraph objects are just enough to construct and gossip the DAG, whereas the consensus objects are used by the consensus methods. Hashgraph objects need to be
persisted, because they are used by the sync methods to enable people to join the network. But consensus objects can live in the inmem store, as they are recalculated locally by every node.

Working on branch issue127
Having update GetParticipantEvents to read from the database, I am still seeing EventCache errors.
It seems to be tied to the Dynamic Membership protocol.
Here's a way to reproduce the issue locally with the docker demo (note that cache-limit is set to 2000):
make nodes=1
make join index=2
make join index=3
make bombard txs=3000
make join index=4

node4 doesn't join succesfully.
But if you start with 3 nodes (instead of joining 1 by 1), and send 9000 txs, the 4 nodes can join no-problem.
make nodes=3
make bombard nodes=3 txs=3000
make join index=4

Now that WireInfo is correctly set/read we have another problem because topologicalIndex is also lost.
The Events are not returned in topological order, which leads to errors when trying to insert (or convert from wire representation) an event whose parents are not yet inserted.
We need to preserve the topological order when reading Events from DB.


add type definition

This PR add type definition.
resolves #205

Thanks for your contribution @musou1500!

Published in 1.5.4


Question about seat_service behaviour

Hello,
I am trying to run seat_service in a container but it looks like it is stuck in trying to connect to the databroker:
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
DataBrokerFeederImpl: Connecting to data broker [databroker:55555] ...
...

I have the following environment  variables set in the container:
CAN=cansim
DAPR_GRPC_PORT=52002
BROKER_ADDR=databroker:55555
RUST_LOG="info,databroker=info,vehicle_data_broker=info"

I am also using the following image: ghcr.io/eclipse/kuksa.val.services/seat_service:v0.1.0
I tried to use telnet and I was able to stablish a connection to the databroker.
Are those env vars enough or is my setup missing something?
Thanks!

If you still need help. Your BROKER_ADDR looks not right. Should be BROKER_ADDR=<IP_ADDRESS>:55555

I am trying to run the databrocker in its container, here is my compose file:
version: "3.9"
services:
  mosquito:
    image: "docker.io/library/eclipse-mosquitto:2.0.14"
    command:
      - "mosquitto"
      - "-c"
      - "/mosquitto-no-auth.conf"
    ports:
      - "1883:1883"
      - "9001:9001"
  databroker:
    image: "ghcr.io/eclipse/kuksa.val/databroker:v0.17.0"
    ports:
      - "55555:55555"
      - "52001:52001"
    environment:
      - "DATABROKER_GRPC_PORT=52001"
      - "RUST_LOG=info,databroker=info,vehicle_data_broker=info"
  seatservice:
    image: "ghcr.io/eclipse/kuksa.val.services/seat_service:v0.1.0"
    ports:
      - "50051:50051"
      - "52002:52002"
    environment:
      # - "VEHICLEDATABROKER_DAPR_APP_ID=vehicledatabroker"
      - "CAN=cansim"
      - "DAPR_GRPC_PORT=52002"
      - "BROKER_ADDR=databroker:55555"
      - "RUST_LOG=info,databroker=info,vehicle_data_broker=info"
    depends_on:
      - "databroker"
    links:
      - "databroker:databroker"
  feedercan:
    image: "ghcr.io/eclipse/kuksa.val.feeders/dbc2val:v0.1.0"
    environment:
      # - "VEHICLEDATABROKER_DAPR_APP_ID==vehicledatabroker"
      # - "DAPR_GRPC_PORT=52001"
      - "LOG_LEVEL=info,databroker=info,dbcfeeder.broker_client=info,dbcfeeder=info"
      - "USECASE=databroker"
      - "CANDUMP_FILE=/data/candumpDefault.log"
      - "DBC_FILE=/data/dbcfileDefault.dbc"
      - "MAPPING_FILE=/data/mappingDefault.yml"
      - "VDB_ADDRESS=databroker:55555"
    volumes:
      - "./deploy/runtime/k3d/volume:/data:z"
    depends_on:
      - "databroker"
    links:
      - "databroker:databroker"
  app:
    build: "./app"
    ports:
      - "5000:5000"
    depends_on:
      - databroker
      - seatservice

Shouldn't BROKER_ADDR point to the container service host in this case?

Hello,
it seems you are not using dapr, in that case you should not set DAPR_GRPC_PORT, as it overrides the port from BROKER_ADDR.
More details here:
https://github.com/eclipse/kuksa.val.services/blob/cf286cf6fe8bf21146254b6e62d4df1239c7aa69/seat_service/src/lib/broker_feeder/collector_client.cc#L51

Oh, that did the trick:
2023-01-25 12:21:25,903 INFO dbcreader: Collecting signals, generating CAN ID whitelist
2023-01-25 12:21:25,903 INFO dbcreader: Found signal in DBC file UIspeed_signed257 in CAN frame id 0x257
2023-01-25 12:21:25,904 INFO dbcreader: Found signal in DBC file RearPower266 in CAN frame id 0x266
2023-01-25 12:21:25,904 INFO dbcreader: Found signal in DBC file DIgear118 in CAN frame id 0x118
2023-01-25 12:21:25,904 INFO dbcreader: Found signal in DBC file ParkingBrake118 in CAN frame id 0x118
2023-01-25 12:21:25,904 INFO dbcreader: Found signal in DBC file VCRIGHT_hvacCabinTempEst in CAN frame id 0x243
2023-01-25 12:21:25,904 INFO dbcreader: Found signal in DBC file UI_SOC in CAN frame id 0x33a
2023-01-25 12:21:25,904 INFO dbcreader: Found signal in DBC file DogModeOn in CAN frame id 0x33b
2023-01-25 12:21:25,904 INFO dbcreader: Found signal in DBC file DogModeTempSet in CAN frame id 0x33b
2023-01-25 12:21:25,905 INFO dbcreader: CAN ID whitelist=[599, 614, 280, 579, 826, 827]
2023-01-25 12:21:25,905 INFO dbcfeeder: Using virtual bus to replay CAN messages (channel: vcan0)
2023-01-25 12:21:25,905 INFO canplayer: Replaying candump from /data/candumpDefault.log
2023-01-25 12:21:26,638 INFO dbcreader: Starting Rx thread
2023-01-25 12:21:26,638 INFO canplayer: Starting Tx thread
2023-01-25 12:21:26,638 INFO dbcfeeder: Connecting to Data Broker using databroker:55555
2023-01-25 12:21:26,641 INFO databroker: Using metadata: (('dapr-app-id', 'vehicledatabroker'),)
2023-01-25 12:21:26,641 INFO dbcfeeder: Connected to data broker
2023-01-25 12:21:26,641 INFO dbcfeeder: Register datapoints
2023-01-25 12:21:26,642 INFO databroker: Vehicle.OBD.Speed was registered with id 3, type 10
2023-01-25 12:21:26,643 INFO databroker: Vehicle.OBD.EngineLoad was registered with id 4, type 10
2023-01-25 12:21:26,643 INFO databroker: Vehicle.Powertrain.Transmission.Gear was registered with id 5, type 2
2023-01-25 12:21:26,643 INFO databroker: Vehicle.Chassis.ParkingBrake.IsEngaged was registered with id 6, type 1
2023-01-25 12:21:26,644 INFO databroker: Vehicle.Cabin.AmbientAirTemperature was registered with id 7, type 10
2023-01-25 12:21:26,644 INFO databroker: Vehicle.Powertrain.Battery.StateOfCharge.Current was registered with id 8, type 10
2023-01-25 12:21:26,644 INFO databroker: Vehicle.Cabin.DogMode was registered with id 9, type 1
2023-01-25 12:21:26,645 INFO databroker: Vehicle.Cabin.DogModeTemperature was registered with id 10, type 10

Thanks :-)

Closing it since the issue was solved.


Deregister Eventeds before dropping

If https://github.com/carllerche/mio/issues/753 is implemented, then tokio-core will have to deregister Eventeds rather than expecting them to be deregistered on Drop. However, I believe we can make this change now (backwards-compatibly).

I sort of disagree, but we can discuss on carllerche/mio#753


out of range for ActiveRecord::Type::Integer with limit 4

I have installed everything as the directions say, but my PKs are still stuck with activerecord thinking they have a 32 bit limit. Any thoughts?
Checked the changelog after doing the upgrade, everything seems to be working fine.

What Rails version, database, etc.

@acook
MySQL
~# mysql --version
mysql  Ver 14.14 Distrib 5.5.44, for debian-linux-gnu (x86_64) using readline 6.3
Server: Localhost via UNIX socket
Server version: 5.5.44-0+deb8u1
Protocol version: 10
MySQL charset: UTF-8 Unicode (utf8)

CREATE TABLE IF NOT EXISTS `authors` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `facebook` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `twitter` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `biography` text COLLATE utf8_unicode_ci,
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=43253 ;

-- --------------------------------------------------------

--
-- Table structure for table `books`
--

CREATE TABLE IF NOT EXISTS `books` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `author_id` bigint(20) DEFAULT NULL,
  `series_id` bigint(20) DEFAULT NULL,
  `featured_id` int(11) DEFAULT NULL,
  `isbn` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `title` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `summary` text COLLATE utf8_unicode_ci,
  `is_public` int(11) DEFAULT NULL,
  `rated` int(11) DEFAULT NULL,
  `downloadcount` int(11) DEFAULT NULL,
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  `series_nbr` varchar(11) COLLATE utf8_unicode_ci DEFAULT NULL,
  `book` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `cover` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `language` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
  `maturity` varchar(5) COLLATE utf8_unicode_ci DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `index_books_on_author_id` (`author_id`) USING BTREE,
  KEY `index_books_on_is_public` (`is_public`) USING BTREE,
  KEY `index_books_on_series_id` (`series_id`) USING BTREE
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=12491981479220 ;

Rails
tuebl@travis:~/tuebl$ rails -v
Rails 4.2.5
tuebl@travis:~/tuebl$ ruby -v
ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux]
tuebl@travis:~/tuebl$ passenger -v
Phusion Passenger version 5.0.21

Can you verify that rails generate bigint_pk:install created the files correctly?

tuebl@travis:~/tuebl$ rails generate bigint_pk:install
DEPRECATION WARNING: The configuration option `config.serve_static_assets` has been renamed to `config.serve_static_files` to clarify its role (it merely enables serving everything in the `public` folder and is unrelated to the asset pipeline). The `serve_static_assets` alias will be removed in Rails 5.0. Please migrate your configuration files accordingly. (called from block in <top (required)> at /home/tuebl/tuebl/config/environments/production.rb:23)
      create  config/initializers/bigint_pk.rb
      create  db/migrate/20151207010812_change_keys_to_bigint.rb
tuebl@travis:~/tuebl$ rake db:migrate
DEPRECATION WARNING: The configuration option `config.serve_static_assets` has been renamed to `config.serve_static_files` to clarify its role (it merely enables serving everything in the `public` folder and is unrelated to the asset pipeline). The `serve_static_assets` alias will be removed in Rails 5.0. Please migrate your configuration files accordingly. (called from block in <top (required)> at /home/tuebl/tuebl/config/environments/production.rb:23)
==<PHONE_NUMBER>0812 ChangeKeysToBigint: migrating ===============================
==<PHONE_NUMBER>0812 ChangeKeysToBigint: migrated (1.6985s) ======================

tuebl@travis:~/tuebl$

RangeError in BooksController#apicreate
12491981479219 is out of range for ActiveRecord::Type::Integer with limit 4

tuebl@travis:~/tuebl$ cat config/initializers/bigint_pk.rb
require 'bigint_pk'

BigintPk.setup do |config|
  # Remove this line or set config.enabled to false to prevent future migrations
  # from defaulting to 64bit ints.  This does not affect the initial migration
  # created during installation.
  #
  config.enabled = true
end

and

tuebl@travis:~/tuebl$ cat db/migrate/20151207010812_change_keys_to_bigint.rb
# This file is auto-generated when running `rails generate bigint_pk:install`
# by the gem rails-bigint-pk (https://github.com/caboteria/rails-bigint-pk).

class ChangeKeysToBigint < ActiveRecord::Migration
  def change
    Rails.application.eager_load!
    ActiveRecord::Base.subclasses.select do |type|
      type.primary_key.present? and connection.table_exists? type.table_name
    end.each do |type|
      BigintPk.update_primary_key type.table_name, type.primary_key

      type.reflect_on_all_associations.select do |association|
        ( association.macro == :belongs_to ) &&
          connection.column_exists?( type.table_name, association.foreign_key )
      end.each do |belongs_to_association|
        BigintPk.update_foreign_key type.table_name, belongs_to_association.foreign_key
      end
    end
  end
end

I appreciate you working with me to figure out this problem, it's creating a huge headache right now :P

I just went through the installation instructions and noticed that they should be
gem 'rails-bigint-pk', '~>1.2.0'

If you're using 1.1.0 (based on the current README which is wrong) then it's likely to not work, but I'm surprised that you didn't have installation problems.  When I tried to install 1.1.0 with Rails 4.2.5 there were gem version errors.  You want to have 1.2.0 which is the current version.
Hopefully this helps, in any case I'll fix the README.

I am using
gem 'rails-bigint-pk', github: 'caboteria/rails-bigint-pk'

Looking at the books table above, it looks as if the PK and some of the FK's (author_id, series_id) are big but one of them (featured_id) is still 32 bits.  Is it possible that not all of your FK references use the "references" declaration?  See https://github.com/caboteria/rails-bigint-pk#gotchas for more info.

@teamcoltra can you post the migration that created featured_id and can you also let us know if that key was created before or after installing rails-bigint-pk (and then running the rails-bigint-pk generator).  Thanks.

@caboteria I almost wonder if we should try to support the add_column case.  Initially I didn't because although it's a bit of a footgun I worried that the code would have to be too clever and we'd accidentally make 64 bit ints for non-fk fields.

@caboteria I'm working with @teamcoltra and it seems that in addition to what you pointed out there's also a tagging gem which is using int for its table's id fields and from the stack traces it seems that the error may be occurring in that gem.
At the very least we might need some documentation mentioning that third party gems will not be automatically migrated to bigint which will inevitably cause problems and produce really terrible error messages. I'm not entirely sure why they wouldn't be picked up as ActiveRecord::Base.subclasses though. Still figuring that out.

I can confirm that the problem seems to be rooted in the fact that the migration code (as shown above) did not actually update every foreign key and even missed some of the primary keys of the tables. I ran the code myself with no difference in the resulting schema dump from the database.


Disabling Network Policy Enforcement After Enabling It

Hello!
I was trying to find any sort of documentation on how you would disable network policy enforcement via the CNI after you've enabled it.
I know to enable the network policy enforcement via the CNI, you can set the following configuration on the CNI:
{ "enableNetworkPolicy": "true" }
We are wondering if there's a way to "backout" having the CNI enforce the network policies in the cluster (in the event something were to happen after we began enforcing them).
I attempted in our testing environment to update the addon and set:
{ "enableNetworkPolicy": "false" }
And that begins to rotate out the aws-node pods in the cluster, but on the first one attempting to come up the aws-eks-nodeagent container goes into a crash backoff loop:
2024-01-24 16:50:29.677598122 +0000 UTC Logger.check error: failed to get caller
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x55cbd278b108]

goroutine 43 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x55cbd3452be0?, 0x55cbd48b73e0?})
        /root/sdk/go1.21.3/src/runtime/panic.go:914 +0x21f
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).configureeBPFProbes(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {0xc00004c678, 0x17}, {0xc00062ef00?, 0x1, 0xc000454240?}, {0xc00073b380, 0x2, ...}, ...)
        /workspace/controllers/policyendpoints_controller.go:258 +0x3c8
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcilePolicyEndpoint(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, 0xc0007bcd00)
        /workspace/controllers/policyendpoints_controller.go:232 +0x7f0
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcile(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {{{0xc000460880, 0x7}, {0xc000468b60, 0x1b}}})
        /workspace/controllers/policyendpoints_controller.go:149 +0x22f
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).Reconcile(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {{{0xc000460880, 0x7}, {0xc000468b60, 0x1b}}})
        /workspace/controllers/policyendpoints_controller.go:130 +0x125
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x55cbd37e1f08?, {0x55cbd37dfd38?, 0xc0006e91a0?}, {{{0xc000460880?, 0xb?}, {0xc000468b60?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004408c0, {0x55cbd37dfd70, 0xc00061a3c0}, {0x55cbd3547580?, 0xc00060ca40?})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316 +0x3c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004408c0, {0x55cbd37dfd70, 0xc00061a3c0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 105
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:223 +0x565

Is there a more proper way to stop having the CNI enforce the network policies?
Thanks!

Environment:

Kubernetes version (use kubectl version): v1.25.16-eks-8cb36c9
CNI Version: v1.15.4-eksbuild.1
Network Policy Agent Version: v1.0.6-eksbuild.1
OS (e.g: cat /etc/os-release): Amazon Linux 2
Kernel (e.g. uname -a): 5.10.199-190.747.amzn2.x86_64

I think this is a duplicate of https://github.com/aws/aws-network-policy-agent/issues/135.

Going to close this one out.  For anyone who happens to stumble on this, @jayanthvn did a good write up on how to disable enforcing the network policies after enabling them (if needed) in the linked issue above:

Delete all NP resources
Set enable-network-policy-controller to false in ConfigMap amazon-vpc-cni (kube-system NS). This will disable the controller.
Set the 'enableNetworkPolicy' parameter to false. This will disable the agents on the nodes.

We had tested this in our environments as well with no issues.


⚠ ClamAV has degraded performance

In a8fd811, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 307 ms

Resolved: ClamAV performance has improved in 5181931.


After upgrading Apple command line tools lib does not compile

👋 as issue title states, after I have upgraded apple command line tools to the latest, I no longer can compile this lib.
Any ideas what is wrong here and how this can be fixed?
Here is my command line tool version:
libpg_query git:(13-latest) pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: <IP_ADDRESS>.1.1663981106
volume: /
location: /
install-time:<PHONE_NUMBER>
groups: com.apple.FindSystemFiles.pkg-group

clang version:
➜  libpg_query git:(13-latest) clang --version
Apple clang version 14.0.0 (clang-14<IP_ADDRESS>)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Here is the output:
➜  libpg_query git:(13-latest) make
compiling src/pg_query.c
compiling src/pg_query_deparse.c
compiling src/pg_query_fingerprint.c
compiling src/pg_query_json_plpgsql.c
compiling src/pg_query_normalize.c
compiling src/pg_query_outfuncs_json.c
compiling src/pg_query_outfuncs_protobuf.c
compiling src/pg_query_parse.c
compiling src/pg_query_parse_plpgsql.c
compiling src/pg_query_readfuncs_protobuf.c
compiling src/pg_query_scan.c
compiling src/pg_query_split.c
compiling src/postgres/src_backend_catalog_namespace.c
compiling src/postgres/src_backend_catalog_pg_proc.c
compiling src/postgres/src_backend_commands_define.c
compiling src/postgres/src_backend_libpq_pqcomm.c
compiling src/postgres/src_backend_nodes_bitmapset.c
compiling src/postgres/src_backend_nodes_copyfuncs.c
compiling src/postgres/src_backend_nodes_equalfuncs.c
compiling src/postgres/src_backend_nodes_extensible.c
compiling src/postgres/src_backend_nodes_list.c
compiling src/postgres/src_backend_nodes_makefuncs.c
compiling src/postgres/src_backend_nodes_nodeFuncs.c
compiling src/postgres/src_backend_nodes_value.c
compiling src/postgres/src_backend_parser_gram.c
In file included from gram.y:46:
In file included from ./src/postgres/include/postgres.h:46:
In file included from ./src/postgres/include/c.h:1330:
In file included from ./src/postgres/include/port.h:17:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netdb.h:91:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netinet/in.h:81:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:425:1: error: expected ';' after top level declarator
__CCT_DECLARE_CONSTRAINED_PTR_TYPES(struct sockaddr, sockaddr);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:587:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
#define __CCT_DECLARE_CONSTRAINED_PTR_TYPES(basetype, basetag)                                      \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:413:1: note: expanded from here
__CCT_DECLARE_CONSTRAINED_PTR_TYPE_3
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:563:14: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPE_3'
        __CCT_DEFER(__CONCAT, basetag,  __CCT_CONTRACT_LIST_TO_TAGGED_SUFFIX_1(kind))
                    ^
In file included from gram.y:46:
In file included from ./src/postgres/include/postgres.h:46:
In file included from ./src/postgres/include/c.h:1330:
In file included from ./src/postgres/include/port.h:17:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netdb.h:91:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netinet/in.h:81:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:425:1: error: expected ';' after top level declarator
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:588:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
__CCT_DECLARE_CONSTRAINED_PTR_TYPE(basetype, basetag, REF);                                         \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:418:1: note: expanded from here
__CCT_DECLARE_CONSTRAINED_PTR_TYPE_4
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:567:11: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPE_4'
                 * __CCT_CONTRACT_TO_ATTR(kind2)                                                        \
                 ^
In file included from gram.y:46:
In file included from ./src/postgres/include/postgres.h:46:
In file included from ./src/postgres/include/c.h:1330:
In file included from ./src/postgres/include/port.h:17:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netdb.h:91:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netinet/in.h:81:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:471:1: error: typedef redefinition with different types ('struct sockaddr_storage *' vs 'struct sockaddr *')
__CCT_DECLARE_CONSTRAINED_PTR_TYPES(struct sockaddr_storage, sockaddr_storage);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:587:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
#define __CCT_DECLARE_CONSTRAINED_PTR_TYPES(basetype, basetag)                                      \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 4 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:540:38: note: expanded from macro '__CCT_CONTRACT_TO_ATTR'
#define __CCT_CONTRACT_TO_ATTR(kind) __CONCAT(__CCT_CONTRACT_ATTR_, kind)
                                     ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:427:1: note: expanded from here
__CCT_CONTRACT_ATTR_582
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:425:1: note: previous definition is here
__CCT_DECLARE_CONSTRAINED_PTR_TYPES(struct sockaddr, sockaddr);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:588:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
__CCT_DECLARE_CONSTRAINED_PTR_TYPE(basetype, basetag, REF);                                         \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 4 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:540:38: note: expanded from macro '__CCT_CONTRACT_TO_ATTR'
#define __CCT_CONTRACT_TO_ATTR(kind) __CONCAT(__CCT_CONTRACT_ATTR_, kind)
                                     ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:419:1: note: expanded from here
__CCT_CONTRACT_ATTR_582
^
In file included from gram.y:46:
In file included from ./src/postgres/include/postgres.h:46:
In file included from ./src/postgres/include/c.h:1330:
In file included from ./src/postgres/include/port.h:17:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netdb.h:91:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netinet/in.h:81:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:471:1: error: expected ';' after top level declarator
__CCT_DECLARE_CONSTRAINED_PTR_TYPES(struct sockaddr_storage, sockaddr_storage);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:587:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
#define __CCT_DECLARE_CONSTRAINED_PTR_TYPES(basetype, basetag)                                      \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:426:1: note: expanded from here
__CCT_DECLARE_CONSTRAINED_PTR_TYPE_3
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:563:14: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPE_3'
        __CCT_DEFER(__CONCAT, basetag,  __CCT_CONTRACT_LIST_TO_TAGGED_SUFFIX_1(kind))
                    ^
In file included from gram.y:46:
In file included from ./src/postgres/include/postgres.h:46:
In file included from ./src/postgres/include/c.h:1330:
In file included from ./src/postgres/include/port.h:17:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netdb.h:91:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netinet/in.h:81:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:471:1: error: typedef redefinition with different types ('struct sockaddr_storage *' vs 'struct sockaddr *')
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:588:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
__CCT_DECLARE_CONSTRAINED_PTR_TYPE(basetype, basetag, REF);                                         \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 4 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:540:38: note: expanded from macro '__CCT_CONTRACT_TO_ATTR'
#define __CCT_CONTRACT_TO_ATTR(kind) __CONCAT(__CCT_CONTRACT_ATTR_, kind)
                                     ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:6:1: note: expanded from here
__CCT_CONTRACT_ATTR_582
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:425:1: note: previous definition is here
__CCT_DECLARE_CONSTRAINED_PTR_TYPES(struct sockaddr, sockaddr);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:588:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
__CCT_DECLARE_CONSTRAINED_PTR_TYPE(basetype, basetag, REF);                                         \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 4 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:540:38: note: expanded from macro '__CCT_CONTRACT_TO_ATTR'
#define __CCT_CONTRACT_TO_ATTR(kind) __CONCAT(__CCT_CONTRACT_ATTR_, kind)
                                     ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:419:1: note: expanded from here
__CCT_CONTRACT_ATTR_582
^
In file included from gram.y:46:
In file included from ./src/postgres/include/postgres.h:46:
In file included from ./src/postgres/include/c.h:1330:
In file included from ./src/postgres/include/port.h:17:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netdb.h:91:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/netinet/in.h:81:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h:471:1: error: expected ';' after top level declarator
__CCT_DECLARE_CONSTRAINED_PTR_TYPES(struct sockaddr_storage, sockaddr_storage);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:588:101: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPES'
__CCT_DECLARE_CONSTRAINED_PTR_TYPE(basetype, basetag, REF);                                         \
                                                                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:580:2: note: expanded from macro '\
__CCT_DECLARE_CONSTRAINED_PTR_TYPE'
        __CCT_DISPATCH(__CCT_DECLARE_CONSTRAINED_PTR_TYPE, basetype, basetag, __VA_ARGS__)
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:548:2: note: expanded from macro '__CCT_DISPATCH'
        __CCT_DISPATCH1(base, __CCT_COUNT_ARGS(__VA_ARGS__), __VA_ARGS__)
        ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:113:26: note: expanded from macro '__CONCAT'
#define __CONCAT(x, y)   x ## y
                         ^
<scratch space>:5:1: note: expanded from here
__CCT_DECLARE_CONSTRAINED_PTR_TYPE_4
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/constrained_ctypes.h:567:11: note: expanded from macro '__CCT_DECLARE_CONSTRAINED_PTR_TYPE_4'
                 * __CCT_CONTRACT_TO_ATTR(kind2)                                                        \
                 ^
6 errors generated.
make: *** [src/postgres/src_backend_parser_gram.o] Error 1

I'm running into the same problem trying to install the pg_query gem. I've tried downgrading my xcode cli tools version and that didn't work either

@ksarunas for my purposes removing all of xcode and reinstalling it fresh seems to have worked. No clue why?
$ sudo rm -rf /Library/Developer/CommandLineTools
$ xcode-select --install

Then once that was done I reset xcode
$ sudo xcode-select -r

@DanDobrick Thanks for confirming that re-installing XCode resolved the matter - @ksarunas could you try to see if that fixes the issue for you as well?
From the error message this appears to be a problem with the system headers rather than a clear issue in the C code of libpg_query itself.

Interesting, I had this problem and removing / re-installing Xcode fixed it temporarily.
But then I had another upgrade to Ventura 13.0.1 and it apparently happened again. So maybe we have a situation where every upgrade will cause this to break again.
This is similar to the situation that macOS Catalina introduced with node-gyp: https://github.com/nodejs/node-gyp/blob/main/macOS_Catalina.md

Hm... removing Xcode and reinstalling didn't work this time 🤔

Oh, maybe this was indeed a problem with libpg_query itself? Seems like it was potentially fixed in 2.2.0 here:

https://github.com/pganalyze/pg_query/issues/263

Maybe I tricked myself before that uninstalling + reinstalling Xcode fixed things?
Or maybe there were 2 issues:

Upgrading to Ventura caused Xcode to be broken - fixed by uninstalling + reinstalling Xcode
pg_query before 2.2.0 has an issue with newer Xcode / Command Line Tools


Included condition to allow @ character on console Input

If the digit is D2 (@ special character) then it will go back one character and the next will replace it
I included a condition to avoid this and include the @ special character as an input character

hey @c3sxr , could you please add some tests for this bug?

Hi @afucher !
I made another push to the forked branch, I included a new class for the email test, applied the changes and returned the version back.
Let me know if there's something missing, I'll be glad to help 👍
Thank you!

Hey @c3sxr for some reason it is not reflecting any new changes here in the Pull Request 😕

Can you please look again, the File changes tab is showing my new changes

Thanks!

Oh those files I saw, my comment was regarding adding a new test that cover the @ bug

Oh you mean on the InquirerUnitTest? Sure let me include it

Can you please take a look and let me know if that looks right?


Not working on Monterey 12.0.1

Monterey likely isn't supported yet, but I still wanted to post this.
Installing and enabling Vimari does nothing. It says it's enabled in the config, but no keyboard shortcuts work, there is no visible change when it's enabled.

I'm also with you. 😢

Same here. Adding a modifier key seems to work (like "modifier": "a", then "a" + "j" works). But without a modifier key, nothing works.

For me, adding anything to the excludedUrls breaks the extension in the way described. I can only get the extension to work if excludedUrls is an empty array.


Add some info in Quantum dir

The readme at https://github.com/ModECI/MDF/tree/main/examples/Quantum is looking pretty empty.
@rpradyumna is there any (preliminary) info you could add here?

Fixed in #81


Ryan403 patch 2

Thank you @ryan403! Could you sign the Contributor License Agreement above? Then I will merge this pull request.


Add deprecation notice to docsite

Instead of dry-rb/dry-rb.org#389.
I linked to dry-core's documentation (just the main page, rather than the future Equalizer page because we'd need to know the version, instead of the main page which redirects properly.
I also updated the examples to use dry/core but not sure if that's right. I think it might make sense to get rid of everything except the deprecation notice?

I think it might make sense to get rid of everything except the deprecation notice?

I agree

Updated to remove everything but the deprecation notice


Basic usage for html md

Hello,
I'm new to node and js.
I am trying to generate the htm and markdown rendering for the streetlights.yml example file.
After installing @asyncpi/generator and html and markdown templates with npm, I ran cli as
$ ag -o ./out streetlights.yml html`
$ ls out
LICENSE  index.js  readme.md

But not able to figure out what to do next to generate html from yml.
Any help is appreciated.
 You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

Something went wrong:
Error: invalid bin entry for package<EMAIL_ADDRESS>key=nopt, value=bin/nopt.js
    at /usr/local/lib/node_modules/npm/node_modules/bin-links/index.js:86:13
    at tryCatcher (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
    at MappingPromiseArray._promiseFulfilled (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/map.js:61:38)
    at MappingPromiseArray.PromiseArray._iterate (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise_array.js:114:31)
    at MappingPromiseArray.init (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise_array.js:78:10)
    at MappingPromiseArray._asyncInit (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/map.js:30:10)
    at _drainQueueStep (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:142:12)
    at _drainQueue (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:131:9)
    at Async._drainQueues (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:147:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:17:14)
    at processImmediate (internal/timers.js:456:21)

Hi @uraala, we just released a new version of the generator that supports remote templates, they are no longer a part of the generator but stored remotely, on npm or git or tar file. So you should no longer refer to remplates as html or markdown but after their npm names like @asyncapi/html-template or @asyncapi/markdown-template. I see we forgot to update README with proper docker example but look at the other examples.
So you should do ag -o ./out streetlights.yml @asyncapi/html-template

Hi @derberg, thanks for the info.
 You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

trace
Something went wrong:
Error: invalid bin entry for package<EMAIL_ADDRESS>key=markdown-it, value=bin/markdown-it.js
    at /usr/local/lib/node_modules/npm/node_modules/bin-links/index.js:86:13
    at tryCatcher (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
    at MappingPromiseArray._promiseFulfilled (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/map.js:61:38)
    at MappingPromiseArray.PromiseArray._iterate (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise_array.js:114:31)
    at MappingPromiseArray.init (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise_array.js:78:10)
    at MappingPromiseArray._asyncInit (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/map.js:30:10)
    at _drainQueueStep (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:142:12)
    at _drainQueue (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:131:9)
    at Async._drainQueues (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:147:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:17:14)
    at processImmediate (internal/timers.js:456:21)

Hope it helps.

@fmvilas you see the error, it is exactly the same I had during the review. I think this template works only on node 13.11 and if you try on other versions, like in Docker we have 12, it fails.
@uraala for this specific template the only workaround for now is to use it with node 13.11, sorry

Thanks for raising the issue, @uraala. I just pushed a fix: https://github.com/asyncapi/generator/pull/273.

Thanks for raising the issue, @uraala. I just pushed a fix: #273.

@fmvilas : welcome!
I really liked how it is possible to define a schema for mqtt events and payloads.
Looking forwards for the same for NATS

@fmvilas you see the error? it is exactly the same I had during the review. I think this template works only on node 13.11 and if you try on other versions, like in Docker we have 12, it fails.
@uraala for this specific template the only workaround for now is to use it with node 13.11, sorry

@derberg : Thanks for the update

@uraala You can probably lend a hand to @jonaslagoni on this template: github.com/asyncapi/ts-nats-template/. The first PR is here: https://github.com/asyncapi/ts-nats-template/pull/1.


fix(cms-api): get file stats from uploaded file in filestorage

@johnnyomair we have to port this for v3 pls :kissing_heart:?


Filtering by relationship does not work, results in an error

Describe the bug
After updating Ash and extensions, filtering by relationship using a generated filter argument results in the following error:
** (ArgumentError) comparison with nil is forbidden as it is unsafe. If you want to check if a value is nil, use is_nil/1 instead

    (ecto 3.11.2) lib/ecto/query/builder.ex:1069: Ecto.Query.Builder.not_nil!/1
    (ash_postgres 1.5.28) lib/expr.ex:939: anonymous fn/3 in AshPostgres.Expr.do_dynamic_expr/6
    (ecto 3.11.2) lib/ecto/query/builder/dynamic.ex:79: Ecto.Query.Builder.Dynamic.expand/3
    (ecto 3.11.2) lib/ecto/query/builder/dynamic.ex:49: Ecto.Query.Builder.Dynamic.fully_expand/2
    (ecto 3.11.2) lib/ecto/query/builder/filter.ex:133: Ecto.Query.Builder.Filter.filter!/7
    (ash_postgres 1.5.28) lib/data_layer.ex:3201: anonymous fn/2 in AshPostgres.DataLayer.add_filter_expression/2
    (elixir 1.16.3) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ash_postgres 1.5.28) lib/data_layer.ex:3110: AshPostgres.DataLayer.filter/4
    (ash 2.21.15) lib/ash/query/query.ex:2853: Ash.Query.maybe_filter/3
    (ash 2.21.15) lib/ash/query/query.ex:2724: Ash.Query.data_layer_query/2
    (ash 2.21.15) lib/ash/actions/read/read.ex:405: anonymous fn/5 in Ash.Actions.Read.do_read/4
    (ash 2.21.15) lib/ash/actions/read/read.ex:537: Ash.Actions.Read.maybe_in_transaction/3
    (ash 2.21.15) lib/ash/actions/read/read.ex:227: Ash.Actions.Read.do_run/3
    (ash 2.21.15) lib/ash/actions/read/read.ex:50: anonymous fn/3 in Ash.Actions.Read.run/3
    (ash 2.21.15) lib/ash/actions/read/read.ex:49: Ash.Actions.Read.run/3
    (my_app 0.1.0) lib/my_app.ex:1: MyApp.read/2
    (ash_graphql 0.28.1) lib/graphql/resolver.ex:447: AshGraphql.Graphql.Resolver.resolve/2
    (absinthe 1.7.6) lib/absinthe/phase/document/execution/resolution.ex:234: Absinthe.Phase.Document.Execution.Resolution.reduce_resolution/1
    (absinthe 1.7.6) lib/absinthe/phase/document/execution/resolution.ex:189: Absinthe.Phase.Document.Execution.Resolution.do_resolve_field/3
    (absinthe 1.7.6) lib/absinthe/phase/document/execution/resolution.ex:174: Absinthe.Phase.Document.Execution.Resolution.do_resolve_fields/6

To Reproduce
There are 2 resources, and there's a belongs_to relationship between them:
defmodule MyApp.ResourceA do
  use Ash.Resource,
    data_layer: AshPostgres.DataLayer,
    extensions: [AshGraphql.Resource, AshArchival.Resource]

  actions do
    read do
      primary? true
    end
  end

  attribues do
    uuid_primary_key :id

    attribue :date, :date do
      allow_nil? false
    end

    attribute :tenant_id, :uuid do
      allow_nil? false
    end
  end

  multitenancy do
    strategy :attribute
    attribute :tenant_id
    global? true
  end

  graphql do
    type :resource_a
  end
end

defmodule MyApp.ResourceA do
  use Ash.Resource,
    data_layer: AshPostgres.DataLayer,
    extensions: [AshGraphql.Resource, AshArchival.Resource]

  actions do
    read do
      primary? true
    end
  end

  attribues do
    uuid_primary_key :id

    attribute :resource_a_id, :uuid do
      allow_nil? false
    end

    attribute :tenant_id, :uuid do
      allow_nil? false
    end
  end

  multitenancy do
    strategy :attribute
    attribute :tenant_id
    global? true
  end

  relationships do
    belongs_to :resource_a, MyApp.ResourceA do
      allow_nil? false
      define_attribute? false
    end
  end

  graphql do
    type :resource_b

    queries do
      list(:resource_b, :read)
    end
  end
end

Then running this query:
query DateFilterQuery($date: String!) {
  resource_b(filter: { resource_a: { date: { eq: $date } } }) {
    id
    tenant_id
  }
}

Results in the error mentioned above.
Expected behavior
Expected not to error in this case and return properly filtered values. This worked for me with Ash 2.18.1/AshGraphql 0.26.9/AshPostgres 1.4.0/AshArchival 0.1.4.
** Runtime

Elixir version 1.16.3
Erlang version 26.1.2
OS MacOS Ventura 13.5.2
Ash version 2.21.15
any related extension versions

AshGraphQL 0.28.1
AshPostgres 1.5.28
AshArchival 0.1.5

Additional context
I'm not sure if this is a AshGraphQL bug or Ash/AshPosgres bug.
I'll be able to start migrating to Ash 3 in a couple of weeks.

Interesting... 🤔 What value is being passed for $date in that case?

A date in the ISO format, for example "2024-07-01".

Oh, sorry, I've just realized, this is for upgrading to a non 3.x version, right?

I feel pretty confident that this issue is either resolved in 3.x or will be much easier to address in 3.x. Is there something forcing you to upgrade to latest 2.x version?

Oh, sorry, I've just realized, this is for upgrading to a non 3.x version, right?

Right:
Ash 2.18.2 -> 2.21.15
AshGraphql 0.26.9 -> 0.28.1
AshPosgres 1.4.0 -> 1.5.28
AshArchival 0.1.4 -> 0.1.5

Is there something forcing you to upgrade to latest 2.x version?

No, I can downgrade back to the latest version that works for me. Just wanted to upgrade to the latest 2.x version to make sure I'm in a good place to start migrating to 3.x

I think that is the best choice for now TBH. Less uncertainty in general :)


Basic Polymer integration

 binds a DAO (currently an eval'd expression - ugh!) to a Polymer
array, and then syncs changes between the DAO and the array. (That means both
upstream changes to the DAO and manipulating the array through your Polymer UI).
Check out polymer/index.html for a todo-list demo app.

I updated my email for this repository to be my @google.com one, so hopefully the CLA thing won't whine about it. In any case, I'm a licensed contributor.

PTAL, all addressed.
Two coding style questions:

My understanding is that if ( foo ) return; and similar are allowed, but if it needs to wrap onto two lines, then it needs braces. Correct?
Loops are spelled for ( var i = 0; i < foo.length; i++ ) {, with the semicolons hugging left and spaced right.

Loops should be spelled for ( var i = 0 ; i < foo.length ; i++ ) {
spaces on both sides of the semicolon

LGTM overall, see if my alternative to eval will work.
Feel free to merge this before documenting foam-select


Added proper TraceID in dummy segments

Issue #, if available:
Description of changes:
Added call to generate proper TraceID in Dummy Segment case. Tested this change with listQueues AWS SDK downstream call.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #223 into master will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #223      +/-   ##
==========================================
+ Coverage   84.53%   84.60%   +0.06%
==========================================
  Files          80       80
  Lines        3130     3131       +1
==========================================
+ Hits         2646     2649       +3
+ Misses        484      482       -2

Impacted Files
Coverage Δ

aws_xray_sdk/core/models/dummy_entities.py
86.66% <100.00%> (+0.30%)
:arrow_up:

aws_xray_sdk/core/sampling/local/reservoir.py
100.00% <0.00%> (+11.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d12c83...2ae9e4d. Read the comment docs.


Disassembly editor is not getting closed automatically after debug session ended

Does this issue occur when all extensions are disabled?: Yes

VSCode Version: 1.62.3
OS: Windows_NT x64 10.0.19042
Steps to Reproduce:

Import any c/c++ project and start debugging
Open disassembly View and terminate the session

Currently we have to close the disassembly editor manually but it should be closed automatically once the debug session ends.

Hello,
In my opinion, it should be a preference, since I do not expect this behavior as a user.
Imo, if it does not open automatically, it should not close automatically as well

Hi @yannickowow , but it doesn't make sense to show disassembly view even when there is no active debug session.
So it makes more sense to close the view after the debug session is terminated.


Feature request: Flag customers when they subscribe to newsletter

Description
We are trying to implement a discount code for new subscribers to our newsletter and want to make sure the account is really subscribed before the discount code applies.
This would be possible within the Rule Builder and the discount-conditions (via customer flag or group)
There is currently no automatic assignment, though.
Example
Customer subscribes to the newsletter, account gets automatically tagged 'newsletter-receiver' or joins group 'newsletter'

Is already fixed by this ticket.


Update README to reflect state with ACD

Looks like ACD isn't really an option for most people, and I think documentation should reflect that.

The state of amazon cloud drive is spelt out in the amazon cloud drive page: https://rclone.org/amazonclouddrive/
This is linked to off the front page of rclone.org
Did you come to rclone through the github page?

Respectfully, I found all the info on this page and the linked forum utterly confusing. The TL;DR is painfully missing in my opinion. It’s basically unusable, and I think even you stated that you have no way of testing it. I find it misleading to claim that recline supports acd, even if it’s technically correct and it does support it (amazon doesn’t support rclone basically). I personally spent a good chunk of time just going through docs and coming to this rather grim realization, which was most succinctly described on the linked GitHub issue.
I think people deserve the simple truth in the most concise way. I think my PR reflects this simply and honestly.
Please forgive me if I misunderstood or misinterpreted anything. I came to rclone specifically for its acd integration.

I'm sorry that the rclone docs caused you to waste time, they aren't as clear as they could be.
Yes you are right ACD is technically supported, but only if you have your own keys which is unlikely.
What I'd like to do is come up with a clear statement of the current situation at the start of https://github.com/ncw/rclone/blob/master/docs/content/amazonclouddrive.md - something like this

Important: Rclone supports Amazon Cloud Drive only if you have your own set of keys.  Unfortunately the amazon cloud drive developer program is now closed to new entries (link to page) so if you don't already have your own set of keys you will not be able to use rclone with amazon cloud drive.

Then link to that statement it in the README.md and https://github.com/ncw/rclone/blob/master/docs/content/about.md (which becomes the rclone.org front page)  with the text "[important note]" or something like that?
What do you think about that?

@ncw it all sounds great overall. It would be a huge improvement to clarity in my opinion. Thanks for being so open about it.
A couple of questions / suggestions though:

I wonder if the google apps proxy is run/maintained by you(?) then is it really still helpful in any way right now? Or could it be deactivated? If it's not run by you, then I would probably remove references to it, unless it's in some kind of historic archive... Otherwise, most of the docs around it might be confusing right now (and a potential leak of credential, even though I trust that it's not storing them anyway, but it still exposes exposure).
Linking to this note/page with [important note], sounds like a good idea too. 👍

I totally understand the natural urge to stick to documentation and code that was definitely useful, and these have some value to show the work and effort that's gone into them. At the same time, that's one of the beauties of version control. History is still there if you go looking :)
Besides that, perhaps the documentation could also focus on some kind of a petition / call-to-action to encourage Amazon to relax those restrictions. I think it would serve the community and rclone better. But that's entirely up to you. I don't know how strongly you feel about this or whether you want to prompt more action in this direction.
(sorry again if this all sounds like couch coaching. I'm a complete outsider and my opinion shouldn't count in any serious way. I sincerely appreciate your openness to have this discussion in the first place)

@ncw perfect! I think things are much(!) clearer now, and I think the call-to-action is very helpeful, and I really hope it leads somewhere.
Just one thing unrelated to this change that I spotted

Since rclone doesn't not currently have its own Amazon Drive credentials...

Thank you so much! I hope to use and contribute to rclone in future, even though my primary use case is not supported (yet!) ;-)

Thanks @ncw. I really appreciate it, and I hope other users would too, despite the fact that ACD isn't very useable for most people (yet!).

@ncw just checked the link from the readme, and the ACD page doesn't seem to show the new updates for me. Maybe it's still not "deployed/compiled" yet?

The website updates at the release time normally, however I'll change it now...