[bug] Error: A critical and unhandled exception has occurred:

Unfortunately, there is no stack trace for some reason.  Also, there is an unhandled promise somewhere which makes debugging even harder.
#### Testing For Type updateEIP ####
-- Testing Must Have --
-- Testing Must Not Have --
!! is type updateEIP
eip-4519.md passed!
A critical and unhandled exception has occurred:
	Message: not found
	Data: (there is no data for this error)(cc @alita-moore, @mryalamanchi)
Error: A critical and unhandled exception has occurred:
	Message: not found
	Data: (there is no data for this error)(cc @alita-moore, @mryalamanchi)
(node:1725) UnhandledPromiseRejectionWarning: A critical and unhandled exception has occurred:
	Message: not found
	Data: (there is no data for this error)(cc @alita-moore, @mryalamanchi)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:1725) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1725) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Error: Process completed with exit code 1.

PR that caused the error (multiple runs of the bot, same error):
https://github.com/ethereum/EIPs/pull/5155/files
It updates some author contacts (email to github) and it modifies some images.  Either could potentially be related to the root cause of the exception.

Found the issue in the PR. Duplicate of #63


parameters files databse

How to contribute to the file parameters database?

@mstaring?

It is already done, thanks!
One should create an account in elastix wiki then wait for approval. I suggest providing a php or java form to fill by the contributor. This makes it easier and faster.


@Size causing java.lang.OutOfMemoryError: Java heap space

Hi random-beans Team,
@Size on String is causing java.lang.OutOfMemoryError: Java heap space.
This bug is not always happening, but during debugging there are always problems.
Example:
public class TestString {
    @Size(min = 2)
    public String str1;
}

Also it is not working at all with validation-api pre 2.0, even if Java heap space doesn't appear @Size is just ignored.
Best regards,
Redid

This bug is not always happening

Yeah, with random beans, we have random bugs 😄 Sorry, I couldn't resist on this..
Anyway, thank you for pointing this out!
Will take a look at this and get back to you asap

Hi,
I'm not able to reproduce the issue even in a debug session.
Here is the type:
public class Person {

    @Size(min = 50, max = 100)
    private String name;

    public String getName() {
        return name;
    }

}

And the (passing) test:
    @Test
    void testSizeOnString() {
        Person person = EnhancedRandom.random(Person.class);
        Assertions.assertThat(person).isNotNull();
        Assertions.assertThat(person.getName().length()).isBetween(50, 100);
    }

Also it is not working at all with validation-api pre 2.0

The previous test uses:
<validation-api.version>1.1.0.Final</validation-api.version>
<hibernate-validator.version>5.4.1.Final</hibernate-validator.version>

and it passes. Can you provide the version of the API and implementation you use ?


Time for some sort of release?

I know there are still issues like #3 but maybe it is time to create a tagged release candidate if not a tagged stable release?

A release would be helpful +1. This is only available on edge, making it unusable for security patches only.


Bugfix: fix strawberry.enum_value handling

Description
Currently strawberry.enum_value causes errors. Here I add a test for the issue, a bugfix and some documentation about the way it works.
note: you cannot differ keys with the same value in a resolver.
This is a limitation by python enum itself (uses singletons for values).
note: I recreate the Enum if an EnumValueDefinition instance is detected, otherwise it breaks the logic
Types of Changes

[ ] Core
[x] Bugfix
[ ] New feature
[ ] Enhancement/optimization
[ ] Documentation

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Thanks for adding the RELEASE.md file!

Here's a preview of the changelog:

Fix enum issues with strawberry.enum_value.
BREAKING: When strawberry.enum_value is used the wrapped Enum is re-created.

Here's the preview release card for twitter:

Here's the tweet text:
🆕 Release (next) is out! Thanks to Alexander for the PR 👏

Get it here 👉 https://github.com/strawberry-graphql/strawberry/releases/tag/(next)

fixes #2179 correctly


Investigate online STT engines

Leading candidates:
Google: https://cloud.google.com/speech-to-text/
Wit.ai: https://wit.ai/


Unable to do an initial load into "bigquery" with any other write_disposition then "append" (in autodetect schema mode)

dlt version
1.0.0
Describe the problem
For some reason that escapes me, at one point I'm no longer able to load any data into BigQuery using a write_disposition other than 'append'. The reason for this is that replace and merge both truncate some tables and for reasons unknown to me, dlt is longer able to detect that the tables don't exist and the pipeline fails.
Expected behavior
The expected behavior is to be able to do an initial load in a non existing dataset using a write_disposition other than append.
Steps to reproduce
Here is my test script:
import dlt
from dlt.common.libs.sql_alchemy import sqltypes
from dlt.destinations.adapters import bigquery_adapter
from dlt.sources import DltSource, incremental
from dlt.sources.sql_database import sql_table

def sql_type_adapter(sql_t):
    if isinstance(sql_t, sqltypes.ARRAY):
        return sqltypes.JSON
    return sql_t

tables_configs: dict[str, dict] = {
    "applicant": {"cursor_path": "updated_at"},
    "application": {},
}

@dlt.source
def apply_db(conn_uri: str) -> DltSource:  # type: ignore[misc]
    for table_name, table_config in tables_configs.items():
        cursor_path = table_config.get("cursor_path")
        t_resource = sql_table(
            credentials=conn_uri,
            table=table_name,
            incremental=incremental(cursor_path=cursor_path) if cursor_path else None,
            detect_precision_hints=True,
            type_adapter_callback=sql_type_adapter,
        )
        bigquery_adapter(t_resource, partition=cursor_path, autodetect_schema=True)  # type: ignore[arg-type]
        assert isinstance(t_resource.columns, dict)
        file_format = (
            "jsonl" if any(col.get("data_type") == "json" for col in t_resource.columns.values()) else "parquet"
        )
        t_resource.apply_hints(file_format=file_format)  # type: ignore[arg-type]
        yield t_resource

if __name__ == "__main__":
    from dlt import Pipeline

    from nxt.config import __config__ as cfg

    pipeline: Pipeline = dlt.pipeline(
        pipeline_name="apply_db",
        destination="bigquery",
        staging="filesystem",
        dataset_name=cfg.apply_db.dataset,
        dev_mode=True,
    )

    source = apply_db(f"{cfg.apply_db.dsn}")
    load_info = pipeline.run(source, write_disposition="merge")

    load_info.raise_on_failed_jobs()
    print(load_info)

    last_trace = pipeline.last_trace
    print(last_trace)

and here is the expected output (I've also adding some additional logging to see the queries that are failing)
+ set -a
+ source .env
++ APPLY_DB__DATASET=drnick_apply
++ APPLY_DB__DSN=postgresql+psycopg2://<truncated>@<truncated>/apply
++ DESTINATION__BIGQUERY__LOCATION=europe-west1
++ DESTINATION__BIGQUERY__TRUNCATE_TABLES_ON_STAGING_DESTINATION_BEFORE_LOAD=false
++ DESTINATION__FILESYSTEM__BUCKET_URL=gs://<truncated>/dlt
+ set +a
++ echo nxt_dlt/tmp_apply_db.py
++ sed 's/\//./g'
++ sed 's/\.py$//'
+ module_name=nxt_dlt.tmp_apply_db
+ poetry run python -m nxt_dlt.tmp_apply_db
2024-09-25 17:43:54,415|[WARNING]|121360|126181426977664|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 17:43:55,235|[WARNING]|121360|126181426977664|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 17:43:55,235|[WARNING]|121360|126181426977664|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name applicant_id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 17:43:55,274|[INFO]|121360|126181426977664|dlt|__init__.py|bind:439|Bind incremental on applicant with initial_value: None, start_value: None, end_value: None
2024-09-25 17:43:55,281|[WARNING]|121360|126181426977664|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 17:43:55,396|[WARNING]|121360|126181426977664|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 17:43:55,396|[WARNING]|121360|126181426977664|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name applicant_id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 17:43:55,514|[DEBUG]|121360|126181426977664|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.267658.apply_db.applicant with file None and actual name /home/drnick/.dlt/pipelines/apply_db/normalize/0666a57b5c0993e7/1727279035.267658/new_jobs/applicant.599a4c07ba.0.typed-jsonl
2024-09-25 17:43:55,518|[DEBUG]|121360|126181426977664|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.267658.apply_db.application with file None and actual name /home/drnick/.dlt/pipelines/apply_db/normalize/0666a57b5c0993e7/1727279035.267658/new_jobs/application.9e7a53daa0.0.typed-jsonl
2024-09-25 17:43:55,572|[DEBUG]|121360|126181426977664|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.267658.apply_db._dlt_pipeline_state with file None and actual name /home/drnick/.dlt/pipelines/apply_db/normalize/0666a57b5c0993e7/1727279035.267658/new_jobs/_dlt_pipeline_state.6e8a7dd556.0.typed-jsonl
2024-09-25 17:43:55,629|[INFO]|121360|126181426977664|dlt|pool_runner.py|run_pool:75|Created none pool with 1 workers
2024-09-25 17:43:55,629|[DEBUG]|121360|126181426977664|dlt|pool_runner.py|run_pool:88|Running pool
2024-09-25 17:43:55,629|[INFO]|121360|126181426977664|dlt|normalize.py|run:263|Running file normalizing
2024-09-25 17:43:55,629|[INFO]|121360|126181426977664|dlt|normalize.py|run:266|Found 1 load packages
2024-09-25 17:43:55,643|[INFO]|121360|126181426977664|dlt|normalize.py|run:289|Found 3 files in schema apply_db load_id<PHONE_NUMBER>.267658
2024-09-25 17:43:55,643|[INFO]|121360|126181426977664|dlt|normalize.py|spool_schema_files:240|Created new load package<PHONE_NUMBER>.267658 on loading volume
2024-09-25 17:43:55,649|[INFO]|121360|126181426977664|dlt|worker.py|_get_items_normalizer:123|A file format for table _dlt_pipeline_state was specified to preferred in the resource so parquet format being used.
2024-09-25 17:43:55,649|[INFO]|121360|126181426977664|dlt|worker.py|_get_items_normalizer:169|Created items normalizer JsonLItemsNormalizer with writer ParquetDataWriter for item format object and file format parquet
2024-09-25 17:43:55,649|[DEBUG]|121360|126181426977664|dlt|worker.py|w_normalize_files:230|Processing extracted items in<PHONE_NUMBER>.267658/new_jobs/_dlt_pipeline_state.6e8a7dd556.0.typed-jsonl in load_id<PHONE_NUMBER>.267658 with table name _dlt_pipeline_state and schema apply_db
2024-09-25 17:43:55,651|[DEBUG]|121360|126181426977664|dlt|items_normalizers.py|__call__:208|Processed 1 lines from file<PHONE_NUMBER>.267658/new_jobs/_dlt_pipeline_state.6e8a7dd556.0.typed-jsonl
2024-09-25 17:43:55,651|[DEBUG]|121360|126181426977664|dlt|worker.py|w_normalize_files:236|Processed file<PHONE_NUMBER>.267658/new_jobs/_dlt_pipeline_state.6e8a7dd556.0.typed-jsonl
2024-09-25 17:43:55,651|[INFO]|121360|126181426977664|dlt|worker.py|_get_items_normalizer:123|A file format for table applicant was specified to parquet in the resource so parquet format being used.
2024-09-25 17:43:55,651|[INFO]|121360|126181426977664|dlt|worker.py|_get_items_normalizer:169|Created items normalizer JsonLItemsNormalizer with writer ParquetDataWriter for item format object and file format parquet
2024-09-25 17:43:55,651|[DEBUG]|121360|126181426977664|dlt|worker.py|w_normalize_files:230|Processing extracted items in<PHONE_NUMBER>.267658/new_jobs/applicant.599a4c07ba.0.typed-jsonl in load_id<PHONE_NUMBER>.267658 with table name applicant and schema apply_db
2024-09-25 17:43:55,655|[DEBUG]|121360|126181426977664|dlt|items_normalizers.py|__call__:208|Processed 1 lines from file<PHONE_NUMBER>.267658/new_jobs/applicant.599a4c07ba.0.typed-jsonl
2024-09-25 17:43:55,655|[DEBUG]|121360|126181426977664|dlt|worker.py|w_normalize_files:236|Processed file<PHONE_NUMBER>.267658/new_jobs/applicant.599a4c07ba.0.typed-jsonl
2024-09-25 17:43:55,656|[INFO]|121360|126181426977664|dlt|worker.py|_get_items_normalizer:123|A file format for table application was specified to parquet in the resource so parquet format being used.
2024-09-25 17:43:55,656|[INFO]|121360|126181426977664|dlt|worker.py|_get_items_normalizer:169|Created items normalizer JsonLItemsNormalizer with writer ParquetDataWriter for item format object and file format parquet
2024-09-25 17:43:55,656|[DEBUG]|121360|126181426977664|dlt|worker.py|w_normalize_files:230|Processing extracted items in<PHONE_NUMBER>.267658/new_jobs/application.9e7a53daa0.0.typed-jsonl in load_id<PHONE_NUMBER>.267658 with table name application and schema apply_db
2024-09-25 17:43:55,658|[DEBUG]|121360|126181426977664|dlt|items_normalizers.py|__call__:208|Processed 1 lines from file<PHONE_NUMBER>.267658/new_jobs/application.9e7a53daa0.0.typed-jsonl
2024-09-25 17:43:55,659|[DEBUG]|121360|126181426977664|dlt|worker.py|w_normalize_files:236|Processed file<PHONE_NUMBER>.267658/new_jobs/application.9e7a53daa0.0.typed-jsonl
2024-09-25 17:43:55,659|[DEBUG]|121360|126181426977664|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.267658.apply_db._dlt_pipeline_state with file None and actual name /home/drnick/.dlt/pipelines/apply_db/load/new/1727279035.267658/new_jobs/_dlt_pipeline_state.51a1f0703d.0.parquet
2024-09-25 17:43:55,661|[DEBUG]|121360|126181426977664|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.267658.apply_db.applicant with file None and actual name /home/drnick/.dlt/pipelines/apply_db/load/new/1727279035.267658/new_jobs/applicant.6d7c3aafc7.0.parquet
2024-09-25 17:43:55,664|[DEBUG]|121360|126181426977664|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.267658.apply_db.application with file None and actual name /home/drnick/.dlt/pipelines/apply_db/load/new/1727279035.267658/new_jobs/application.de61105182.0.parquet
2024-09-25 17:43:55,667|[INFO]|121360|126181426977664|dlt|worker.py|w_normalize_files:244|Processed all items in 3 files
2024-09-25 17:43:55,967|[INFO]|121360|126181426977664|dlt|normalize.py|update_schema:86|Updating schema for table _dlt_pipeline_state with 1 deltas
2024-09-25 17:43:55,968|[INFO]|121360|126181426977664|dlt|normalize.py|update_schema:86|Updating schema for table applicant with 1 deltas
2024-09-25 17:43:55,968|[INFO]|121360|126181426977664|dlt|normalize.py|update_schema:86|Updating schema for table application with 1 deltas
2024-09-25 17:43:55,968|[DEBUG]|121360|126181426977664|dlt|normalize.py|map_parallel:152|0 tasks still remaining for<PHONE_NUMBER>.267658...
2024-09-25 17:43:55,968|[INFO]|121360|126181426977664|dlt|normalize.py|spool_files:194|Table _dlt_pipeline_state has seen data for a first time with load id<PHONE_NUMBER>.267658
2024-09-25 17:43:55,968|[INFO]|121360|126181426977664|dlt|normalize.py|spool_files:194|Table applicant has seen data for a first time with load id<PHONE_NUMBER>.267658
2024-09-25 17:43:55,968|[INFO]|121360|126181426977664|dlt|normalize.py|spool_files:194|Table application has seen data for a first time with load id<PHONE_NUMBER>.267658
2024-09-25 17:43:55,969|[INFO]|121360|126181426977664|dlt|normalize.py|spool_files:200|Saving schema apply_db with version 1:2
2024-09-25 17:43:55,970|[INFO]|121360|126181426977664|dlt|normalize.py|spool_files:216|Committing storage, do not kill this process
2024-09-25 17:43:55,971|[INFO]|121360|126181426977664|dlt|normalize.py|spool_files:222|Extracted package<PHONE_NUMBER>.267658 processed
2024-09-25 17:43:55,971|[INFO]|121360|126181426977664|dlt|pool_runner.py|run_pool:101|Closing processing pool
2024-09-25 17:43:55,971|[INFO]|121360|126181426977664|dlt|pool_runner.py|run_pool:104|Processing pool closed
2024-09-25 17:43:57,355|[INFO]|121360|126181426977664|dlt|pool_runner.py|run_pool:75|Created thread pool with 20 workers
2024-09-25 17:43:57,355|[DEBUG]|121360|126181426977664|dlt|pool_runner.py|run_pool:88|Running pool
2024-09-25 17:43:57,355|[INFO]|121360|126181426977664|dlt|load.py|run:612|Running file loading
2024-09-25 17:43:57,355|[INFO]|121360|126181426977664|dlt|load.py|run:615|Found 1 load packages
2024-09-25 17:43:57,355|[INFO]|121360|126181426977664|dlt|load.py|run:621|Loading schema from load package in<PHONE_NUMBER>.267658
2024-09-25 17:43:57,363|[INFO]|121360|126181426977664|dlt|load.py|run:623|Loaded schema name apply_db and version 2
2024-09-25 17:43:57,365|[WARNING]|121360|126181426977664|dlt|type_mapping.py|to_db_datetime_type:56|Column flags for timezone or precision are not yet supported in this destination. One or both of these flags were used in column 'birth_date'. in table 'applicant'.
2024-09-25 17:43:57,365|[WARNING]|121360|126181426977664|dlt|type_mapping.py|to_db_datetime_type:56|Column flags for timezone or precision are not yet supported in this destination. One or both of these flags were used in column 'created_at'. in table 'applicant'.
2024-09-25 17:43:57,365|[WARNING]|121360|126181426977664|dlt|type_mapping.py|to_db_datetime_type:56|Column flags for timezone or precision are not yet supported in this destination. One or both of these flags were used in column 'updated_at'. in table 'applicant'.
2024-09-25 17:43:57,365|[INFO]|121360|126181426977664|dlt|utils.py|_init_dataset_and_update_schema:157|Client for bigquery will start initialize storage
2024-09-25 17:43:59,480|[INFO]|121360|126181426977664|dlt|utils.py|_init_dataset_and_update_schema:175|Client for bigquery will update schema to package schema
2024-09-25 17:43:59,896|[INFO]|121360|126181426977664|dlt|job_client_impl.py|update_stored_schema:177|Schema with hash fUkdRFw+jB9hkEkr1v6vY65J184GXGOBgiG4I6dMrG0= not found in the storage. upgrading
2024-09-25 17:43:59,896|[WARNING]|121360|126181426977664|dlt|sql_client.py|execute_sql:218|TEMP: Executing query:
SELECT table_name,column_name,data_type,is_nullable
    FROM `<truncated>`.`drnick_apply_20240925034353`.INFORMATION_SCHEMA.COLUMNS
WHERE table_name IN (%s,%s,%s,%s,%s) ORDER BY table_name, ordinal_position;
2024-09-25 17:44:00,505|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 6 updates for _dlt_version in apply_db
2024-09-25 17:44:00,506|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 7 updates for application in apply_db
2024-09-25 17:44:00,506|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 8 updates for _dlt_pipeline_state in apply_db
2024-09-25 17:44:00,507|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 5 updates for _dlt_loads in apply_db
2024-09-25 17:44:00,508|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 15 updates for applicant in apply_db
2024-09-25 17:44:00,509|[WARNING]|121360|126181426977664|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: CREATE TABLE IF NOT EXISTS `<truncated>`.`drnick_apply_20240925034353`.`_dlt_version` (
`version` INT64 NOT NULL,
`engine_version` INT64 NOT NULL,
`inserted_at` TIMESTAMP NOT NULL,
`schema_name` STRING NOT NULL,
`version_hash` STRING NOT NULL,
`schema` STRING NOT NULL);
CREATE TABLE IF NOT EXISTS `<truncated>`.`drnick_apply_20240925034353`.`_dlt_pipeline_state` (
`version` INT64 NOT NULL,
`engine_version` INT64 NOT NULL,
`pipeline_name` STRING NOT NULL,
`state` STRING NOT NULL,
`created_at` TIMESTAMP NOT NULL,
`version_hash` STRING ,
`_dlt_load_id` STRING NOT NULL,
`_dlt_id` STRING NOT NULL);
CREATE TABLE IF NOT EXISTS `<truncated>`.`drnick_apply_20240925034353`.`_dlt_loads` (
`load_id` STRING NOT NULL,
`schema_name` STRING ,
`status` INT64 NOT NULL,
`inserted_at` TIMESTAMP NOT NULL,
`schema_version_hash` STRING );
2024-09-25 17:44:01,582|[WARNING]|121360|126181426977664|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: INSERT INTO `<truncated>`.`drnick_apply_20240925034353`.`_dlt_version`(`version`, `engine_version`, `inserted_at`, `schema_name`, `version_hash`, `schema`) VALUES (%s, %s, %s, %s, %s, %s);
2024-09-25 17:44:03,883|[INFO]|121360|126181426977664|dlt|utils.py|_init_dataset_and_update_schema:157|Client for bigquery will start initialize storage for staging dataset
2024-09-25 17:44:05,216|[INFO]|121360|126181426977664|dlt|utils.py|_init_dataset_and_update_schema:175|Client for bigquery will update schema to package schema for staging dataset
2024-09-25 17:44:05,524|[INFO]|121360|126181426977664|dlt|job_client_impl.py|update_stored_schema:177|Schema with hash fUkdRFw+jB9hkEkr1v6vY65J184GXGOBgiG4I6dMrG0= not found in the storage. upgrading
2024-09-25 17:44:05,524|[WARNING]|121360|126181426977664|dlt|sql_client.py|execute_sql:218|TEMP: Executing query:
SELECT table_name,column_name,data_type,is_nullable
    FROM `<truncated>`.`drnick_apply_20240925034353_staging`.INFORMATION_SCHEMA.COLUMNS
WHERE table_name IN (%s,%s,%s) ORDER BY table_name, ordinal_position;
2024-09-25 17:44:06,132|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 6 updates for _dlt_version in apply_db
2024-09-25 17:44:06,133|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 7 updates for application in apply_db
2024-09-25 17:44:06,133|[INFO]|121360|126181426977664|dlt|job_client_impl.py|_create_table_update:606|Found 15 updates for applicant in apply_db
2024-09-25 17:44:06,134|[WARNING]|121360|126181426977664|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: CREATE TABLE IF NOT EXISTS `<truncated>`.`drnick_apply_20240925034353_staging`.`_dlt_version` (
`version` INT64 NOT NULL,
`engine_version` INT64 NOT NULL,
`inserted_at` TIMESTAMP NOT NULL,
`schema_name` STRING NOT NULL,
`version_hash` STRING NOT NULL,
`schema` STRING NOT NULL);
2024-09-25 17:44:06,651|[WARNING]|121360|126181426977664|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: INSERT INTO `<truncated>`.`drnick_apply_20240925034353_staging`.`_dlt_version`(`version`, `engine_version`, `inserted_at`, `schema_name`, `version_hash`, `schema`) VALUES (%s, %s, %s, %s, %s, %s);
2024-09-25 17:44:08,401|[INFO]|121360|126181426977664|dlt|utils.py|_init_dataset_and_update_schema:183|Client for bigquery will truncate tables for staging dataset
2024-09-25 17:44:08,690|[WARNING]|121360|126181426977664|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: TRUNCATE TABLE `<truncated>`.`drnick_apply_20240925034353_staging`.`application`;
TRUNCATE TABLE `<truncated>`.`drnick_apply_20240925034353_staging`.`applicant`;
2024-09-25 17:44:09,001|[INFO]|121360|126181426977664|dlt|pool_runner.py|run_pool:101|Closing processing pool
2024-09-25 17:44:09,002|[INFO]|121360|126181426977664|dlt|pool_runner.py|run_pool:104|Processing pool closed
Traceback (most recent call last):
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 220, in _execute
    rows = client.query_and_wait(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 3601, in query_and_wait
    return _job_helpers.query_and_wait(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_job_helpers.py", line 509, in query_and_wait
    return job_retry(do_query)()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_job_helpers.py", line 450, in do_query
    response = retry(client._call_api)(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 833, in _call_api
    return call()
           ^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/<truncated>/queries?prettyPrint=false: Not found: Table <truncated>:drnick_apply_20240925034353_staging.application was not found in location europe-west1 at [1:1]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/destinations/sql_client.py", line 338, in _wrap_gen
    return (yield from f(self, *args, **kwargs))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/destinations/impl/bigquery/sql_client.py", line 238, in execute_query
    curr.execute(query, db_args, job_config=self._session_query or self._default_query)
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 496, in with_closed_check
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 189, in execute
    self._execute(
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 226, in _execute
    raise exceptions.DatabaseError(exc)
google.cloud.bigquery.dbapi.exceptions.DatabaseError: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/<truncated>/queries?prettyPrint=false: Not found: Table <truncated>:drnick_apply_20240925034353_staging.application was not found in location europe-west1 at [1:1]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 572, in load
    runner.run_pool(load_step.config, load_step)
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/common/runners/pool_runner.py", line 89, in run_pool
    while _run_func():
          ^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/common/runners/pool_runner.py", line 82, in _run_func
    run_metrics = run_f.run(cast(TExecutor, pool))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/load/load.py", line 637, in run
    self.load_single_package(load_id, schema)
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/load/load.py", line 526, in load_single_package
    applied_update = init_client(
                     ^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/load/utils.py", line 136, in init_client
    _init_dataset_and_update_schema(
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/load/utils.py", line 187, in _init_dataset_and_update_schema
    job_client.initialize_storage(truncate_tables=truncate_tables)
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/destinations/job_client_impl.py", line 163, in initialize_storage
    self.sql_client.truncate_tables(*truncate_tables)
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/destinations/sql_client.py", line 125, in truncate_tables
    self.execute_many(statements)
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/destinations/sql_client.py", line 182, in execute_many
    ret.append(self.execute_sql(sql_fragment, *args, **kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/destinations/impl/bigquery/sql_client.py", line 219, in execute_sql
    with self.execute_query(sql, *args, **kwargs) as curr:
  File "/home/drnick/repos/github/pyenv/pyenv/versions/3.11.9/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/destinations/sql_client.py", line 340, in _wrap_gen
    raise self._make_database_exception(ex)
dlt.destinations.exceptions.DatabaseUndefinedRelation: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/<truncated>/queries?prettyPrint=false: Not found: Table <truncated>:drnick_apply_20240925034353_staging.application was not found in location europe-west1 at [1:1]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/drnick/repos/analytics/42next-modeling/nxt_dlt/tmp_apply_db.py", line 54, in <module>
    load_info = pipeline.run(source, write_disposition="merge")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 223, in _wrap
    step_info = f(self, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 268, in _wrap
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 711, in run
    return self.load(destination, dataset_name, credentials=credentials)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 223, in _wrap
    step_info = f(self, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 163, in _wrap
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 268, in _wrap
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.11/site-packages/dlt/pipeline/pipeline.py", line 579, in load
    raise PipelineStepFailed(
dlt.pipeline.exceptions.PipelineStepFailed: Pipeline execution failed at stage load when processing package<PHONE_NUMBER>.267658 with exception:

<class 'dlt.destinations.exceptions.DatabaseUndefinedRelation'>
400 POST https://bigquery.googleapis.com/bigquery/v2/projects/<truncated>/queries?prettyPrint=false: Not found: Table <truncated>:drnick_apply_20240925034353_staging.application was not found in location europe-west1 at [1:1]

Operating system
Linux
Runtime environment
Local
Python version
3.11
dlt data source
sql_database
dlt destination
Google BigQuery
Other deployment details
If an change the write_disposition to append, everything works fine.
import dlt
from dlt.common.libs.sql_alchemy import sqltypes
from dlt.destinations.adapters import bigquery_adapter
from dlt.sources import DltSource, incremental
from dlt.sources.sql_database import sql_table

def sql_type_adapter(sql_t):
    if isinstance(sql_t, sqltypes.ARRAY):
        return sqltypes.JSON
    return sql_t

tables_configs: dict[str, dict] = {
    "applicant": {"cursor_path": "updated_at"},
    "application": {},
}

@dlt.source
def apply_db(conn_uri: str) -> DltSource:  # type: ignore[misc]
    for table_name, table_config in tables_configs.items():
        cursor_path = table_config.get("cursor_path")
        t_resource = sql_table(
            credentials=conn_uri,
            table=table_name,
            incremental=incremental(cursor_path=cursor_path) if cursor_path else None,
            detect_precision_hints=True,
            type_adapter_callback=sql_type_adapter,
        )
        bigquery_adapter(t_resource, partition=cursor_path, autodetect_schema=True)  # type: ignore[arg-type]
        assert isinstance(t_resource.columns, dict)
        file_format = (
            "jsonl" if any(col.get("data_type") == "json" for col in t_resource.columns.values()) else "parquet"
        )
        t_resource.apply_hints(file_format=file_format)  # type: ignore[arg-type]
        yield t_resource

if __name__ == "__main__":
    from dlt import Pipeline

    from nxt.config import __config__ as cfg

    pipeline: Pipeline = dlt.pipeline(
        pipeline_name="apply_db",
        destination="bigquery",
        staging="filesystem",
        dataset_name=cfg.apply_db.dataset,
        dev_mode=True,
    )

    source = apply_db(f"{cfg.apply_db.dsn}")
    load_info = pipeline.run(source, write_disposition="append")

    load_info.raise_on_failed_jobs()
    print(load_info)

    last_trace = pipeline.last_trace
    print(last_trace)

output
+ set -a
+ source .env
++ APPLY_DB__DATASET=drnick_apply
++ APPLY_DB__DSN=postgresql+psycopg2://<truncated>@<truncated>/apply
++ DESTINATION__BIGQUERY__LOCATION=europe-west1
++ DESTINATION__BIGQUERY__TRUNCATE_TABLES_ON_STAGING_DESTINATION_BEFORE_LOAD=false
++ DESTINATION__FILESYSTEM__BUCKET_URL=gs://<truncated>/dlt
+ set +a
++ echo nxt_dlt/tmp_apply_db.py
++ sed 's/\//./g'
++ sed 's/\.py$//'
+ module_name=nxt_dlt.tmp_apply_db
+ poetry run python -m nxt_dlt.tmp_apply_db
2024-09-25 18:10:14,530|[WARNING]|123420|124623998454656|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 18:10:15,372|[WARNING]|123420|124623998454656|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 18:10:15,372|[WARNING]|123420|124623998454656|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name applicant_id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 18:10:15,437|[INFO]|123420|124623998454656|dlt|__init__.py|bind:439|Bind incremental on applicant with initial_value: None, start_value: None, end_value: None
2024-09-25 18:10:15,444|[WARNING]|123420|124623998454656|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 18:10:15,495|[WARNING]|123420|124623998454656|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 18:10:15,495|[WARNING]|123420|124623998454656|dlt|schema_types.py|sqla_col_to_column_schema:132|A column with name applicant_id contains unknown data type UUID which cannot be mapped to `dlt` data type. When using sqlalchemy backend such data will be passed to the normalizer. In case of `pyarrow` and `pandas` backend, data types are detected from numpy ndarrays. In case of other backends, the behavior is backend-specific.
2024-09-25 18:10:15,611|[DEBUG]|123420|124623998454656|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.430451.apply_db.applicant with file None and actual name /home/drnick/.dlt/pipelines/apply_db/normalize/4c8372df47dd3c7c/1727280615.430451/new_jobs/applicant.4188c90bfb.0.typed-jsonl
2024-09-25 18:10:15,615|[DEBUG]|123420|124623998454656|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.430451.apply_db.application with file None and actual name /home/drnick/.dlt/pipelines/apply_db/normalize/4c8372df47dd3c7c/1727280615.430451/new_jobs/application.032e52f4dc.0.typed-jsonl
2024-09-25 18:10:15,732|[DEBUG]|123420|124623998454656|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.430451.apply_db._dlt_pipeline_state with file None and actual name /home/drnick/.dlt/pipelines/apply_db/normalize/4c8372df47dd3c7c/1727280615.430451/new_jobs/_dlt_pipeline_state.98f9cbbe51.0.typed-jsonl
2024-09-25 18:10:15,811|[INFO]|123420|124623998454656|dlt|pool_runner.py|run_pool:75|Created none pool with 1 workers
2024-09-25 18:10:15,811|[DEBUG]|123420|124623998454656|dlt|pool_runner.py|run_pool:88|Running pool
2024-09-25 18:10:15,811|[INFO]|123420|124623998454656|dlt|normalize.py|run:263|Running file normalizing
2024-09-25 18:10:15,811|[INFO]|123420|124623998454656|dlt|normalize.py|run:266|Found 1 load packages
2024-09-25 18:10:15,825|[INFO]|123420|124623998454656|dlt|normalize.py|run:289|Found 3 files in schema apply_db load_id<PHONE_NUMBER>.430451
2024-09-25 18:10:15,826|[INFO]|123420|124623998454656|dlt|normalize.py|spool_schema_files:240|Created new load package<PHONE_NUMBER>.430451 on loading volume
2024-09-25 18:10:15,831|[INFO]|123420|124623998454656|dlt|worker.py|_get_items_normalizer:123|A file format for table _dlt_pipeline_state was specified to preferred in the resource so parquet format being used.
2024-09-25 18:10:15,831|[INFO]|123420|124623998454656|dlt|worker.py|_get_items_normalizer:169|Created items normalizer JsonLItemsNormalizer with writer ParquetDataWriter for item format object and file format parquet
2024-09-25 18:10:15,831|[DEBUG]|123420|124623998454656|dlt|worker.py|w_normalize_files:230|Processing extracted items in<PHONE_NUMBER>.430451/new_jobs/_dlt_pipeline_state.98f9cbbe51.0.typed-jsonl in load_id<PHONE_NUMBER>.430451 with table name _dlt_pipeline_state and schema apply_db
2024-09-25 18:10:15,832|[DEBUG]|123420|124623998454656|dlt|items_normalizers.py|__call__:208|Processed 1 lines from file<PHONE_NUMBER>.430451/new_jobs/_dlt_pipeline_state.98f9cbbe51.0.typed-jsonl
2024-09-25 18:10:15,832|[DEBUG]|123420|124623998454656|dlt|worker.py|w_normalize_files:236|Processed file<PHONE_NUMBER>.430451/new_jobs/_dlt_pipeline_state.98f9cbbe51.0.typed-jsonl
2024-09-25 18:10:15,832|[INFO]|123420|124623998454656|dlt|worker.py|_get_items_normalizer:123|A file format for table applicant was specified to parquet in the resource so parquet format being used.
2024-09-25 18:10:15,833|[INFO]|123420|124623998454656|dlt|worker.py|_get_items_normalizer:169|Created items normalizer JsonLItemsNormalizer with writer ParquetDataWriter for item format object and file format parquet
2024-09-25 18:10:15,833|[DEBUG]|123420|124623998454656|dlt|worker.py|w_normalize_files:230|Processing extracted items in<PHONE_NUMBER>.430451/new_jobs/applicant.4188c90bfb.0.typed-jsonl in load_id<PHONE_NUMBER>.430451 with table name applicant and schema apply_db
2024-09-25 18:10:15,836|[DEBUG]|123420|124623998454656|dlt|items_normalizers.py|__call__:208|Processed 1 lines from file<PHONE_NUMBER>.430451/new_jobs/applicant.4188c90bfb.0.typed-jsonl
2024-09-25 18:10:15,836|[DEBUG]|123420|124623998454656|dlt|worker.py|w_normalize_files:236|Processed file<PHONE_NUMBER>.430451/new_jobs/applicant.4188c90bfb.0.typed-jsonl
2024-09-25 18:10:15,836|[INFO]|123420|124623998454656|dlt|worker.py|_get_items_normalizer:123|A file format for table application was specified to parquet in the resource so parquet format being used.
2024-09-25 18:10:15,836|[INFO]|123420|124623998454656|dlt|worker.py|_get_items_normalizer:169|Created items normalizer JsonLItemsNormalizer with writer ParquetDataWriter for item format object and file format parquet
2024-09-25 18:10:15,836|[DEBUG]|123420|124623998454656|dlt|worker.py|w_normalize_files:230|Processing extracted items in<PHONE_NUMBER>.430451/new_jobs/application.032e52f4dc.0.typed-jsonl in load_id<PHONE_NUMBER>.430451 with table name application and schema apply_db
2024-09-25 18:10:15,839|[DEBUG]|123420|124623998454656|dlt|items_normalizers.py|__call__:208|Processed 1 lines from file<PHONE_NUMBER>.430451/new_jobs/application.032e52f4dc.0.typed-jsonl
2024-09-25 18:10:15,839|[DEBUG]|123420|124623998454656|dlt|worker.py|w_normalize_files:236|Processed file<PHONE_NUMBER>.430451/new_jobs/application.032e52f4dc.0.typed-jsonl
2024-09-25 18:10:15,839|[DEBUG]|123420|124623998454656|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.430451.apply_db._dlt_pipeline_state with file None and actual name /home/drnick/.dlt/pipelines/apply_db/load/new/1727280615.430451/new_jobs/_dlt_pipeline_state.61eb001e99.0.parquet
2024-09-25 18:10:15,841|[DEBUG]|123420|124623998454656|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.430451.apply_db.applicant with file None and actual name /home/drnick/.dlt/pipelines/apply_db/load/new/1727280615.430451/new_jobs/applicant.0776018b8d.0.parquet
2024-09-25 18:10:15,844|[DEBUG]|123420|124623998454656|dlt|data_item_storage.py|close_writers:81|Closing writer for<PHONE_NUMBER>.430451.apply_db.application with file None and actual name /home/drnick/.dlt/pipelines/apply_db/load/new/1727280615.430451/new_jobs/application.1c2261d59c.0.parquet
2024-09-25 18:10:15,847|[INFO]|123420|124623998454656|dlt|worker.py|w_normalize_files:244|Processed all items in 3 files
2024-09-25 18:10:16,147|[INFO]|123420|124623998454656|dlt|normalize.py|update_schema:86|Updating schema for table _dlt_pipeline_state with 1 deltas
2024-09-25 18:10:16,147|[INFO]|123420|124623998454656|dlt|normalize.py|update_schema:86|Updating schema for table applicant with 1 deltas
2024-09-25 18:10:16,148|[INFO]|123420|124623998454656|dlt|normalize.py|update_schema:86|Updating schema for table application with 1 deltas
2024-09-25 18:10:16,148|[DEBUG]|123420|124623998454656|dlt|normalize.py|map_parallel:152|0 tasks still remaining for<PHONE_NUMBER>.430451...
2024-09-25 18:10:16,148|[INFO]|123420|124623998454656|dlt|normalize.py|spool_files:194|Table _dlt_pipeline_state has seen data for a first time with load id<PHONE_NUMBER>.430451
2024-09-25 18:10:16,148|[INFO]|123420|124623998454656|dlt|normalize.py|spool_files:194|Table applicant has seen data for a first time with load id<PHONE_NUMBER>.430451
2024-09-25 18:10:16,149|[INFO]|123420|124623998454656|dlt|normalize.py|spool_files:194|Table application has seen data for a first time with load id<PHONE_NUMBER>.430451
2024-09-25 18:10:16,150|[INFO]|123420|124623998454656|dlt|normalize.py|spool_files:200|Saving schema apply_db with version 1:2
2024-09-25 18:10:16,154|[INFO]|123420|124623998454656|dlt|normalize.py|spool_files:216|Committing storage, do not kill this process
2024-09-25 18:10:16,156|[INFO]|123420|124623998454656|dlt|normalize.py|spool_files:222|Extracted package<PHONE_NUMBER>.430451 processed
2024-09-25 18:10:16,156|[INFO]|123420|124623998454656|dlt|pool_runner.py|run_pool:101|Closing processing pool
2024-09-25 18:10:16,156|[INFO]|123420|124623998454656|dlt|pool_runner.py|run_pool:104|Processing pool closed
2024-09-25 18:10:17,550|[INFO]|123420|124623998454656|dlt|pool_runner.py|run_pool:75|Created thread pool with 20 workers
2024-09-25 18:10:17,550|[DEBUG]|123420|124623998454656|dlt|pool_runner.py|run_pool:88|Running pool
2024-09-25 18:10:17,550|[INFO]|123420|124623998454656|dlt|load.py|run:612|Running file loading
2024-09-25 18:10:17,551|[INFO]|123420|124623998454656|dlt|load.py|run:615|Found 1 load packages
2024-09-25 18:10:17,551|[INFO]|123420|124623998454656|dlt|load.py|run:621|Loading schema from load package in<PHONE_NUMBER>.430451
2024-09-25 18:10:17,559|[INFO]|123420|124623998454656|dlt|load.py|run:623|Loaded schema name apply_db and version 2
2024-09-25 18:10:17,560|[WARNING]|123420|124623998454656|dlt|type_mapping.py|to_db_datetime_type:56|Column flags for timezone or precision are not yet supported in this destination. One or both of these flags were used in column 'birth_date'. in table 'applicant'.
2024-09-25 18:10:17,560|[WARNING]|123420|124623998454656|dlt|type_mapping.py|to_db_datetime_type:56|Column flags for timezone or precision are not yet supported in this destination. One or both of these flags were used in column 'created_at'. in table 'applicant'.
2024-09-25 18:10:17,560|[WARNING]|123420|124623998454656|dlt|type_mapping.py|to_db_datetime_type:56|Column flags for timezone or precision are not yet supported in this destination. One or both of these flags were used in column 'updated_at'. in table 'applicant'.
2024-09-25 18:10:17,561|[INFO]|123420|124623998454656|dlt|utils.py|_init_dataset_and_update_schema:157|Client for bigquery will start initialize storage
2024-09-25 18:10:20,236|[INFO]|123420|124623998454656|dlt|utils.py|_init_dataset_and_update_schema:175|Client for bigquery will update schema to package schema
2024-09-25 18:10:20,543|[INFO]|123420|124623998454656|dlt|job_client_impl.py|update_stored_schema:177|Schema with hash gwDhtUuS9ludM6P7mtVF5qPkEb4CL0hmHT92Cmx5lDE= not found in the storage. upgrading
2024-09-25 18:10:20,544|[WARNING]|123420|124623998454656|dlt|sql_client.py|execute_sql:218|TEMP: Executing query:
SELECT table_name,column_name,data_type,is_nullable
    FROM `<truncated>`.`drnick_apply_20240925041014`.INFORMATION_SCHEMA.COLUMNS
WHERE table_name IN (%s,%s,%s,%s,%s) ORDER BY table_name, ordinal_position;
2024-09-25 18:10:21,158|[INFO]|123420|124623998454656|dlt|job_client_impl.py|_create_table_update:606|Found 15 updates for applicant in apply_db
2024-09-25 18:10:21,159|[INFO]|123420|124623998454656|dlt|job_client_impl.py|_create_table_update:606|Found 6 updates for _dlt_version in apply_db
2024-09-25 18:10:21,160|[INFO]|123420|124623998454656|dlt|job_client_impl.py|_create_table_update:606|Found 5 updates for _dlt_loads in apply_db
2024-09-25 18:10:21,161|[INFO]|123420|124623998454656|dlt|job_client_impl.py|_create_table_update:606|Found 8 updates for _dlt_pipeline_state in apply_db
2024-09-25 18:10:21,162|[INFO]|123420|124623998454656|dlt|job_client_impl.py|_create_table_update:606|Found 7 updates for application in apply_db
2024-09-25 18:10:21,162|[WARNING]|123420|124623998454656|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: CREATE TABLE IF NOT EXISTS `<truncated>`.`drnick_apply_20240925041014`.`_dlt_version` (
`version` INT64 NOT NULL,
`engine_version` INT64 NOT NULL,
`inserted_at` TIMESTAMP NOT NULL,
`schema_name` STRING NOT NULL,
`version_hash` STRING NOT NULL,
`schema` STRING NOT NULL);
CREATE TABLE IF NOT EXISTS `<truncated>`.`drnick_apply_20240925041014`.`_dlt_loads` (
`load_id` STRING NOT NULL,
`schema_name` STRING ,
`status` INT64 NOT NULL,
`inserted_at` TIMESTAMP NOT NULL,
`schema_version_hash` STRING );
CREATE TABLE IF NOT EXISTS `<truncated>`.`drnick_apply_20240925041014`.`_dlt_pipeline_state` (
`version` INT64 NOT NULL,
`engine_version` INT64 NOT NULL,
`pipeline_name` STRING NOT NULL,
`state` STRING NOT NULL,
`created_at` TIMESTAMP NOT NULL,
`version_hash` STRING ,
`_dlt_load_id` STRING NOT NULL,
`_dlt_id` STRING NOT NULL);
2024-09-25 18:10:22,607|[WARNING]|123420|124623998454656|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: INSERT INTO `<truncated>`.`drnick_apply_20240925041014`.`_dlt_version`(`version`, `engine_version`, `inserted_at`, `schema_name`, `version_hash`, `schema`) VALUES (%s, %s, %s, %s, %s, %s);
2024-09-25 18:10:25,151|[INFO]|123420|124623998454656|dlt|utils.py|_init_dataset_and_update_schema:157|Client for filesystem will start initialize storage
2024-09-25 18:10:25,448|[INFO]|123420|124623998454656|dlt|utils.py|_init_dataset_and_update_schema:175|Client for filesystem will update schema to package schema
2024-09-25 18:10:26,151|[INFO]|123420|124623998454656|dlt|load.py|resume_started_jobs:290|Found 0 that are already started and should be continued
2024-09-25 18:10:26,151|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:389|Will complete 0 for<PHONE_NUMBER>.430451
2024-09-25 18:10:26,151|[INFO]|123420|124623998454656|dlt|load.py|start_new_jobs:273|Will load additional 3, creating jobs
2024-09-25 18:10:26,155|[INFO]|123420|124623998454656|dlt|load.py|submit_job:168|Will load file<PHONE_NUMBER>.430451/new_jobs/_dlt_pipeline_state.61eb001e99.0.parquet with table name _dlt_pipeline_state
2024-09-25 18:10:26,161|[INFO]|123420|124623998454656|dlt|load.py|submit_job:168|Will load file<PHONE_NUMBER>.430451/new_jobs/applicant.0776018b8d.0.parquet with table name applicant
2024-09-25 18:10:26,165|[INFO]|123420|124623998454656|dlt|load.py|submit_job:168|Will load file<PHONE_NUMBER>.430451/new_jobs/application.1c2261d59c.0.parquet with table name application
2024-09-25 18:10:26,548|[INFO]|123420|124622313883328|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:26,560|[INFO]|123420|124622758479552|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:26,561|[INFO]|123420|124622536181440|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:27,167|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:389|Will complete 3 for<PHONE_NUMBER>.430451
2024-09-25 18:10:27,167|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job _dlt_pipeline_state.61eb001e99.parquet
2024-09-25 18:10:27,171|[INFO]|123420|124623998454656|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:27,171|[INFO]|123420|124623998454656|dlt|load.py|create_followup_jobs:368|Job _dlt_pipeline_state.61eb001e99.parquet CREATED a new FOLLOWUP JOB /tmp/_dlt_pipeline_state.61eb001e99.0.reference placed in new_jobs
2024-09-25 18:10:27,172|[INFO]|123420|124623998454656|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:27,172|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:458|Job for _dlt_pipeline_state.61eb001e99.parquet completed in load<PHONE_NUMBER>.430451
2024-09-25 18:10:27,172|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job applicant.0776018b8d.parquet
2024-09-25 18:10:27,176|[INFO]|123420|124623998454656|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:27,177|[INFO]|123420|124623998454656|dlt|load.py|create_followup_jobs:368|Job applicant.0776018b8d.parquet CREATED a new FOLLOWUP JOB /tmp/applicant.0776018b8d.0.reference placed in new_jobs
2024-09-25 18:10:27,177|[INFO]|123420|124623998454656|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:27,177|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:458|Job for applicant.0776018b8d.parquet completed in load<PHONE_NUMBER>.430451
2024-09-25 18:10:27,177|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job application.1c2261d59c.parquet
2024-09-25 18:10:27,181|[INFO]|123420|124623998454656|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:27,182|[INFO]|123420|124623998454656|dlt|load.py|create_followup_jobs:368|Job application.1c2261d59c.parquet CREATED a new FOLLOWUP JOB /tmp/application.1c2261d59c.0.reference placed in new_jobs
2024-09-25 18:10:27,182|[INFO]|123420|124623998454656|dlt|path_utils.py|prepare_datetime_params:112|current_datetime is not set, using timestamp from load package
2024-09-25 18:10:27,182|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:458|Job for application.1c2261d59c.parquet completed in load<PHONE_NUMBER>.430451
2024-09-25 18:10:27,183|[INFO]|123420|124623998454656|dlt|load.py|start_new_jobs:273|Will load additional 3, creating jobs
2024-09-25 18:10:27,183|[INFO]|123420|124623998454656|dlt|load.py|submit_job:168|Will load file<PHONE_NUMBER>.430451/new_jobs/applicant.0776018b8d.0.reference with table name applicant
2024-09-25 18:10:27,184|[INFO]|123420|124623998454656|dlt|load.py|submit_job:168|Will load file<PHONE_NUMBER>.430451/new_jobs/_dlt_pipeline_state.61eb001e99.0.reference with table name _dlt_pipeline_state
2024-09-25 18:10:27,184|[INFO]|123420|124623998454656|dlt|load.py|submit_job:168|Will load file<PHONE_NUMBER>.430451/new_jobs/application.1c2261d59c.0.reference with table name application
2024-09-25 18:10:28,185|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:389|Will complete 3 for<PHONE_NUMBER>.430451
2024-09-25 18:10:28,185|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job applicant.0776018b8d.reference
2024-09-25 18:10:28,185|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:396|job applicant.0776018b8d.reference still running
2024-09-25 18:10:28,185|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job _dlt_pipeline_state.61eb001e99.reference
2024-09-25 18:10:28,185|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:396|job _dlt_pipeline_state.61eb001e99.reference still running
2024-09-25 18:10:28,185|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job application.1c2261d59c.reference
2024-09-25 18:10:28,185|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:396|job application.1c2261d59c.reference still running
2024-09-25 18:10:28,185|[INFO]|123420|124623998454656|dlt|load.py|start_new_jobs:273|Will load additional 0, creating jobs
2024-09-25 18:10:29,186|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:389|Will complete 3 for<PHONE_NUMBER>.430451
2024-09-25 18:10:29,186|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job applicant.0776018b8d.reference
2024-09-25 18:10:29,186|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:396|job applicant.0776018b8d.reference still running
2024-09-25 18:10:29,186|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job _dlt_pipeline_state.61eb001e99.reference
2024-09-25 18:10:29,186|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:396|job _dlt_pipeline_state.61eb001e99.reference still running
2024-09-25 18:10:29,186|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job application.1c2261d59c.reference
2024-09-25 18:10:29,186|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:396|job application.1c2261d59c.reference still running
2024-09-25 18:10:29,186|[INFO]|123420|124623998454656|dlt|load.py|start_new_jobs:273|Will load additional 0, creating jobs
2024-09-25 18:10:30,187|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:389|Will complete 3 for<PHONE_NUMBER>.430451
2024-09-25 18:10:30,187|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job applicant.0776018b8d.reference
2024-09-25 18:10:30,189|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:458|Job for applicant.0776018b8d.reference completed in load<PHONE_NUMBER>.430451
2024-09-25 18:10:30,190|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job _dlt_pipeline_state.61eb001e99.reference
2024-09-25 18:10:30,192|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:458|Job for _dlt_pipeline_state.61eb001e99.reference completed in load<PHONE_NUMBER>.430451
2024-09-25 18:10:30,192|[DEBUG]|123420|124623998454656|dlt|load.py|complete_jobs:392|Checking state for job application.1c2261d59c.reference
2024-09-25 18:10:30,192|[INFO]|123420|124623998454656|dlt|load.py|complete_jobs:458|Job for application.1c2261d59c.reference completed in load<PHONE_NUMBER>.430451
2024-09-25 18:10:30,193|[INFO]|123420|124623998454656|dlt|load.py|start_new_jobs:273|Will load additional 0, creating jobs
2024-09-25 18:10:30,193|[WARNING]|123420|124623998454656|dlt|sql_client.py|execute_sql:218|TEMP: Executing query: INSERT INTO `<truncated>`.`drnick_apply_20240925041014`.`_dlt_loads`(`load_id`, `schema_name`, `status`, `inserted_at`, `schema_version_hash`) VALUES(%s, %s, %s, %s, %s);
2024-09-25 18:10:32,324|[INFO]|123420|124623998454656|dlt|load.py|complete_package:495|All jobs completed, archiving package<PHONE_NUMBER>.430451 with aborted set to False
2024-09-25 18:10:32,324|[INFO]|123420|124623998454656|dlt|pool_runner.py|run_pool:101|Closing processing pool
2024-09-25 18:10:32,325|[INFO]|123420|124623998454656|dlt|pool_runner.py|run_pool:104|Processing pool closed
Pipeline apply_db load step completed in 14.77 seconds
1 load package(s) were loaded to destination bigquery and into dataset drnick_apply_20240925041014
The filesystem staging destination used gs://<truncated>/dlt location to stage data
The bigquery destination used None@<truncated> location to store data
Load package<PHONE_NUMBER>.430451 is LOADED and contains no failed jobs
Run started at 2024-09-25 16:10:15.378843+00:00 and COMPLETED in 16.95 seconds with 4 steps.
Step extract COMPLETED in 0.35 seconds.

Load package<PHONE_NUMBER>.430451 is EXTRACTED and NOT YET LOADED to the destination and contains no failed jobs

Step normalize COMPLETED in 0.42 seconds.
Normalized data for the following tables:
- _dlt_pipeline_state: 1 row(s)
- applicant: 41 row(s)
- application: 41 row(s)

Load package<PHONE_NUMBER>.430451 is NORMALIZED and NOT YET LOADED to the destination and contains no failed jobs

Step load COMPLETED in 16.14 seconds.
Pipeline apply_db load step completed in 14.77 seconds
1 load package(s) were loaded to destination bigquery and into dataset drnick_apply_20240925041014
The filesystem staging destination used gs://<truncated>/dlt location to stage data
The bigquery destination used None@<truncated> location to store data
Load package<PHONE_NUMBER>.430451 is LOADED and contains no failed jobs

Step run COMPLETED in 16.95 seconds.
Pipeline apply_db load step completed in 14.77 seconds
1 load package(s) were loaded to destination bigquery and into dataset drnick_apply_20240925041014
The filesystem staging destination used gs://<truncated>/dlt location to stage data
The bigquery destination used None@<truncated> location to store data
Load package<PHONE_NUMBER>.430451 is LOADED and contains no failed jobs

Additional information
I can start removing all of the type hinting and bigquery tuning I can try to isolate the change that brought this, but I really need the options that I'm setting :sweat_smile:

@Pipboyguy  Update when I remove the BigQuery auto-detection then everything works fine.
...
bigquery_adapter(t_resource, partition=cursor_path) # autodetect_schema=True)  # type: ignore[arg-type]
...

And FYI my current ugly workaround is doing my own table detection like so
def missing_destination_tables(pipeline: Pipeline, table_names: Iterable[str]) -> set[str]:
    with pipeline.sql_client() as c:

        def table_exists(table: str) -> bool:
            q_name = c.make_qualified_table_name(table)
            try:
                c.execute_sql(f"SELECT 1 from {q_name}")
                return True
            except DestinationUndefinedEntity:
                return False

        return {table for table in table_names if not table_exists(table)}
...
if __name__ == "__main__":
    from dlt import Pipeline

    from nxt.config import __config__ as cfg

    pipeline: Pipeline = dlt.pipeline(
        pipeline_name="apply_db",
        destination="bigquery",
        staging="filesystem",
        dataset_name=cfg.apply_db.dataset,
    )

    tables_to_append = missing_destination_tables(pipeline, tables_configs.keys())
    print(f"Missing tables: {tables_to_append}")

    source = apply_db(f"{cfg.apply_db.dsn}")
    for t_name, rsrc in source.resources.items():
        if t_name not in tables_to_append and tables_configs[t_name].get("cursor_path") is None:
            rsrc.apply_hints(write_disposition="merge")

    load_info = pipeline.run(source)

We have the same problem.
Can confirm it works with write_disposition="append" and staging="filesystem". However, we need merge/replace.

@neuromantik33 what you did is really clever, not ugly :) a temporary switch to "append" (not explicit but automatic, for the sake of the first load) is probably the only way to go. the mechanism behind merge and replace require synchronized schemas in staging and final dataset.
tbh. I do not understand why your example is not failing on a second run when you swithc back to merge because staging table will not be created...
it will take a little bit of time to fix it. not sure it will make into release next week.

tbh. I do not understand why your example is not failing on a second run when you switch back to merge because staging table will not be created...

@rudolfix You are correct. The only write_disposition that I'm using atm is replace. merge presented too many tables in the staging dataset that I just couldn't create. It's a bit tricky as I've seen what the root cause is detailed in a slack thread where I pinged you.
Thanks in advance. I'll be happy to lend a hand if need be or test whatever you might come up with.

+1 reported


Actually transfer views

I really like the elegance of your API and the approach of associating pairs of views using the heroID. Sometimes, there are situations where I want to actually transfer a specific view instance from one view controller to another. This is needed for complex views with opaque state such as MKMapView or EAGLView. I'd love to see support for such use cases!
I would envision to associate pairs of container views with a similar heroID mechanism and then insert the view into the paired container view at the start of the transition. Of course it's a bit tricky to install new constraints etc. so we'd need an afterTransfer configuration block or something similar where we could setup constraints.
I'm sure this would be a very useful addition! What do you think?

I would like to support this as well. I had thought about the idea of actually animating the original views when I built Hero. No good solutions came up during that time, So I decided to make Hero simple.
At this point, I am still not sure whats the best way of doing this.
I thought about having a transferable modifier. and some delegate callback on the view controller like this:
func targetStateForTransferable(view:UIView) -> HeroTargetState?

Suppose we have two view controllers: A & B
A can have an AVPlayerView with the modifier transferable.
When transitioning to B, Hero can ask B if it can accept the AVPlayerView. If B returns a non-nil HeroTargetState, then Hero can animate according to it. B can also keep the reference of the AVPlayerView at this time.
There is a delegate callback already when the transition is ended.
func heroDidEndTransition()

Thats where B can setup its constraints.
I am currently adding HeroTargetState to the framework. It basically holds view state informations and can be constructed from HeroModifiers.
Let me know what you think about this solution.
Luke

So you would keep the transferable view in the A.view hierarchy for the duration of the transition and let B take it over and insert it in its view hierarchy after the transition is finished, right?
Sounds reasonable though it might be useful to support the other way as well to transfer the view at the beginning of the transition. In my concrete use case this is needed because B needs to control the map view during the transition. This might also be needed to achieve a specific view layering during the transition.

The view will not be in A nor B's view hierarchy. It will be in a container view that Hero uses internally. But B can control its properties any time during the transition.

OK, makes sense.

B will have to manually insert the view when the transition finishes. Since Hero won't be able to know the location to insert nor the layout system that B uses. But during the transition Hero handles the layout for you. It animates from what A has to the targetState that B gave.
This will probably be build into a HeroPlugin. Not into the main Hero components.
In fact, currently you can control animation like this by building a custom HeroPlugin yourself. I haven't written any documentations on HeroPlugin yet since it is still under development. But the feature is there and you can play around with it.

@futuretap This is implemented in 0.1.5. Checkout the new video player example in the example project. I will have documentation available soon.

Can you document this? I am looking at the example code for the VideoPlayer transition in v1.1.0 and I cannot see it transitioning the actual live view of the AVVideoPlayer... the cell in the Examples Scene shows a UIImageView as the source view for the videoPlayer. You skip the opening scene so it appears as if the movie has started, but doesn't actually transfer any views between VCs.

Any update on this? Has this feature been implemented?


speed up import by re-using stream and block objects

What current issue(s) does this address, or what feature is it adding?
the new blockchain import was apparently slow. The changes introduced give a 2x improvement
How did you solve this problem?
re-use objects instead of re-creating them for each block to process.
How did you make sure your solution works?
manual testing
Are there any special changes in the code that we should be aware of?
no
Please check the following, if applicable:

[ ] Did you add any tests?
[X] Did you run make lint?
[ ] Did you run make test?
[X] Are you making a PR to a feature branch or development rather than master?
[ ] Did you add an entry to CHANGELOG.rst? (if not, please do)
can I skip for this one? pretty please 🙏

Coverage remained the same at 79.46% when pulling d53027e86985628556fbd5e91ae6a40e11ab2a70 on ixje:import_speed_improvement into 0c58c1d7df0c6241dcd057bbcfa36148598e4e72 on CityOfZion:development.

👍


Added a Lt. Nascimento quote and fix the JSON

@LucsFreitas your branch has merge conflicts pls resolve and pr again


Speed up query performances by disabling color computation

Closes #282
If a query returns many objects, the computation of color slows down enormously the query execution time. For 20,000 objects one goes from 15 seconds to more than 180 seconds (timeout error).
It is available only when using the objects endpoint.

We should do better. Let's take this query:
import requests
import pandas as pd

# Get all MPC objects for 2022-03-03
r = requests.post(
  'https://fink-portal.org/api/v1/latests',
  json={
    'class': 'Solar System candidate',
    'n': '100000',
    'startdate': '2022-03-03',
    'stopdate': '2022-03-04'
  }
)

# Format output in a DataFrame
pdf = pd.read_json(r.content)

With the fix, the total time to execute the query is 30 seconds (21,712 objects returned). Among this, the HBase query time is only 6 seconds!
916976 INFO  (HBaser.HBaseClient            : 275) : Searching for key: , search: key:key:Solar System MPC_2459641.5,key:key:Solar System MPC_2459642.5, filter: *, interval: 0 ms -<PHONE_NUMBER>361 ms, id/time: false/false
916976 INFO  (HBaser.HBaseClient            : 337) : Searching for key: , search: {key:key=Solar System MPC_2459641.5,Solar System MPC_2459642.5}, filter: *, interval: 0 ms -<PHONE_NUMBER>361 ms, id/time: false/false
916976 INFO  (HBaser.HBaseClient            : 943) : Merging * +  => *
916977 INFO  (HBaser.HBaseClient            : 464) : Performing range scan
922850 INFO  (HBaser.HBaseClient            : 513) : 21712 entries found
922850 INFO  (HBaser.HBaseClient            : 519) : 21712 results found in 5874ms
977182 INFO  (HBaser.HBaseClient            : 742) : Setting limit 100000

Which means the remaining 24 seconds (80%!) are spent in data manipulation in the server...


Term tags and aliases not translated

There are no translated fields for term tags or aliases, add these.

Should be fixed by 6d1fb1b


Add ArgoCD argocd.argoproj.io/sync-wave annotation to generated Kubernetes resources

Whenever the mode of KPOps is configured to Argo the manifestation process should add the argocd.argoproj.io/sync-wave annotation to determine the deployment order of the resources.
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

All applications except for the KafkaTopics should have a sync-wave of 1

Closed with #541


With the Beta from HA 0.110 following have changed

NOTE: For a feature to be created it has to be doable in the Amazon app because we are basically replicating the app for HA. If you can't find it in the app, we probably can't create it here.
Is your feature request related to a problem? Please describe.
in preparation for HA 0.110 the following errors are shown and make changes to the integration necessary
AlarmControlPanel is deprecated, modify AlexaAlarmControlPanel to extend AlarmControlPanelEntity
SwitchDevice is deprecated, modify AlexaMediaSwitch to extend SwitchEntity
SwitchDevice is deprecated, modify DNDSwitch to extend SwitchEntity
SwitchDevice is deprecated, modify ShuffleSwitch to extend SwitchEntity
SwitchDevice is deprecated, modify RepeatSwitch to extend SwitchEntity
MediaPlayerDevice is deprecated, modify AlexaClient to extend MediaPlayerEntity
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Thanks. This is a duplicate of #699


adding toctree using doctrees

This is an attempt at adding the toctree at the doctree level rather than at the page source level.
The doctrees do seem to be inserted properly, but for some reason Sphinx is still saying that it doesn't find the pages that are referenced in the doctrees...I need to debug this
Should look into what Sphinx is doing here: https://github.com/sphinx-doc/sphinx/blob/82d49bafc50835f4934f05567977d31f029c53de/sphinx/environment/init.py#L552

closing this as I don't think it's the right pattern to use for now


New AccountLoader Prototype

prototype for #997

Gm frens. pls have a look at this prototype and let me know what you think, if at all and how we might move forward with this.
@archseer. I fully agree that my previous suggestion for a new loader api was not idiomatic so I tried sth new
This new AccountLoader deserializes the data once and once only - when it’s created.
There is no load/load_mut/init. Instead it uses Deref/DerefMut. But this time Deref/DerefMut are actually as cheap as one expects as they just have to return a pointer to the stored deserialized data.
It uses a little bit of unsafe code, specifically a pointer to T. This is safe because the underlying &[u8] will never be freed while T is still pointing at them. Maybe there is a way to do it without this raw pointer but I havent found one yet.
Benefits of this new api are

same as all other accounts, just deref and derefmut
reduced compute unit cost (with the current api, every load deserializes the entire data structure again, this doesn’t cost much because it’s just reinterpreting the bytes but it’s still a little bit of unnecessary work)
compile time guarantees:

// WITH LOAD - RUNTIME ERROR

pub fn update_foo_second(ctx: Context<UpdateFooSecond>, second_data: u64) -> ProgramResult {
        let mut foo = ctx.accounts.foo.load_mut()?;
        let bar = ctx.accounts.foo.load()?;
        foo.second_data = second_data;
        Ok(())
    }

1) zero-copy
       Updates a a second zero copy account field:
     Error: failed to send transaction: Transaction simulation failed: Error processing Instruction 0: instruction tries to borrow reference for an account which is already borrowed

// WITHOUT LOAD - COMPILATION ERROR

pub fn update_foo_second(ctx: Context<UpdateFooSecond>, second_data: u64) -> ProgramResult {
        let foo = &mut ctx.accounts.foo;
        let bar = &ctx.accounts.foo;
        foo.second_data = second_data;
        Ok(())
    }

error[E0502]: cannot borrow `ctx.accounts.foo` as immutable because it is also borrowed as mutable
  --> programs/zero-copy/src/lib.rs:32:19
   |
31 |         let foo = &mut ctx.accounts.foo;
   |                   --------------------- mutable borrow occurs here
32 |         let bar = &ctx.accounts.foo;
   |                   ^^^^^^^^^^^^^^^^^ immutable borrow occurs here
33 |         foo.second_data = second_data;
   |         --- mutable borrow later used here

there would still be a couple things to do and this is just to show what I had imagined in my head. that said, all zero-copy tests still pass with no modification except removing the load calls. pls let me know what you think. would appreciate your feedback too @armaniferrante @fanatid

There's some other tests that need to be updated. E.g., tests/chat and others.

There's some other tests that need to be updated. E.g., tests/chat and others.

yes, this PR is really just to showcase how it could look like and get some feedback. still a lot to do and think about

We need reload function. Data in AccountInfo can be changed in CPI, so we need a way to update deserialized representation. (probably we need save too, Account have only reload 🤔)

@fanatid
ha I assumed that the reference to the T would keep pointing to the correct data because it's just pointing to the underlying &[u8] in the end so it would update automatically. we should definitely add a test for that to the zero-copy tests
but it should be possible to have both right? before the cpi, user can use the deref/derefmut features. after the cpi call, they can call a function (refresh?) that refreshes the reference. We could even do this automatically and then have a constraint (norefresh) that disables this for a particular (Account)Loader account if user really wants to save compute (although it's really not much they save, it's just a pointer cast after all)

ha I assumed that the reference to the T would keep pointing to the correct data because it's just pointing to the underlying &[u8]

Oh, indeed, I think you are right about pointer and underlaying data! But would be good to have a test with CPI.

Closing as stale.


Not all attachments are pushed to the collection

I have a problem with fetching all attachments from the email (v4 - I don't know abount other versions). In example I have 7 attachments on the email, but randomly getting 5 or 6 attachments. I don't know, maybe problem is with text attachments, because I have a problem only with text attachments, but this patch solvs the problem. I can't reproduce this problem and can't share my issue, because it is private data. You can try to get email message with 10 attachments with different types, maybe you will reproduce it. Is possible to fast merge this pull request?
There is a hotfix and tag for v4 which I using in project to solve this problem https://github.com/AdrianKuriata/php-imap/tree/4.1.2-hotfix https://github.com/AdrianKuriata/php-imap/tree/<IP_ADDRESS>

Hi @AdrianKuriata ,
thanks for your time and effort to investigate this issue. I'm indeed not able to reproduce this issue myself, but I don't see any harm in merging your pr.
Thanks again and best regards,


Switch to Reline?

https://www.ruby-lang.org/en/news/2023/12/25/ruby-3-3-0-released/

ext/readline is retired

We have reline that is pure Ruby implementation compatible with ext/readline API. We rely on reline in the future. If you need to use ext/readline, you can install ext/readline via rubygems.org with gem install readline-ext.
We no longer need to install libraries like libreadline or libedit.

🙏 this would be so great


Fixed broken links

Fix to issue #647

@cmosnick Im closing this in favor of #647  - here your editing outside the master branch on a page thats been depcreated. Thanks for pointing this out for us though! 👌


Allegheny County, PA types of arrests by police force

Submission notes
Looking for types of arrests by police in each municipality.
Location
Allegheny County
Requirements
Arrest records
How to help
Leave a comment if you'd like to help. Do you have ideas, questions, or need any more information to get started?

Work has begun on this issue, but we need someone else to pick up the baton!
How to progress this issue

[ ] Review the collected data referenced here
[ ] Leave a comment on this issue

Is there enough information at the sources referenced to understand the types of arrests being made in agencies in the county?
If not, are there other potential sources for arrest records?


[3.0] Add SPACESHIP_DIR_TRUNC_PREFIX, a prefix before cwd when truncated

A version of #280 for the 3.0 branch
When the cwd is truncated by SPACESHIP_DIR_TRUNC instead of showing:
a/b/c

It will show:
.../a/b/c

The prefix is configurable of course and can be disabled by setting it to an empty string.
I also modified SPACESHIP_DIR_TRUNC_REPO to use SPACESHIP_DIR_TRUNC_PREFIX and made it not truncate for a repository that is below the root directory. In addition, I optimized its code to use Zsh constructs instead of command substitutions.

cc @maximbaz

Thanks for PR. Let's do not include this feature in 3.0. I just don't want to make next major release too complex, because it already takes too much time to make it ready. We can release this later as a minor release after 3.0.

Re-based due to https://github.com/denysdovhan/spaceship-prompt/pull/385. It was fixed differently here by https://github.com/denysdovhan/spaceship-prompt/pull/281/commits/62d654c981639da6c0a81a0fd9819fe9d5772405.

Sorry I didn't pay attention to this PR for a long time.
My point on this: I don't like the idea of cluttering prompt with .../. I'd like to have it disabled by default. Prompt looks much cleaner without it.
BTW, instead of using ..., we can try …, which is much cleaner and shorter.

@denysdovhan I wasn't sure originally whether to use the Unicode ellipsis (…), but since Spaceship already uses other exotic Unicode chars and the Unicode ellipsis is quite older than them it should be OK.
In terms of whether to enable this by default or not, I guess that's up to you. Maybe @maximbaz @salmanulfarzy want to pitch in?
The facility to disable it is already there, you simply set SPACESHIP_DIR_TRUNC_PREFIX to be empty:
SPACESHIP_DIR_TRUNC_PREFIX=""

I added some more documentation so that even if you choose to disable it the user still has examples for how to enable it, or on the other hand, disable it.

In terms of whether to enable this by default or not, I guess that's up to you. Maybe @maximbaz @salmanulfarzy want to pitch in?

Whatever gets this PR finally merged 🙂 I personally find this feature very intuitive and I use it, but I don't mind enabling it locally if it's disabled by default.

I prefer it disabled by default, Giving a cleaner look. But prefix is much useful when current path is longer than SPACESHIP_DIR_TRUC.

I prefer it disabled by default for git directories, giving a cleaner look. But I find prefix to be much useful when current path is longer than SPACESHIP_DIR_TRUC on non-git directories.

There is currently only an option to enable or disable it for both outside and inside a repo. I guess it can be split into two separate options, though I think that makes is slightly more complex for users to enable/disable.
I will start at least by setting it to disabled by default, hopefully the instructions I left on how to enable it are enough.

Just wanted to ask if there are any updates on this or anything blocking the PR from being merged?

I see, thanks, I just came here because of #343 (for which there is a workaround). If the main reason for holding off the PR is to avoid possible breaking changes, I would perhaps look into pulling #384 since that seems more innocuous.
My two cents about the default behavior: I like … and I don't think it should be enabled by default.

Rebased.

@salmanulfarzy https://github.com/denysdovhan/spaceship-prompt/pull/417 should fix the ".git directory" case.


How to use date picker

Hello, I´m new here and I would like to know how do I implement a date picker on my website, I don't know if I have to download pickadate.js libraries or not and if i have to which files do I have to link to my webpage?
Tanks

No need to download pickadate.js
http://codepen.io/Dogfalo/pen/zGNvMm


skip the ldd test in case of fedora

Required items, please complete
Change(s):

See guidance below

Reason for Change(s):

See guidance below

Version Updated:

Required only for Detections/Analytic Rule templates
See guidance below

Testing Completed:

See guidance below

Checked that the validations are passing and have addressed any issues that are present:

See guidance below

Guidance <- remove section before submitting

Before submitting this PR please ensure that you have read the following sections and filled out the changes, reason for change and testing complete sections:
Thank you for your contribution to the Microsoft Sentinel Github repo.

Details of the code changes in your submitted PR.  Providing descriptions for pull requests ensures there is context to changes being made and greatly enhances the code review process.  Providing associated Issues that this resolves also easily connects the reason.

Change(s):

Updated syntax for XYZ.yaml

Reason for Change(s):

New schema used for XYZ.yaml
Resolves ISSUE #1234

Version updated:

Yes
Detections/Analytic Rule templates are required to have the version updated

The code should have been tested in a Microsoft Sentinel environment that does not have any custom parsers, functions or tables, so that you validate no incorrect syntax and execution functions properly.  If your submission requires a custom parser or function, it must be submitted with the PR.

Testing Completed:

Yes/No/Need Help

Note: If updating a detection, you must update the version field.

Before the submission has been made, please look at running the KQL and Yaml Validation Checks locally.
https://github.com/Azure/Azure-Sentinel#run-kql-validation-locally

Checked that the validations are passing and have addressed any issues that are present:

Yes/No/Need Help

Note: Let us know if you have tried fixing the validation error and need help.

References:

Guidance for Detection checks
General contribution guidance
PR validation troubleshooting

Hi @dvir-ms, we are working on this validation failure and will provide you with an update as soon as possible.

Hi @udidekel, please provide your feedback on this. Thanks


Purpose of "--" in lint-fix command?

npm run lint -- --fix
What does the -- do here?

This was kind of tough to search blind, but it suddenly dawned on me what this probably did, and a quick Google search confirmed.
-- passes the --fix argument to the npm run lint command, instead of it being an argument on the npm run lint-fix command.
https://stackoverflow.com/questions/11580961/sending-command-line-arguments-to-npm-script


Update zzseotk.php

This change adds support for Leo Theme custom blog module, "blogleoblog".

This change adds support for Leo Theme custom blog module, "blogleoblog".

 Fix CMS Category link

see #34

Leo Blog support merged into a customised hack-branch
@jelicesv please try if it works for you .. I think the only place where $id_lang has to be not-null could be the Blog class instantiation .. please check
Thank you


Issue after running 'vagrant provision' in Windows 10 machine

Issue Type

Bug Report / Support Request

Your Environment

Vagrant 1.9.7
cmd: 'vboxmanage' is not recognized as an internal or external command,
operable program or batch file.
cmd: 'ansible' is not recognized as an internal or external command,
operable program or batch file.

Your OS

Windows (10, build 15063.540, version 1703)

Full console output
https://gist.github.com/JuanGongora/3b3cba59212adf844ffdc6567f571705
Summary
I am new to Drupal VM, so I was following the instructions from this blog post that was showing how to work with a Windows machine.
I got stuck after running vagrant up and then as the post described, followed up with vagrant provision to see if the error would be resolved. Unfortunately the error remained, which was saying that a PHP version higher than 5.6.31 is required for various installations. However the config.yml file inside the drupal-vm folder states in the blog to use php_version: "5.6".  So I'm not sure what to do.
I currently have this version of PHP installed in my machine:
PHP 7.1.6 (cli) (built: Jun  8 2017 01:52:52) ( NTS MSVC14 (Visual C++ 2015) x64 )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies

Do I have to add some sort of additional dependency that will switch it to my current version from within vargrant at a certain point, or some other file in the drupal-vm folder?
Thanks for your time,
Juan

Problem 1 - Installation request for doctrine/cache v1.7.0 -> satisfiable by doctrine/cache[v1.7.0]. - doctrine/cache v1.7.0 requires php ~7.1 -> your PHP version (5.6.31) does not satisfy that requirement.

Seems PHP 7.1 is required so you'll need to set php_version: "7.1"

Thanks oxyc!


TFRS - Director is not able to accept compliance report

Describe the Bug
Director is unable to accept a compliance report.
Expected Behaviour
The system should accept the report and the corresponding credit transaction should be initiated. A 'validation' type credit transaction if credits are issued or a 'reduction' type credit transaction if credits are used to offset a net debit position.
Actual Behaviour
The system throws an error after clicking the Accept button.
Implications
Blocking the director from processing reports and issuing credits into the market.
Steps To Reproduce
Steps to reproduce the behaviour:
User/Role: IDIR user / Director role

Go to the Compliance & Exclusion Reports page
Click on the "For Director Review" filter
Click on one of the compliance reports that is ready for the Director's review and decision (one in the filtered list)
Click on the Accept button
See error

Example Report in Test:
https://tfrs-test.apps.silver.devops.gov.bc.ca/compliance_reporting/edit/317/schedule-assessment
Screenshot of error in Prod:

Developer Notes

Check if we're receiving api response
if we're not then handle the exception.
if we're receiving the api response but code bug in React components.

Is working, but want to retest after issues related to 2273 are resolved.

Is working, but want to retest after issues related to 2273 are resolved.


Fixed TO-DO header formatting

Removes the hashes and makes it so that it is a header

This also added another commit that changes the lists to use markdowns versions opposed to embedding HTML.


console.openshift.io cluster role permission in cluster_role.yaml but not csv

Appears that the console.openshift.io permissions in cluster_role.yaml, added in #1177, is not being included in the manifest (clusterserviceversion yaml).

May I assign this to you, @rubenvp8510 ?

Yes :) no problem


Bump Dependencies #4417145034

Bumps dependencies for product-is. Link : https://github.com/wso2/product-is/actions/runs/4417145034

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.jenkins-is-staging seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


How to get Runtime Caching of root "/", mixed with precache of assets.

In my Nuxt application I always SSR the home page, which has a constantly updating list of posts. I'd like to cache it at Runtime, so that if a user reloads while offline, they get the latest version they've seen. At the same time, I'd like most of the rest of my assets to be precached. I can't seem to get this to happen.
My config looks something like this:
export default defineNuxtConfig({
  nitro: {
   // Disabled, as I don't want to serve a prerendered page that could be days-stale:
   // prerender: {
   //   crawlLinks: false,
   //   routes: ['/'],
   // },
  },
  modules: ['@vite-pwa/nuxt'],
  pwa: {
    // Added scope/base mostly from browsing the Elk & Vitesse repos:
    scope: '/',
    base: '/',
    manifest: {   ...  },  //
    injectRegister: 'auto',
    registerType: 'autoUpdate',
    // Disabled, since I don't want to prerender (but I might have misunderstood what this does):
    // registerWebManifestInRouteRules: true,
    workbox: {
      // Successfully precaches other assets:
      globPatterns: ['**/*.{js,css,html,png,svg,ico}'],
      // Try to Runtime Cache the home page:
      runtimeCaching: [{
        urlPattern: "/",
        handler: 'NetworkFirst',
      }]
      // Don't use navigationFallback because "/" isn't in our precache manfiest:
      // navigationFallback: '/'
    },
  },
})

If I uncomment the commented lines, I can get it working - however then I'm stuck with prerendering, and what I want is to have the client cache a SSR page. But when I try runtime caching, caching is flakey, because workbox gets a:
non-precached-url: non-precached-url :: [{"url":"index.html"}]

But if it does build the Cache, it's lacking the root. Here's the difference with prerendering and not-prerendering for my Application Cache:

With Prerender
Trying to Runtime Cache

17 entries (including "/")
16 entries (missing "/")

Is there something I'm missing?

You need at least 1 prerendered route to work offline using generateSW strategy.

You can try navigateFallback: undefined and let nuxt error do its work, not tested.

Thanks for the quick reply! I'll try that and get back

setting navigateFallback: undefined seemed to work! Thanks! I'll double check in prod

Works great! Thanks for the quick response, closing this 👍


Type explainer can't parse methods ending in '?' and '!'

It would be nice to be able to document methods as @param foo [#all?] but the type explainer currently fails on methods like this. I realize it is really hard to parse ruby methods so I'm definitely not asking for something comprehensive, but adding ? and ! would be nice.
Steps to reproduce
$ irb
>> require 'yard'
>> YARD::Tags::TypesExplainer::Parser.parse('#all?')
SyntaxError: invalid character at ?
from /Users/johnbackus/.gem/ruby/2.3.1/gems/yard-0.9.8/lib/yard/tags/types_explainer.rb:154:in `block in parse'
>> YARD::Tags::TypesExplainer::Parser.parse('#gsub!')
SyntaxError: invalid character at !
from /Users/johnbackus/.gem/ruby/2.3.1/gems/yard-0.9.8/lib/yard/tags/types_explainer.rb:154:in `block in parse'

I would be happy to do a PR fixing this

For context, I opened this issue because I am currently using the type explainer for backus/yardcheck. I realize it is private API and I intend on replacing it with my own parser, but I figured this would still be something YARD might want to fix

You can actually probably use CodeObjects::METHODNAMEMATCH to parse most method names.

Cool. I can open a PR in the future to support that. Any interest in making the type explainer API public at some point?

I don't see where it's marked private. You should be able to use it already.

Oh I assumed that it was private I guess because all of the namespaces within it are marked @private but that makes sense. Cool. I'm already using it in Yardcheck so glad to know it is public API

Ah ok I see what I was concerned about before. TypeExplainer.explain! is public but I need to actually invoke the parser and get back objects. explain! just gives me a string

Root issue is fixed. As far as making it public I'd be more open to it if it were documented. If you wanted to contribute docs that would help.


Update numpy/pandas for better 3.12 support

Related Issues

fixes #issue-number add the addressed issue as link

Proposed Changes:

How did you test it?

Notes for the reviewer

Checklist
[] I have updated the related issue with new insights and changes
[] I added unit tests and updated the docstrings
[] I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
[] I documented my code
[] I ran pre-commit hooks and fixed any issue

Seeing variants of https://github.com/python-poetry/poetry/issues/7611 annoyingly


Fix parsing of command

Fix #45 , Fix #47
Under the current rule for parsing of command, whether the command is a subexpression or not, line breaks are allowed between command arguments.
This causes the parser to fail to correctly detect the end of a command or pipeline.
This pr changes the rules to allow line breaks between command arguments only when the command is a subexpression.
After this change, the highlights looks like follows

nice work! we appreciate all your efforts here.

nice :ok_hand:
after updating, it looks much better :pray:
i just have the whole content of the string interpolation that is white, i.e. ($env.HOME)

@amtoine isn't that controlled by highlights.scm? I'm not positive but I thought it was.

@amtoine isn't that controlled by highlights.scm? I'm not positive but I thought it was.

it might not be related to that PR yeah, because it's there in other places, i.e. $env.HOME never appear to highlight properly in my Neovim.
i did update both the parser and the query file :wink:

what i'm suggesting is that the highlights.scm file is very old and it probably needs a good updating too.