Example needed for KeSetHardwareCounterConfiguration

This page need either more details or at least some code example on how to use this API. Currently using any index value in the hardware counter structure always return success, making it hard to understand how to use the API.
Additionally, the Hardware Counter struct documentation related to this need a better explanation on how to find the index of the counters we are interested in.
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c27dbd8c-d6b0-8259-5093-bf3a7651b546
Version Independent ID: 7aea9698-0ece-2126-dee5-dfbaa402ab7a
Content: KeSetHardwareCounterConfiguration function (ntddk.h) - Windows drivers
Content Source: wdk-ddi-src/content/ntddk/nf-ntddk-kesethardwarecounterconfiguration.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @tedhudek
Microsoft Alias: tedhudek

To add more details, I have been trying to figure out how to use this API but I cannot seem to figure out the correct index value needed for the hardware counter (this documentation is the only information available online about this API it seems)

I don't have much experience here, but it looks like this ties into hardware counter profiling, which is described at https://docs.microsoft.com/en-us/previous-versions/windows/desktop/hcp/hcp-reference.  Can you use the Threadprofiling APIs to get counter info?  For example -- https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-querythreadprofiling.  I added both links to the ref page (should be live soon).
I checked the driver samples repo but did not see any examples of KeSetHardwareCounterConfiguration.
You could also try the forums, if this is still an issue:

Windows Hardware WDK and Driver Development
OSR's ntdev List

Thanks for reaching out and good luck!

This does not actually fix the documentation problem described by the OP. Calling QueryThreadProfiling simply tells you if you have previously called EnableThreadProfiling on a thread. It does not help you figure out how to call KeSetHardwareCounterConfiguration.
KeSetHardwareCounterConfiguration is supposed to control which hardware counters are enabled for profiling. According to the documentation, you can supposedly call it to enable hardware performance counters that will then be returned by future calls to ReadThreadProfilingData,
However, nobody (to the best of my knowledge) has ever figured out how to get this to work, because there is no documentaiton from Microsoft about how to correctly call KeSetHardwareCounterConfiguration. You can call it all you want - and if you pass a properly formatted array and correct count, it will even return ERROR_SUCCESS - but you will still not get any counters coming out of ReadThreadProfilingData.
So this issue should be reopened. Somebody needs to publish a working example of KeSetHardwareCounterConfiguration so people outside Microsoft can actually use this API. As it currently stands, the KeSetHardwareCounterConfiguration/EnableThreadProfiling/ReadThreadProfilingData documentation clearly states that you can enable and retrieve performance counters, but lack sufficient explanation for anyone to figure out how to make that actually happen.
- Casey


Reference link of content is updated in md file

In version 3.1 md file, reference link of the content updated

Duplicate of PR #3408  , but thank you for the work!


Expense not yet paid out

Describe the bug
Hi! Not sure if this is the right place to report this, but this expense on the Manyverse OpenCollective has not yet been paid out, although it has been some weeks already, and the invoice date has passed too. I do this every month, but this month the payment hasn't been sent yet.

@staltz I already forwarded internally but the best channel for that is<EMAIL_ADDRESS>

Closing as conversation has moved to the other channel.


[Feature request] Ability to reclaim unused space

I've noticed that after Windows Updates, bunch of files deleted, I can't really do much with my VMs hard drive to make it smaller. For example Parallels can reclaim unused space on every power off and I'm wondering if it's possible to implement that here.

This appears to be a duplicate of #3460 ?

Duplicate of #3460


Backport double sided material spec

backport of pull request #410

Codecov Report

Merging #418 (9361861) into sdf9 (0e77816) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             sdf9     #418      +/-   ##
==========================================
+ Coverage   86.25%   86.26%   +0.01%
==========================================
  Files          59       59
  Lines        9181     9191      +10
==========================================
+ Hits         7919     7929      +10
  Misses       1262     1262

Impacted Files
Coverage Δ

src/Material.cc
98.07% <100.00%> (+0.13%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e77816...9361861. Read the comment docs.

some odd windows test failures after merging with sdf9:

 https://build.osrfoundation.org/job/sdformat-ci-pr_any-windows7-amd64/1686/testReport/

1: [----------] 2 tests from DOMActor
1: [ RUN      ] DOMActor.LoadActors
  1/172 Test   #1: INTEGRATION_actor_dom ..................................Exit code 0xc0000417
***Exception:   0.09 sec

35: [----------] 1 test from DOMWorld
35: [ RUN      ] DOMWorld.LoadLights
 35/172 Test  #35: INTEGRATION_light_dom ..................................Exit code 0xc0000417
***Exception:   0.08 sec

81: [----------] 5 tests from DOMWorld
81: [ RUN      ] DOMWorld.NoName
81: No HOME defined in the environment. Will not log.
81: [       OK ] DOMWorld.NoName (47 ms)
81: [ RUN      ] DOMWorld.Duplicate
81: [       OK ] DOMWorld.Duplicate (59 ms)
81: [ RUN      ] DOMWorld.LoadIncorrectElement
81: [       OK ] DOMWorld.LoadIncorrectElement (62 ms)
81: [ RUN      ] DOMWorld.Load
 81/172 Test  #81: INTEGRATION_world_dom ..................................Exit code 0xc0000417
***Exception:   0.24 sec

85: [----------] 7 tests from DOMActor
85: [ RUN      ] DOMActor.DefaultConstruction
85: [       OK ] DOMActor.DefaultConstruction (0 ms)
85: [ RUN      ] DOMActor.CopyConstructor
85: [       OK ] DOMActor.CopyConstructor (0 ms)
85: [ RUN      ] DOMActor.CopyAssignmentOperator
85: [       OK ] DOMActor.CopyAssignmentOperator (0 ms)
85: [ RUN      ] DOMActor.MoveConstructor
85: [       OK ] DOMActor.MoveConstructor (0 ms)
85: [ RUN      ] DOMActor.MoveAssignment
85: [       OK ] DOMActor.MoveAssignment (0 ms)
85: [ RUN      ] DOMActor.CopyAssignmentAfterMove
85: [       OK ] DOMActor.CopyAssignmentAfterMove (0 ms)
85: [ RUN      ] DOMActor.Add
 85/172 Test  #85: UNIT_Actor_TEST ........................................Exit code 0xc0000417
***Exception:   0.03 sec

@osrf-jenkins run tests please

now the windows build is failing with an internal compiler error:

 https://build.osrfoundation.org/job/sdformat-ci-pr_any-windows7-amd64/1687/

I'm going to take that build machine offline and try again

@osrf-jenkins run tests please


Fix test_observations_by_known_user

This test was broken by the merger of PR #858
(Post merger, users must login in order to GET observations_by_user.)

Thanks, Joe.
On Wed, Feb 23, 2022 at 9:08 AM Joseph D. Cohen @.***>
wrote:

This test was broken by the merger of PR #858
https://github.com/MushroomObserver/mushroom-observer/pull/858
(Post merger, users must login in order to GET observations_by_user.)
You can view, comment on, or merge this pull request online at:
https://github.com/MushroomObserver/mushroom-observer/pull/871
Commit Summary

da9a4d1
https://github.com/MushroomObserver/mushroom-observer/pull/871/commits/da9a4d17b8c510c5ecb4faeb94d86106dbc42ef0
Fix test that was broken by merger of PR #858

File Changes
(1 file
https://github.com/MushroomObserver/mushroom-observer/pull/871/files)

M test/controllers/observer_controller_test.rb
https://github.com/MushroomObserver/mushroom-observer/pull/871/files#diff-1f21ebcbd530f50a48f065f398b8a9d3e5f8c31a7c62ea8062fd62c631ec5e9e
(2)

Patch Links:

https://github.com/MushroomObserver/mushroom-observer/pull/871.patch
https://github.com/MushroomObserver/mushroom-observer/pull/871.diff

—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/pull/871, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNJP77GC77D37TEU6PTU4TS4HANCNFSM5PEOUL7A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


PBS being driven by a C script

I am looking to run 3 C scripts followed by a Python script iteratively over a number of snapshots.  The C scripts are parallelized and the Python script is not (the Python script just handles setting up ini files and cleaning up for the next snapshot iteration).
Ideally I would like to have a master C script to handle running each of these other scripts via the system() command. This way I can explicitly handle the return values of each of my scripts.
However since these would all be deployed to largemem, I need to think about the correct way to request processors/memory from PBS.  For example, should it look like
#!/bin/bash
#PBS -q largemem
#PBS -l nodes=2:ppn=32
#PBS -l walltime=01:00:00:00
#PBS -l pmem=1000gb

/Path/to/driver_script

And then inside the driver script have
// Loop over Snapshots //
exit_code =  system(mpirun -np 64 /path/to/first_script first_ini.ini);
// Handle exit code //

exit_code =  system(mpirun -np 64 /path/to/second_script second_ini.ini);

// etc...

Will the driver_script be executed once and the 64 processors wait idly until they're called with the mpirun call? Do I need to deal with nasty synchronization (please no)?
I definitely will give this implementation a shot myself but I thought I'd throw it open and see if anyone has had experiencing with executing multiple (parallel) codes within the scheduling system.
Cheers!

You do not need the "exit_code" handling explicitly that way - you should be able to handle the exit codes directly from the PBS script.
Check out the example script provided by NCI for self-submitting jobs: here

You're exactly correct.  Shell provides a way to check the exit code of the previous command with $?.  The following script now correctly runs, checking the exit code of each script along the way.
#!/bin/bash
# specify the queue name
#PBS -q sstar
# resource requests
#PBS -l nodes=1:ppn=4
#PBS -l walltime=00:20:00:00
#PBS -N self_consistent_27_60
#PBS -l pmem=8gb

# Load Modules
module load gcc/6.4.0
module load fftw/x86_64/gnu/3.3.3-openmpi-psm
module load gsl/x86_64/gnu/1.9
module load python/x86_64/gnu/3.5.2

LowSnap=27
HighSnap=27

NUMPROC=4

for ((snap_idx = LowSnap; snap_idx < HighSnap+ 1; ++snap_idx))
{
  echo "Running Snapshot ${snap_idx}"
  mpirun -np ${NUMPROC} /path/to/first_script first_ini.ini > /path/to/log.log
  exit_code=$?
  if [ ${exit_code} -ne 0 ]; then
    echo "First code exited with errorcode ${exit_code}"
    exit ${exit_code}
  fi

  # Repeat for all other scripts.

} # Loop over snapshots.

Closing the issue as everything is now gucci.


Import ZTF DR22 on Epyc

Raw data is here: /data3/epyc/data3/hats/raw/ztf/lc_dr22
The result catalog should go here: /data3/epyc/data3/hats/catalogs/ztf_dr22/ztf_lc
I haven't checked checksums of the files, failures are possible.

Per the README, checked the checksums of the files.  This operation has not yet completed, probably because I set its nice value so low (in order not to interfere):
nohup md5sum -c checksum.md5 > ~/md5-output.log &
# Process ID is 127216
renice 20 127216

At the time of this writing, it is still running, and one of the parquet files has failed its checksum so far:
wc -l checksum.md5 ~/md5-output.log
  176561 checksum.md5
  105392 /astro/users/dtj1s/md5-output.log
  281953 total

grep -v ' OK$' ~/md5-output.log
./0/field000385/ztf_000385_zr_c11_q2_dr22.parquet: FAILED

The import also failed with this message:
RuntimeError: Some mapping stages failed. See logs for details.

The script I used:
from dask.distributed import Client
from hats_import.pipeline import pipeline_with_client
from hats_import.catalog.arguments import ImportArguments

def main(input_path: str, output_path: str = "./output", output_artifact="test_cat"):
    args = ImportArguments(
        sort_columns="objectid",
        ra_column="objra",
        dec_column="objdec",
        input_path=input_path,
        output_artifact_name=output_artifact,
        output_path=output_path,
        file_reader="parquet",
    )
    with Client(n_workers=10, memory_limit="10GB", threads_per_worker=2) as client:
        pipeline_with_client(args, client)

if __name__ == "__main__":
    main(
        "/data3/epyc/data3/hats/raw/ztf/lc_dr22",
        "/data3/epyc/data3/hats_import/catalogs/ztf_dr22",
        "ztf_lc",
    )

The most common errors printed to the console were of this form:
Failed MAPPING stage with file /data3/epyc/data3/hats/raw/ztf/lc_dr22/1/index.html?C=N;O=D
  worker address: tcp://<IP_ADDRESS>:38628
Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
Failed MAPPING stage with file /data3/epyc/data3/hats/raw/ztf/lc_dr22/1/index.html?C=N;O=D
2024-12-03 19:20:09,841 - distributed.worker - ERROR - Compute Failed

I will try to resume the import in a way that specifically only includes *.parquet files.

Verification of checksums concluded, and only one file was found that failed.  Restored this file by rereading it from the source and verifying the checksum:
curl https://irsa.ipac.caltech.edu/data/ZTF/lc/lc_dr22/0/field000385/ztf_000385_zr_c11_q2_dr22.parquet -o fixme.parquet
md5sum fixme.parquet
12b316eac3768bfac393c503ff94f55e  # correct
cp fixme.parquet ./0/field000385/ztf_000385_zr_c11_q2_dr22.parquet

Modifying the script to use explicit glob patterns to identify the Parquet files only.  Note also the change in output path, now possible due to corrected permissions.
import glob

from dask.distributed import Client
from hats_import.pipeline import pipeline_with_client
from hats_import.catalog.arguments import ImportArguments

def main(input_path: str, output_path: str = "./output", output_artifact="test_cat"):
    # Cannot use input_path directly because the input directory contains many, many
    # non-parquet files, which clog up the pipeline.  Rather, use the glob module
    # to create an authoritative and complete list of parquet files.
    print(f"Reading {input_path} for *.parquet files")
    parquet_list = glob.glob(f"{input_path}/**/*.parquet", recursive=True)
    print(f"Parquet files found: {len(parquet_list)}")
    args = ImportArguments(
        sort_columns="objectid",
        ra_column="objra",
        dec_column="objdec",
        # input_path=input_path,
        input_file_list=parquet_list,
        output_artifact_name=output_artifact,
        output_path=output_path,
        file_reader="parquet",
    )
    with Client(n_workers=10, memory_limit="10GB", threads_per_worker=2) as client:
        pipeline_with_client(args, client)

if __name__ == "__main__":
    main(
        "/data3/epyc/data3/hats/raw/ztf/lc_dr22",
        "/data3/epyc/data3/hats/catalogs/ztf_dr22",
        "ztf_lc",
    )

With a hint from @delucchi-cmu, created batched indexes of the Parquet files, to reduce the amount of file I/O, and to compensate for the fact that the files' size distribution is logarithmic:
def prepare_batches(input_path: str, n_batches: int, prefix: str) -> list[str]:
    """
    Given a directory to search for `.parquet` files, find them all,
    get their sizes, and sort them into n_batches files, each of which
    will contain roughly an equal number of bytes.  These batches
    will be written to a family of files that all start with `prefix`
    and are numbered.
    """
    print(f"Reading {input_path} for *.parquet files")
    parquet_list = glob.glob(f"{input_path}/**/*.parquet", recursive=True)
    print(f"Parquet files found: {len(parquet_list)}")
    file_sizes = [Path(p).stat().st_size for p in parquet_list]
    print(f"Total size of the files: {sum(file_sizes)}")
    print(f"Sorting into {n_batches} batches")
    batches = binpacking.to_constant_bin_number(
        list(zip(file_sizes, parquet_list)), n_batches, weight_pos=0
    )
    # Now write out the batch files
    batch_files = []
    for i, b in enumerate(batches):
        batch_file = Path(f"{prefix}_{i:03d}.batch")
        batch_file.write_text("\n".join([fname for sz, fname in b]) + "\n")
        batch_files.append(batch_file)
    return batch_files

def main(input_path: str, n_batches: int, output_path: str = "./output", output_artifact="test_cat"):
    batch_files = prepare_batches(input_path, n_batches, "./batch")
    args = ImportArguments(
        sort_columns="objectid",
        ra_column="objra",
        dec_column="objdec",
        # input_path=input_path,
        input_file_list=batch_files,
        output_artifact_name=output_artifact,
        output_path=output_path,
        file_reader="indexed_parquet",
    )
    # Attempt to scale splitting phase.
    with Client(n_workers=8, memory_limit="32GB", threads_per_worker=2) as client:
        pipeline_with_client(args, client)

This caused the planning, mapping, and binning phases to complete much more quickly, each phase on the order of 20 to 30 minutes.  The splitting phase is still taking a long time (~8h projected) but appears to be on track to complete without errors or warnings.

Uh oh.
Planning  : 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:07<00:00,  1.88s/it]
Binning   : 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:11<00:00,  5.85s/it]
Splitting : 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2683/2683 [7:38:01<00:00, 10.24s/it]
Reducing  :  13%|█████████████████████████▊                                                                                | 1443/10839 [43:50<2:19:43,  1.12it/s]
Failed REDUCING stage for shard: 5 1168
  worker address: tcp://<IP_ADDRESS>:38087
Unexpected number of objects at pixel (Order: 5, Pixel: 1168). Expected 340518, wrote 326169
Failed REDUCING stage for shard: 5 1168
  worker address: tcp://<IP_ADDRESS>:38087
Unexpected number of objects at pixel (Order: 5, Pixel: 1168). Expected 340518, wrote 326169

Will try restarting in case it was a question of resources.

No such luck.  Error is stable, and as such, this particular attempt appears to be at an end.
Failed REDUCING stage for shard: 5 1168
  worker address: tcp://<IP_ADDRESS>:33080
Unexpected number of objects at pixel (Order: 5, Pixel: 1168). Expected 340518, wrote 326169
Failed REDUCING stage for shard: 5 1168
  worker address: tcp://<IP_ADDRESS>:33080
Unexpected number of objects at pixel (Order: 5, Pixel: 1168). Expected 340518, wrote 326169
2024-12-09 15:43:57,157 - distributed.worker - ERROR - Compute Failed
Key:       reduce_pixel_shards-3d9b695a9bba43de503ca033ebd7a136

I have another scheme for batching the files that keeps the fields together.  I had originally expected that it was only an optimization, but perhaps there is something about it that lets it get past this obstacle.  This is the code for creating it:
def write_batches(parquet_file_sizes: list[tuple[int, str]]):
    total_size = sum([x[0] for x in parquet_file_sizes])
    bin_size = total_size // n_bins
    pat = re.compile(r'^.*/field(\d+)/.*$')
    field = None
    batch_count = 0
    bytes_in_bin = 0
    f_out = None

    def cycle_batch(done = False):
        nonlocal bytes_in_bin, batch_count, f_out
        if f_out is not None:
            f_out.close()
            print("Closing batch", batch_count, "containing", bytes_in_bin, "bytes")
        bytes_in_bin = 0
        batch_count += 1
        if not done:
            f_out = open(f"{prefix}_{batch_count:03d}.batch", "w")

    cycle_batch()
    while parquet_file_sizes:
        file_sz, file_name = parquet_file_sizes.pop(0)
        next_field = pat.match(file_name).group(1)
        if bytes_in_bin > bin_size and next_field != field:
            cycle_batch()
        f_out.write(file_name + '\n')
        field = next_field
        bytes_in_bin += file_sz
    cycle_batch(done=True)

Used the above to create index files with a limit of 1.5GiB apiece.  This ran smoothly for planning and mapping, but splitting, even though it seems to be continuing, has run into this error.  That batch has 128 parquet files in it, most of which are about 20MiB, and which only contains files from field 478; the files are limit_1_5G_99{5,6,7}.batch.
Failed SPLITTING stage with file limit_1_5G_996.batch
  worker address: tcp://<IP_ADDRESS>:34155
Unsupported cast from list<item: double> to double using function cast_double
2024-12-10 13:53:44,159 - distributed.worker - ERROR - Compute Failed
Key:       split_pixels-efbca3cbd64256d07453ff2ef83e6274
State:     executing
Task:  <Task 'split_pixels-efbca3cbd64256d07453ff2ef83e6274' split_pixels(, ...)>
  File "/astro/users/dtj1s/.conda/envs/dtj1s-py3.12/lib/python3.12/site-packages/hats_import/catalog/map_reduce.py", line 198, in split_pixels
    raise exception
  File "/astro/users/dtj1s/.conda/envs/dtj1s-py3.12/lib/python3.12/site-packages/hats_import/catalog/map_reduce.py", line 170, in split_pixels
    for chunk_number, data, mapped_pixels in _iterate_input_file(
                                             ^^^^^^^^^^^^^^^^^^^^
  File "/astro/users/dtj1s/.conda/envs/dtj1s-py3.12/lib/python3.12/site-packages/hats_import/catalog/map_reduce.py", line 52, in _iterate_input_file
    for chunk_number, data in enumerate(file_reader.read(input_file, read_columns=read_columns)):
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/astro/users/dtj1s/.conda/envs/dtj1s-py3.12/lib/python3.12/site-packages/hats_import/catalog/file_readers.py", line 431, in read
    for batch in input_dataset.to_batches(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_dataset.pyx", line 3806, in _iterator
  File "pyarrow/_dataset.pyx", line 3424, in pyarrow._dataset.TaggedRecordBatchIterator.__next__
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status

Splitting completed after 6h but then the import stopped and failed, re-reporting the error above.
Reran this, but it errored out exactly the same way on exactly the same batch, limit_1_5G_996.batch.  This contains 128 files from field 478.
Reading the schema out of all of these files, 126 of them look like this:
objectid: int64
filterid: int8
fieldid: int16
rcid: int8
objra: float
objdec: float
nepochs: int64
hmjd: list<item: double>
  child 0, item: double
mag: list<item: float>
  child 0, item: float
magerr: list<item: float>
  child 0, item: float
clrcoeff: list<item: float>
  child 0, item: float
catflags: list<item: int32>
  child 0, item: int32
__index_level_0__: int64
-- schema metadata --
pandas: '{"index_columns": ["__index_level_0__"], "column_indexes": [{"na' + 1688

and two of them look like this:
objectid: int64
filterid: int8
fieldid: int16
rcid: int8
objra: float
objdec: float
nepochs: int64
hmjd: double
mag: float
magerr: float
clrcoeff: float
catflags: int32
__index_level_0__: uint32
-- schema metadata --
pandas: '{"index_columns": ["__index_level_0__"], "column_indexes": [{"na' + 1663

The two files in that batch with the list<double> schema are:
/data3/epyc/data3/hats/raw/ztf/lc_dr22/0/field000478/ztf_000478_zg_c15_q3_dr22.parquet
/data3/epyc/data3/hats/raw/ztf/lc_dr22/0/field000478/ztf_000478_zr_c15_q1_dr22.parquet

There are two other batches with field 478, and all of the files in those batches use list<double>.

Other batches that contain non-list schema:
limit_1_5G_201.batch
limit_1_5G_691.batch
limit_1_5G_935.batch
limit_1_5G_988.batch
limit_1_5G_996.batch
limit_1_5G_1000.batch
limit_1_5G_1207.batch
limit_1_5G_2231.batch
limit_1_5G_3962.batch
limit_1_5G_3970.batch

I don't know why these others didn't cause the same problem.  Luck of the draw?

Found 129 empty parquet files.
empty_parquet.txt

Job finished, getting to 100%.  Tested the output catalog:
from hats.io.validation import is_valid_catalog
output_path = "/data3/epyc/data3/hats/catalogs/ztf_dr22"
ztf_dr22_path = output_path + "/" + "ztf_lc"
print(is_valid_catalog(ztf_dr22_path)
True

Catalog is valid.


feat: add IGetWeatherQuality_2

Owner

[x] Nickname(Tuya): 洛竹

Description
Please provide a brief summary of the changes. If it is related to an issue, please bring the issue number, like fixes #${number}.
Package

[x] tuya-panel-api

Type of change
Please select the relevant option.

[x] New feature (non-breaking change which adds functionality)

Codecov Report

Merging #80 (880596d) into main (c3b68af) will decrease coverage by 0.03%.
The diff coverage is 28.57%.

@@            Coverage Diff             @@
##             main      #80      +/-   ##
==========================================
- Coverage   61.02%   60.99%   -0.04%
==========================================
  Files         157      157
  Lines        6020     6025       +5
  Branches     1374     1375       +1
==========================================
+ Hits         3674     3675       +1
- Misses       2342     2346       +4
  Partials        4        4

Impacted Files
Coverage Δ

packages/tuya-panel-api/src/common/device.ts
50.00% <28.57%> (-7.90%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3b68af...880596d. Read the comment docs.

hi I am so sorry . I think my kids has done some thing while I get coffee . so sorry

hi I am so sorry . I think my kids has done some thing while I get coffee . so sorry

???


Add view option: List, Medium Icons, Large Icons [Gssoc'23]

hey @srivastavaritik I would like to add the feature to view files and folders in List, Medium Icons, and Large Icons. Please assign me this task to make this contribution possible under Gssoc'23.

Hi, @codesfromvikrant , you may work on this and please refer to the Contribution guidelines on the readme once.
All the best!


[Coverage] Disable forced emission for unmapped decls

Disable forced SILGen for lazy functions which are not used for code
coverage reporting.
As a drive-by, fix the forced emission logic for functions profiled for
PGO.
This helps address a compile-time issue (r://39332957).

@swift-ci Please smoke test

@swift-ci Please smoke test

Hi @shahmishal, this PR has passed smoke testing but I'm unable to merge it. Any clues as to what's going on?
Linux results: https://ci.swift.org/job/swift-PR-Linux-smoke-test/6225/
macOS results: https://ci.swift.org/job/swift-PR-osx-smoke-test/6554/

@swift-ci Please smoke test Linux platform

The fallback Linux smoke testing job has been running for 3 hours now. I’ll try again.

@swift-ci Please smoke test Linux platform


Cannot create new cf.EventDispatcher() when loading with ES6 Module Bundler

Hi,
I am trying to replicate the voice examples on your website. I am loading conversational-form using Webpack on my project with import cf from 'conversational-form'. When I try to instantiate a new cf.EventDispatcher() I get the following error __WEBPACK_IMPORTED_MODULE_0_conversational_form___default.a.EventDispatcher is not a constructor. I'm kind of a newbie in using Webpack as a module bundler so I might be missing some steps, but I'd say the error occurs because the EventDispatcher class is not exported and thus not importable with Webpack.
Thanks in advance for your help :)
Jordi

Hi @jorditarrida. I'm not exactly sure if you hit a bug or what the problem might be. Let me have a look in the coming days and report back.

@jorditarrida did you ever figure this out?

@jenssogaard nope, if you could check on it that'd be great :)

@jorditarrida alright, I'll have a look.

I've got the same issue !

@jorditarrida @tonyyb I experience the same issue. I'll try to debug and figure out what is wrong.

I've got the same issue ,any updates on it

@Rchanger no solution yet. I'm still trying to figure it out.

problem is,
export function exports cf.ConversationalForm, not cf

@will3 indeed, that is the issue.
@Rchanger @tonyyb  @jorditarrida so the issue is that if I change the export to just cf then existing declarations breaks:
Before:
var myCF = new cf.startTheConversation({})
After:
var myCF = new cf.ConversationalForm.startTheConversation({})
So for the sake of SemVer I see no other option than to wait for a v1.0 release as it is a breaking change. Any other suggestions on how to go about it will be warmly received.

any update on this?
Running into the same problem when using meteor - using this fix removed the EventDispatcher error for me, however while the form is created, I cannot modify it with any of the functions (options in startTheConversation are not applied, no event listener added...)

I now loaded from the external script and I get the form version and options printed to the console, as well as seeing/being able to use the dialogue. However, none of my options (e.g. robot picture) are applied, and I still get the cf.EventDispatcher is not a constructor error.

@diedea the solution should be clear - fixing the export as described above. However, it must be in a 1.0 beacuse of SemVer. I'll try to get a 1.0 out very soon.
Regarding Meteor - any chance you could put together a very simple example repo for me to look at?

Thanks for your help @jenssogaard - I published a small exemplatory repo here. A real beginner here, so I feel I don't have a lot of insight.

Any workaround to solve this while 1.0 isn't out.

@rafafotes, unfortunately, you will have to wait until 1.0 is released which will hopefully happen sometime next month.

Has this been solved? Im still getting the same error 'cf.EventDispatcher is not a constructor' im using nuxt.js in case it has any relevance.

Hi @edlgg. We have had a bit of a delay on rolling out v1.0.0 where this issue is fixed. I hope it will be released in the next 4 weeks. I'm sorry for the delay.

No worries I! I managed to do what I was trying to do anyways. Thank you for developing this project. It is very useful.

When will v1.0.0 be released? Still getting the error EventDispatcher is not a constructor

@rahulrawat1210 I hope it will be released within 2-3 weeks.


Login feedback for users

Logging in

✅incorrect password
✅incorrect email
✅email/account doesn't exist

Signing up

✅email already exists
✅passwords don't match
✅password is too short

All use cases tested and were correct.
I have only one thing. Validations are core features and will be used for different things. Therefore please move the validation folder into the core folder.

All use cases tested and were correct. I have only one thing. Validations are core features and will be used for different things. Therefore please move the validation folder into the core folder.

LoginPage and LoginData are now renamed to AuthenticationPage and AuthData.


Android app crashes when choose image from Gallery (Android API 28 )

Bug report

Summary

Android app crashes when choose image from Gallery (Android API 28 ). I test Android API 29 and 30 and it works fine.
 ImagePicker.launchImageLibrary(options, (image) => {
        console.log("image ", image)
        const source = { uri: image.uri };
        this.setState({imagePath: source})
        this.setState({image: image})
      });

Environment info

react-native info output:
# paste it here

OS: macOS 10.15.3
CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Memory: 19.32 MB / 32.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 10.15.0 - /usr/local/bin/node
Yarn: 1.12.3 - /usr/local/bin/yarn
npm: 6.4.1 - /usr/local/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
SDKs:
iOS SDK:
Platforms: iOS 13.5, DriverKit 19.0, macOS 10.15, tvOS 13.4, watchOS 6.2
Android SDK:
API Levels: 23, 24, 25, 26, 27, 28, 29
Build Tools: 23.0.1, 25.0.0, 25.0.2, 25.0.3, 26.0.1, 26.0.2, 27.0.1, 27.0.3, 28.0.3, 29.0.2
System Images: android-23 | Google APIs Intel x86 Atom, android-25 | Google APIs Intel x86 Atom, android-25 | Google Play Intel x86 Atom, android-26 | Google APIs Intel x86 Atom, android-28 | Google APIs Intel x86 Atom, android-28 | Google Play Intel x86 Atom, android-29 | Google Play Intel x86 Atom, android-30 | Google Play Intel x86 Atom
IDEs:
Android Studio: 4.0 AI-193.69<IP_ADDRESS>14223
Xcode: 11.5/11E608c - /usr/bin/xcodebuild
npmPackages:
react: ~16.9.0 => 16.9.0
react-native: ~0.61.4 => 0.61.5
npmGlobalPackages:
create-react-native-app: 1.0.0
react-native-cli: 2.0.1
react-native-device-info: 2.0.0
react-native-fabric: 0.5.2
react-native-gesture-handler: 1.4.1
react-native-git-upgrade: 0.2.7
react-native-maps: 0.25.0
react-native-permissions: 2.0.2
react-native-reanimated: 1.2.0
Library version: 2.3.3
Steps to reproduce

ImagePicker.launchImageLibrary(options, (image) => {
console.log("image ", image)
const source = { uri: image.uri };
this.setState({imagePath: source})
this.setState({image: image})
});
Describe what you expected to happen:
App doesn't crash
Reproducible sample code

我也遇到程序奔溃问题，从安卓手机图库选择图片大小超过10兆，程序崩溃。

Mine does not crash but it does not return an image and closes the image picker! Only some users have this problem! Did you find a solution?

@rezasazesh I switched to react-native-image-crop-picker library, it doesn't crash.

Sorry we could not respond in time.
Version 2.x.x is not longer supported, please upgrade to 3.x.x
I hope the issue is solved, if not reply here or create new issue with details I am happy to fix it.


fix(@opentelemetry/api:test): Major upgrade of webpack to 5.85.0 and use xxhash64 hash function

Which problem is this PR solving?
Fixes errors when trying to run unit tests on Mac OS with Node v18 and above.
Fixes #4321
Short description of the changes
Upgrade webpack from 4.46.0 to 5.85.0
Type of change

[X] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Ran all tests locally.  All tests now pass instead of throwing errors about unsupported openssl functions
Checklist:

[X] Followed the style guidelines of this project
[n/a] Unit tests have been added
[ ] Documentation has been updated

Unclear whether this requires a CHANGELOG.md update -- since it a bug fix to unit tests.

Clearly, webpack-5.x is more than I can deal with... I defer to someone else who understands webpack/karma/browser runners better than I do.


Devices without logo Sonoff do not work with SonoffLAN when update to v2.3

I found that all devices (even made by Sonoff) but if do not have logo on eWelink app will be not works anymore with SonoffLAN. Here are Sonoff POW and tx3c...

More photos.

I have 9 devices - not one of which is made by Sonoff (and none of which show a Sonoff logo in eweLink) - they all work absolutely fine in SonoffLAN for me...so perhaps you're doing something wrong ?

I have 9 devices - not one of which is made by Sonoff (and none of which show a Sonoff logo in eweLink) - they all work absolutely fine in SonoffLAN for me...so perhaps you're doing something wrong ?

@ianfretwell All of my devices work very from beginning. This issuse just only happen when I upgrade HACS and SonoffLAN to v2.3.0. I dont know why. I try to reinstall couple times but no luck.

Possibly you are on LAN mode, with reload option disabled or set to once. Try deleting the .json file, or put reload: always, then restart HA. You can revert back after that.

I'm seeing the exact same thing:
I've got a sonoff pow (not R2). It works fine in cloud mode, but doesn't show up at all in local mode. I've tried nuking .sonoff.json and I've got:
sonoff:
username<EMAIL_ADDRESS>password: YOURGUESS
mode: local
reload: always
force-update: [temperature, power]
in my configuration.yaml
The eWelink app says:
Name: Sonoff pow
Current Version: 2.6.1
Manufacturer: Maker
Model: Sonoff Pow
FW Name: PSC-B01-GL

Is problem still actual?

Is problem still actual?


Channel balance

Hmm, can't reproduce the test failure.

Thanks for the PR!
I've finished an initial round of review. As I mentioned in the comments above, if you change ChannelBalance to sum up the available local balance rather than the total capacity, the test should pass. I'll also shortly push that modification I mentioned w.r.t OpenChannel.

I've push a few commits to patch up the temporary hack within the networkHarness's OpenChannel method. It now targets nodes according to their global lightnignID rather than the relative local peerID.


Is thread safety a requirement?

I'm trying to get Refile to work with Ceph but found that the Fog provider doesn't work with it. Also, fog-radosgw only supports provisioning but not usage. I found that aws-s3 works well with Ceph and plan on wrapping that with a Refile provider and then later on submit it as part of this Github org. However, it's not thread safe. For my purposes, that's not a problem since I can just use the process forking feature of Puma to run my Rails app but I wasn't sure if that's OK with the rest of the community.

Which part do you think is not thread safe?

Here's how it's used:
AWS::S3::Base.establish_connection!(
        :server            => 'objects.dreamhost.com',
        :use_ssl           => true,
        :access_key_id     => 'my-access-key',
        :secret_access_key => 'my-secret-key'
)

AWS::S3::Service.buckets.each do |bucket|
        puts "#{bucket.name}\t#{bucket.creation_date}"
end

AWS::S3::Bucket.create('my-new-bucket')

More at: http://docs.ceph.com/docs/hammer/radosgw/s3/ruby/

I'm still not following which part you think is not thread safe?

Is it the AWS::S3 gem, or something in Refile? Which part in particular?

@jnicklas sorry for not being clear. I meant that the aws-s3 gem was not thread safe. I wanted to wrap that with a refile-ceph gem/provider but then it would not be thread safe (because aws-s3 is not thread safe) and I wanted to check if that's alright if i submitted it anyway.

You know we already have an S3 gem, right? https://github.com/refile/refile-s3 The underlying API is the same, so if it works with aws-s3 it should work with aws-sdk as well.

Yes, I have and that works fine when using S3, but not with Ceph. And it's not the fault of refile-s3 but simply because aws-sdk works with the API in a way that Ceph doesn't understand:
2.3.0 :011 >   Aws.config[:ssl_ca_bundle] = 'config/certs.pem'
 => "config/certs.pem"

2.3.0 :014 > c = Aws::S3::Resource.new endpoint: "https://<REDACTED PRIVATE URL>", access_key_id: "BDCVIRVEWSGN9NWVW1QL", secret_access_key: "550sEvjtxBFuPz10L8xWASrSkMZ3gmc6aykzLlCb", region: "us-east-1", http_wire_trace: true
 => #<Aws::S3::Resource>

2.3.0 :015 > c.buckets.each{|b| p b}
opening connection to <REDACTED PRIVATE URL>:443...
opened
starting SSL for <REDACTED PRIVATE URL>:443...
SSL established
<- "GET / HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.2.29 ruby/2.3.0 x86_64-darwin15 resources\r\nX-Amz-Date: 20160323T202845Z\r\nHost: <REDACTED PRIVATE URL>\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nAuthorization: AWS4-HMAC-SHA256 Credential=BDCVIRVEWSGN9NWVW1QL/20160323/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2499cfb578cd0d45385a9e58c1e91e6650321552094e6f5a3fe8c0411c47c1e3\r\nContent-Length: 0\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "Server: nginx/1.6.3\r\n"
-> "Date: Wed, 23 Mar 2016 20:28:42 GMT\r\n"
-> "Content-Type: application/xml\r\n"
-> "Content-Length: 81\r\n"
-> "Connection: keep-alive\r\n"
-> "x-amz-request-id: tx000000000000000011afe-0056f2fc7a-259f5-default\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "\r\n"
reading 81 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error>InvalidArgument</Error>"
read 81 bytes
Conn keep-alive
Aws::S3::Errors::InvalidArgument:
        from /path/to/gems/aws-sdk-core-2.2.29/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
        from /path/to/gems/aws-sdk-core-2.2.29/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
        from /path/to/gems/aws-sdk-core-2.2.29/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
        from /path/to/gems/aws-sdk-core-2.2.29/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
        from /path/to/gems/aws-sdk-core-2.2.29/lib/seahorse/client/plugins/response_target.rb:21:in `call'
        from /path/to/gems/aws-sdk-core-2.2.29/lib/seahorse/client/request.rb:70:in `send_request'
        from /path/to/gems/aws-sdk-core-2.2.29/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
        from /path/to/gems/aws-sdk-resources-2.2.29/lib/aws-sdk-resources/request.rb:24:in `call'
        from /path/to/gems/aws-sdk-resources-2.2.29/lib/aws-sdk-resources/operations.rb:127:in `all_batches'
        from /path/to/gems/aws-sdk-resources-2.2.29/lib/aws-sdk-resources/collection.rb:18:in `each'
        from /path/to/gems/aws-sdk-resources-2.2.29/lib/aws-sdk-resources/collection.rb:18:in `each'
        from (irb):15
        from /path/to/.rvm/rubies/ruby-2.3.0/bin/irb:11:in `'
2.3.0 :016 >

Turns out I may not need to do this. I downgraded aws-sdk to 2.1.x and tried it in irb and looks like it's working. I'll go ahead and see if everything's good when I used that with refile-s3. In the meantime, closing this issue. Thanks!

Downgrading aws-sdk to 2.1.x works! I was able to connect and upload files after I used the following config:
require "refile-s3"

Aws.config[:ssl_ca_bundle] = "path/to/ceph/cert" # Necessary if the ceph server uses a self-signed cert

Refile.configure do |c|
  opts = {
    endpoint: "https://url.of.ceph.instance",
    access_key_id: "accesskeyhere",
    secret_access_key: "secretkeyhere",
    bucket: "mybucket",
    region: "us-east-1",  # required by aws-sdk but doesn't really mater to ceph
    force_path_style: true
  }

  c.cache = Refile::S3.new(prefix: "cache", **opts)
  c.store = Refile::S3.new(prefix: "store", **opts)
end

Thanks @jnicklas and team for this excellent gem!


feat: #3798 Replaced classnames with clsx

Description
Linked issue: close #3798
Problem
See issue #3798
Changes
Replaced classnames with clsx
Screenshots

To reviewers

Contribution checklist

[x] I have followed the contributing guidelines.
[x] I have added sufficient test coverage for my changes.
[x] I have formatted my code with Prettier and checked for linting issues with ESLint for code readability.

Thanks for the review.
I removed classnames from global in rollup.config.mjs.
The clsx did not affect the build result whether it was present in globals or not.

Thanks this looks good.


[SPARK-26986][ML][FOLLOWUP] Add JAXB reference impl to build for Java 9+

What changes were proposed in this pull request?
Remove a few new JAXB dependencies that shouldn't be necessary now.
See https://github.com/apache/spark/pull/23890#issuecomment-468299922
How was this patch tested?
Existing tests

Test build #102866 has started for PR 23923 at commit b69f2d1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8354/
Test PASSed.

Test build #102866 has finished for PR 23923 at commit b69f2d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/102866/
Test PASSed.

Merged to master