Meilisearch bulk index benchmark

Goals
The aim of this PR is to run a benchmark of a bulk index scenario in Meilisearch, using one of three methods: sync, async, and async + multiproc, per @sanders41 in #15.
Three scripts are created for this purpose, included in this PR. The meilisearch official (sync) python client is also included in requirements.txt.
Results
cd scripts

Case 1: 1 run
Sync
python bulk_index_sync.py -b 1

Bulk index took 2.2454 seconds
Async
python bulk_index_async.py -b 1

Bulk index took 1.5590 seconds
Multiprocessor async
python bulk_index_multiproc_async.py -b 1

Bulk index took 3.5335 seconds

Case 2: 10 runs
Sync
python bulk_index_sync.py -b 10

Bulk index took 22.3101 seconds
Async
python bulk_index_async.py -b 10

Bulk index took 16.0275 seconds
Multiprocessor async
python bulk_index_multiproc_async.py -b 10

Bulk index took 22.5052 seconds

Case 3: 100 runs
Sync
python bulk_index_sync.py -b 100

Bulk index took 231.9254 seconds
Async
python bulk_index_async.py -b 100

Bulk index took 165.9993 seconds
Multiprocessor Async
python bulk_index_multiproc_async.py -b 100

Bulk index took 228.9810 seconds

Observations

It's clear that the meilisearch-python-async client's async coroutines outperform the version on all counts, and this difference will become more and more prominent as the dataset gets larger and larger (no question that async is very valuable in this scenario)
Surprisingly, the async + multiprocessing results as per the code in #15 are not as performant as the pure async version. I can understand that firing up the multiprocessor does involve its own overhead, and this slows down the code

In addition, there is added overhead due to the sync for loop -- I ran the benchmark in a sync for loop to mimic a real-world scenario where you have many files to ingest, and typically these are ingested in a blocking fashion (using aiofiles, as discussed, has its own issues with consistency across different OSes and file systems)

I'd have also thought that running the benchmark in a loop over 100 runs would show that case 3 (async + multiprocessing) would show much better performance than both other cases, but this isn't happening. Despite instantiating the multiprocessor at the start, and looping through the data multiple times after, why is it still taking so long to attach to the running event loop? Perhaps this isn't ideal on regular hardware in any case, as combining threads with cores in Python seems to be hoping for too much.

@sanders41 Do you think there's anything fundamentally off about the code in the async + multiprocessing case? I'm a bit surprised at the fact that the performance is comparable to the sync meilisearch Python client -- at the very least, I'd have expected the performance to be in between the sync and async versions, closer to async than sync.
The way these numbers look, I'm tempted to just remove the code you originally suggested in #15 altogether -- the async version is super intuitive, blazing fast, and is extremely readable (thus maintainable in a real setting).
I'd love to hear your thoughts when you've had the chance to take a quick stab at it. Hopefully, the code and benchmark should be reproducible enough on your system the way it's been written. Cheers!

I should add, although these benchmarks were run on an M2 mac, I've also tried running similar code for indexing both Meilisearch and Elasticsearch on regular intel CPU machines, and I think there are a host of issues with the way green threads are available on various systems.. MacOS arm64 architectures seem to have by far the most "free" threads (several thousand) at given time, so combining a multi-process  + async approach doesn't work as well on intel-based Linux systems.
Based on my experience in real settings, I feel like just using async alone is so much easier to develop, test and maintain. Curious to hear how your experience has been, @sanders41!

I tried running a cProfile but got some errors running the script and haven't had a chance to figure out why. My guess is the overhead of starting the processes is negating the performance benefit bringing it in line with the sync client. This is probably even more true with the speed ups from Pydantic 2.
I agree with you that async alone is easier. What I do usually is reach for multiprocessing once performance becomes an issue. Once a dataset gets that big the overhead usually becomes worth it.
I wonder if you would see any difference by using add_documents_in_batches (available in both the sync and async clients) instead of doing the chunking yourself. It looks like roughly the same idea so might not make a difference, but it may be worth a try just to see.

Updates

I tried it with update_documents_in_batches as suggested, and indeed, it's faster! Clearly, it helps to avoid doing pure-Python operations wherever possible.
Added tqdm.asyncio to see async batch progress
Reduced verbosity and redundant functions

Timing: 1 Run
Sync
python bulk_index_sync.py -b 1
Finished updating database index settings
  0%|                                      | 0/1 [00:00<?, ?it/s]Finished run 1 of benchmark
100%|██████████████████████████████| 1/1 [00:01<00:00,  1.56s/it]
Bulk index took 2.2505 seconds

Async
python bulk_index_async.py -b 1
Finished updating database index settings
100%|██████████████████████████████| 1/1 [00:00<00:00,  1.25it/s]
Finished running benchmarks
Bulk index took 1.5184 seconds

Timing: 10 runs
Sync
python bulk_index_sync.py -b 10
Finished updating database index settings
100%|████████████████████████████| 10/10 [00:15<00:00,  1.54s/it]
Bulk index took 16.1129 seconds

Async
python bulk_index_async.py -b 10
Finished updating database index settings
100%|████████████████████████████| 10/10 [00:09<00:00,  1.10it/s]
Finished running benchmarks
Bulk index took 9.7602 seconds

Timing: 100 Runs
Sync
python bulk_index_sync.py -b 100
Finished updating database index settings
100%|██████████████████████████| 100/100 [02:43<00:00,  1.64s/it]
Bulk index took 164.3521 seconds

Async
There were memory issues when ingesting batches of 10k at a time for these many runs. As a result, smaller batches were supplied. The run time when using a smaller batch size of 2k is slightly larger than when using a batch size as 5k, which makes sense.
python bulk_index_async.py -b 100 --chunksize 5000
Finished updating database index settings
100%|███████████████████████████████████| 100/100 [01:32<00:00,  1.08it/s]
Finished running benchmarks
Bulk index took 93.1814 seconds

python bulk_index_async.py -b 100 --chunksize 2000
Finished updating database index settings
100%|████████████████████████| 100/100 [01:40<00:00,  1.01s/it]
Finished running benchmarks
Bulk index took 101.4677 seconds

In general, it's recommended to optimize the batch size to be as large as permissible when using update_documents_in_batches.

I decided against going forward with the multiproc + async approach due to unnecessary complexity. The code is harder to reason about, and not very readable. The async method works great! As does uploading the documents in batches (assuming the batches are of a reasonably small size such that Rust's multi-threaded execution can happen smoothly in the background without running out of memory). I'll have to tune this more carefully in a real world case! Thanks for the tips @sanders41 👊🏽

Do you think the Rust memory issue was caused by how meilisearch-python-async was sending things, or was it purely on the Meilisearch side?
If you think it was related to the client feel free to open an issue with your findings and I'll see if I can come up with a solution.

@sanders41 I actually don't know what the error could be -- you may be right; the error could be with how the async client is sending the batches concurrently to the backend. There's a gigantic stack trace whose error messages make no sense to me when I run the async version with a batch size of 10k. Do you think I should file an issue? I'm not sure if I can create an MRE with any other random data.
python bulk_index_async.py -b 100 --chunksize 10000

Only a part of the stack trace is shown below.
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 43, in _send_request
    response = await http_method(path, json=body, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1885, in put
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.WriteError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
              ^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 948, in update_documents_in_batches
    return await gather(*batches)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 904, in update_documents
    response = await self._http_requests.put(url, documents)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 73, in put
    return await self._send_request(self.http_client.put, path, body, content_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 55, in _send_request
    raise MeilisearchError(str(err))  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
meilisearch_python_async.errors.MeilisearchError: MeilisearchError. Error message: .
Task exception was never retrieved
future: <Task finished name='Task-4' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at /Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/tqdm/asyncio.py:75> exception=MeilisearchError('')>
Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/asyncio/selector_events.py", line 1082, in _write_ready
    n = self._sock.send(self._buffer)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 55] No buffer space available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1212, in receive
    raise self._protocol.exception
anyio.BrokenResourceError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 31, in read
    with map_exceptions(exc_map):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 43, in _send_request
    response = await http_method(path, json=body, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1885, in put
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
              ^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 948, in update_documents_in_batches
    return await gather(*batches)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 904, in update_documents
    response = await self._http_requests.put(url, documents)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 73, in put
    return await self._send_request(self.http_client.put, path, body, content_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 55, in _send_request
    raise MeilisearchError(str(err))  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
meilisearch_python_async.errors.MeilisearchError: MeilisearchError. Error message: .
Task exception was never retrieved
future: <Task finished name='Task-5' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at /Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/tqdm/asyncio.py:75> exception=MeilisearchError('')>
Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/asyncio/selector_events.py", line 1082, in _write_ready
    n = self._sock.send(self._buffer)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 55] No buffer space available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1212, in receive
    raise self._protocol.exception
anyio.BrokenResourceError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 31, in read
    with map_exceptions(exc_map):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 43, in _send_request
    response = await http_method(path, json=body, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1885, in put
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
              ^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 948, in update_documents_in_batches
    return await gather(*batches)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 904, in update_documents
    response = await self._http_requests.put(url, documents)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 73, in put
    return await self._send_request(self.http_client.put, path, body, content_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 55, in _send_request
    raise MeilisearchError(str(err))  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
meilisearch_python_async.errors.MeilisearchError: MeilisearchError. Error message: .
Task exception was never retrieved
future: <Task finished name='Task-6' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at /Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/tqdm/asyncio.py:75> exception=MeilisearchError('')>
Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/asyncio/selector_events.py", line 1082, in _write_ready
    n = self._sock.send(self._buffer)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 55] No buffer space available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1212, in receive
    raise self._protocol.exception
anyio.BrokenResourceError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 31, in read
    with map_exceptions(exc_map):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 43, in _send_request
    response = await http_method(path, json=body, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1885, in put
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
              ^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 948, in update_documents_in_batches
    return await gather(*batches)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/index.py", line 904, in update_documents
    response = await self._http_requests.put(url, documents)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 73, in put
    return await self._send_request(self.http_client.put, path, body, content_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prrao/.pyenv/versions/3.11.2/lib/python3.11/site-packages/meilisearch_python_async/_http_requests.py", line 55, in _send_request
    raise MeilisearchError(str(err))  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
meilisearch_python_async.errors.MeilisearchError: MeilisearchError. Error message: .

My initial hunch was that the larger the batch size, the more data that's concurrently being processed by Rust on the Meilisearch side, and considering it's all multi-threaded + async Rust, there's only so many processes that can be handled with the given amount of memory. It could explain why there are absolutely no issues with smaller batch sizes.

The stack trace does make it look like it's on the Meilisearch side. My first thought was maybe it was sending too many requests at once, but a larger batch size would actually send less requests. You are probably correct in your hunch.
Will you go ahead and open an issue? You can just reference this discussion, no need for an MRE. I may not be able to do anything about the issue itself, but I'm wondering if there is a way I could use exception groups in Python 3.11+ to give a better error message without slowing things down and making it messy trying to figure out which Python version is being used.
Really I'm not sure if exceptioins groups could give a better message or not, I haven't used them yet since in all my libraries I'm supporting 3.8+. I figure it's at least worth looking into though.


Turing support

Why is Ampere or Ada (RTX 3000 and RTX 4000 series) required to support this?
Turing (RTX 2000 series) has INT4 tensor cores.

Hi, Marlin does not use any INT4 tensor cores, 4-bit weights are decompressed on-the-fly and then the actual computation is carried out in FP16. The reason Turning is not support is that Marlin heavily relies on the cp.async instruction which was introduced with compute capability 8.0; this allows explicitly fetching global memory in the background while doing other work at the same time, which is crucial to reach peak performance in an FP16xINT4 setting. While you could probably reuse quite some work of Marlin for writing a Turing kernel, some significant changes will likely be necessary.


Update expensify system emails

Details

Adds more of the system emails that we should be blocking from being added to policies via front-end validation. I updated the list to match what is in expensify-common here and to be in the same order.
I found this while working on https://github.com/Expensify/Expensify/issues/174181.
Tests

Sign into a new dot account
Create a workspace if one doesn't exist already
Go to the workspace settings and click "manage members"
Click the "invite" button
Try adding any of these system emails to a policy
Ensure that you don't see the option to select the email and add it
Try adding any other email address to the policy
Ensure that you are able to add it

What you see when trying to add a non-system email:

What you see when you try to add a system email:

QA Steps

Sign into a new dot account
Create a workspace if one doesn't exist already
Go to the workspace settings and click "manage members"
Click the "invite" button
Try adding any of these system emails to a policy
Ensure that you don't see the option to select the email and add it
Try adding any other email address to the policy
Ensure that you are able to add it

System emails:

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Tested On

[x] Web
[ ] Mobile Web
[ ] Desktop
[ ] iOS
[ ] Android

Screenshots

Web

Mobile Web

Desktop

iOS

Android

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by @MonilBhavsar in version: 1.1.15-18 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by @roryabraham in version: 1.1.16-10 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Fix: codecov badge

Codecov Report

Merging #14 (0e01ae6) into dev (1dd3f5d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev      #14   +/-   ##
=======================================
  Coverage   90.10%   90.10%
=======================================
  Files          12       12
  Lines         192      192
=======================================
  Hits          173      173
  Misses         19       19

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1dd3f5d...0e01ae6. Read the comment docs.


Style Photogrammetry Models with Styling Language

This came out of this forum discussion. If you try this Sandcastle, transparency works! But if you set the photogrammetry variable to true, you'll see it has no effect.
@lilleyse mentioned this was because the styling only applies to features, and this photogrammetry tileset doesn't have any features. A work-around would be to add batch ids where all the values are 0 (see https://github.com/AnalyticalGraphicsInc/3d-tiles/tree/master/specification/TileFormats/Batched3DModel#batch-table), however this is kind of wasteful.
It would be nice if we handled this case where there would be an implicit single feature that would be styled, perhaps?

Congratulations on closing the issue! I found these Cesium forum links in the comments above:
https://groups.google.com/d/msg/cesium-dev/bpBtoVUQgwA/Yw327kv8BAAJ
If this issue affects any of these threads, please post a comment like the following:

The issue at https://github.com/AnalyticalGraphicsInc/cesium/issues/7149 has just been closed and may resolve your issue. Look for the change in the next stable release of Cesium or get it now in the master branch on GitHub https://github.com/AnalyticalGraphicsInc/cesium.

I am a bot who helps you make Cesium awesome! Contributions to my configuration are welcome.
:earth_africa: :earth_americas: :earth_asia:


GAE standard bookshelf - step 2

An adaptation of the flex one.
Tested with devserver + cloudsql, appengine + cloudsql, devserver + datastore, appengine + datastore.

Comments addressed. PTAL

Updated to use the old appengine plugin - PTAL


Errors trying to convert mapnik wiki

Expectation
the markdown, downloaded from https://github.com/mapnik/mapnik/wiki
Result
Unhandled rejection Error: Unknown language: "csv"
Please report this to https://github.com/chjj/marked.
at Object.highlight (/usr/local/lib/node_modules/github-wikito-converter/node_modules/highlight.js/lib/highlight.js:443:13)
at Renderer.mainRenderer.code (/usr/local/lib/node_modules/github-wikito-converter/dist/markdown.js:34:79)
at Parser.tok (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:999:28)
at Parser.parse (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:944:17)
at Function.Parser.parse (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:931:17)
at marked (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:1252:19)
at Markdown.convertMarkdownString (/usr/local/lib/node_modules/github-wikito-converter/dist/markdown.js:109:14)
at Markdown.convertMarkdownFile (/usr/local/lib/node_modules/github-wikito-converter/dist/markdown.js:116:19)
at WikiConverter. (/usr/local/lib/node_modules/github-wikito-converter/dist/wiki-converter.js:87:41)
at Array.forEach (native)
at WikiConverter.computePages (/usr/local/lib/node_modules/github-wikito-converter/dist/wiki-converter.js:83:27)
at WikiConverter.getPages (/usr/local/lib/node_modules/github-wikito-converter/dist/wiki-converter.js:96:14)
at HtmlWriter.write (/usr/local/lib/node_modules/github-wikito-converter/dist/html-writer.js:33:34)
at WikiConverter. (/usr/local/lib/node_modules/github-wikito-converter/dist/wiki-converter.js:60:43)
at tryCatcher (/usr/local/lib/node_modules/github-wikito-converter/node_modules/bluebird/js/main/util.js:26:23)
at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/github-wikito-converter/node_modules/bluebird/js/main/promise.js:510:31)
What was attempted
gwtc ../mapnik.wiki/

Can you be more specific on what you was done here? When I go to the wiki I don't know what was downloaded. Also, not sure what gwtc ../mapnik.wiki/ does as it relates to Marked.
It appears a .md file or string was not passed in - instead a .csv file or string was passed in; Marked doesn't parse CSV files; so, that would not be a problem with Marked.

Expectation
the markdown, downloaded from https://github.com/mapnik/mapnik/wiki

I'm guessing you meant https://github.com/mapnik/mapnik/wiki/CSV-Plugin

Result
Unhandled rejection Error: Unknown language: "csv"
Please report this to https://github.com/chjj/marked.
at Object.highlight (/usr/local/lib/node_modules/github-wikito-converter/node_modules/highlight.js/lib/highlight.js:443:13)
at Renderer.mainRenderer.code (/usr/local/lib/node_modules/github-wikito-converter/dist/markdown.js:34:79)
at Parser.tok (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:999:28)
at Parser.parse (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:944:17)
at Function.Parser.parse (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:931:17)
at marked (/usr/local/lib/node_modules/github-wikito-converter/node_modules/marked/lib/marked.js:1252:19)
...

As you can see from the bold part, the error was raised inside the highlight.js file, being called from the markdown.js file of the github-wikito-converter module. This is not caused by marked.
What happened is that inside the github-wikito-converter module a new code renderer was written, and it uses highlight.js to do syntax formatting of code. Whenever a language unknown to highlight.js is
given, it raises an exception. In this case, csv.

Closing based on further clarification from @Feder1co5oave. @TeAmigo, if you believe this is not a valid assessment, please let us know, and I can reopen it.


"rancher start" should provide better error when service is already running (and not return entire object

Rancher Version:Master
Docker Version:1.10.3, 1.12.0
Rancher CLI version: v0.0.1
OS and where are the hosts located? (cloud, bare metal, etc):Ubuntu, Cloud
Environment Type: (Cattle/Kubernetes/Swarm/Mesos): Cattle
Steps to Reproduce:

Create a rancher-cli set up
Create service/containers on rancher.
On an running container/service run the ./rancher start command. It returns "Action [activate] not available" and the entire object. Entire object should not be displayed.

soumyas-MacBook-Pro:rancher-v0.0.1 soumya$ ./rancher -w start mynewservice
Action [activate] not available on [&{1s61 service map[serviceExposeMaps:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/serviceexposemaps containerStats:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/containerstats self:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61 account:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/account consumedbyservices:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/consumedbyservices consumedservices:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/consumedservices environment:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/environment instances:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/instances] map[removeservicelink:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=removeservicelink upgrade:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=upgrade addservicelink:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=addservicelink deactivate:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=deactivate update:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=update restart:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=restart remove:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=remove setservicelinks:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=setservicelinks]}]
FATA[0001] Action [activate] not available on [&{1s61 service map[serviceExposeMaps:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/serviceexposemaps containerStats:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/containerstats self:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61 account:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/account consumedbyservices:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/consumedbyservices consumedservices:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/consumedservices environment:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/environment instances:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/instances] map[removeservicelink:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=removeservicelink upgrade:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=upgrade addservicelink:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=addservicelink deactivate:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=deactivate update:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=update restart:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=restart remove:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=remove setservicelinks:http://<IP_ADDRESS>:8080/v1/projects/1a5/services/1s61/?action=setservicelinks]}]
soumyas-MacBook-Pro:rancher-v0.0.1 soumya$

Verified with v0.1.0-rc2. On a running service, performing a start will give an error message.
soumyas-MacBook-Pro:rancher-v0.1.0-rc2 soumya$ ./rancher start newservice
error 1s1: start/activate not currently available on service 1s1
1s1


Incompatible with mod 'Neofetch'

Hello, when trying to add 'TooManyEmotes' to my modpack I found I was unable to access the terminal menu:

Entering keyword "emotes":

No emotes menu is returned:

I narrowed down the conflict to be the mod Neofetch.

Mods installed when testing for conflict:

Please could you assess and potentially fix this conflict with Neofetch. As mentioned, I was attempting to add 'TooManyEmotes' to my modpack. And as shown below, Neofetch works perfectly with many other mods which interact with the terminal, 'TooManyEmotes' is the only mod I had an issue with.

Other terminal mods I normally use in my modpack with Neofetch:

P.S. Thank you for your work on such top tier mods!

Let me see if I can reproduce this and see what the issue is. Thanks for letting me know!

I believe I got this issue fixed! I will be pushed in either 1.7.17, or 1.8, whichever update I end up releasing. Keep your eyes out for it!


Org Code Updates and Changes - Future Onboarding

Title of ticket: Org Code Updates - Future Onboarding
Description
Upcoming onboarding will require some changes in existing codes from AXIS, with new codes being made for new Ministries
Dependencies
Are there any dependencies?
DOD

[x] ADD new public body code for Ministry of Housing (AXIS Public Body Code: HSG)
[x] UPDATE Existing public body code for EMB to EMC
[x] UPDATE Ministry name for EMC/EMB to "Ministry of Emergency Management and Climate Readiness"
[ ] REMOVE Linkage from EMC/EMB to PSS in stats for Redash - it is now its own Ministry
[x] UPDATE Existing public body code for LWR to WLR
[x] UPDATE Ministry name for LWR/WLR to "Ministry of Water, Land and Resource Stewardship"
[x] ADD new action office "COR" as Corrections Branch
[ ] Link Corrections Branch to PSS reporting (COR is sub-org of PSS)

Note: adding a task to create a new action office in AXIS (Andrew will handle this part) called COR - which we will then use as a child public body to the Ministry of Public Safety (PSS)

@m-prodan : Could you please let me the divisions required for HSG, WLR , EMC and  COR?

@abin-aot - as discussed in standup, Housing (HSG) falls under Justice / Health Processing Team.

Thanks @KyEggleston . I pushed changes to TEST. Please have a look. The unlinking changes of PSS  EMC and linking on COR- PSS can be done as data get pushed after this test to TEST EDW.

Thanks @abin-aot, and the updated codes/names look good in Test- I'll make a note of the remaining two items (unlinking PSS -EMC and linking COR-PSS) when I set up their reporting. I don't think there's anything else to do here, so I'll close.
cc: @m-prodan

@KyEggleston @m-prodan , I am adding this to Justice 6.1 release ! Thoughts?

@abin-aot - makes sense to me!

Prod Redash queries/dashboards made for MAG, PSS, EMC and HSG.
PSS queries include procorg 'COR'.
@abin-aot - let me know if anything needs to be done to ensure these reports are correct (outside the standard query/dashboard creation process)


Remove todo for swarm-camel with next release of archetypes

See
https://github.com/fabric8io/fabric8-forge/blob/master/fabric8-forge-maven-plugin/src/main/java/io/fabric8/forge/maven/DownloadArchetypesMojo.java#L144
Next release of archetypes has a fix in the quickstart, which we can then upgrade and remove this todo in the plugin

That would be the release after 2.2.191


logging future warnings

whenever we have to change the API, we display a FutureWarning to users so they have a chance to update their code. However, we don't know the proportion of them that already switched.
It'd be better to log these events, so we know when users need more time to switch because we want to ship new features as fast as possible without compromising stability.
We can add a telemetry.deprecation_warning method that uses warnings.warn (with FutureWarning) and captures the event.

Do you mean every time we show the future warnings, meanwhile we will log the event to PostHog?

a function that will show a FutureWarning, but also log the event to posthog

We created the @deprecated.*  decorators when we were mostly working in sklearn-evaluation. Ironically, now that we shifted to JupySQL, these decorators are not very relevant since the user interacts with JupySQL via the %sql magic, instead of importing and calling regular Python functions. So what I'm proposing is something simpler like this:
from warnings import warn

def deprecation_warning(message):
    warn(message, FutureWarning)
    # also log the event to posthog

Later, we can decide if we also add posthog calls to the @deprecated decorators, but for now, we don't have to.


python inference code not different from "normal" whipser?

The python inference code provided seems the same as "normal" whisper. So where is the speedup coming from? Flash attention?

I also had the same thoughts about this, and I think you are right, the Flash Attention mechanism and the use of float16 or bfloat16 with transformers is the magic behind the faster implementation.


Fix Baits IndexOutOfBoundsException

This is a temporary solution until @sarhatabaot can take a good look at the code.
Waiting for confirmation of this working.

No confirmation from the person testing it, but also no message saying it's broken. Should be fine to merge


Update iOS library version constant to 14.0.6

in gatsby-config.js:
IOS_VERSION: '14.0.6',

I think this is only necessary for android.

removed zap 👍


setfit model performance advice

Hello gents,
I was hoping I can get a second opinion about a situation I am facing while using setfit for a multi class classification use case.
The dataset is small with 255 samples across 9 classes. It suffers from both ambiguity (overlapping) and class imbalance. I partially remedied the overlaps by merging classes together. The imbalance is still present but the new dataset version is such as , except for one class, all the other have at least 10 samples per class as shown here below:

I am performing a stratified split training 80%, validation 10%, testing 10%. The train, validation and test splits are preserving the initial class imbalance observed in the dataset post-merge.
My model is an embeddings model with a logistic regression head, constructed using this code:
model_body = SentenceTransformer("BAAI/bge-base-en-v1.5")

# Choose Logistic Regression as the classification head
model_head = LogisticRegression(class_weight="balanced")

# Create a SetFit model, combining the feature extractor and classification head
model = SetFitModel(model_body, model_head)
model.labels = categories
# labels = list(set(dataset['train']['label']))

# Load a SetFit model from Hub
model: SetFitModel = SetFitModel.from_pretrained(
    "BAAI/bge-small-en-v1.5",
)

I am then using the following code (with the latest hyper parameters choice) to train the setfit model:
# Create Training Arguments
args = TrainingArguments(
    # When an argument is a tuple, the first value is for training the embeddings,
    # and the latter is for training the differentiable classification head:
    batch_size=(32, 2),
    num_iterations=10,
    num_epochs=(5, 16),
    body_learning_rate=(1e-5, 1e-5),
    head_learning_rate=2e-2,
    end_to_end=True,
    show_progress_bar=False,
    report_to="none",
    logging_strategy="steps",
    logging_steps=50,
    eval_steps=50,
    output_dir= f"{current_path}/checkpoints",
    logging_dir = f"{current_path}/runs",
    seed=40
)

# Create Trainer
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=dataset['train'],
    eval_dataset=dataset['validation'],
    metric="accuracy",
)

# Train and evaluate
trainer.train()

This is the logging output I get during training:

the evaluation step results in an accuracy score of 0.6153846153846154
I am plotting the embedding and eval embedding loss curves:

when I predict against the test split , I get the following results:

Looking at these results but especially the embedding and eval embedding curves, it's obvious the training routine needs improvement. I was initially suffering from a eval embedding loss curve that was flat, but the problem was in the splitting and that problem is now solved. But the curves shouldn't look like this.
I also suspect the classifier might not be learning enough (it's just a suspicion based on digging at the predictions it made against the test split).
I welcome any helpful suggestions.
PS:

I have already tried suggested hyperparameters, I read on other GitHub issues here like:
batch_size=(16, 2)
num_epochs=(3,16),
body_learning_rate=(2e-5, 1e-5),
head_learning_rate=1e-2,

but I got this:

and worse evaluation and test prediction scores

already tried optuna but do not seem to get anywhere.

I've improved it slightly but still does not look like how it should be.
The new training arguments look like this:
# Create Training Arguments
args = TrainingArguments(
    # When an argument is a tuple, the first value is for training the embeddings,
    # and the latter is for training the differentiable classification head:
    batch_size=(32, 2),
    num_iterations=5,
    num_epochs=(3, 16),
    body_learning_rate=(2.5e-5, 1e-5),
    head_learning_rate=1.5e-2,
    end_to_end=True,
    show_progress_bar=False,
    report_to="none",
    logging_strategy="steps",
    logging_steps=50,
    eval_steps=50,
    output_dir= f"{current_path}/checkpoints",
    logging_dir = f"{current_path}/runs",
    seed=40
)

# Create Trainer
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=dataset['train'],
    eval_dataset=dataset['validation'],
    metric="accuracy",
)

# Train and evaluate
trainer.train()

I am using number samples = 15 (10 gives me the same results).
The run's logs are as follows:

The loss curves now look a bit better

eval accuracy rate is 53.8
and the precision/recall/F1  on the test split gives me :

It can't seem to perform better than this on this dataset. I highly suspect the dataset quality to be the main culprit.
Goes to show the quality of a set can either positively impact a model's performance or completely constrain it.

I solved my own problem.
1- I had left parameters in there which were only applicable to a differentiable head.
2- I was still using pre v1.0.0 migration syntax. I actually ignored the migration guide and relied on reading code snippets off of some of the GitHub issues.
3- Fixed the main culprit: data . performed data curation including annotation corrections, class merging and data augmentation.
Although the dataset is still small sized and variability is to be expected in deployment/production. Here are the revised test set performances on this multi class problem.

could you show your codes what you did to improve the performance?

@chewbm05
I fixed how I was calling setfit and dropped function arguments that are only useful, if I was using a differentiable head (I am using a logicalRegression Head).  This helped: https://huggingface.co/docs/setfit/en/how_to/v1.0.0_migration_guide
But also reading through the documentation properly: https://huggingface.co/docs/setfit/v1.0.3/en/index
My specific code won't probably help you because  your data is different then my data so the changes you need to implement and hyperparameter values you use would probably need to be different.
However here are some of the changes that somewhat helped out in "my case":
The main change is data transformation:
I slightly augmented the underrepresented classes in the dataset using an LLM
so the data imbalance went down from severe to low-to-mid level.
The number of samples per class was still nowhere close to where I would have liked it to be  (100 samples per class). I was still at 40-50 per class.
I merged classes that were extremely semantically close (almost similar) that the model was getting confused by.
The hyper-parameter changes below would probably not have been necessary, If the dataset wasn't still suffering from some quality issues.
a) I passed weight="balanced" to the logistic regression head, and C=0.1
 model_head = LogisticRegression(class_weight='balanced', C=0.1)

C is the opposite of regularization strength, you can read more here: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
I specified C manually based on a conversation. You can probably use grid search to find C. I  I haven't yet quantified the impact from the C hyper parameter alone (vs the combination of setting C and Class_weight), and not sure it's even used here. But when I made that combo change, I noticed a positive impact, so I left it as is.
b) used a slightly unusual body learning rate because I suspected the model wasn't learning the sample data enough:
body_learning_rate=8e-6

I used an epoch of 1 and a batch size of 8.
c) Tried some non-default sampling strategies:
The default sampling strategy in set-fit is oversampling.
I tried:

a 'unique'  sampling_strategy while setting a  warmup_proportion (I used 0.0.3) as training arguments.
the now deprecated num_iterations by setting the value to 35.
Both approaches seemed to help in my case.

You can read more on sampling strategies here: https://huggingface.co/docs/setfit/v1.0.3/en/conceptual_guides/sampling_strategies
Setting some of these hyper parameters can be more easily automated using optuna, once you figure out what ranges to use for a smaller sized dataset that has quality issues. I wouldn't say my approach was highly scientific but it worked.
And again, fix the dataset problems and the performance should improve considerably. Models can't make data problems disappear. Garbage in, Garbage out.


[core] Allow creation dates with a length of 0

Some torrents contain a creation date key with a zero-length string as its value. Currently, MonoTorrent will throw an exception when trying to parse these.
Let me know if there's another way you'd want to handle this (I'm guessing invalid) case. I'm not sure if a tool creates these or what, but I've seen it enough that I think something should be done here. I know torrent clients don't choke on this like MonoTorrent does, so maybe we can handle this more generically (e.g. skip parsing a key if its value's length is 0) rather than special casing the creation date.

I'm +1 on this a fix for the specific issue of creation dates as it cannot cause side-effects.
Whether or not this can be added as a generic fix probably depends on whether or not anything in the Torrent parsing code checks for the existence of specific keys without also ensuring the value is valid.
If there are no occurrences of this, I'd be +1 on making this a generic fix, where all empty keys are quietly skipped. I haven't reviewed this yet.
I did do a quick review to see if this could cause a user visible change, but didn't see anywhere that would occur :)

Would you be up for augmenting this to skip all keys whose values are bencodedstrings and those strings are empty?
(I think that's the only case that matters?)

Sure, closing in favor of #584 due to more fitting branch name.


Cannot change the body background via theme configuration

A question / suggestion or a potential bug.

I'm trying to configure a theme and when I try to change the background color the change does not get applied.

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Expected Behavior
With this theme configuration, I'm expecting the body background color to be #ff0000 but it's actually #cccccc.
export default {
  palette: {
    type: 'light',
    primary: {
      main: '#00AAE1',
      dark: '#143C8C',
      contrastText: '#fff',
    },
    secondary: {
      main: '#64B42D',
      dark: '#008732',
      contrastText: '#fff',
    },
    error: {
      main: '#BD0043',
      contrastText: '#fff',
    },
    divider: '#D7D6D5',
    background: {
      paper: '#fff',
      default: "#ff0000"
    },
  },

  typography: {
    // Use the system font over Roboto.
    fontFamily: 'Avenir Next, Roboto,"Helvetica Neue",Arial,sans-serif',
    htmlFontSize: 16,
  },
};

const theme = createMuiTheme(uiThemeConfig);

Current Behavior
The body's background color remains unchanged.
Steps to Reproduce (for bugs)
I found that having the <CssBaseline /> makes the background color possible.
https://codesandbox.io/s/kkpw0y1vm5
But without it doesn't work.
https://codesandbox.io/s/q4kvw1kj4q
I must have missed this in the documentation to be a mandatory part of the setup. If so, then please point me to it :)
Your Environment

Tech
Version

Material-UI
1.0.0-beta.38

React
16.2.0

I found that having the  makes the background color possible.
https://codesandbox.io/s/kkpw0y1vm5

But without it doesn't work.
https://codesandbox.io/s/q4kvw1kj4q

I must have missed this in the documentation to be a mandatory part of the setup. If so, then please point me to it :)

I'm trying to configure a theme and when I try to change the background color the change does not get applied.

@mnemanja It's CssBaseline responsibility to change the background color of the page. This component applies side effects. What you are describing sounds like the expected output. I'm not sure how we could improve the situation. Any idea?

@oliviertassinari thanks for the awesomely quick response! 👍
Just adding it to the documentation as an expectation would be fine I'd say :)

For reference, the CssBaseline documentation page.

I would suggest a link with a description that the CssBaseline is needed in certain situation (for example changing a body's background color) here https://material-ui-next.com/customization/themes/

I have made some changes. I hope it's clearer. The theme page is already very dense. I want to offload as much information as possible to keep the important part.

Well done sir :)

Is this stuff still applicable? I tried following the CSSBaseline page, and my dev tools say:
Uncaught Error: Cannot find module '@material-ui/core/CssBaseline

I probably don't understand something about the syntax. Why is there an @ symbol there when most of the other import statements I've seen so far don't have one?

@HebaruSan Upgrade to v1.

I simply ran "npm install material-ui". Why wouldn't I have the latest version after that?

@HebaruSan The main package was moved to @material-ui/core.

Well that isn't going to confuse anyone!!
Seriously though, is that going to be fixed? Is there at least an issue tracking it? Getting an old version when we install the name of your project (and only finding out that happened later when asking questions on GitHub) is pretty much the opposite of the principle of least surprise.

Meanwhile pages like this still have the old version:
https://v0.material-ui.com/#/customization/themes
Where did the theme stuff move to?

Meanwhile pages like this still have the old version:

@HebaruSan I don't understand.

import React from 'react';
import darkBaseTheme from 'material-ui/styles/baseThemes/darkBaseTheme';
import MuiThemeProvider from 'material-ui/styles/MuiThemeProvider';
import getMuiTheme from 'material-ui/styles/getMuiTheme';
import AppBar from 'material-ui/AppBar';

I ran into this issue and eventually realized that I was wrapping <CssBaseline> around the <MuiThemeProvider> and not the other way around. This was causing <CssBaseline> to not be aware of the theme configuration.
👎
        <CssBaseline>
            <MuiThemeProvider theme={muiTheme}>
              <App />
            </MuiThemeProvider>
          </CssBaseline>

👍
            <CssBaseline>
              <App />
            </CssBaseline>
          </MuiThemeProvider>```

When i use ThemeProvider Higher CssBaseline and Override theme.overrides.body
children:
null
theme:
{…}
breakpoints:
{…}
direction:
"ltr"
mixins:
{…}
overrides:
{…}
MuiCssBaseline:
{…}
body:
{…}
background:
"#fff!important"

@chayakornwc
palette: {
  background: {
    default: 'newBackgroundColour'
  }
}

The theme structure can be found here: https://material-ui.com/customization/default-theme/

@chayakornwc
palette: {
  background: {
    default: '#fff'
  }
}

The theme structure can be found here: https://material-ui.com/customization/default-theme/

I did solving about that solution. that's not work :(

@chayakornwc CSSBaseline needs to be inside the ThemeProvider. If you still can't get it to work if you create a StackOverflow question with a reproducible example, I'd be happy to have a look.

@chayakornwc CSSBaseline needs to be inside the ThemeProvider. If you still can't get it to work if you create a StackOverflow question with a reproducible example, I'd be happy to have a look.

Thanks!


Added Chat Box Button

Issue:

All the changes are done ............... please kindly check this

Great thank you!


Scan radius decreased from 100m -> 70m

Has pokeminer been amended to account for the changes yet?
Also are there any plans on changing the algorithm to a hexbased search algorithm?

This is addressed in #72 I had the same thought process seeing that last night, and there is another issue already about the hexagonal scan #57

Long story short: amend it yourself by using LAT_GAIN and LON_GAIN.
Using hexagons seems like a more challenging task than just creating a grid, and I'll definitely look into it, since who doesn't like challenges?

I went with default values suggested elsewhere, and unless something else submitted changed the workers scanning, I went from 200 points per cycle to 1k... we could use a way to determine proper values.

That would probably be accurate if you don't want to miss anything, otherwise the javascript posted in #72 would be the closest thing to help you figure out what you want.

That's just an insane increase, I'd have to create quadruple the amount of accounts I was using before...

I got +120 workers for 7 square km area. Is that overkill? Not sure if i can expand the area without loosing  real time info of pokemons.

I'd say it's an overkill. I'm using similar number for 130 square km.


Support for file input field / Dropzone.js?

Hi, what would be the chance, or is it already possible, to integrate a file upload input field type, or something like Dropzone.js (which angular-formly already supports)?
Thanks!

Hi,
Nice idea!

DropzoneJS would be nice to integrate in easyFormGenerator.
As you mentioned, it plays well in angular formly so it should play well too in easyFormGenerator.

The only thing I have to think about is :

what and how I'll give access to configuration

The main configuration parameter is clearly the url.
The good news is :
EasyFormGenerator new version — v1.1.0 — is coming (this weekend or if I don't have enough time at least following week).
This version v1.1.0 was long time consuming since it was a complete ES6 migration.
So
I will have pleasure working on integrating DropZoneJS in the following version v1.1.1.

v1.1.1 won't exist but v1.2.0 instead.
it will be a better support for ui-bootrap up to date versions (I mean when directive name changed to uib-*).
Then I take care (v1.2.1) of DropZoneJS it is very required.

I think Dropzone will not be able to integrate (don't worry I have a fallback solution).

Explanations:
Existing example with Dropzone in angular formly is more a hack than a regular formly template.
there is no:
formlyConfig.setType({})

// or if in config:
formlyConfigProvider.setType({})

But it remain in html (here is what I call hack since it breaks the principle of angular formly to generate from by a model not html)
      <form ng-submit="vm.onSubmit()" name="vm.form" novalidate>
        <formly-form model="vm.model" fields="vm.fields" options="vm.options" form="vm.form">
          <!-- drop zone is here it is not in vm.fields model...  -->
           <button dropzone="dropzoneConfig">
                    Drag and drop files here or click to upload
           </button>
          <button type="submit" class="btn btn-primary submit-button" ng-disabled="vm.form.$invalid">Submit</button>
          <button type="button" class="btn btn-default" ng-click="vm.options.resetModel()">Reset</button>
        </formly-form>
      </form>

fallback solution since I would like to have an upload control in easy Form Generator:
Good news is I found an example of file upload in angular formly
other way to implement file upload in angular formly
otherwise:
if someone reading has an implement of dropzone as a real angular formly type I will be interested to have an eye on it.

v1.2.0 is now released, I will have more time to focus on upload file control.
let's go on!

I don't find any working example that could fit in easyFormGenerator.
Until something works, I will go on refactoring easyFormGenerator using angular 1.5 components (and all new angular2 like stuffs)

to why not being able to be more dynamic in adding new control
to be ready to migrate to angular2 some day (hoping angular formly will migrate it would be great).

@MacKentoch Yeh that sounds more beneficial to extend it so dynamic controls can be added.


Window controls display incorrectly on hidden title bar in Windows 11

On Windows 11 (21H2 build 22000.168) running Notion-repackaged 2.0.16 (rev 2) results in the window control buttons becoming oversized and bleeding into the app itself. This only seems to happen for me when the title bar is set to hidden.
Screenshot

dup of https://github.com/notion-enhancer/notion-enhancer/issues/291


False positive for image used for logo or favicon

mkdocs.yml uses img/Cat_Icon.png as logo and favicon.
theme:
    logo: img/Cat_Icon.png
    favicon: img/Cat_Icon.png

However, img/Cat_Icon.png is flagged as unused:
INFO     -  The following files exist in the docs directory, but may be unused:
              - img/Cat_Icon.png

Below the minimal example updated with logo and favicon
Minimal_example_with_favicon.zip

This is working as intended. The plugin only searches in the page content, not in the rendered template (footer, header, navigation), for better performance.
The htmlproofer also doesn't do this by default, also because of performance reasons:

This defaults to off because it is much slower and often redundant to repeat for every single page.

However, they have a validate_rendered_template option to explicitly turn this on. I might add a similar option in the future.

Thank you for the explanations. I am closing the ticket.

I've added a note to the Readme for clarification.


[pt-br] Add docs/tasks/access-application-cluster/configure-dns-cluster.md

Changes
Brazilian portuguese translation of the Configure DNS for a Cluster
Related Docs.
This document, configure DNS for a cluster, does a reference to another document that I am translating too but it is a WIP at the moment.
https://github.com/kubernetes/website/pull/38481
Related Issue
https://github.com/kubernetes/website/issues/13939

Hi @PauloGoncalvesLima Thank you for your contribution! 🎉
If you need help, please don't hesitate to ask us in Portuguese slack channel #kubernetes-doc-pt

Hi, @MrErlison Thank you for reviewing my PR.
I have made the changes that were proposed. 😄
And I will certainly ask for help on the slack channel if I have any doubts about what would be the translation of some words.

/cc @stormqueen1990

/retitle [pt-br] Add docs/tasks/access-application-cluster/configure-dns-cluster.md

/cc @MrErlison

/LGTM

/approve

Thank you @stormqueen1990 and @MrErlison  for reviewing my PR. ❤


Downloading age restricted videos no longer works

from pytube import YouTube

url = 'https://www.youtube.com/watch?v=irauhITDrsE'
yt = YouTube(url)
yt.streams.first().download()

  File "test.py", line 14, in <module>
    yt = YouTube(url)
  File "/home/guy/Documents/pytube/pytube/__main__.py", line 88, in __init__
    self.prefetch_init()
  File "/home/guy/Documents/pytube/pytube/__main__.py", line 96, in prefetch_init
    self.prefetch()
  File "/home/guy/Documents/pytube/pytube/__main__.py", line 162, in prefetch
    raise VideoUnavailable('This video is unavailable.')
pytube.exceptions.VideoUnavailable: This video is unavailable.

submitted PR https://github.com/nficano/pytube/pull/372


Configure the extension

First of all, thank you very much for such useful extension!
Is it possible to configure it? If so, how? Could you please document this?
How to configure the extension to:

use node_modules instead of jspm.io CDN?
use esm.sh, unpkg.com instead of jspm.io?
not add es-module-shims?
not add <!-- Generated by @jspm/generator VSCode Extension - https://github.com/jspm/jspm-vscode --> comment?

What about https://code.visualstudio.com/docs/getstarted/settings#_extension-settings?

As far as I understand, there is no any config file for JSPM CLI like .jspmrc. Right?

These are excellent points, thanks for putting some thought to this.
In general, JSPM detects configurations from the existing import map file, including handling of whitespace and the provider and environment.
We don't currently define a .jspmrc, but this is something we could introduce for the CLI and extension by supporting it in the generator project.
I tend to agree that might well be the best approach here. Would you be interested in working on this as a feature at https://github.com/jspm/generator?

I wouldn't mind working on adding support for .jspmrc. But that would require agreeing on the format of that file.

In general, JSPM detects configurations from the existing import map file, including handling of whitespace and the provider and environment.

I think in 50% of the use cases of this extension, the import map isn't there yet. I mean users who are using the VSCode extension to create import map for the first time. :)
I've started working on a pull request for VSCode extension that adds additional prompts to existing ones (which provider to use, whether to use es-module-shims, etc.).
But I'm blocked by #17. @guybedford Could you please take a look?


Dodgy signals

I'm seeing a few dodgy looking signals, inferred where there is no sign of a peak. E.g.:

Here's another one...

I'll have a look into this one...

There seems to be quite a lot of these @alimanfoo ....
Any idea why the signal detection algorithm would have got worse? ....

60-85 Mb

Uncalled signal on the left looks more real than called one on the right:

I've seen a couple like this...?


Android BC Wallet Nightly Test Job Cancellation Remediation

History: Since the Aries Mobile Test Harness has moved from Hyperledger to OWF as the Mobile Test Harness, The nightly regression test runs for BC Wallet on the Android platform has been getting cancelled by github.  Logs are left empty after the cancellation making it very difficult to diagnose. On average jobs would be cancelled after 40 minutes of execution time.
The differences between Hyperledger and OWF seems to be in the size of the runners. Hyperledger uses large runners and OWF uses standard. So somewhere along the way github is not liking the use of CPU or Memory (disk size is the same between large and standard runners).
A local runner was configured for the Android job out of an Apple M3 Macbook with no cancellation issues over multiple runs. Someones personal work laptop as a nightly runner isn’t deemed a longterm solution.
The differences between the iOS job, which does not get cancelled in the same environment, and the Android job, is the fact that with the BC Wallet Android tests, an NGROK container is stood up and runs tunnels for the AATH test agent Issuer and verifier.  The iOS job utilizes the Sauce Labs Tunnel instead of NGROK. This is becasue the Android app, when using the Sauce Labs tunnel has untranslated and unreachable invitation URLs in the QR Code (it references host.docker.internal). There is a way to fix this by setting a non-production android:usesCleartextTraffic="true" setting in the Android app, however that was deemed unreasonable as we want to test the app in its releasable state.
After some substantial investigation including trying to use a newer Sauce Labs Tunnel, and using a lighter weight ngrok container, all failed to rectify the situation.
To get the Android tests running again, one of, or a combination of the following, must be chosen as the path forward. This is ordered from easiest and fastest to get working to hardest and longest.

We set the android:usesCleartextTraffic="true" setting on the Android app so the Sauce Tunnel can be used instead of ngrok.
Have a dedicated local runner in the bcgov environment ready for nightly regression execution of the Android tests. Maybe @WadeBarnes  can comment on the feasibility of this option and if it is the second easiest solution in this stack?
We use the BC Wallet Repo in bcgov to run the tests instead of in the OWF Mobile Test Harness repo
We segment the Android tests into separate jobs, and then attempt to take all the test results from the different jobs and show it as one set of results in Allure Reports.
We move the BC Wallet tests from the Mobile Test Harness repo and into the BC Wallet repo and pull in the test harness when the job runs. This is similar to 3 above except that in 3 everything including the BC Wallet tests are pulled from the test harness repo. (Keep in mind that the BC Wallet tests were kept in the  test harness repo as examples for the community)
We Move the BC Wallet tests off of the AATH test agents and onto Traction services. These will not require tunnels to communicate with the app in Sauce Labs as Traction is publicly accessible. This doesn’t totally get us off the hook. Since the Mobile Test Harness is an Open Source solution being used by others in the community, it may be the case that they would require using AATH agents and would still need to be supported in the Mobile Test Harness itself.
Any other ideas?

The biggest concern with self hosting runners is security, not to say it can't be done, it would just take some time to design a secure environemnt based on the recommendations.


MNT: Ignore deprecation warnings

Ignore deprecation warnings from importlib and jsii that we can't control. I put in a PR to jsii to fix it upstream https://github.com/aws/jsii/pull/3986, but until that is fixed, just ignore them for now.

The upstream PR has already been merged, so that was much faster than I expected. I don't know when they'll push a new release, but we could pin to that as well if it is soon.


Display of Harvested Blocks

17 or more Harvested Blocks cannot be displayed. It need to add a scroller or something.
It might be useful to have a link from the block number to the explorer.

Blocks presentation fixed.
@yilmazbahadir can we add link to explorer when clicking on block height?

This shouldn't be valid anymore (with the latest release)


don't be so greedy

Thanks for submitting an pull request to pcf-pipelines.
To speed up the process of reviewing your pull request please provide us with:

Recently there has been some format changes in the pivnet yaml file for GCP where the opsman tarball is located under the US directory for all regions example:

OpsMan 2.4
$> cat OpsManager2.4-build.192onGCP.yml
us: ops-manager-us/pcf-gcp-2.4-build.192.tar.gz
eu: ops-manager-us/pcf-gcp-2.4-build.192.tar.gz
asia: ops-manager-us/pcf-gcp-2.4-build.192.tar.gz
This results in grep matching more then one occurrence of the pattern.  This leads to the following incorrect new yml file from being created.
disk_image_url: ops-manager-us/pcf-gcp-2.3-build.305.tar.gz
ops-manager-us/pcf-gcp-2.3-build.305.tar.gz
ops-manager-us/pcf-gcp-2.3-build.305.tar.gz
Notice the 2 additional values being inserted in the key value pair.
The proposed pull request would limit the regex matching to one occurrence, when grep is run.

@pivotal-ben-chacko Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@pivotal-ben-chacko Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@pivotal-ben-chacko Thank you for signing the Contributor License Agreement!

merged via command line.
This is being merged in because this is a breaking change for current pipelines. This will be included in the next (last) release of pcf-pipelines. In the future, please use platform-automation for all your automation needs. Pivotal Documentation


[Snyk] Fix for 1 vulnerabilities

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to fix one or more vulnerable packages in the `rubygems` dependencies of this project.
Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

Gemfile

⚠ Warning
Failed to update the Gemfile.lock, please update manually before merging.

Vulnerabilities that will be fixed
With an upgrade:

Severity
Priority Score (*)
Issue
Breaking Change
Exploit Maturity

661/1000  Why? Recently disclosed, Has a fix available, CVSS 7.5
NULL Pointer Dereference SNYK-RUBY-NOKOGIRI-3052880
No
No Known Exploit

(*) Note that the real score may have changed since the PR was raised.
Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings
📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

@colinbruce it was this repo that had problems with doorkeeper bumps last time i believe. Do we need to revisit?

@colinbruce it was this repo that had problems with doorkeeper bumps last time i believe. Do we need to revisit?
Can't see why it has even bumped door keeper when the dependency needed addressing is nokogiri

I hate these Snyk updates... I rarely understand or see the point of them!

 I rarely understand or see the point of them!

I'm not sure they're useful.
They don't update the lock file either 😤.
Personally I'd rather use bundler-audit and it force us to update things ourselves through the power of failed CI builds.
I'm all for using tools consistently across teams, but we're probably paying for this Snyk service and it can't do what it says on the tin (when it comes to rubygems anyway) 🤯

I think this will result in a battle between Snyk and Dependabot - whoever wins we lose!
Without the lock below 5.6 Dependabot will want to update the gem - see #796 🤦♂


[C#, XP] Application crashes during RakPeerInterface::connect

I ported the project via SWIG to C#. The provided sample runs fine on machines like win7, win10 but crashes on XP as soon as i try to connect the client to a running server (for simplicity both running on the same system!). The sample has not been modified.
It's working if the client cannot find the sever (e.g. due to not firing up the server...). I tried to catch all possible exception which can be catched in C#, but no luck here. Just the typical XP prompt (xyz has encountered a problem and needs to close. Send Error Report, Don't Send)
The port was done via visual studio 2015 with the toolset vs140_xp (otherwise it won't even start the application due to missing XP support).

I have no experience with using RakNet on Windows XP, but the initial stand out difference between those operating systems is that I'd assume your Win7 and Win10 are both 64bit and your Win XP is likely 32bit (?) If so, I'd be suspicious that you're running into some type size differences.


Add flow type declarations

Closes #32

@marudy here's the PR that I had mentioned! Let me know if you want any changes :)

@sanjaypojo thanks for the PR. Can you maybe add a quick note in README as well ? Also did you test your additions locally to see if they work as expected? If yes we can release it soon.
Thanks!

@marudy I've tested the additions locally and it works fine! With regards to usage, flow types should just automatically work. Where in the readme should we add it? Didn't find anything for TS, but they should probably go together

@sanjaypojo merged! I'd put it some description in README under The Package -> Updates. If we add that as well, it will be awesome 👍

@sanjaypojo released new version 1.4.0 that contains this . Thanks for the help 👍

Awesome, thanks for the update @marudy :)


How to use classical "if" control

I was trying the teleportation circuit, which requires control using classical registers. The code is the following:
`include "qelib1.inc";
qreg q[5];
creg c0[1];
creg c1[1];
creg c2[1];
h q[0];
t q[0];
h q[0];
cx q[0],q[1];
h q[0];
measure q[0] -> c0[0];
measure q[1] -> c1[0];
if (c0==1)
x q[2];
if (c1==1)
z q[2];
measure q[2] -> c2[0];
`
And I got error:

Error in line 14: Operations are disabled

Which refers to the "if" operation. How should I use a classical register to control a quantum gate?

@taojiastanford what backend are you using. If is not supported in real devices yet.

Dear Dr. Gambetta,
Thank you for your reply!
I was using the composer linked to ibmqx2, but this error also disables  "simulate".
Sincerely,
Tao Jia

Tao Jia
PhD Candidate
Stanford University
Tel:<PHONE_NUMBER>

On Aug 24, 2017, at 6:12 PM, Jay Gambetta<EMAIL_ADDRESS>wrote:
@taojiastanford what backend are you using. If is not supported in real devices yet.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@jaygambetta
Is the following control-if supported in the real device? I have been using it with the simulator:
queryStateCircuit.x(qr[N]).c_if(cr, 0)
queryStateCircuit.h(qr[N]).c_if(cr, 0)

@rraymondhp all the qasm simulators should support this.

I also found that I could not create multiple classical registers. The following code:

include "qelib1.inc";
qreg q[5];
creg c0[1];
creg c1[1];
h q[0];
t q[0];
h q[0];
cx q[0],q[1];
h q[0];
measure q[0] -> c0[0];
measure q[1] -> c1[0];

This does not have compile error, but when I tried to simulate it I found "Generic error".
Am I supposed to only use one classical register?

Hello @taojiastanford
Sorry for the late response.
I confirmed that the code can be compiled and can run with the simulators. I believe you got the error message because you executed the qasm code (with QASM editor instead of the QISKit tutorial) using the topology of the real device on IBM Q experience, right? Is it correct?
If so, the error (bug) is due to the limitation of commands used to operate the real device. You are right that at the moment one is supposed to use one type of classical register.  The error message should report this clearer instead of "Generic Error" message.
Thank you very much for reporting this.

Hi Tao,
Thanks for your contribution. We really appreciate it.  Please just call me Raymond.
Meanwhile, about your questions, the real devices ibmqx2 and ibmqx3 has connectivity limitations: not all circuits that can be run on real devices, even though they can be run on simulators.   For example, if you use real devices, a measurement will end the execution of your program. In this case, the teleportation code, as you gave me, cannot run the classically controlled gates.
For the number of shots, at the moment, there is an upper limit for one submission, but I think you can trivially combine two submissions with N1 shots and N2 shots, and essentially get N1 + N2 shots of experiments, and so on.
About the error for the experiment, I believe this is a bug report to be handled by IBM Q experience (the web interface). I will forward your suggestion to the forum here:
https://quantumexperience.ng.bluemix.net/qx/community
There is a tutorial on quantum teleportation here, that allows you to write the program with python interface:
https://nbviewer.jupyter.org/github/QISKit/qiskit-tutorial/blob/master/2_quantum_information/quantum_teleportation.ipynb

@taojiastanford and @rraymondhp I think this issue is solved. I will close the thread but please open it again if there is more.
For the bug can you use the bug tracker on the IBM Q experience website.
Thanks

@rraymondhp the link you shared does not work now. Can you send another one, or share the code?

Hi @KishoreSShenoy
Please try these links Teleportation / nbviewer-Teleportation


unsafe-eval in dev

I just added crossbuilder in a nested folder in another of my repositories (so it uses parent babelrc, eslintrc,etc.), so this problem could be related to my specific config, but wanted to share anyway.
When loading the extension and inspecting the background console, I get this error:
redux-devtools-extension.js:2 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'self' http://localhost:3000 'unsafe-eval'". Either the 'unsafe-inline' keyword, a hash ('sha256-CPbkA6xlx6EbHFrsB6YD4F99pTLewMUVM3xYm1Pt5hM='), or a nonce ('nonce-...') is required to enable inline execution.
e.injectOptions @ redux-devtools-extension.js:2(anonymous function) @ redux-devtools-extension.js:2(anonymous function) @ redux-devtools-extension.js:2target.(anonymous function) @ extensions::SafeBuiltins:19safeCallbackApply @ extensions::sendRequest:21handleResponse @ extensions::sendRequest:74
configureStore.js:33 GET STATE (configure, callback) {
  (0, _getStoredState2.default)(configure, function (store) {
    (0, _crossmessaging.onConnect)(function () {
      return { name: 'init', state: { counter: store.get… true

As the warning advises, I added the hash to manifest.dev.json, and it actually works fine (adding the unsafe-inline threw an error).
Is this a suitable solution? I thought the hash was a hash of the actual inline script, but when I make small modifications to the Counter component, the extension still injects it fine, so I assume I'm missing something.

error seemed similar to this, but looking @ the commit @jhen0409  references, it's not clear to me what it would affect in crossbuilder

upon closer inspection, it looks like the hash works for just injecting into https, but the description in the issue I linked above still applies--that is, it looks like the context is falling under the page's CSP (so CORs doesn't work, for example).
Is there a way to fix this? up to & including disabling hot-reload on https?

Seems like in the last versions of Chrome were added some restrictions for inlining scripts in the extension, so we cannot inject Redux DevTools Extension anymore. Using the hash is not a reliable solution because you'll have to change it everytime there's something new in Redux DevTools Extension.  I'll figure out how better to handle it, and will update the extension.
For now you can just comment these two lines.

A possible solution would be to eval the Redux DevTools code like for the injected:
const httpRequest = new XMLHttpRequest();
httpRequest.open('GET', 'chrome-extension://lmhkpmbekcpmknklioeibfkpmmfibljd/js/redux-devtools-extension.js');
httpRequest.send();
httpRequest.onreadystatechange = function () {
  if (httpRequest.readyState === XMLHttpRequest.DONE && httpRequest.status === 200) {
    eval(httpRequest.responseText);
  }
};

It would work in our case because the store is created asynchronously. But for most of other cases it will be executed after the store is created, thus DevTools enhancer will not be applied.

Thinking over, the best option would be to use remote-redux-devtools for extensions. So, we'll start remotedev-server together with webpack's developer server. It would be easier and more reliable than fetching and injecting Redux DevTools extension (we did it before because remote-redux-devtools wasn't released yet). In 3.0 there will be possible to inspect local and remote instances together in Chrome devpanel.
@jhen0409 what do you think? If you agree, I'll add a deprecation warning in redux-devtools-extension.js. I guess this is breaking react-chrome-extension-boilerplate as well.

what do you think? If you agree, I'll add a deprecation warning in redux-devtools-extension.js. I guess this is breaking now react-chrome-extension-boilerplate as well.

Yeah it's breaking the boilerplate now. I agreed for use remote-redux-devtools instead. 👍


Cursor blinking after reset in :term?

Steps to reproduce

Open vim in xterm.
:term.
type reset (or just printf '\ec').
See cursor blinking.

Expected behaviour
That it should not blink. (in a normal xterm it doesn't normally blink either.)
Version of Vim
9.0.1337
Environment
xterm.
I'm not sure why there's a difference between running inside and outside :term.
As a workaround I just XTerm.VT100.cursorBlink: never as in https://superuser.com/a/1639614/577463 .
Logs and stack traces
No response

Steps to reproduce

Open vim in xterm.
:term.
type reset (or just \ec).
See cursor blinking.

Expected behaviour
That it should not blink. (in a normal xterm it doesn't normally blink
either.)

I cannot reproduce it with "reset".  Using "printf '\ec'" does make the
cursor blink.  What did you expect that to do?

Version of Vim
9.0.1337
Environment
xterm.

What version?  I'm using 377.

I'm not sure why there's a difference between running inside and
outside :term.
As a workaround I just XTerm.VT100.cursorBlink: never as in https://superuser.com/a/1639614/577463 .

Perhaps there is something else in your setup that makes the blinking
happen?  I'm not sure where to check, xterm uses various X resources.
--
LAUNCELOT: I am, sir. I am a Knight of King Arthur.
FATHER:    'Mm ... very nice castle, Camelot ... very good pig country....
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///


App context/vhosts domains in the upstream config

As a developer I want to use keycloak-proxy as a standard reverse proxy. That means I want to use app context in the upstream setting, for example: https://domain.com/app-context/ or I want to use it with the domain, where virtual hosts are enabled (Host header needs to be part of the upstream request).
Will it be possible?

As of 2.1.0, you can now set the Host: header in your headers:.

Sure, it's partial improvement. But some legacy apps need also context (/something).

Probably that means prefixing req.URL.Path with the r.endpoint.Path in that same bit of code.  I don't actually have a use-case for this at the moment, so I'm not currently motivated to slog through the escaping and other issues.  I did find an example of reverse proxy code here, and translating what it does comes out to:
req.URL.Path = singleJoiningSlash(r.endpoint.Path, req.URL.Path)

But I am uncertain what happens to any URL escaping in the paths, since this is manipulating Path and not RawPath.  Close reading of the net/url/url.go source suggests that the above is probably correct, but it will alter the form of some escaped URLs.
(The code sample I linked also included some query string manipulation (to add query params from the reverse proxy target); that code would probably work here as well; at least it manipulates encoded paths.  But I'm not clear on the use case for always adding query params, vs. the one for a path prefix.)
Anyway, you can try adding that code to the source and see if it works for you.

Hm.  By the way, it looks like X-Forwarded-Host was supposed to be the original URL host, but the way the forwarding code is written, it's been sending the target host instead.  The req.Header manipulations should be before the URL manipulation.

@jangaraj thanks for taking your time reporting this. The repository keycloak-proxy was moved to Keycloak organization and renamed to keycloak-gatekeeper. If you still think this issue is valid, please read https://www.keycloak.org/community.html and report your bug/feature request to https://issues.jboss.org/browse/KEYCLOAK.