Implement publisher information api

We need an api to fetch details of publisher information.
Api:
/api/profile/publisher/<name>
Response:
{
  "title": "", //like display name
  "joined":"",
  "writeup:"some details writeup about the publisher",
  "contact": {
<EMAIL_ADDRESS>   "phone":"1234-1111"
  },
  "country": "somewhere on earth"
}

tasks:

[ ] Add contact, writeup, country, title in publisher model
[ ] Add controller for profile information

fixed in #205

@subhankarb why was this as /api/profile/publisher? That does not make a lot of sense to me - this should surely be at /api/publisher/{name} - can you explain why it is not.
Also I want to have at least a days notice on issues so they can be reviewed before they are implemented - this stuff should be in at the beginning of the sprint wherever possible.

@rufuspollock The thought behind the naming prefixed with /api/profile is that:
We need profile information for both user and publisher.
So both api would be:

For publisher /api/profile/publisher/{name}
For user /api/profile/user/{name}
[We are using Auth0 for using only verification. Every time user logged in or signup the db gets synced. ]


Can't build on 1.6

   Compiling regex_macros v0.1.21 (https://github.com/rust-lang-nursery/regex#6f428bc5)
/Users/yongqli/.cargo/git/checkouts/regex-5ba7da685e6a7b39/master/regex_macros/src/lib.rs:510:36: 510:54 error: unresolved name `ast::PatWildSingle` [E0425]
/Users/yongqli/.cargo/git/checkouts/regex-5ba7da685e6a7b39/master/regex_macros/src/lib.rs:510                 node: ast::PatWild(ast::PatWildSingle),
                                                                                ^~~~~~~~~~~~~~~~~~
/Users/yongqli/.cargo/git/checkouts/regex-5ba7da685e6a7b39/master/regex_macros/src/lib.rs:507:19: 511:16 note: in this expansion of vec! (defined in <std macros>)
/Users/yongqli/.cargo/git/checkouts/regex-5ba7da685e6a7b39/master/regex_macros/src/lib.rs:510:36: 510:54 help: run `rustc --explain E0425` to see a detailed explanation

rustc version is rustc 1.6.0-dev on2015-11-01

https://github.com/rust-lang/rust/commit/3468b8d42c0c5a8510a2810747221354dbbb8eff

I just noticed that this change has not yet landed in the latest (as of right now) rust-nightly-x86_64-unknown-linux-gnu.tar.gz in https://static.rust-lang.org/dist/2015-11-01/ (rustc 1.6.0-nightly (1a2eaffb6 2015-10-31))

This particular issue is easily fixed by
diff --git a/regex_macros/src/lib.rs b/regex_macros/src/lib.rs
index 5fe05bb..52de6ed 100644
--- a/regex_macros/src/lib.rs
+++ b/regex_macros/src/lib.rs
@@ -507,7 +507,7 @@ fn exec<'t>(
             pats: vec!(P(ast::Pat{
                 id: ast::DUMMY_NODE_ID,
                 span: self.sp,
-                node: ast::PatWild(ast::PatWildSingle),
+                node: ast::PatWild,
             })),
             guard: None,
             body: body,

Which then reveals another error
src/lib.rs:531:48: 531:60 error: no method named `parse_expr` found for type `syntax::parse::parser::Parser<'_>` in the current scope
src/lib.rs:531     let entry = cx.expander().fold_expr(parser.parse_expr());
                                                              ^~~~~~~~~~~~
src/lib.rs:532:23: 532:33 error: the type of this value must be known in this context
src/lib.rs:532     let regex = match entry.node {

Fixed by #128 in f76921


Ensure OTP prompting works correctly

Specifically:

first publish is attempted without --otp
once the "one-time password" error is thrown the user is prompted for the OTP
once entered, the OTP is used to publish the previously failed package again and to publish all future packages

Also fixes a fairly gnarly issue with this.eachWorkspace (ensuring it runs serially vs in parallel, due to process.chdir).

👍


[v1.x] fixing breaking change introduced in #17123 when batch_axis=0

Description
earlier split_and_save on batch_axis=0 would take in CSRNDArray and return CSRNDArray but now its not doing the same and returns NDArray. This causes change in external API behaviour. More details in #19268. Change was introduced in #17123
Checklist
Essentials

[x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage
[x] Code is well-documented

Behaviour After Fix
import scipy.sparse as sps
import mxnet.ndarray.sparse as mxsps
import mxnet as mx
x = mxsps.csr_matrix(sps.coo_matrix(([2.0], ([99], [999]))).tocsr(), ctx=mx.gpu(0))
y = mx.gluon.utils.split_and_load(x, (mx.gpu(0), mx.gpu(0)))
print(x)
print(y)

output:
[20:54:45] ../src/base.cc:84: Upgrade advisory: this mxnet has been built against cuDNN lib version 7501, which is older than the oldest version tested by CI (7600).  Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.

<CSRNDArray 100x1000 @gpu(0)>
[
<CSRNDArray 50x1000 @gpu(0)>,
<CSRNDArray 50x1000 @gpu(0)>]

@sxjscience

I think we may need to add a test case. Also, we should better add a comment in the code base that links to the issue. The root cause is that slice_axis has not supported sparse ndarray.

@mxnet-bot run ci [all]

@sxjscience can you review again ?


Shouldn't be less than sign?!

https://github.com/karpetrosyan/hishel/blob/34231cc421a74fbe1d40f11a7e548347576de12a/hishel/_async/_storages.py#L236C86-L236C86

DELETE FROM cache WHERE datetime(date_created, '+{self._ttl} seconds') > datetime()
it's correct
x = creation time
y = current time
remove if x + ttl > y

    async with self._lock:
        await self._connection.execute("DELETE FROM cache WHERE key = ?", [key])
        serialized_response = self._serializer.dumps(response=response, request=request, metadata=metadata)
        await self._connection.execute("INSERT INTO cache(key, data) VALUES(?, ?)", [key, serialized_response])
        await self._connection.commit()
    await self._remove_expired_caches()  (here)

x = 2023-11-29 12:26:08
ttl = 86400
x + ttl = 2023-11-30 12:26:08
y = 2023-11-29 12:26:08
it will be removed!
sorry, if i am missing something?

OOps, it seems I have missed something!
I am wondering how it passes the test cases that we have.
Goot catch, thank you :pray:
Do you want to submit a PR that resolves this?

Yes, I will submit a PR.

Resolved in #119


Add extra admin modules

[ ] Module with amount applications confirmed, invited, applied...
[ ] Module with confirmed amount travel reimbursement and invited travel reimbursement.
[ ] Module with tshirts confirmed (number and %)
[ ] Module with tshirts invited (number and %)
[ ] Module with Food restrictions confirmed (number and %)
[ ] Module with Food restrictions invited (number and %)

Continuation on #30


JS Exception: TypeError: undefined is not an object (evaluating 'modules.length')

When I change some code (I have hot-reloading enabled), my app crashes to homescreen often. Logcat shows me:
Got JS Exception: TypeError: undefined is not an object (evaluating 'modules.length')
Is there something I can do about this, it make my development really slow :(.

I traced it down to the following: When my React class inherits from a super class, e.g.
class Base extends React.Component {
}

class App extends Base {
}

and I update something inside App, react-native will crash.

Closing as it seems to be a duplicate of https://github.com/facebook/react-native/issues/7199#issuecomment-216247419


[BUG] Couldn't resolve the package 'flutter_gen'

:information_source: Info
msix version: 3.16.7
flutter doctor output:
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.22.0, on Microsoft Windows [Version 10.0.22631.3447], locale en-US)
[✓] Windows Version (Installed version of Windows is version 10 or higher)
[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.dev/docs/get-started/install/windows#android-setup for detailed instructions).
      If the Android SDK has been installed to a custom location, please use
      `flutter config --android-sdk` to update to that location.

[✓] Chrome - develop for the web
[✓] Visual Studio - develop Windows apps (Visual Studio Community 2022 17.9.7)
[!] Android Studio (not installed)
[✓] Connected device (3 available)
[✓] Network resources

:speech_balloon: Description
Enter a description of your problem here.
flutter run does work
flutter run msix:create fails
dart run msix:create fails
Any logs, error and assets you have that can help to find this bug.
github action with logs: https://github.com/wim07101993/qr_code_generator/actions/runs/9133440507/job/25116922893
To run the tool with extended logs use the -v argument, for example: lutter pub run msix:create -v
:scroll: Pubspec.yaml
ame: qr_code_generator
description: Generates different kinds of QR-codes, like a link or a payment request.
publish_to: 'none'
version: 0.4.0

environment:
  sdk: '>=3.0.0 <4.0.0'

dependencies:
  ansicolor: ">=2.0.0 <3.0.0"
  auto_route: ">=8.0.0 <9.0.0"
  behaviour: ">=3.0.0 <4.0.0"
  collection: ">=1.0.0 <2.0.0"
  download: ">=1.0.0 <2.0.0"
  file_picker: ">=8.0.0 <9.0.0"
  flex_color_picker: ">=3.0.0 <4.0.0"
  flutter:
    sdk: flutter
  flutter_app_base:
    git:
      url: https://github.com/wim07101993/flutter_app_base.git
      ref: 921445b60c8acab050ae9f365078a4208c6c1f13
  flutter_fox_logging: ">=2.0.0 <3.0.0"
  flutter_localizations:
    sdk: flutter
  function_tree: ">=0.0.0 <1.0.0"
  get_it: ">=7.0.0 <8.0.0"
  intl: ">=0.0.0 <1.0.0"
  path: ">=1.0.0 <2.0.0"
  path_provider: ">=2.0.0 <3.0.0"
  qr: ">=3.0.0 <4.0.0"
  qr_flutter: ">=4.0.0 <5.0.0"
  share_plus: ">=9.0.0 <10.0.0"
  shared_preferences: ">=2.0.0 <3.0.0"

dev_dependencies:
  auto_route_generator:
  build_runner:
  flutter_test:
    sdk: flutter
  lint:
  msix: ">=3.0.0 <4.0.0"

flutter:
  uses-material-design: true
  generate: true

msix_config:
  display_name: QR-generator
  publisher_display_name: wim07101993
  publisher: CN=C359B0E3-0A9C-4C4C-9037-E1B393CA187E
  identity_name: 38383wim07101993.QR-generator
  msix_version: <IP_ADDRESS>
  languages: nl-be, en-gb
  store: true
  logo_path: logo_without_background.png

Same problem here
it s like when running
dart run msix:create with last SDK remove flutter_gen package reference.
flutter pub get restore link to this package

It was an issue with the flutter sdk I think. There were problems with the build_runner as well. Upgrading to flutter 3.22.1 solves the problem.


Prefer Trusted Publishers

ssia

coverage: 100.0%. remained the same
when pulling 177bb8da9c2be330865937df1fbbff768049e479 on trusted-publishers
into a596621512addf9f4005f22d75a3b942040a846d on main.


connect timeout error for Ollama server when trying to insert embeddings for a large number of documents

🐛 Describe the bug
Using the following code (that works for smaller number of documents -- tested with LIMIT 200)
    .setModel(new Ollama({ modelName: "llama3.2", baseUrl: ollama }))
    .setEmbeddingModel(
        new OllamaEmbeddings({
            model: 'nomic-embed-text',
            baseUrl: ollama
        })
    )
    .setVectorDatabase(new LibSqlDb({ path: './zai.db' }))
    .build();

async function loadData() {
    const result = await db.execute("SELECT fulltext FROM t LIMIT 20000");

    for (const row of result.rows) {
        app.addLoader(new TextLoader({ text: row.fulltext }));
    }
}

loadData();

I got the following error
node:internal/deps/undici/undici:13178
      Error.captureStackTrace(err);
            ^

TypeError: fetch failed
    at node:internal/deps/undici/undici:13178:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async post (file:///Users/punkish/Projects/zai/node_modules/ollama/dist/shared/ollama.cddbc85b.mjs:117:20)
    at async Ollama.embed (file:///Users/punkish/Projects/zai/node_modules/ollama/dist/shared/ollama.cddbc85b.mjs:407:22)
    at async RetryOperation._fn (/Users/punkish/Projects/zai/node_modules/p-retry/index.js:50:12) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: ::1:11434)
      at onConnectTimeout (node:internal/deps/undici/undici:2331:28)
      at node:internal/deps/undici/undici:2283:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:2315:13)
      at process.processImmediate (node:internal/timers:483:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

Node.js v20.16.0

Ollama is running and is available on port 11434

Do you have an idea of how many documents it can embed safely before this error appears? This timeout seems to be based on the capacity of the machine itself. I tried in two different machines and got two different sizes before timeout. I am thinking it may make sense to provide an optional parameter to specify the chunk size which will break the insert into different groups inserted sequentially.

an update:

loading 5000 articles took: 14:32.024 (m:ss.mmm), created a 197MB db with 45592 vectors
loading 10000 crashed leaving a 120MB db with 27895 vectors

I am thinking it may make sense to provide an optional parameter to specify the chunk size which will break the insert into different groups inserted sequentially.

From what I understand, input text has to be tokenized and chunked so that embeddings of the right size can be generated. This level of chunking of my input text, the individual articles, is already happening internally managed by embedjs.  My problem is that I am throwing a very large number of articles at it. But, this is what I am unable to understand -- the model generating the embeddings, in my case ollama, should never have to worry about that if there is a queue handling the way the input text is being fed to it.
Using rough calculations, if 5000 articles took ~15 mins, a million articles will take about 300 mins, or 5 hours, to generate embeddings.


Optimize JS bundles

Related to https://github.com/matrix-org/matrix-public-archive/issues/132
Bundle efficiency
Based on what I was seeing in https://github.com/matrix-org/matrix-public-archive/pull/175 trying to get the bundles to work; I couldn't properly get the chunks working as I ideally wanted where each entry point would be bundled up with no chunks except for a vendor chunk that included hydrogen-view-sdk
And I know the current chunking makes entry-client-room-alias-hash-redirect.js inefficient by just making it small and pulling in a separate redirect-if-room-alias-in-hash chunk which seems completely unnecessary and could be bundled into itself without an extra import.
Any combination of manualChunks and experimentalMinChunkSize options I tried with Vite just didn't do the trick.
Also see https://vitejs.dev/guide/build.html#chunking-strategy
Bundle size
The bundles seem pretty big at the moment and there is probably lots of opportunity to prune things down. Are we even tree-shaking?
Hopefully in https://github.com/matrix-org/matrix-public-archive/pull/175, we figure out how to minify the JS.
Dev notes
A way to visualize JS bundles:

DevTools: A better to way to visualize your JavaScript, https://umaar.com/dev-tips/270-devtools-lighthouse-treemap/ (direct link to YouTube)

From https://pagespeed.web.dev/analysis/https-archive-matrix-org/lcerqhz1xk?form_factor=mobile

Reduce unused JavaScript (1.05s estimated savings)
Reduce unused JavaScript and defer loading scripts until they are required to decrease bytes consumed by network activity. Learn how to reduce unused JavaScript. LCP

URL
Transfer Size
Potential Savings

matrix.org 1st Party
179.0 KiB
163.5 KiB

/assets/AvatarVie….js(archive.matrix.org)
179.0 KiB
163.5 KiB

…node_modules/hydrogen-view-sdk/lib-build/hydrogen.cjs.js
159.3 KiB
148.5 KiB

<EMAIL_ADDRESS>10.9 KiB
10.8 KiB

…node_modules/dompurify/dist/purify.js
5.6 KiB
2.6 KiB

…node_modules/off-color/cjs/index.js
0.9 KiB
0.7 KiB


fix bato not loading chapter images

Fixes Bato having problems loading images.

hold off on merging this one, bato pushed a new update & now doesn't feel like working again.

Ready to go now, imgAccListString seems to no longer be provided.


Expose Round Number to Contract Runtime

Certain smart contracts will want time locks based on round height rather than the wall clock. We need to expose the round height to the smart contract runtime.
~We should add the previous switch block's round for an "absolute" round number, pretending that no rounds were skipped after the switch block.~

Reassigning to @fizyk20 for now.
I think this only makes sense for Zug, not for Highway, since in Highway the round number is subjective.
Not sure whether it's needed at all. We should discuss whether smart contracts need to know the round number.

Wouldn't block height be better and more universal? It is well defined at the stage at which deploys are executed, too.

Yes, I believe this issue came up when we were considering basing block rewards on this somehow. Maybe it's just not needed anymore?

This user story is a thing we intend to do, but its in the wrong epic. This would better fall into the epic for contract runtime enhancements because this enables functionality for smart contract to leverage.

@RitaMAllenCA seems like this should be assigned to some EE-related epic per comment above

Superseded by issue #4853, therefore closing.


Potential Discrepancy in Yield Factor Value in tmodel_const.py

Description
Hello,
While reviewing the tmodel_const.py file, I noticed that the yld (Yield Factor) is currently set to 0.17. However, based on the reference paper (Table 1 of Li et al., 2014), the yield factor should be 0.6.
Here is the specific line in question:
yld: float = 0.17  # y, Yield_factor (-)
"""Yield_factor (:math:`y`, 0.17, -)"""

I’m not entirely sure if this is an intentional adjustment or a possible typo, but I thought it would be helpful to raise it here for clarification.
Thank you in advance for looking into this. I really appreciate all the hard work that goes into creating this project!
Best,
Yiran
Type of Change

[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist
N/A
Further checks
N/A

Yup - that is indeed a typo from the table, duplicating the value of the default "ratio of fine-root mass to foliage area". Good spot.


Detected blocking call inside the event loop

Describe the bug
Home Assistant since a few version detects blocking calls.
One is related to botocore and surfaces via aiobotocore.
I opened a PR to fix it, but I was redirected here ( https://github.com/aio-libs/aiobotocore/pull/1120 )
Expected Behavior
Avoid blocking calls inside the event loop
Current Behavior
There are blocking calls inside the event loop
Reproduction Steps

Install Home Assistant >= 2024.6.0
Configure AWS integration
Check log

Possible Solution
None
Additional Information/Context
2024-07-17 18:41:54.986 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to listdir with args ('/usr/local/lib/python3.12/site-packages/botocore/data/sns',) inside the event loop by integration 'aws' at homeassistant/components/aws/notify.py, line 38: return await session.get_available_regions(service) (offender: /usr/local/lib/python3.12/site-packages/botocore/loaders.py, line 363: for dirname in os.listdir(possible_path):), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A"integration%3A+aws"
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#listdir
Traceback (most recent call last):
2024-07-17 18:41:54.994 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open with args ('/usr/local/lib/python3.12/site-packages/botocore/data/sns/2010-03-31/service-2.json.gz', 'rb') inside the event loop by integration 'aws' at homeassistant/components/aws/notify.py, line 38: return await session.get_available_regions(service) (offender: /usr/local/lib/python3.12/gzip.py, line 192: fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A"integration%3A+aws"
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#open
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/src/homeassistant/homeassistant/main.py", line 223, in
sys.exit(main())
File "/usr/src/homeassistant/homeassistant/main.py", line 209, in main
exit_code = runner.run(runtime_conf)
File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
SDK version used
1.34.131
Environment details (OS name and version, etc.)
Home Assistant Core (docker container): Alpine Linux v3.20.1

Thanks for reaching out. Boto3/Botocore does not use an event loop, that is all on the aiobotocore side. (Documentation on the Boto3 event system can be found here).
Regarding the discussion in https://github.com/aio-libs/aiobotocore/pull/1120 around the service models, the team has already made optimizations to reduce the size of the models. There are also open issues like https://github.com/boto/botocore/issues/2365 and https://github.com/boto/boto3/issues/3197 tracking the request to implement further improvements. On the topic of multithreading you can refer to the documentation for sessions (not thread safe) and clients.
We cannot guarantee compatibility with third-party software like Home Assistant or aiobotocore. For requests that directly involve Boto3/Botocore you can +1 (👍) existing issues or share a code snippet and logs to highlight the problem you're experiencing.


Feature/new type bug

Deployed in version 4.80
https://www.nuget.org/packages/CompareNETObjects/4.80.0


Upgrade eclipse paho mqtt library

upgrade from 1.4.1 to 1.4.2
To inlcude this fix https://github.com/eclipse/paho.mqtt.golang/issues/597 from 1.4.2

Thanks :+1:


Use 'type' field from nominatim response for display name

Fixes #41
This also cleans up some duplicated logic.

This looks fine to me during self-review. I don't have anyone to review frontend stuff anyway. Merging.


Typescript errors for inserting with Kysely

Describe the bug
db.insertInto('my_table').values(<some values>) throws errors in Typescript since it expects the presence of fields and methods from XataRecord.
To Reproduce

Create a simple schema
Use xata pull to codegen the xata.ts file
Example code snippet from generated file:export type MyTable = InferredTypes["my_table"];
export type MyTableRecord = MyTable & XataRecord;

export type DatabaseSchema = {
  my_table: MyTableRecord;
};

Build an insertInto statement with Kyselydb.insertInto('my_table').values({...myValues})

Will throw TypeScript errors for type ... is missing the following properties from type ... id, xata, getMetadata, toSerializable, and 4 more

Those missing properties come from XataRecord

Expected behavior

Accept object without XataRecord properties to insert into a table

Software version
    "dependencies": {
<EMAIL_ADDRESS>"^0.30.1",
<EMAIL_ADDRESS>"^0.2.1",
        "kysely": "^0.27.4",
        "zod": "^3.23.8",
        "zod-validation-error": "^3.4.0"
    },

Additional context
The docs only show an example for selecting a table entry. An insertInto example would be a great addition.

I just run into this issue too, setting up Kysely with Xata and NextJS.
My desired partial type from form data is missing the following properties from type:
replace
xata_id
xata_createdat
xata_updatedat
and 5 more...


Add in a Fuzzer for KCL's parser and lexer

This found https://github.com/KittyCAD/modeling-app/pull/2491 and I want to chip away at covering more of the input parsers with a fuzzer as time goes on - this was just the easiest one to do first. I'm thinking of using something a bit more structured long-term, but for now this'll do.

already in the repo at https://github.com/KittyCAD/modeling-app/tree/main/src/wasm-lib/kcl/fuzz


Wk/taco 378 preserve explicitly set transfer-encoding header

preserve explicitly set transfer-encoding header for proxied h2 to h1 requests

Coverage decreased (-0.09%) to 44.426% when pulling 62b4fac34c04faea10a5925ac8fcc1b65697ec9e on wk/TACO-378 into 1cb796b4f9607e6d504373dcbf3e63c39c05992c on master.


Successful INSERT returns undefined recordset in Promise

When a INSERT query is successful, and a promise is returned, the promise's value is undefined.

This is probably correct behavior, need to specify records to be returned using the OUTPUT clause.


Upgrade to Hyper 1.0 & Axum 0.7

Current Status: This is ready for CI & review!
This is a continuation of PR #1595 by @ikrivosheev.
TODO:

[X] update imports to use -util crates (done in #1595)
[X] update AddrStream and AddrIncoming (done in #1595)
[X] fix hyper-timeout connector (done in #1595)
[X] use axum Request and Response in transport (via TowerToHyperService in Client)
[X] change poll_trailers to use poll_frame. Still some work to do here about preserving data and trailers if poll_frame returns a frame type we aren't expecting.
[X] Correctly handle data & trailer frames for server.
[X] fix connect, Connect (no more hyper::client::service::Connect) (using hyper-util)
[X] Get around "Implementation of From not general enough" compiler issue.
[X] Correctly implement graceful shutdown (maybe based on hyper-util, but also not too hard to do independently, following axum's example.)
[ ] Pass through max_pending_accept_reset_streams (needs a release with PR #102 in hyper-util)
[ ] Correctly implement http2_only (needs PR #111 in hyper-util.
[ ] Figure out what to do about hyper_util ConnectError
[X] Add socket2 dependency to support TCP SO_NODELAY and SO_KEEPALIVE
[X] Compatibility of other tonic-* crates (e.g. tonic-web)
[X] Examples
[X] Check over Docs
[ ] Version Update

I'd like to propose that this be reviewed as-is, and that we add follow-up issues for the remaining todos, which will all require PRs to land and then releases to happen for hyper and/or hyper-utils.

works perfectly after cargo update

@alexrudy FYI https://github.com/hyperium/hyper-util/pull/102 is merged now.

called Result::unwrap() on an Err value: hyper::Error(User(ManualUpgrade))

@alexrudy Example h2c failed.

Thanks! https://github.com/hyperium/tonic/pull/1670/commits/5a579e63cadbe83c9dbe903472490c1d2397c904 contains the fix.

@alexrudy and @ikrivosheev thanks for all your work on this!
I'm going to be helping @LucioFranco out with tonic reviews, and obviously this is a desirable change. As such, I've made an initial pass of the changes here. Here are some thoughts:

Since this is already such a large change, I'd like to avoid intertwining the changes that are necessary to upgrade hyper/axum with any other changes that can be done independently. For example, reformatting of Cargo.toml files, upgrading to a different nightly, reordering imports and other small changes like adding type annotations. To facilitate review I think all of those changes should either be reverted in this PR or submitted as part of separate PRs (with one commit or PR per logical change) where we can judge them properly -- the goal is to make this PR as small and focused as possible.
I think it might make sense for now to just delete any of the examples that rely on warp, probably in a separate PR? While warp was one of the newer options when tonic got started, today it's much less popular than Axum -- and by virtue of doing this in a separate commit/PR it should be relatively straightforward to bring back those examples once warp upgrades to hyper/http 1.
I would like to avoid further entangling of Axum and Tonic proper. Ideally this change should be neutral in how much of Axum is used in -- if that's hard for some reason let's discuss that separately? Especially for client-only users, we should be working in a direction where it's possible to avoid depending on Axum.

@alexrudy FYI hyper-util v0.1.4 should contain the API you need abovex

@alexrudy as I can see, hyper-util v0.1.4 is released. Can you upgrade the PR?
https://crates.io/crates/hyper-util

@djc Thanks so much for your initial comments.
Overall, I tried to minimally scope the changes here based on what was required to make the tests pass with hyper >= 1.
For instance:
* The change in the nightly compiler version from `2024-02-06` to `2024-04-16` is required to make Axum >= 0.7 compile on nightly (it otherwise requires `#![feature(diagnostic_namespace)]` to compile, and we can't change the nightly features requested by a dependency IIUC.

Makes sense. So can we bump this independently from the Axum upgrade or is this required to be "exactly" in sync? IMO this is one of these things that seems conceptually separate enough that it would be nice to deal with in a separate PR if possible.

* There are type annotations added where type inference was no longer sufficient to allow code to compile - e.g. in the h2c/server.rs example, due to the move from calling hyper's server directly to calling `TcpListener:bind()`, type inference is no longer able to resolve the `FromStr::Err` type (which will get transformed to `Box<dyn Error>`, and in `tls_rustls` examples, types have to be added because rustls and rustls_pemfile have changed their public API types, which in turn are required upgrades (I think) for compatibility with `hyper-rustls` for `hyper` 1.0.

That's fair! I'll point out detailed examples on my next, more detailed round of review.

I will fix the cargo.toml formatting (this came from my IDE's auto formatter, and I should have caught that before).

👍

For warp – I'm happy to delete those examples entirely, either in this PR or another one - deleting them is a pre-requisite to the CI tests passing.

Yes, let's take care of this in a separate PR.

This also brings up @LucioFranco's point about the changed error type, which we might want to leave failing: that sounds fine by me, I don't know of a good way though to leave a test in an expected failure state in Rust. Also, just letting the test fail will be problematic, as CI is currently configured to fail-fast, meaning that one failure will cause a lot of other tests to be skipped entirely. We could change that setting as well, if that is what is desired.
On Axum: I don't think that this PR increases the degree to which the two libraries are intertwined - there is some private adaptor code added, but the tonic::transport::service::router part has always been built on the axum::Router class - are you suggesting that we should be implementing our own routing? That feels like a separate endeavor to me. Or was there somewhere else that this PR ends up intertwining the two further? From my perspective, I thought it was pretty neutral about how much of Axum is required by tonic.

Sounds good -- I'll take a look in more detail on my next read through.

Finally, it seems like you both have a preference for a re-written history here to break commits into more logical chunks - I'll take a stab at that as well, I had defaulted to preserving history to start, but am fine with either strategy.

I think this repo has a squash merge strategy so doing one logical change per PR might be better. In other projects I help maintain I usually prefer to rebase and keep a clean history of atomic commits that deal with one conceptual change each -- exactly because in my opinion this facilitates review which is usually in shortest supply in open source projects. However, this is just fundamentally a large change (and I'm not advocating we split it up by doing piecemeal changes that try to paper over http 0.2/1.0 differences somehow).

Overall, @djc, I would love specific comments on parts of this PR that are really separable - or the decision that we'd rather merge a series of PRs, some of which contain the pre-requistites to the upgrade to hyper-1.0, and the final one that contains the final upgrade.

@djc (just about Axum & Nightly)

Makes sense. So can we bump this independently from the Axum upgrade or is this required to be "exactly" in sync? IMO this is one of these things that seems conceptually separate enough that it would be nice to deal with in a separate PR if possible.

We can bump the nightly version separately, thought it is then a one line change. I'll put up a PR for that - it is just a change which is (IMO) purely motivated by finding the minimum nightly where we can compile Axum.

@djc PR to remove warp examples

Nightly bump in a separate PR: https://github.com/hyperium/tonic/pull/1709

The separate PRs have been merged -- will go through this again once it has been rebased.

I've both rebased this and tried to split the commits into roughly atomic parts - hopefully that makes it easier to review. Because of the large surface of api changes in hyper and http, I'm not sure if any of these remaining commits work as separate PRs since they all require the hyper and http upgrades.

After a quick look, it appears that there are some unrelated changes included. Although I have not checked everything, here are some of them:

The version style in Cargo.toml manifest such as 0.26 -> 0.26.0. If you want to propose these kind of changes, they should be issued as a separated pull request (but it is not the maintainer's preffered style: https://github.com/hyperium/tonic/pull/1589#discussion_r1608126077)
There are some unnecessary version updating which seems not to be unnecessary:  2.0 -> 2.1.1, and so on.
The kind of the changes like dropping logging feature of tokio-rustls is not related to this migration.
The rebasing resolution seems to be broken, seeing the tls implementation.

For very large changes of this kind, it is necessary and important to reduce the scope of interest of the changes required in a single review as much as possible.
The impact of updating dependencies must also be considered in conjunction with the existing design. For example, dealing with tonic's Extensions, which is one of the effects of the breaking changes to the http crate, should not be handled simply, but should be handled by considering the necessity of the type in the first place. By the change of #1710 first, the responsibility for changing the update itself will be slightly reduced.
Or other example, since hyper no longer provides a Body type, I think it would be a good to have tonic's own Body type instead of BoxBody, like Axum does, in order to make the handling of the Body type in tonic more robust and ease to maintain, and it is possible to be addressed before updating hyper version. After that adopting it to hyper 1 can be addressed (Of course, whether this is actually possible requires separate consideration).
Basically, large changes like this should be broken up into smaller scope changes. For example, the scale of transport implementations is large, making it difficult to update them incrementally and therefore difficult to review.
The purpose of the following pull request is to improve both the user's convenience and the maintainability of this project itself, and I believe that these should be reflected first before migrating to hyper 1.

#1572
#1574
#1630

Hi @tottoto,
Thanks for the comments!
This is a particularly painful kind of review for me, since I'm not sure that it consistently moves this PR towards being ready to merge, so I'll address your comments one-by-one.

The version style in Cargo.toml manifest such as 0.26 -> 0.26.0. If you want to propose these kind of changes, they should be issued as a separated pull request (but it is not the maintainer's preffered style: https://github.com/hyperium/tonic/pull/1589#discussion_r1608126077)

For the sake of getting this to merge, I'll revert that part of the change, despite the .0 being meaningful - I'm not sure what @djc meant by that comment originally?

There are some unnecessary version updating which seems not to be unnecessary:  2.0 -> 2.1.1, and so on.

I've reverted this particular version change. I'm not sure where the "and so on." comes from though - I have really tried to whittle down the dependencies. It is true that a lot of the tls-specific libraries get bumped, they are bumped to maintain compatibility with hyper-rustls (used in the examples) which requires the minor version bump.

The kind of the changes like dropping logging feature of tokio-rustls is not related to this migration.

Happy to put this back in. I'm not sure what function it is providing right now, since it isn't used by tonic.

The rebasing resolution seems to be broken, seeing the tls implementation.

I'm not sure what you mean by this - can you point to something specific for me?
Overall, I'm a little frustrated to keep coming back to this PR to find that folks are annoyed that it is so large - it is of course large, the scope of API changes between hyper 0.14 and hyper 1 is pretty huge. I also think that the desire to keep this PR as small as possible is pretty heavily in conflict with the desire to "consider dependencies" re. things like #1710, (which I'm fine with, but @djc mentioned there that they'd rather wait to land that until this lands).
The same applies to a Body type for tonic - that feels like a decent-sized API change which deserves its own PR, not something to be wedged into this giant morass.

Basically, large changes like this should be broken up into smaller scope changes. For example, the scale of transport implementations is large, making it difficult to update them incrementally and therefore difficult to review.

This really makes absolutely no sense to me - the transport changes have to land in-sync with the other changes because they are all required by the upgrade to hyper 1. Doing them separately would not make much sense. Alternatively, we could try to introduce some of the changes required for this PR in transport ahead of time, but that is rife with potential problems, since the API changes between hyper 0.14 and hyper 1 are so large, that writing the same code for hyper 0.14 is considerable additional effort which doesn't (at least to me, contributing on my free time) feel like it will pay off.
It also seems that you would prefer your list of PRs to be merged before this one:

The purpose of the following pull request is to improve both the user's convenience and the maintainability of this project itself, and I believe that these should be reflected first before migrating to hyper 1.

That's fine, but perhaps not a useful contribution here. In this thread, I think I've demonstrated that I'm very willing and responsive to rebasing this (massive) PR on top of additional changes to tonic. If you have a different view of how the project prioritizes reviews, maybe we could discuss that on Discord?
I think I feel a particular frustration here because you are the 3rd person to "take a quick look" and provide review comments which are difficult to take significant actions on. As well, although I'm probably mis-reading your comment, it comes off to me as pretty condescending to explain, yet again, that changes should be broken into small parts to make it easier on reviewers. I think that @djc and I have been having that conversation back-and-forth considering the tradeoffs up thread. Comments like "basically,  large changes like this should be broken up into smaller scope changes" read like you are trying to educate someone with limited experience in the software world. I suspect you mean well, but in this case, I'd really encourage you to assume that the folks on the other end of your messages are knowledgable and competent developers, especially when they've clearly put a lot of time and effort into what they are building.
Again, thanks for the comments - I'm really looking forward to getting something like this merged, since it does also greatly impact my own user experience with tonic (since it prevents me from upgrading numerous other libraries which depend on hyper 1 now), which is a large part of my own motivation here.

This is a particularly painful kind of review for me, since I'm not sure that it consistently moves this PR towards being ready to merge, so I'll address your comments one-by-one.

The version style in Cargo.toml manifest such as 0.26 -> 0.26.0. If you want to propose these kind of changes, they should be issued as a separated pull request (but it is not the maintainer's preffered style: #1589 (comment))

For the sake of getting this to merge, I'll revert that part of the change, despite the .0 being meaningful - I'm not sure what @djc meant by that comment originally?

I'm not sure exactly what part of the Cargo documentation leads you to believe that the trailing zero in 0.26.0 is meaningful, but I'm pretty sure it's not in practice. Both 0.26.0 and 0.26 mean the same thing, namely that the accepted version range is >= 0.26.0, < 0.27.0. This might be different for the case where the leading number is 0 because Cargo treats 0.x somewhat specially (in the sense that 0.1.x and 0.2.x are treated as semver-incompatible but 1.1.x and 1.2.x are treated as compatible).

There are some unnecessary version updating which seems not to be unnecessary:  2.0 -> 2.1.1, and so on.

I've reverted this particular version change. I'm not sure where the "and so on." comes from though - I have really tried to whittle down the dependencies. It is true that a lot of the tls-specific libraries get bumped, they are bumped to maintain compatibility with hyper-rustls (used in the examples) which requires the minor version bump.

The abundance of examples is unfortunate in my view -- IMO examples have diminishing returns as the number grows, and given that the maintenance efforts of examples is non-zero, increasing numbers of examples over time can have a net-negative value to the project (in particular for examples that drag in dependencies beyond what the libraries need anyway). So I think it would be good to have a discussion on what examples we ought to keep which might reduce the changes needed in this PR, but I also don't necessarily want to block this PR on those changes.

The kind of the changes like dropping logging feature of tokio-rustls is not related to this migration.

Happy to put this back in. I'm not sure what function it is providing right now, since it isn't used by tonic.

The logging feature is forwarded to rustls in the end, so it provides a feature that downstream users might belong on. I think defaulting to enabled makes sense here, but primarily I agree that this PR should not change whether we enable it or not because the change is unrelated to the change this PR seeks to make.

The rebasing resolution seems to be broken, seeing the tls implementation.

I'm not sure what you mean by this - can you point to something specific for me?
Overall, I'm a little frustrated to keep coming back to this PR to find that folks are annoyed that it is so large - it is of course large, the scope of API changes between hyper 0.14 and hyper 1 is pretty huge. I also think that the desire to keep this PR as small as possible is pretty heavily in conflict with the desire to "consider dependencies" re. things like #1710, (which I'm fine with, but @djc mentioned there that they'd rather wait to land that until this lands).
The same applies to a Body type for tonic - that feels like a decent-sized API change which deserves its own PR, not something to be wedged into this giant morass.

Agreed that these changes should go in separate PRs. I think the main question is whether they should be merged before this PR or other -- though at the moment I'm inclined to agree that it probably doesn't make sense to merge them before this PR.

Basically, large changes like this should be broken up into smaller scope changes. For example, the scale of transport implementations is large, making it difficult to update them incrementally and therefore difficult to review.

This really makes absolutely no sense to me - the transport changes have to land in-sync with the other changes because they are all required by the upgrade to hyper 1. Doing them separately would not make much sense. Alternatively, we could try to introduce some of the changes required for this PR in transport ahead of time, but that is rife with potential problems, since the API changes between hyper 0.14 and hyper 1 are so large, that writing the same code for hyper 0.14 is considerable additional effort which doesn't (at least to me, contributing on my free time) feel like it will pay off.
It also seems that you would prefer your list of PRs to be merged before this one:

The purpose of the following pull request is to improve both the user's convenience and the maintainability of this project itself, and I believe that these should be reflected first before migrating to hyper 1.

I think it's interesting to consider changes that isolate the impact of the specific dependencies used for the default transport implementation, but given that most downstream users are still likely to end up wanting to use the default transport implementation I feel like this is the more important PR.

Again, thanks for the comments - I'm really looking forward to getting something like this merged, since it does also greatly impact my own user experience with tonic (since it prevents me from upgrading numerous other libraries which depend on hyper 1 now), which is a large part of my own motivation here.

Thanks for your mature response to comments and sorry for being unable to spend more time on this PR -- tonic is just a small part of my job's dependency stack so while I want to keep things moving forward I have to be deliberate in how I spend my time on this. That said, as mentioned before this is a high priority for me and will definitely keep spending time on reviewing this.

I think I feel a particular frustration here because you are the 3rd person to "take a quick look" and provide review comments which are difficult to take significant actions on.

I do not know what makes you frustrated, but I said a quick look to make it clear that I did not give a completely comprehensive review. And it is difficult to review for large changes that include changes that seem unrelated  (I read this entire pull request, of course). Additionally, it does not matter that anybody be a 3rd person for reviewing.  Recommended that you understand that receiving multiple similar points means that that point of view is considered important and needs to be addressed. I cannot do anything about difficulties to take significant action on makes you frustrated. Because your emotions are your own. Since it contains inherently difficult issues, it is inevitable that the content pointed out in reviews will be difficult. What I can say is that expressing frustration over unfavorable comments for you is not a good idea for OSS because it evokes fear that makes people hesitant to review. I believe that OSS contributors participate in the hope that OSS will become better.
When making large-scale changes (which is not a bad thing in itself; there are times when it is inherently inevitable, and I think this will be the case), we should judge the necessity of the changes more strictly than usual as possible to reduce the cognitive area required in the review. In this regard, your suggestions in this pull request are painful to review. I understand that you are tired of what is taking up your time, but I encourage you to realize that you are making unkind suggestions in reviews.

As well, although I'm probably mis-reading your comment, it comes off to me as pretty condescending to explain, yet again, that changes should be broken into small parts to make it easier on reviewers.

Yes. It is not the intended meaning. It seems that you are reading too deeply and are reading into things that are not written. It doesn't matter whether you're a seasoned developer or not, and I'm not talking about it. I am commenting on the code. You look  be reluctant to basically make the changes smaller. In any case, at least I'm commenting on what I feel is necessary in the context, even if others have already pointed it out. In other words, at least from my point of view, it doesn't seem to be of sufficient quality in this respect. When we developers write code ourselves, we fully understand the scope of the code, so we tend to misunderstand some changes (including small refactoring) as trivial changes and underestimate the negative impact it will have on reviews. We tend to misunderstand, but the time it takes to write code has little to do with its quality.

The rebasing resolution seems to be broken, seeing the tls implementation.

I'm not sure what you mean by this - can you point to something specific for me?

Can you explain the need for the following changes to this pull request?
tonic/src/transport/service/tls.rs of https://github.com/hyperium/tonic/pull/1670/files#diff-f8078e6eaa6dfb7a56bc0ee9d2b6c345c231ebaa4c76d58e54faa95bf43945c4

@djc Thanks for the clarification about the version numbers - I see your point now. Its too bad that the current state of the Cargo.toml files for tonic is pretty inconsistent on this front.

The abundance of examples is unfortunate in my view -- IMO examples have diminishing returns as the number grows, and given that the maintenance efforts of examples is non-zero, increasing numbers of examples over time can have a net-negative value to the project (in particular for examples that drag in dependencies beyond what the libraries need anyway). So I think it would be good to have a discussion on what examples we ought to keep which might reduce the changes needed in this PR, but I also don't necessarily want to block this PR on those changes.

Happy to consider removing tls_rustls in a separate PR - that would reduce the changes required to the tls code elsewhere, although this issue is likely to re-emerge as users who want to leverage hyper-rustls start to upgrade.

@djc Super helpful comments - especially since I've been knee-deep in this PR, and some of it I inherited from the previous attempts. Thanks for taking the time. I've spent some time minimizing the formatting changes etc. that you commented on.

I've gotten about halfway through the commits and will continue reviewing likely tomorrow depending on time. Left some feedback but mostly questions. Overall looks great. Thank you so much for breaking this up into commits, it has made it so much easier to review!

@LucioFranco I think I have addressed your comments!


Update easy-rsa and remove workaround

https://github.com/OpenVPN/easy-rsa/releases/tag/v3.0.7
https://github.com/angristan/openvpn-install/pull/469/files

Hi, there is also Easy-TLS, an extension script to Easy-RSA.
Please see: https://github.com/TinCanTech/easy-tls
Sorry for the spam


Send all locha-p2p RuntimeEvents to Java

Almost finished, Java method signatures need to be updated in order for this PR to be ready for review.

@luisan00 comments addressed. Thanks for the review :wink:


fix(button): add class around the icon to fix bug

fix #1883
Create follow up issue in React.

Deploy preview for pf-next ready!
Built with commit 479025e005ced13f88dd32e1c9b7f3385d6276cb
https://deploy-preview-1890--pf-next.netlify.com

@mcoker  updated

looks like you need to update https://github.com/christiemolloy/patternfly-next/blob/master/src/patternfly/components/NotificationBadge/examples/notification-badge-simple-example.hbs since the merge

updated @mcoker

:tada: This PR is included in version 2.12.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[21209] Add v2.14.2 release notes

Description

This PR adds the v2.14.2 release notes to Fast DDS Docs master

Contributor Checklist

[x] Commit messages follow the project guidelines.
N/A: Code snippets related to the added documentation have been provided.
[x] Documentation tests pass locally.
N/A: Applicable backports have been included in the description.

Reviewer Checklist

[x] The PR has a milestone assigned.
[x] The title and description correctly express the PR's purpose.
[x] Check contributor checklist is correct.
[ ] CI passes without warnings or errors.

@richiprosima please test_3 this

@richiprosima please test_3 this


systemctl restart opennebula doesnt restart onehem

Description
systemctl restart opennebula doesnt restart onehem service. That needs restarting with systemctl restart opennebula-hem.
Expected behavior
As with the other opennebula services I would have expected this to restart as well eg after an upgrade
Details

Affected Component: hem
Version: 5.10.0 & .1

Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

It works the same way scheduler works:
https://github.com/OpenNebula/one/blob/99fd76c1567e82ef3df3b3e07ae0b36a1167690c/share/pkgs/services/systemd/opennebula.service#L5
They both are required for OpenNebula to start but they are different services, you may want to restart OpenNebula without restarting them.


Allow dataset parsing to extract options from dataset (e.g. labels, colors, ...)

Feature Proposal
I'm feeding this type of data-set to Chart.js:
dataset= [
  data: [
     ['people' => 'name' , 'x-axis' => 'yyyy-dd', 'y-axis' => 35 ],
     ['people' => 'other name' , 'x-axis' => 'yyyy-dd', 'y-axis' => 35 ],
  ],
  'parsing': [ 'xAxisKey':  'x-axis', 'yAxisKey' : 'y-axis', ],
];

Unfortunately, 'parsing' only allow for 'xAxisKey' and 'yAxisKey' parsing. It would be interesting to allow for things like label (and color, etc.) to be extracted from data columns.
The goal is to create a stacked bar chart with each 'people' value having a different color using the autocolors plugin.
Possible Implementation
No response

You can use scriptable options for those and get the value from the data.

Hi, where can i find a working example of this feature ? My current solution is to create a dataset for each 'people' value, but it is not optimum.

https://www.chartjs.org/docs/master/samples/scriptable/bar.html


[arte] Move to v2 API

  Boilerplate (own code, improvement)

I am the original author of this code and I am willing to release it under Unlicense

What is the purpose of your pull request?

Improvement

The v1 API code was removed for simplicity, even though it still mostly works. Not anymore. Not not anymore. Not not not anymore, the v1 API is now completely down (404).
I have left a number of ‘XXX’ comments in, please take a look before merging.
The testcases for this extractor are stale. I have not updated them, other than throwing away upload_date, which the updated code no longer extracts.
Fixes or subsumes:

#3086
#3428
#3502
ytdl-org/youtube-dl#29640
ytdl-org/youtube-dl#29653
ytdl-org/youtube-dl#29870
ytdl-org/youtube-dl#30816
ytdl-org/youtube-dl#30878

To settle before merging:

[x] Whether to keep v1 API extraction (it looks like Arte has pulled the plug on the v1 API, so no, it seems) (probably not worth the trouble if even Arte themselves neglect it; plus, text subtitles in v2 API > burned-in subtitles in v1 API)
[x] Update testcases (I think good enough for now; videos may expire in the future, though)
[x] Language preference inferred from version label
[x] All the XXX comments

[x] Non-HLS protocols
[x] nulls for videos that are no longer available
[x] Whether rights is a good upload_date replacement (I say good enough)
[x] id vs providerId (decided to use providerId after all)
[x] extract chapters from stream['segments'] (deferred this one)

[x] Is stream['versions'] always a one-element list? (answer seems to be yes; even the host’s own player seems to assume so)
[x] Handling of https://api.arte.tv/api/player/v2/config/fr/LIVE
[x] Whether to also extract unlisted non-streaming mp4 URLs (my answer: no, #3521 should take care of this)

https://static-cdn.arte.tv/replay/_next/static/chunks/d3664e6f.d2e93ead7bcebcfd.js can be mined for more insights, probably. It’s pretty readable once reformatted. Apparently other possible protocol values include HLS, HTTPS and RTMP.

https://api.arte.tv/api/player/v2/config/fr/LIVE and https://api.arte.tv/api/player/v2/config/de/LIVE should extract correctly within their permitted geographical areas; unfortunately, the geoblocking is too sophisticated to fool with XFF.

I wouldn’t necessarily say I’m done with it, but I think I can invite some initial reviews.

From their player:
                const l = ee(`arte_vp.preferredVersion_${a.options.lang}`);
                if (l && !i) {
                    const e = a.filter(e => e.get("versions")[0].eStat.ml5 === l ? e : null);
                    e.length > 0 && (r = new ht(e));
                }

I was trying to show to how to add tests... (881f0e0aee210055311e5689d1cb7bbca25bce6a) but then had to rebase it on top of your force-push 😢. I believe I've already told u this, but please don't force-push in the middle of review. I understand doing it for PRs like #2173, but here it has no benefit!

I hope this can be merged soon

The recent v1 API breakage may be only temporary. Still, given that the web front-end has migrated away from it already, I doubt they are going to make it a priority to fix it. The response body is:
{
  "videoJsonPlayer": {
    "customMsg": {
      "msg": "Cannot resolve argument $videoJsonPlayerService of \"App\\Controller\\V1\\VideoApiController::getConfigJsonAction()\": Invalid JSON in env var \"URL_LIVE\": Syntax error",
      "type": "error"
    }
  }
}

from which we can also infer that Arte is a PHP shop.

I've just tested the patch.
$ yt-dlp -F https://www.arte.tv/de/videos/100225-000-A/sagrada-familia-antoni-gaudis-meisterwerk/
ID                           EXT  RESOLUTION FPS │   FILESIZE   TBR PROTO  │ VCODEC        VBR ACODEC     MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
VOF-STF-program_audio_0-VOF  m3u8 audio only     │                  m3u8_n │ audio only        unknown    [fr] [FR] Französisch
...
VA-STA-program_audio_0-VA    m3u8 audio only     │                  m3u8_n │ audio only        unknown    [de] [DE] Deutsch
...
VOF-STF-2326                 mp4  1280x720    25 │ ~  1.52GiB 2326k m3u8_n │ avc1.4d401f 2326k video only [FR] Französisch
...
VA-STA-2326                  mp4  1280x720    25 │ ~  1.52GiB 2326k m3u8_n │ avc1.4d401f 2326k video only [DE] Deutsch

Please consider EXT for audio to be: mp4, since the mp4 file (with MD5 checksum) is available. The file size, bit-rate and codec can then also be displayed.
$ yt-dlp -f VA-STA-program_audio_0-VA --get-url https://www.arte.tv/de/videos/100225-000-A/sagrada-familia-antoni-gaudis-meisterwerk/
[now:]
https://arte-cmafhls.akamaized.net/am/cmaf/100000/100200/100225-000-A/220414121352/medias/100225-000-A_aud_VA.m3u8
[expected:]
https://arte-cmafhls.akamaized.net/am/cmaf/100000/100200/100225-000-A/220414121352/medias/100225-000-A_aud_VA.mp4

EXT for video is mp4, but for now URL with m3u8 extension is returned instead:
$ yt-dlp -f VA-STA-2326 --get-url https://www.arte.tv/de/videos/100225-000-A/sagrada-familia-antoni-gaudis-meisterwerk/
[now:]
https://arte-cmafhls.akamaized.net/am/cmaf/100000/100200/100225-000-A/220414121352/medias/100225-000-A_v720.m3u8
[expected:]
https://arte-cmafhls.akamaized.net/am/cmaf/100000/100200/100225-000-A/220414121352/medias/100225-000-A_v720.mp4

The language acronym of audio tracks is printed twice: [fr] [FR] / [de] [DE].
Thanks!
P.S. Hopefully, the patch could be ported back to youtube-dl.

The recent v1 API breakage may be only temporary.

arte did it already at the end of August 2021.

I am kind of reluctant to also extract the non-streaming .mp4 URL. It is not listed in the manifest....

Unfortunately not.
@styroll has just tested the m3u8 files and it turned out ffmpeg cannot handle them: https://github.com/mediathekview/MServer/issues/791#issuecomment-1105349202.
Looking for the error "sidx reference_type 1 is not implemented" reveals following issues:

https://github.com/Catch-up-TV-and-More/plugin.video.catchuptvandmore/issues/543
https://github.com/xbmc/xbmc/issues/20039

But no upstream ffmpeg bug report at the first glance.

Only excerpts of a few minutes seem to be downloadable though.

@Tamaranch context?

For example
https://www.arte.tv/fr/videos/070794-000-A/sergio-leone-une-amerique-de-legende/
https://www.arte.tv/fr/videos/062284-000-A/cinq-femmes-cinq-cultures-une-histoire/
are only available as excerpts.
It's happened to me before I think, but rarely twice in a row.

I'm unsubbing from this thread. Ping me when I'm needed

@Tamaranch

https://www.arte.tv/fr/videos/070794-000-A/sergio-leone-une-amerique-de-legende/

On the web page: "Extrait (3 min)", thus no full video is available, yt-dlp cannot do anything about it.

Yes, indeed, sorry for the noise. I don't go through the site to get the links, so I didn't see the info, but I should have gone to check.

I had tried an earlier version of this PR (few days ago) and that didn't work very well... the precent display when downloading acted completely weird (stayed at close to 0% first... despite downloading something - and then suddenly it increased with impossible speeds)... also the downloaded file was apparently broken (much bigger than what -F told it would be, and playback with mpv just failed).
I've just tried again with the current version of arte.py and hls.py video works now, the precent-display seems also reasonable.
However:

the video has no audio (I tried: yt-dlp https://www.arte.tv/de/videos/093039-006-F/wildnis-europa/ -f VOA-2328)
the display of the expected total size varies while downloading (though that might be just expected)
the downloaded files Wildnis_Europa__[ArteTV:093039-006-F].de.vtt and Wildnis_Europa__[ArteTV:093039-006-F].mp4 have the executable bit set
the downloader seems to prefer the version for hearing impaired (which is probably not an optimal default):

$ yt-dlp https://www.arte.tv/de/videos/093039-006-F/wildnis-europa/ -F
[ArteTV] 093039-006-F: Downloading JSON metadata
[ArteTV] 093039-006-F: Downloading m3u8 information
[ArteTV] 093039-006-F: Downloading m3u8 information
[ArteTV] 093039-006-F: Downloading m3u8 information
[info] Available formats for 093039-006-F:
ID                           EXT RESOLUTION FPS │   FILESIZE   TBR PROTO  │ VCODEC        VBR ACODEC     MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
VF-program_audio_0-VF        mp4 audio only     │                  m3u8_n │ audio only        unknown    [fr] [FR] Französisch
VOA-program_audio_0-VOA      mp4 audio only     │                  m3u8_n │ audio only        unknown    [de] [DE] Deutsch
VOA-STMA-program_audio_0-VOA mp4 audio only     │                  m3u8_n │ audio only        unknown    [de] [UT] Deutsch (Hörgeschädigte)
VF-430                       mp4 384x216     25 │ ~163.28MiB  430k m3u8_n │ avc1.42e00d  430k video only [FR] Französisch
VF-934                       mp4 640x360     25 │ ~354.75MiB  934k m3u8_n │ avc1.4d401e  934k video only [FR] Französisch
VF-1629                      mp4 768x432     25 │ ~618.70MiB 1629k m3u8_n │ avc1.4d401e 1629k video only [FR] Französisch
VF-2328                      mp4 1280x720    25 │ ~883.99MiB 2328k m3u8_n │ avc1.4d401f 2328k video only [FR] Französisch
VOA-430                      mp4 384x216     25 │ ~163.28MiB  430k m3u8_n │ avc1.42e00d  430k video only [DE] Deutsch
VOA-934                      mp4 640x360     25 │ ~354.75MiB  934k m3u8_n │ avc1.4d401e  934k video only [DE] Deutsch
VOA-1629                     mp4 768x432     25 │ ~618.70MiB 1629k m3u8_n │ avc1.4d401e 1629k video only [DE] Deutsch
VOA-2328                     mp4 1280x720    25 │ ~883.99MiB 2328k m3u8_n │ avc1.4d401f 2328k video only [DE] Deutsch
VOA-STMA-430                 mp4 384x216     25 │ ~163.28MiB  430k m3u8_n │ avc1.42e00d  430k video only [UT] Deutsch (Hörgeschädigte)
VOA-STMA-934                 mp4 640x360     25 │ ~354.75MiB  934k m3u8_n │ avc1.4d401e  934k video only [UT] Deutsch (Hörgeschädigte)
VOA-STMA-1629                mp4 768x432     25 │ ~618.70MiB 1629k m3u8_n │ avc1.4d401e 1629k video only [UT] Deutsch (Hörgeschädigte)
VOA-STMA-2328                mp4 1280x720    25 │ ~883.99MiB 2328k m3u8_n │ avc1.4d401f 2328k video only [UT] Deutsch (Hörgeschädigte)

the video has no audio (I tried: yt-dlp https://www.arte.tv/de/videos/093039-006-F/wildnis-europa/ -f VOA-2328)

WYGIWYAF, VOA-2328 is a video-only format. You need to pass both video and audio formats for merging: e.g. -f VOA-program_audio_0-VOA+VOA-2328 or even --audio-multistreams -f VOA-program_audio_0-VOA+VF-program_audio_0-VF+VOA-2328 to merge both languages’ audio. Or just leave the defaults as they are.

the display of the expected total size varies while downloading (though that might be just expected)

Yes, expected, this is just an estimate; though I expect a resolution to #3521 to ameliorate this.

the downloader seems to prefer the version for hearing impaired (which is probably not an optimal default):

Subtitles for the hearing-impaired are the only ones they offer for that video, apparently. There is no way around this. The audio and video streams are identical within a language version. I don’t see how the HI version is preferred otherwise.

the downloaded files Wildnis_Europa__[ArteTV:093039-006-F].de.vtt and Wildnis_Europa__[ArteTV:093039-006-F].mp4 have the executable bit set

Apply 98804d034d04d21cbeb8cd43d1e1d90f1cdae836 as well.

WYGIWYAF, VOA-2328 is a video-only format. You need to pass both video and audio formats for merging: e.g. -f VOA-program_audio_0-VOA+VOA-2328 or even --audio-multistreams -f VOA-program_audio_0-VOA+VF-program_audio_0-VF+VOA-2328 to merge both languages’ audio. Or just leave the defaults as they are.

 There is no way around this. The audio and video streams are identical within a language version. I don’t see how the HI version is preferred otherwise.

Same confusion than above... completely forgot that yt-dlp selects+downloads video and audio separately O:-)
Well than I can add the the code from this PR seems to work :-)

WRT subtitles: some videos on this site offer several (separate) subtitle files of the same format and language, and I don't think there currently is a way to specify a particular one. For example, this video offers 2 different French subtitles: subtitles for the German original version, and closed-captions for the hearing impaired matching I think the French dub more closely. Neither are burnt into the video stream BTW, they're available as separate files. When specifying --sub-langs fr or even --sub-langs all, the CC for hearing impaired is the one downloaded by default. I don't know how to download the French subtitles (the ones matching the German OV) file, if ever possible.
[info] Available subtitles for 047941-000-A:
Language Formats
fr       vtt, vtt, vtt
de       vtt, vtt

Unfortunately this video will be taken down in a couple of days, but (after a cursory search) I couldn't find another one with two distinct subtitles in the same language that would remain up longer. I've seen this case several times in the past on that site though so this is not an outlier case, simply less common.

Ive cherry picked https://github.com/yt-dlp/yt-dlp/pull/3302/commits/23c565604a5497dc141ae2b562f2467617b8856a

This patch seems to work (applied on master as of the date of this post, minus hls.py which has a conflict), but there is a bug when returning json (with -J): the top-level "url" field (at the same level than "extractor", "extractor_key", ...) pointing to the stream URL is missing. Example invocation:
yt-dlp https://www.arte.tv/en/videos/105617-003-A/twist/ -J

@fstirlitz I think there is still a bug with your fix : it produces too big files.
For example, with the latest Karambolage, when I downladed it last Sunday (1st May), I got a 162.7 MiB file.
MediaInfo data:
Complete name                            : /.../Karambolage.mp4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (isom/mp42)
File size                                : 163 MiB
Duration                                 : 11 min 7 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 2 045 kb/s
Encoded date                             : UTC 2022-04-29 03:15:17
Tagged date                              : UTC 2022-04-29 03:15:17

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile<EMAIL_ADDRESS>Format settings                          : CABAC / 3 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 3 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 11 min 7 s
Bit rate mode                            : Variable
Bit rate                                 : 1 915 kb/s
Maximum bit rate                         : 2 200 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.083
Stream size                              : 152 MiB (94%)
Encoded date                             : UTC 2022-04-29 03:15:19
Tagged date                              : UTC 2022-04-29 03:15:19
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Codec configuration box                  : avcC

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 11 min 7 s
Bit rate mode                            : Variable
Bit rate                                 : 125 kb/s
Maximum bit rate                         : 425 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 9.97 MiB (6%)
Encoded date                             : UTC 2022-04-29 03:15:19
Tagged date                              : UTC 2022-04-29 03:15:19

But with your fixed arte.py extractor, I get a 313.7 MiB file:
General
Complete name                            : /.../Karambolage V2.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 314 MiB
Duration                                 : 11 min 8 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 3 935 kb/s
Writing application                      : Lavf59.16.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile<EMAIL_ADDRESS>Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Format settings, GOP                     : M=4, N=50
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 11 min 8 s
Source duration                          : 11 min 8 s
Bit rate mode                            : Variable
Bit rate                                 : 3 668 kb/s
Maximum bit rate                         : 2 200 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 49.903 FPS
Minimum frame rate                       : 25.000 FPS
Maximum frame rate                       : 12 800.000 FPS
Original frame rate                      : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.080
Stream size                              : 292 MiB (93%)
Source stream size                       : 292 MiB (93%)
Language                                 : English
mdhd_Duration                            : 668824
Codec configuration box                  : avcC

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 11 min 8 s
Bit rate mode                            : Constant
Bit rate                                 : 258 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 20.5 MiB (7%)
Language                                 : French
Default                                  : Yes
Alternate group                          : 1

As you can see, the streams rates are different. Plus, when I run a download, it begins with downloading at max speed for a 16 GiB video file, then it slows down to 4-5 MiB/s and the estimated total size go down as well, and it downloads segment by segment. Is it downloading the video twice? Is it a API issue, or your fix that must be refined?

@bubbleguuum works fine here, weird
@Trit34 I am getting the ~160 mb file for your example download
I should however add that I had to edit lines 9 and 64 of hls.py regarding the Cryptodome_AES stuff, removing it/commenting out

Hi, didnt read the code, but I just needed an arteTV video today, so I tested @fstirlitz branch and it worked beautifully! Thanks everyone !

@bubbleguuum works fine here, weird

I applied this PR as a patch and running this test, does not give me the "url" json field at the root level of the JSON:
yt-dlp https://www.arte.tv/en/videos/105617-003-A/twist/ -J
I'm just pasting the end of the JSON:
....
 "timestamp" :<PHONE_NUMBER>,
   "title" : "Twist",
   "upload_date" : "20220411",
   "vbr" : 2325.512,
   "vcodec" : "avc1.4d401f",
   "webpage_url" : "https://www.arte.tv/en/videos/105617-003-A/twist/",
   "webpage_url_basename" : "twist",
   "webpage_url_domain" : "arte.tv",
   "width" : 1280
}

The "url" field is missing. All other extractors have it.

So Xontic, how have you done this?

This patch enabled me to download https://www.arte.tv/fr/videos/100809-000-A/l-histoire-oubliee-des-femmes-au-foyer/ Thanks!

fyi on this branch when geofenced i'm getting the following error:
yt-dlp  https://www.arte.tv/fr/videos/100809-000-A/l-histoire-oubliee-des-femmes-au-foyer/ -J
ERROR: not enough values to unpack (expected 2, got 1)

However once not geofenced it works

@clickot Probably because FFmpeg doesn't support WebVTT blocks other than cue blocks.

So, v2022.05.18 fixed the HLS issue, but the arte.py extractor was not fixed (no mention for it in the changelog, and JSON 404 error again). Replacing it with the fstirlitz version makes it work again.

Thank you very much @fstirlitz for making Arte work again! (tested 1b2803d230c3)
I am not really a yt-dlp expert and I wonder how the "naive" user is supposed to use this now. Previously I just selected one format code (with the help of --list-formats) of a muxed stream and got video, audio and embedded subtitles in a single file.
Now I tried:
python3 -m yt_dlp -f "[width=768][language=fr]" --write-subs  https://www.arte.tv/de/videos/104432-040-A/stadt-land-kunst/

(That link will expire soon, but I guess it's the same for many.)
But I got an error that such format does not exist. I also tried other spellings from what I see in the format list, but I could not make any of them work). Looking at the video-only files with mediainfo tells me that the language is reported as English. Maybe that's the reason? (Of course the language of a video-only file does not really matter.)
So I tried:
python3 -m yt_dlp -f VOF-STMF-1630+VOF-STMF-program_audio_0-VOF --write-subs --sub-langs fr https://www.arte.tv/de/videos/104432-040-A/stadt-land-kunst/

That worked nicely.
If that is really the simplest way to use it now, maybe a comment at the top of the extractor file might help? Even if I don't know how a really naive user would find that. But I don't think there is any extractor-specific help or documentation or did I miss that?

What is holding this PR to be merged ?

Confirmed. This is because the Italian version’s manifest fails to download. I added a workaround for this so that it at least doesn’t prevent downloading other versions.

Haven't reviewed the code, but I also just successfully downloaded a video using this branch.

Listing formats with -F only lists formats that are either audio only or video only, but not both.
Downloading an Arte video merges video + audio track accordingly after downloading both the separate video and audio streams.
But doesn't Arte provide at all a HLS stream with combined audio and video that would allow direct streaming without requiring downloading + combining first ?

But doesn't Arte provide at all a HLS stream with combined audio and video ...

arte does even provide direct mp4 files with combined video, audio and subtitles.
For example: https://www.arte.tv/de/videos/038827-001-A/der-stoff-aus-dem-der-kosmos-ist-1-4/.
Using API bearer token downloading https://api.arte.tv/api/opa/v3/videoStreams?&limit=100&programId=038827-001-A
And then searching for HD videos with embedded audio tracks and burned-in subtitles:
$ jq -r '.videoStreams[] | select(.protocol == "HTTPS" and .bitrate == 2200 and .channel == "DE") | .url' /tmp/arte.videostreams.v3.page.1.json
https://arteptweb-a.akamaihd.net/am/ptweb/038000/038800/038827-001-A_SQ_0_VA_06786932_MP4-2200_AMM-PTWEB_1np8Jf9aCu.mp4
https://arteptweb-a.akamaihd.net/am/ptweb/038000/038800/038827-001-A_SQ_0_VF-STF_06786923_MP4-2200_AMM-PTWEB_1np8If9aCu.mp4
https://arteptweb-a.akamaihd.net/am/ptweb/038000/038800/038827-001-A_SQ_0_VF-STMF_06786928_MP4-2200_AMM-PTWEB_1np9Ef9aJx.mp4

The URLs correspond to DE, FR, FR+FR subtitles.

@andreygursky

But doesn't Arte provide at all a HLS stream with combined audio and video ...

arte does even provide direct mp4 files with combined video, audio and subtitles.
(...)
Using API bearer token downloading https://api.arte.tv/api/opa/v3/videoStreams?&limit=100&programId=038827-001-A

awesome find. but where does one get the api token from?

@uahim, you can find it e.g. here: https://github.com/RSS-Bridge/rss-bridge/issues/524#issuecomment-304857517.

Would be awesome if this patch could be extended so yt-dlp could download these mp4 !

Hallo all specialists!
Sorry I am very naive user. I only can copy a script and but it in tampermonkey, and ist worked in the past to download arte, it was great.
Now I cant handle yt-dlp. Is there a new script for arte downloader?? Please let me know.
Till now I use MeditheView 13.8.1. and it works also very good to download all series, films, dokus. Waiting for an elegant script.
Thanks Boobooblue

with these improvements, is possible to create a mkv with all the subs and all the languages from arte? how the yt-dlp command could look like?

first of all, many thanks to fstirlitz and @andreygursky for the extractor and token/mp4 information, respectively.
however, I believe this is getting a little off topic here but I'll try to answer the questions still:
@boobooblue
try searching gist.github.com, GreasyFork or elsewhere for working UserJS that would do this in-browser. while I think it is technically possible, I haven't seen a working solution just yet (e.g. Arte-7-Downloader) because it apparently requires wrting the script from scratch.
@felisucoibi
it is possible, but as far as I know not automatically. you'll have download all streams and subtitles separetly, then use something like mkvmerge to build one big mkv file from the video and audio streams. additionally as fstirlitz pointed out above, ffmpeg does not support webvtt so you'll have convert them to srt or any other supported format first. I don't know about mkv, probably the same problem.
you can try
yt-dlp -F <your video> and then yt-dlp -f video-format+audio-format1+audio-format2+audio-format3 <your video>  etc, though I haven't tried myself.
and generally reading this thread here and yt-dlp --help  should be a good start.
hope this helps answering the non-extractor related issues here.

Thanks Stephane Ascoet, yes MediathekView works great its easy and elegant and it works also for arte france.
Stephane: The only sad thing is the fact that nobody told me about it earlier!
Sometimes the good things are coming late but they come....  :-). All the best

Hello, I've noticed some issues when I attempt to download a video with subtitle.
yt-dlp --verbose --write-subs --sub-langs "fr" https://www.arte.tv/fr/videos/040374-011-A/the-killing-saison-1-11-20/

returns this error
[debug] Command-line config: ['--verbose', '--write-subs', '--sub-langs', 'fr', 'https://www.arte.tv/fr/videos/040374-011-A/the-killing-saison-1-11-20/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.05.18 [b14d52355] (zip)
[debug] Python version 3.8.10 (CPython 64bit) - Linux-5.4.0-121-generic-x86_64-with-glibc2.29
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.0.1-1 (setts), ffprobe 5.0.1-1
[debug] Optional libraries: Crypto-3.10.4, certifi-2019.11.28, mutagen-1.30, secretstorage-2.3.1, sqlite3-2.6.0, websockets-9.0.1
[debug] Proxy map: {}
[debug] [ArteTV] Extracting URL: https://www.arte.tv/fr/videos/040374-011-A/the-killing-saison-1-11-20/
[ArteTV] 040374-011-A: Downloading JSON metadata
[ArteTV] 040374-011-A: Downloading m3u8 information
[ArteTV] 040374-011-A: Downloading m3u8 information
[ArteTV] 040374-011-A: Downloading m3u8 information
[ArteTV] 040374-011-A: Downloading m3u8 information
[ArteTV] 040374-011-A: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Downloading subtitles: fr
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 040374-011-A: Downloading 1 format(s): VF-STF-2326+VF-STF-program_audio_0-VF
[info] Writing video subtitles to: The Killing - Saison 1 (11_20) [040374-011-A].fr.vtt
[debug] Invoking hlsnative downloader on "https://arte-cmafhls.akamaized.net/am/cmaf/040000/040300/040374-011-A/220601051357/medias/040374-011-A_st_VF-MAL.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1
[download] Destination: The Killing - Saison 1 (11_20) [040374-011-A].fr.vtt
[download] 100.0% of ~99.53KiB at  912.51KiB/s ETA 00:00 (frag 1/1)ERROR: Parse error at position 51469 (near '           <c.white.')
Traceback (most recent call last):
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1415, in wrapper
    return func(self, *args, **kwargs)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1499, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1556, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2667, in process_video_result
    self.process_info(new_info)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2925, in process_info
    sub_files = self._write_subtitles(info_dict, temp_filename)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3892, in _write_subtitles
    self.dl(sub_filename, sub_copy, subtitle=True)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2866, in dl
    return fd.download(name, new_info, subtitle)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/downloader/common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/downloader/hls.py", line 345, in real_download
    self.download_and_append_fragments(
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 519, in download_and_append_fragments
    result = append_fragment(
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 478, in append_fragment
    self._append_fragment(ctx, pack_func(frag_content, frag_index))
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/downloader/hls.py", line 260, in pack_fragment
    for block in webvtt.parse_fragment(frag_content):
  File "/home/nate/.local/bin/yt-dlp/yt_dlp/webvtt.py", line 385, in parse_fragment
    raise ParseError(parser)
yt_dlp.webvtt.ParseError: Parse error at position 51469 (near '           <c.white.')

Arte appears to be using some alternative webvtt syntax that yt-dlp can't parse.
Apart from that, i've been using your branch from some months now and this is the first problem I run into. Thanks a lot for your time and contribution.

https://arte-cmafhls.akamaized.net/am/cmaf/040000/040300/040374-011-A/220601051357/medias/040374-011-A_st_VF-MAL.vtt, lines 1568 onwards:

00:27:16.760 --> 00:27:18.400 line:83%

          <c.white.bg_black>NBL ne répond pas.</c>

Spurious blank line. A problem well known from other video hosts. It increasingly grinds me down; who knows if eventually I am going to give in and submit a patch allowing it everywhere. It’s not like even YouTube’s WebVTT is 100% standards-compliant either. (Though that ‘even’ might be giving Google too much credit.)
In this particular case the problem can be mitigated (well, punted to the player, really) by
--- i/yt_dlp/downloader/hls.py
+++ w/yt_dlp/downloader/hls.py
@@ -251,7 +251,7 @@ def is_ad_fragment_end(s):
             #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)

-        if is_webvtt:
+        if is_webvtt and len(fragments) > 1:
             def pack_fragment(frag_content, frag_index):
                 output = io.StringIO()
                 adjust = 0

which avoids invoking the parser entirely if there is only a single fragment. Your player may still fail to read the subtitle (most don’t even support STYLE blocks either, so you will still have to post-process it yourself).

Thank you very much for your answer, it solved my problem.
Once I have the STYLE block removed and the "align:center" replaced by "align:middle", FFMEPG doesn't have any problem burning the subtitles into the video.
Thanks again!

Thanks a lot for this fix! Any chance this might get merged soon? Are there still blocking issues?

Hi @pukkandan,
thank you for the work and for the merge.
I'm writing because I noticed the last (merged) version of this extractor fails to obtain video titles sometimes:
$ python3 -m yt_dlp 'https://www.arte.tv/fr/videos/110371-000-A/la-chaleur-supplice-des-arbres-de-rue/'
[ArteTV] 110371-000-A: Downloading JSON metadata
[ArteTV] 110371-000-A: Downloading m3u8 information
[ArteTV] 110371-000-A: Downloading m3u8 information
WARNING: Extractor failed to obtain "title". Creating a generic title instead
[info] 110371-000-A: Downloading 1 format(s): VOF-2332+VOF-program_audio_0-VOF
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 26
[download] Destination: ArteTV video #110371-000-A [110371-000-A].fVOF-2332.mp4
[download] 100% of 40.35MiB in 00:xx
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 26
[download] Destination: ArteTV video #110371-000-A [110371-000-A].fVOF-program_audio_0-VOF.mp4
[download] 100% of 2.39MiB in 00:xx
[Merger] Merging formats into "ArteTV video #110371-000-A [110371-000-A].mp4"
Deleting original file ArteTV video #110371-000-A [110371-000-A].fVOF-program_audio_0-VOF.mp4 (pass -k to keep)
Deleting original file ArteTV video #110371-000-A [110371-000-A].fVOF-2332.mp4 (pass -k to keep)

Whereas the patched version I was using before this recent merge did get the title correctly:
$ yt-dlp 'https://www.arte.tv/fr/videos/110371-000-A/la-chaleur-supplice-des-arbres-de-rue/'
[ArteTV] 110371-000-A: Downloading JSON metadata
[ArteTV] 110371-000-A: Downloading m3u8 information
[ArteTV] 110371-000-A: Downloading m3u8 information
[info] 110371-000-A: Downloading 1 format(s): VOF-2332+VOF-program_audio_0-VOF
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 26
[download] Destination: La chaleur, supplice des arbres de rue [110371-000-A].fVOF-2332.mp4
[download] 100% of 40.35MiB in 00:xx
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 26
[download] Destination: La chaleur, supplice des arbres de rue [110371-000-A].fVOF-program_audio_0-VOF.mp4
[download] 100% of 2.39MiB in 00:xx
[Merger] Merging formats into "La chaleur, supplice des arbres de rue [110371-000-A].mp4"
Deleting original file La chaleur, supplice des arbres de rue [110371-000-A].fVOF-2332.mp4 (pass -k to keep)
Deleting original file La chaleur, supplice des arbres de rue [110371-000-A].fVOF-program_audio_0-VOF.mp4 (pass -k to keep)

Is there a tiny bug that was left in the new v2 API extractor?

In the original PR, the title for https://www.arte.tv/pl/videos/100103-000-A/usa-dyskryminacja-na-porodowce was coming out wrong. This issue happened when fixing that.
f640e42ffa4049aa702f707be8a6c4472af9cbeb gives correct title for both cases

@pukkandan for this video below the title seems to be incorrect on current master (871a8929bcc3) and also if I revert f640e42ffa40. Looks like it picks up the title of the series instead of the title of the episode now? I would have expected "Wie werden wir Krieg los?" instead of "42 - Die Antwort auf fast alles"
$ yt-dlp -vU -f VOA-STA-428 "https://www.arte.tv/de/videos/104841-005-A/wie-werden-wir-kriege-los/"
[debug] Command-line config: ['-vU', '-f', 'VOA-STA-428', 'https://www.arte.tv/de/videos/104841-005-A/wie-werden-wir-kriege-los/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.07.18 [135f05ef6] (zip)
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Python 3.10.5 (CPython 64bit) - Linux-5.18.0-2-amd64-x86_64-with-glibc2.33 (glibc 2.33)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.0.1-3 (setts), ffprobe 5.0.1-3, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.45.1, pyxattr-0.7.2, secretstorage-3.3.2, sqlite3-2.6.0, websockets-10.2
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.07.18, Current version: 2022.07.18
yt-dlp is up to date (2022.07.18)
[debug] [ArteTV] Extracting URL: https://www.arte.tv/de/videos/104841-005-A/wie-werden-wir-kriege-los/
[ArteTV] 104841-005-A: Downloading JSON metadata
[ArteTV] 104841-005-A: Downloading m3u8 information
[ArteTV] 104841-005-A: Downloading m3u8 information
[ArteTV] 104841-005-A: Downloading m3u8 information
[ArteTV] 104841-005-A: Downloading m3u8 information
[ArteTV] 104841-005-A: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] 104841-005-A: Downloading 1 format(s): VOA-STA-428
[debug] Invoking hlsnative downloader on "https://arte-cmafhls.akamaized.net/am/cmaf/104000/104800/104841-005-A/220728160500/medias/104841-005-A_v216.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 297
[download] Destination: 42 - Die Antwort auf fast alles [104841-005-A].mp4
[download] 100% of 63.26MiB in 04:31
[debug] ffprobe command line: ffprobe -hide_banner -show_format -show_streams -print_format json 'file:42 - Die Antwort auf fast alles [104841-005-A].mp4'

Open a new issue/PR for this. Related discussion https://github.com/yt-dlp/yt-dlp/commit/f640e42ffa4049aa702f707be8a6c4472af9cbeb#r79676490


'pad block corrupted'' Exception in Rijndael Decryption

I've described my situation here
losks like PaddedBufferedBlockCipher works not correct with  RijndaelEngine(256)

Answered at stackoverflow.


Turbolinks.reload()

Please make a method that behaves just like window.location.reload() which would likewise:

reload the current document
preserve the current state of the current document (scroll position, etc.)
not create a new entry in the history

but wouldn't have to:

re-run all javascript code (because it isn't a full page reload)
make the user look at the page being rendered from scratch

and ideally would:

clear the current page cache via Turbolinks.clearCache()

This can be useful for a use case like this:

An activate/deactivate user button is present in a "user list" page. When the button is clicked, the button should change in both color and text. If the deactivated user is at the bottom part of the page, Turbolinks.reload() would automatically scroll down to the same location, without flashing the cache of the current page.

You don't have to do this, but I might as well give "asking for a feature a shot". Thanks in advance!

You can do Turbolinks.visit(window.location) to "reload" the current page.

To preserve the scroll position, and prevent creating a new history entry, you could try the following:
;(function () {
  var scrollPosition

  document.addEventListener('turbolinks:load', function () {
    if (scrollPosition) {
      window.scrollTo.apply(window, scrollPosition)
      scrollPosition = null
    }
  }, false)

  Turbolinks.reload = function () {
    scrollPosition = [window.scrollX, window.scrollY]
    Turbolinks.visit(window.location, { action: 'replace' })
  }
})()

To prevent the page from flickering as it scrolls from the top to the desired position, add the no-preview cache directive in the page's head:
<meta name="turbolinks-cache-control" content="no-preview">

Closing as solved

@domchristie / @javan ,
i am trying to use Turbolinks.visit(window.location) but it does not seem to work and page does not reload.
It works if i provide a different page url in the application.
any ideas on what I could be missing?
Thanks.

I find Turbolinks.visit() will refresh the current page


Update cache action with correct snippet for Java-Maven example.

Code of Conduct

[X] I have read and agree to the GitHub Docs project's Code of Conduct

What article on docs.github.com is affected?
https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
What part(s) of the article would you like to see updated?
Caching dependencies
You can cache your dependencies to speed up your workflow runs. After a successful run, your local Maven repository will be stored in a cache. In future workflow runs, the cache will be restored so that dependencies don't need to be downloaded from remote Maven repositories. You can cache dependencies simply using the setup-java action or can use cache action for custom and more advanced configuration.
YAML
``steps:
  - uses: actions/checkout@v3
  - name: Set up JDK 11
    uses: actions/setup-java@v3
    with:
      java-version: '11'
      distribution: 'adopt'
      cache: maven
  - name: Build with Maven
    run: mvn --batch-mode --update-snapshots verify

Additional information
This can be corrected by updating as below
- name: Cache local Maven repository   uses: actions/cache@v3   with:     path: ~/.m2/repository     key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}     restore-keys: |       ${{ runner.os }}-maven-

@vinzydev Thanks so much for opening an issue! I'll triage this for the team to take a look :eyes:

Hi @vinzydev, thanks for opening this.
The documented workflow in that section already contains a caching configuration using the cache: maven option with the actions/setup-java@v3 action (you can see the caching docs for that action here). So, there is no need to manually use the cache action in this instance.


ref: More deep error messages

Without proposed changes, errors on start are very hard to debug.
For example, if i start app with
spoof-dpi -system-proxy=true -debug
I will get:

But with given changes error message is less difficult to understand

(whole problem was with no admin privileges)

(if pkg/errors is too big to include in project, i can rewrite code using std errors.Joins)

Can't it be done with fmt.Errorf()?

@xvzc you are right! ty. I've rewrited code using fmt.Errorf


Bower is case sensitive but you can't have two packages with "same" name on Windows

There is issue when you want to use packages with "same" name (i.e. Sortable and sortable) which differs only in case.
Try:
bower install Sortable
bower install sortable

Now you have only sortable installed.
Is there any workaround or fix?

Yes:
bower install sortable1=Sortable
bower install sortable2=sortable


[HW] James Colvin

Name: James Colvin
Email<EMAIL_ADDRESS>Linkedin Profile: https://www.linkedin.com/in/colvinjames/
Attach the homework screenshots below for both step II and step III:
Step 2:

Step 3:

Thank you James here is your badge
https://api.badgr.io/public/assertions/3R5nU0dgRISFjWYpum0leA?identity__email=james.colvin%40datastax.com


Use binary mode since we return bytecode not text

Using the output of gluac makes the bytecode corrupted because of \r\n on Windows.
This PR changes streams to binary mode to prevent such problems.

Looks good 👍

Looks good 👍