api_key not used in deliver download_metadata

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description
I want to skip the 2FA when running download_metadata. To do it, I wrote a line which calls app_store_connect_api_key and after that download_metadata(app_identifier: bundle_id). According to the docs, the api_key must be automatically used, but it returns an error:
Missing password for user<EMAIL_ADDRESS>and running in non-interactive shell

Command executed
fastlane test2fa

the lane is like this:
  lane :test2fa do
    bundle_id = "com.xxx.yyy"

    app_store_connect_api_key(
      key_id: ENV["APP_STORE_CONNECT_KEY_ID"],
      issuer_id: ENV["APP_STORE_CONNECT_ISSUER_ID"],
      key_filepath: ENV["APP_STORE_CONNECT_KEY_PATH"],
      is_key_content_base64: false,
      duration: 1200,
      in_house: false
    )

    download_metadata(app_identifier: bundle_id)
  end

Complete output when running fastlane, including the stack trace and command used

 fastlane test2fa
[✔] 🚀
[14:58:30]: fastlane detected a Gemfile in the current directory
[14:58:30]: However, it seems like you didn't use `bundle exec`
[14:58:30]: To launch fastlane faster, please use
[14:58:30]:
[14:58:30]: $ bundle exec fastlane test2fa
[14:58:30]:
[14:58:30]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
+-----------------------------------------+---------+----------------------------------+
|                                     Used plugins                                     |
+-----------------------------------------+---------+----------------------------------+
| Plugin                                  | Version | Action                           |
+-----------------------------------------+---------+----------------------------------+
| fastlane-plugin-firebase_app_distribut  | 0.2.5   | firebase_app_distribution_login  |
| ion                                     |         | firebase_app_distribution        |
+-----------------------------------------+---------+----------------------------------+
14:58:31: --- Step: update_fastlane ---
14:58:31: -----------------------------
14:58:31: Looking for updates for fastlane...
14:58:31: It seems that your Gem directory is not writable by your current user.
14:58:31: fastlane would need sudo rights to update itself, however, running 'sudo fastlane' is not recommended.
14:58:31: If you still want to use this action, please read the documentation on how to set this up:
14:58:31: https://docs.fastlane.tools/actions/update_fastlane/
14:58:31: ------------------------------
14:58:31: --- Step: default_platform ---
14:58:31: ------------------------------
14:58:31: ------------------------------
14:58:31: --- Step: default_platform ---
14:58:31: ------------------------------
14:58:31: ------------------------------
14:58:31: --- Step: default_platform ---
14:58:31: ------------------------------
14:58:31: Driving the lane 'ios test2fa' 🚀
14:58:31: -------------------
14:58:31: --- Step: is_ci ---
14:58:31: -------------------
14:58:31: ---------------------------------------
14:58:31: --- Step: app_store_connect_api_key ---
14:58:31: ---------------------------------------
14:58:31: ------------------------------------------------
14:58:31: --- Step: fastlane deliver download_metadata ---
14:58:31: ------------------------------------------------
14:58:31: $ fastlane deliver download_metadata
[14:58:32]: ▸ [14:58:32]: fastlane detected a Gemfile in the current directory
[14:58:32]: ▸ [14:58:32]: However, it seems like you didn't use bundle exec
[14:58:32]: ▸ [14:58:32]: To launch fastlane faster, please use
[14:58:32]: ▸ [14:58:32]:
[14:58:32]: ▸ [14:58:32]: $ bundle exec fastlane deliver download_metadata
[14:58:32]: ▸ [14:58:32]:
[14:58:32]: ▸ [14:58:32]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
[14:58:33]: ▸ [14:58:33]: Successfully loaded 'XXX/fastlane/Deliverfile' 📄
[14:58:33]: ▸ +----------------------------+--------------------------------------+
[14:58:33]: ▸ |               Detected Values from './Deliverfile'                |
[14:58:33]: ▸ +----------------------------+--------------------------------------+
[14:58:33]: ▸ | copyright                  | 2021, XXXX	                          |
[14:58:33]: ▸ | automatic_release          | false                                |
[14:58:33]: ▸ | skip_screenshots           | true                                 |
[14:58:33]: ▸ | submit_for_review          | true                                 |
[14:58:33]: ▸ | run_precheck_before_submit | false                                |
[14:58:33]: ▸ | submission_information     | {:export_compliance_uses_encryption  |
[14:58:33]: ▸ |                            | =>false,                             |
[14:58:33]: ▸ |                            | :export_compliance_encryption_updat  |
[14:58:33]: ▸ |                            | ed=>false,                           |
[14:58:33]: ▸ |                            | :add_id_info_limits_tracking=>true,  |
[14:58:33]: ▸ |                            | :add_id_info_serves_ads=>true,       |
[14:58:33]: ▸ |                            | :add_id_info_tracks_action=>false,   |
[14:58:33]: ▸ |                            | :add_id_info_tracks_install=>false,  |
[14:58:33]: ▸ |                            | :add_id_info_uses_idfa=>true}        |
[14:58:33]: ▸ +----------------------------+--------------------------------------+
[14:58:33]: ▸ [14:58:33]: Login to App Store Connect<EMAIL_ADDRESS>[14:58:33]: ▸ -------------------------------------------------------------------------------------
[14:58:33]: ▸ Please provide your Apple Developer Program account credentials
[14:58:33]: ▸ The login information you enter will be stored in your macOS Keychain
[14:58:33]: ▸ You can also pass the password using the FASTLANE_PASSWORD environment variable
[14:58:33]: ▸ See more information about it on GitHub: https://github.com/fastlane/fastlane/tree/master/credentials_manager
[14:58:33]: ▸ -------------------------------------------------------------------------------------
[14:58:33]: ▸ Looking for related GitHub issues on fastlane/fastlane...
[14:58:33]: ▸ Found no similar issues. To create a new issue, please visit:
[14:58:33]: ▸ https://github.com/fastlane/fastlane/issues/new
[14:58:33]: ▸ Run fastlane env to append the fastlane environment to your issue
[14:58:33]: ▸ /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:135:in ask_for_login': [!] Missing password for user<EMAIL_ADDRESS>and running in non-interactive shell (RuntimeError) [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:56:in password'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:383:in login' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:355:in login'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/tunes/spaceship.rb:24:in login' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/client.rb:56:in login'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/spaceship.rb:85:in login' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:40:in login'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:22:in initialize' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in new'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in block (2 levels) in run' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:186:in run' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:17:in start'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/cli_tools_distributor.rb:114:in take_off' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/bin/fastlane:23:in <top (required)>'
[14:58:33]: ▸ from /usr/local/bin/fastlane:23:in load' [14:58:33]: ▸ from /usr/local/bin/fastlane:23:in '
[14:58:33]: ▸ /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:135:in ask_for_login': Missing password for user<EMAIL_ADDRESS>and running in non-interactive shell (RuntimeError) [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:56:in password'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:383:in login' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:355:in login'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/tunes/spaceship.rb:24:in login' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/client.rb:56:in login'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/spaceship.rb:85:in login' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:40:in login'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:22:in initialize' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in new'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in block (2 levels) in run' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:186:in run' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:17:in start'
[14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/cli_tools_distributor.rb:114:in take_off' [14:58:33]: ▸ from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/bin/fastlane:23:in <top (required)>'
[14:58:33]: ▸ from /usr/local/bin/fastlane:23:in load' [14:58:33]: ▸ from /usr/local/bin/fastlane:23:in '
+------------------+-------------+
|          Lane Context          |
+------------------+-------------+
| DEFAULT_PLATFORM | ios         |
| PLATFORM_NAME    | ios         |
| LANE_NAME        | ios test2fa |
+------------------+-------------+
[14:58:33]: Exit status of command 'fastlane deliver download_metadata' was 1 instead of 0.
[14:58:32]: fastlane detected a Gemfile in the current directory
[14:58:32]: However, it seems like you didn't use bundle exec
[14:58:32]: To launch fastlane faster, please use
[14:58:32]:
[14:58:32]: $ bundle exec fastlane deliver download_metadata
[14:58:32]:
[14:58:32]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
[14:58:33]: Successfully loaded '/XXXXX/fastlane/Deliverfile' 📄
+----------------------------+--------------------------------------+
|               Detected Values from './Deliverfile'                |
+----------------------------+--------------------------------------+
| copyright                  | 2021                                 |
| automatic_release          | false                                |
| skip_screenshots           | true                                 |
| submit_for_review          | true                                 |
| run_precheck_before_submit | false                                |
| submission_information     | {:export_compliance_uses_encryption  |
|                            | =>false,                             |
|                            | :export_compliance_encryption_updat  |
|                            | ed=>false,                           |
|                            | :add_id_info_limits_tracking=>true,  |
|                            | :add_id_info_serves_ads=>true,       |
|                            | :add_id_info_tracks_action=>false,   |
|                            | :add_id_info_tracks_install=>false,  |
|                            | :add_id_info_uses_idfa=>true}        |
+----------------------------+--------------------------------------+
[14:58:33]: Login to App Store Connect<EMAIL_ADDRESS>Please provide your Apple Developer Program account credentials
The login information you enter will be stored in your macOS Keychain
You can also pass the password using the FASTLANE_PASSWORD environment variable
See more information about it on GitHub: https://github.com/fastlane/fastlane/tree/master/credentials_manager
Looking for related GitHub issues on fastlane/fastlane...
Found no similar issues. To create a new issue, please visit:
https://github.com/fastlane/fastlane/issues/new
Run fastlane env to append the fastlane environment to your issue
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:135:in ask_for_login': [!] Missing password for user<EMAIL_ADDRESS>and running in non-interactive shell (RuntimeError) 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:56:in password'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:383:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:355:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/tunes/spaceship.rb:24:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/client.rb:56:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/spaceship.rb:85:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:40:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:22:in initialize' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in new'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in block (2 levels) in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'
from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:186:in run' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:17:in start'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/cli_tools_distributor.rb:114:in take_off' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/bin/fastlane:23:in <top (required)>'
from /usr/local/bin/fastlane:23:in load' 	from /usr/local/bin/fastlane:23:in '
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:135:in ask_for_login': Missing password for user<EMAIL_ADDRESS>and running in non-interactive shell (RuntimeError) 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:56:in password'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:383:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:355:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/tunes/spaceship.rb:24:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/client.rb:56:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/spaceship.rb:85:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:40:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:22:in initialize' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in new'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in block (2 levels) in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'
from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:186:in run' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:17:in start'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/cli_tools_distributor.rb:114:in take_off' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/bin/fastlane:23:in <top (required)>'
from /usr/local/bin/fastlane:23:in load' 	from /usr/local/bin/fastlane:23:in '
+------+------------------------------------+-------------+
|                    fastlane summary                     |
+------+------------------------------------+-------------+
| Step | Action                             | Time (in s) |
+------+------------------------------------+-------------+
| 1    | update_fastlane                    | 0           |
| 2    | default_platform                   | 0           |
| 3    | default_platform                   | 0           |
| 4    | default_platform                   | 0           |
| 5    | is_ci                              | 0           |
| 6    | app_store_connect_api_key          | 0           |
| 💥   | fastlane deliver download_metadata | 2           |
+------+------------------------------------+-------------+
+---------------------------+--------------+----------------+
|                 Plugin updates available                  |
+---------------------------+--------------+----------------+
| Plugin                    | Your Version | Latest Version |
+---------------------------+--------------+----------------+
| firebase_app_distribution | 0.2.5        | 0.2.7          |
+---------------------------+--------------+----------------+
[14:58:33]: To update all plugins, just run
[14:58:33]: $ bundle exec fastlane update_plugins
[14:58:33]: fastlane finished with errors
Looking for related GitHub issues on fastlane/fastlane...
Traceback (most recent call last):
26: from /usr/local/bin/fastlane:23:in <main>' 	25: from /usr/local/bin/fastlane:23:in load'
24: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/bin/fastlane:23:in <top (required)>' 	23: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/cli_tools_distributor.rb:122:in take_off'
22: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/commands_generator.rb:41:in start' 	21: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/commands_generator.rb:352:in run'
20: from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!' 	19: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!'
18: from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command' 	17: from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run'
16: from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call' 	15: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/commands_generator.rb:108:in block (2 levels) in run'
14: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/command_line_handler.rb:36:in handle' 	13: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/lane_manager.rb:47:in cruise_lane'
12: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/runner.rb:45:in execute' 	11: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/runner.rb:45:in chdir'
10: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/runner.rb:49:in block in execute' 	 9: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/lane.rb:33:in call'
8: from Fastfile:43:in block (2 levels) in parsing_binding' 	 7: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/fast_file.rb:216:in sh'
6: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/fast_file.rb:222:in sh' 	 5: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/actions/actions_helper.rb:69:in execute_action'
4: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/fast_file.rb:223:in block in sh' 	 3: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/helper/sh_helper.rb:16:in sh_no_action'
2: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/helper/sh_helper.rb:80:in sh_control_output' 	 1: from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/ui.rb:17:in method_missing'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/interface.rb:153:in shell_error!': [!] Exit status of command 'fastlane deliver download_metadata' was 1 instead of 0. (FastlaneCore::Interface::FastlaneShellError) [14:58:32]: fastlane detected a Gemfile in the current directory [14:58:32]: However, it seems like you didn't use bundle exec`
[14:58:32]: To launch fastlane faster, please use
[14:58:32]:
[14:58:32]: $ bundle exec fastlane deliver download_metadata
[14:58:32]:
[14:58:32]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
[14:58:33]: Successfully loaded '/XXX/fastlane/Deliverfile' 📄
+----------------------------+--------------------------------------+
|               Detected Values from './Deliverfile'                |
+----------------------------+--------------------------------------+
| copyright                  | 2021                                 |
| automatic_release          | false                                |
| skip_screenshots           | true                                 |
| submit_for_review          | true                                 |
| run_precheck_before_submit | false                                |
| submission_information     | {:export_compliance_uses_encryption  |
|                            | =>false,                             |
|                            | :export_compliance_encryption_updat  |
|                            | ed=>false,                           |
|                            | :add_id_info_limits_tracking=>true,  |
|                            | :add_id_info_serves_ads=>true,       |
|                            | :add_id_info_tracks_action=>false,   |
|                            | :add_id_info_tracks_install=>false,  |
|                            | :add_id_info_uses_idfa=>true}        |
+----------------------------+--------------------------------------+
[14:58:33]: Login to App Store Connect<EMAIL_ADDRESS>Please provide your Apple Developer Program account credentials
The login information you enter will be stored in your macOS Keychain
You can also pass the password using the FASTLANE_PASSWORD environment variable
See more information about it on GitHub: https://github.com/fastlane/fastlane/tree/master/credentials_manager
Looking for related GitHub issues on fastlane/fastlane...
Found no similar issues. To create a new issue, please visit:
https://github.com/fastlane/fastlane/issues/new
Run fastlane env to append the fastlane environment to your issue
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:135:in ask_for_login': [!] Missing password for user<EMAIL_ADDRESS>and running in non-interactive shell (RuntimeError) 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:56:in password'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:383:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:355:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/tunes/spaceship.rb:24:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/client.rb:56:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/spaceship.rb:85:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:40:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:22:in initialize' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in new'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in block (2 levels) in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'
from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:186:in run' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:17:in start'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/cli_tools_distributor.rb:114:in take_off' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/bin/fastlane:23:in <top (required)>'
from /usr/local/bin/fastlane:23:in load' 	from /usr/local/bin/fastlane:23:in '
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:135:in ask_for_login': Missing password for user<EMAIL_ADDRESS>and running in non-interactive shell (RuntimeError) 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/credentials_manager/lib/credentials_manager/account_manager.rb:56:in password'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:383:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/client.rb:355:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/tunes/spaceship.rb:24:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/client.rb:56:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/spaceship/lib/spaceship/connect_api/spaceship.rb:85:in login' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:40:in login'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/runner.rb:22:in initialize' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in new'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:163:in block (2 levels) in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'
from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' 	from /Library/Ruby/Gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:186:in run' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/deliver/lib/deliver/commands_generator.rb:17:in start'
from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/fastlane/lib/fastlane/cli_tools_distributor.rb:114:in take_off' 	from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.181.0/bin/fastlane:23:in <top (required)>'
from /usr/local/bin/fastlane:23:in load' 	from /usr/local/bin/fastlane:23:in '

Environment

🚫 fastlane environment 🚫
Stack

Key
Value

OS
11.2.3

Ruby
2.6.3

Bundler?
false

Git
git version 2.24.3 (Apple Git-128)

Installation Source
/usr/local/bin/fastlane

Host
macOS 11.2.3 (20D91)

Ruby Lib Dir
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib

OpenSSL Version
LibreSSL 2.8.3

Is contained
false

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcode.app/Contents/Developer/

Xcode Version
12.4

System Locale

Variable
Value

LANG
es_ES.UTF-8
✅

LC_ALL

LANGUAGE

fastlane gems

Gem
Version
Update-Status

fastlane
2.181.0
✅ Up-To-Date

Loaded fastlane plugins:

Plugin
Version
Update-Status

fastlane-plugin-firebase_app_distribution
0.2.5
🚫 Update available

Loaded gems

Gem
Version

did_you_mean
1.3.0

slack-notifier
2.3.2

rouge
2.0.7

xcpretty
0.3.0

terminal-notifier
2.0.0

unicode-display_width
1.6.1

terminal-table
1.8.0

plist
3.5.0

public_suffix
2.0.5

addressable
2.7.0

multipart-post
2.0.0

word_wrap
1.0.0

tty-screen
0.7.1

tty-cursor
0.7.1

tty-spinner
0.9.3

artifactory
3.0.15

babosa
1.0.3

colored
1.2

highline
1.7.10

commander-fastlane
4.4.6

excon
0.72.0

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20190701

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

faraday
1.0.1

faraday_middleware
1.0.0

fastimage
2.1.7

gh_inspector
1.1.3

mini_magick
4.10.1

naturally
2.2.0

rubyzip
2.3.0

security
0.1.3

xcpretty-travis-formatter
1.0.0

dotenv
2.7.5

bundler
2.0.2

simctl
1.6.8

jwt
2.1.0

uber
0.1.0

declarative
0.0.10

declarative-option
0.1.0

representable
3.0.4

retriable
3.1.2

mini_mime
1.0.2

httpclient
2.8.3

google-api-client
0.38.0

google-cloud-errors
1.0.0

google-cloud-core
1.5.0

digest-crc
0.4.1

google-cloud-storage
1.25.1

emoji_regex
1.0.1

jmespath
1.4.0

aws-partitions
1.326.0

aws-eventstream
1.1.0

aws-sigv4
1.1.4

aws-sdk-core
3.98.0

aws-sdk-kms
1.33.0

aws-sdk-s3
1.67.1

nanaimo
0.3.0

colored2
3.1.2

claide
1.0.3

CFPropertyList
3.0.2

atomos
0.1.3

xcodeproj
1.19.0

json
2.3.0

multi_json
1.14.1

signet
0.14.0

os
1.0.1

memoist
0.16.2

googleauth
0.13.1

google-cloud-env
1.3.3

forwardable
1.2.0

logger
1.3.0

stringio
0.0.2

ipaddr
1.2.2

openssl
2.1.2

ostruct
0.1.0

strscan
1.0.0

date
2.0.0

fileutils
1.1.0

etc
1.0.1

io-console
0.4.7

zlib
1.0.0

libxml-ruby
3.1.0

rexml
3.1.9

psych
3.1.0

mutex_m
0.1.0

webrick
1.4.2

fastlane-plugin-firebase_app_distribution
0.2.5

generated on: 2021-04-23

Hi @ManueGE
can you try below once, please
lane :test2fa do
    bundle_id = "com.xxx.yyy"

    app_store_connect_api_key(
      key_id: ENV["APP_STORE_CONNECT_KEY_ID"],
      issuer_id: ENV["APP_STORE_CONNECT_ISSUER_ID"],
      key_filepath: ENV["APP_STORE_CONNECT_KEY_PATH"],
      is_key_content_base64: false,
      duration: 1200,
      in_house: false
    )

    download_metadata(
       api_key: Actions.lane_context[SharedValues::APP_STORE_CONNECT_API_KEY]
       app_identifier: bundle_id
    )
  end

Thanks for your help @crazymanish , but still not working. It seems that download_metadata doesn't accept api_key as an argument.
Could not find option 'api_key' in the list of available options: app_identifier, force

Hi @ManueGE
Thanks!
Can you guide me, where is the implementation of download_metadata are you using any Fastlane plugin or your custom lane?

are you using below deliver download_metadata in your custom action or custom lane? I would like to know how you are using that
https://github.com/fastlane/fastlane/blob/f55ec210827f1f7e6f3b77f4b19f73f1154d23f6/deliver/lib/deliver/commands_generator.rb#L154

This is the whole code of my lane:
lane :test2fa do
    bundle_id = "com.xxx.yyy"

    app_store_connect_api_key(
      key_id: ENV["APP_STORE_CONNECT_KEY_ID"],
      issuer_id: ENV["APP_STORE_CONNECT_ISSUER_ID"],
      key_filepath: ENV["APP_STORE_CONNECT_KEY_PATH"],
      is_key_content_base64: false,
      duration: 1200,
      in_house: false
    )

    download_metadata(
       api_key: lane_context[SharedValues::APP_STORE_CONNECT_API_KEY],
       app_identifier: bundle_id
    )
  end

Okay cool, thanks!
I will come back to you...I will test myself first, how to call download_metadata command using API-key inside a lane

Hi @ManueGE
When I am trying download_metadata in the Fastfile test lane, it did not work...please see my logs are different from you.
I am very curious, how download_metadata is working in your Fastfile? Are you using any Fastlane plugin or any custom action?
please help me, how are you running download_metadata exactly?

.
.

 It is a custom action we have, I was sure it was a built-in action, but obviously I was wrong.
Probably making fixes on this lane will work. I will post the solution when ready.
Thanks a lot for pointing me into the right direction.


SailGraph and Neo4j error.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdf:Description rdf:about="http://example.org/selfvertex">
<ex:self xmlns:ex="http://example.org/" rdf:resource="http://example.org/selfvertex"/>
</rdf:Description>
</rdf:RDF>
Here is how to reproduce the problem :
gremlin> neo = new Neo4jGraph("/Users/alex/Desktop/neo-test")
==>neo4jgraph[EmbeddedGraphDatabase [/Users/alex/Desktop/neo-test]]
gremlin>
gremlin>
gremlin> sailGraph = new SailGraph(new GraphSail(neo))
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/alex/Documents/autocat/gremlin/gremlin-groovy-1.4/target/gremlin-groovy-1.4-standalone/lib/gossip-1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/alex/Documents/autocat/gremlin/gremlin-groovy-1.4/target/gremlin-groovy-1.4-standalone/lib/slf4j-log4j12-1.5.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
==>sailgraph[graphsail]
gremlin>
gremlin>
gremlin> stream = new FileInputStream(new File("/Users/alex/selfvertex.owl"))
/Users/alex/selfvertex.owl (No such file or directory)
Display stack trace? [yN]
gremlin>
gremlin> stream = new FileInputStream(new File("/Users/alex/Desktop/selfvertex.owl"))
==>java.io.FileInputStream@3484fdbc
gremlin>
gremlin>
gremlin> sailGraph.loadRDF(stream, "", "rdf-xml", null)
==>null
gremlin>
gremlin>
gremlin> sailGraph.clear()
Node[2] not found.
Display stack trace? [yN] y
org.neo4j.graphdb.NotFoundException: Node[2] not found.
at org.neo4j.kernel.impl.core.NodeManager.getNodeForProxy(NodeManager.java:477)

at org.neo4j.kernel.impl.core.NodeProxy.getRelationships(NodeProxy.java:72)

at com.tinkerpop.blueprints.pgm.impls.neo4j.Neo4jVertex.getOutEdges(Neo4jVertex.java:43)

at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.removeEdge(GraphSailConnection.java:423)

at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.deleteEdgesInIterator(GraphSailConnection.java:397)

at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.clearInternal(GraphSailConnection.java:362)

at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.clearInternal(GraphSailConnection.java:345)

at org.openrdf.sail.helpers.SailConnectionBase.clear(SailConnectionBase.java:420)

at com.tinkerpop.blueprints.pgm.impls.sail.SailGraph.clear(SailGraph.java:327)

at com.tinkerpop.blueprints.pgm.TransactionalGraph$clear.call(Unknown Source)

at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)

at groovysh_evaluate.run(groovysh_evaluate:32)

at groovysh_evaluate$run.call(Unknown Source)

at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)

at groovysh_evaluate$run.call(Unknown Source)

at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:67)

at org.codehaus.groovy.tools.shell.Interpreter$evaluate.call(Unknown Source)

at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)

at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)

at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)

at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)

at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1054)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)

at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:267)

at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)

at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1054)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)

at com.tinkerpop.gremlin.groovy.console.Console.<init>(Console.java:50)

at com.tinkerpop.gremlin.groovy.console.Console.<init>(Console.java:57)

at com.tinkerpop.gremlin.groovy.console.Console.main(Console.java:62)

By doing the same kind of program in a junit test, I have another exception :
Caused by: java.lang.NullPointerException
at com.tinkerpop.blueprints.pgm.impls.neo4j.Neo4jGraph.autoStopTransaction(Neo4jGraph.java:426)
at com.tinkerpop.blueprints.pgm.impls.neo4j.Neo4jGraph.removeVertex(Neo4jGraph.java:264)
at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.removeEdge(GraphSailConnection.java:425)
at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.deleteEdgesInIterator(GraphSailConnection.java:397)
at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.clearInternal(GraphSailConnection.java:362)
at com.tinkerpop.blueprints.pgm.oupls.sail.GraphSailConnection.clearInternal(GraphSailConnection.java:345)
at org.openrdf.sail.helpers.SailConnectionBase.clear(SailConnectionBase.java:420)
at com.tinkerpop.blueprints.pgm.impls.sail.SailGraph.clear(SailGraph.java:327)

Here is the example file used:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdf:Description rdf:about="http://example.org/selfvertex">
<ex:self xmlns:ex="http://example.org/" rdf:resource="http://example.org/selfvertex"/>
</rdf:Description>
</rdf:RDF>
It seems that an illegalstatexception (due to node already deleted) should be throwed, catched by GraphSailConnection#removeEdge and finally ignored.
The call to "this.autoStopTransaction(TransactionalGraph.Conclusion.FAILURE);" does not behave as expected. "tx.get()" returns null and if I understand correctly it means that there is no transaction, whereas there should be one at this point.
The nullpointerexception happens in the junit test as the code is encapsulated in a manual mode transaction, like this:
sailGraph.setMaxBufferSize(0);
sailGraph.startTransaction();
sailGraph.clear();
sailGraph.stopTransaction(Conclusion.SUCCESS);
Anyway, the nullpointerexception and notfoundexception seem to be due to the particular self vertex described in the file.

Is there a workaround for clearing the graph ?

btw,
are you able to recreate the error using Tinkergraph?

Interesting... with TinkerGraph everything seems ok.

do you have selfvertex.owl somewhere to reproduce this? Seems like some oddity in the Neo4j subsystem then ...

Here is the rdf file : http://paste.pocoo.org/show/523388/

Thanks.  I have looked into this, and it is definitely a Neo4jGraph-specific issue.  An RDFS inferencing test which previously passed for GraphSail-on-Neo4jGraph, now fails as well, and with the same stack trace (starting at GraphSailConnection.deleteEdgesInIterator).  Apparently the recent move to Neo4j 1.5 has broken some assumption which is true of TinkerGraph but no longer true of Neo4jGraph.  Still investigating.

Could it be that there is a new constructor argument
new Neo4jGraph(neo4jRawGraph, isFresh) that is (not) used somewhere?
/peter
On Tue, Dec 20, 2011 at 7:24 AM, Joshua Shinavier
<EMAIL_ADDRESS>wrote:

Thanks.  I have looked into this, and it is definitely a Neo4jGraph-specific issue.  An RDFS inferencing test which previously passed for GraphSail-on-Neo4jGraph, now fails as well, and with the same stack trace (starting at GraphSailConnection.deleteEdgesInIterator).  Apparently the recent move to Neo4j 1.5 has broken some assumption which is true of TinkerGraph but no longer true of Neo4jGraph.  Still investigating.

Reply to this email directly or view it on GitHub:
https://github.com/tinkerpop/blueprints/issues/196#issuecomment-3215158

I'm not using the service-based constructor, and neither is Alex apparently...

I have found a way to clear the graph without getting any error: by passing in manual mode on the transactions operated on the Neo4jGraph, as follows:
gremlin> neo = new Neo4jGraph("/Users/alex/Desktop/neo-test")
21 d?c. 2011 09:34:52 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery
INFO: Non clean shutdown detected on log [/Users/alex/Desktop/neo-test/nioneo_logical.log.1]. Recovery started ...
21 d?c. 2011 09:34:52 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery
INFO: Non clean shutdown detected on log [/Users/alex/Desktop/neo-test/index/lucene.log.1]. Recovery started ...
21 d?c. 2011 09:34:53 org.neo4j.kernel.impl.transaction.TxManager init
INFO: Unresolved transactions found, recovery started ...
21 d?c. 2011 09:34:53 org.neo4j.kernel.impl.transaction.TxManager init
INFO: Recovery completed, all transactions have been resolved to a consistent state.
==>neo4jgraph[EmbeddedGraphDatabase [/Users/alex/Desktop/neo-test]]
gremlin>
gremlin>
gremlin>
gremlin> sailGraph = new SailGraph(new GraphSail(neo))
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/alex/Documents/autocat/gremlin/gremlin-groovy-1.4/target/gremlin-groovy-1.4-standalone/lib/gossip-1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/alex/Documents/autocat/gremlin/gremlin-groovy-1.4/target/gremlin-groovy-1.4-standalone/lib/slf4j-log4j12-1.5.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
==>sailgraph[graphsail]
gremlin>
gremlin>
gremlin>
gremlin> neo.setMaxBufferSize(0);
==>null
gremlin> neo.stopTransaction(TransactionalGraph.Conclusion.SUCCESS);
==>null
gremlin>
gremlin> neo.startTransaction()
==>null
gremlin> sailGraph.getRawGraph().getConnection().clear()
==>null
gremlin>
gremlin> neo.stopTransaction(TransactionalGraph.Conclusion.SUCCESS);
==>null
Hope, it will give some lights for the resolution of the problem
Alex

Mmh,
just tested this, but seems to work as expected, please confirm.


representation and processing of expenses

When a worker submits an expense to their employer.
Are they invoices in a way?


LSTM prediction is numerically inconsistent for the last few instances.

The predictions you get may differ slightly depending on input length and position within it.  E.g., if you have 11 instances of input, you get one answer for the first 8, and a different answer for the last 3.
I write "may" as it happens to me with probability around 0.4. "Slightly" means in the order of the least significant bits of the float32 mantissa.
System information

Yes, I have written custom code, supplied below as a reprex in R using keras.
Tried on two platforms, with identical results.
Platform A:
Linux Ubuntu Ubuntu 16.04.5 LTS
TensorFlow version:
VERSION                        "1.7.0"
GIT_VERSION                    "v1.7.0-3-g024aecf414"
COMPILER_VERSION               "4.8.4"
Python version: 2.7.12

Platform B:

Linux Ubuntu 14.04.6 LTS
TensorFlow version: tried both
VERSION                        "1.12.0"
GIT_VERSION                    "v1.12.0-0-ga6d8ffae09"
COMPILER_VERSION               "4.8.5"
Python version: 2.7.6

both:

TensorFlow installed from binary.
Not a mobile device.
CUDA/cuDNN version: Not used.
GPU model and memory:  Not used

Describe the current behavior
If the first dimension of x is n, "row" i will get one value if 0 <= i < (n&-4), but a possibly different value for (n&-4) <= i < n. (These C++/Python style 0-based indices.  For R, 1-based, it's 0 < i <= bitwAnd(n, -4) versus bitwAnd(n, -4) < i <= n.)
Describe the expected behavior
Reproducible prediction from same input instance, independent of row number or input length.  I use generalized "row" for a slice of a tensor with a given fixed first index, e.g., x[i,,] or pred[i,].
Code to reproduce the issue
This is a reprex written in R.  I'd be happy to port to other languages if that's preferable.
options(digits=8)
fake <- function(shape_) {                        # arbitrary but reproducible
   array(seq_len(prod(shape_)) %% 2.71 - 1.04, shape_)
}

library(keras)
shape <- c(30,5)
model <- keras_model_sequential() %>%
   layer_lstm(units=2, input_shape=shape) %>%
   set_weights(list(fake(c(5, 8)), fake(c(2, 8)), fake(8)))

n <- 11                                           # not a multiple of 4
x <- array(rep(fake(shape), each=n), c(n, shape)) # n copies of identical input
p <- model %>% predict(x)                         # all predictions should match
p                                                 # but last n%%4 rows differ
#>             [,1]        [,2]
#>  [1,] 0.46561426 -0.22865930
#>  [2,] 0.46561426 -0.22865930
#>  [3,] 0.46561426 -0.22865930
#>  [4,] 0.46561426 -0.22865930
#>  [5,] 0.46561426 -0.22865930
#>  [6,] 0.46561426 -0.22865930
#>  [7,] 0.46561426 -0.22865930
#>  [8,] 0.46561426 -0.22865930
#>  [9,] 0.46561423 -0.22865926
#> [10,] 0.46561423 -0.22865926
#> [11,] 0.46561423 -0.22865926
(t(p)-p[1,]) * 2**26                              # the difference is low bits
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11]
#> [1,]    0    0    0    0    0    0    0    0   -2    -2    -2
#> [2,]    0    0    0    0    0    0    0    0    3     3     3
stopifnot(t(p)==p[1,])                            # all *should* be equal
#> Error in eval(expr, envir, enclos): t(p) == p[1, ] are not all TRUE

##in contrast...
x12 <- array(rep(fake(shape), each=12), c(12, shape))
p12 <- model %>% predict(x12)
stopifnot(t(p12)==p12[1,])                         # ...all is well for n == 12

Created on 2019-07-25 by the reprex package (v<IP_ADDRESS>00)
Other info / logs

This is not Build/Installation or Bug/Performance issue. Please post this kind of support questions at Stackoverflow. There is a big community to support and learn from your questions. GitHub is mainly for addressing bugs in installation and performance. Thanks!

This is a bug report.  (It is not about build, installation, or performance.  It is about correctness.)
It is a small reproducible example illustrating a bug.  (In our real code, we trained a bigger model of thousands of samples, only to find that the trained model behaved oddly.)  The same input should give the same response.  And it does, except if the length of the input isn't divisible by 4; then the remaining 1 to 3 instances differ.  To make this obvious, I repeated the same input 11 times in my reprex.
This bug in keras or tensorflow also manifests like this when applying to time series:   If predicting n days, the first n-1 predictions should match what you get if you had 1 less data of data and were generating just n-1 predictions.  But the prediction on the historical data does change!

@Quiigi Can you provide a standalone code in python to reproduce the issue? Thanks!

After looking at python basics, here's the same example as above, ported to python.
import numpy as np
from keras.models import Sequential
from keras.layers import LSTM

def fake(shape_):                                # arbitrary but reproducible
    f = np.reshape(range(np.prod(shape_)), shape_, order="F") + 1
    return f % 2.71 - 1.04

shape = (30,5)
model = Sequential()
model.add(LSTM(units=2, input_shape=shape))
model.set_weights([fake((5, 8)), fake((2, 8)), fake(8)])

n = 11                                           # not a multiple of 4
x= np.broadcast_to(fake(shape), (n,)+shape)      # n copies of identical input
p = model.predict(x)                             # all predictions should match
p == p[1]                                        # but last n%4 rows differ
assert (p == p[1]).all()                         # fails iff n%4 > 0
(p-p[1]) * 2**26                                 # the difference is low bits

Thanks for reporting the issue. Let me take a look.

@qlzh727 What did you find?

Sorry for the late reply.
I was able to reproduce the issue, and I think it is somehow happening when batch_size that is not a perfect 2^n number. I can see the value difference between batch 0-3 and 4-6. If I change the batch size to 9, then the difference is between 0-7 and 8, same for batch size 17.
The cause of this might be numerical instability for numerical libraries. Also, given the fact that the diff is so small, this is usually ignored in the unit test (the default atol and rtol for numpy assert allclose() is 1e-6). In fact if I change assert in the code to np.allclose(), the issue goes away.
Could you give more details about why this issue is concerning you in your application, and what's the specific problem it causes?

We are predicting financial time series.  In this application, snooping future
data is an insidious problem: critical to avoid, but at times subtle and hard
to notice.  It's even possible that it is avoided in research, only to creep in
in production use.  For example, you need to use info on FX rates to get
information into a common currency.  The live data source for FX rates
might "snap" the prices at a different time each day than your research
data source and inadvertently "cheat".
We have found a "snoop test" to be a useful tool: every day, we generate
predictions not just for "tomorrow", but also for the past few days.  Then
a live check can compare predictions on the overlapped days (all but the
last day).  Specifically, when predicting n days, the first n-1 predictions
should match what you get if you had 1 less day of data and were generating
just n-1 predictions.  This property is useful in testing and checking to
verify it catch bugs where a backtest accidentally snoops into future.  If
on a new day your prediction changes for an old day, regardless of how
small the change, the cause might be snooped data, that was not available
back at the original date.  But keras/tensorflow has a bug causing the final
1, 2or 3 predictions to change as you stack on additional data, even though
no change is made to the historical data!
The numerical difference is very small, on the order of floating point
noise, and ordinarily completely insignificant.  But it might be very
significant in terms of tending to falsely inflating your (small) edge in
accurately predicting financial time series. Also: if you snoop the future
data, the error you get is also very small.  Finally, this issue makes it
hard to to use our "snoop test" idea because it introduces many false
alarms, that are generally pretty indistinguishable from the real thing.
Ideally, the fix should be in the library.  The way "multiple of 4" gets
into it makes us think it is some sort of batch optimization that is flawed
in some way.  If there was a clean fix at that level, it would be ideal.
(It doesn't have to be a perfect 2^n number; any multiple of 4 is immune,
e.g., 12 in my original test case.  We use the default batch size, 32.)
We have a workaround: we tack on additional 0, 1, 2, or 3 irrelevant data
elements when predicting, to force the length to always be multiple of 4.
Then we discard as many from the prediction.  Possibly this sort of fix
could be pushed into keras, wrapped around the predict code, but hopefully
there is a more elegant solution.

Thanks for the detailed explanation.
After some debug, the lowest op I can track that cause the difference was the "recurrent_activation" function (sigmoid), where the inputs with same value will produce slight different result. The underlying implementation of sigmoid for CPU goes to Eigen, which I don't have any knowledge.
Adding @rmlarsen who is the expert for Eigen in TF team for this issue.

Thank you for the update!
Trying to replicate your debugging, I wasn't able to find a node called "recurrent_activation" or "sigmoid" in my tensorflow graph.  The closest I see is "Tanh", and its output (Tanh:0) shows the tiny discrepancies at the end.  I see the issue the nodes feeding it directly, "add_5", and idirectly, MatMul_6, BiasAdd_2, MatMul_2.  Notably, the inputs to the latter (Enter and TensorArrayReadV3) are clean.  So I would guess the difference starts around there.

I have keras.__version__ '2.2.4', and there's no "recurrent_v2.py".
At https://github.com/tensorflow/tensorflow/blob/3a9580e2c7ed4473eb63438c6f408a49e87f417d/tensorflow/python/keras/layers/recurrent.py#L2103
I see
               recurrent_activation='hard_sigmoid',
(And activation is 'tanh', resulting presumably in the node from where I was able to trace back the mod 4 issue to matrix multiplication.

In my case, the inconsistency originates in MatMul.  It's possible that activation functions 'sigmoid' or 'hard_sigmoid' have a similar issue.  This reduced example demonstrates the issue:
import numpy as np
import tensorflow as tf
a = np.broadcast_to(np.float32([.6, -.8, -.3, 0]), (5,4))
b = np.float32([[8, 1], [6, 4], [-9, 1], [0, 0]])
tf.matmul(a,b).eval(session=tf.Session())

My output:
2019-10-28 16:13:24.184178: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
array([[ 2.6999998, -2.9      ],
       [ 2.6999998, -2.9      ],
       [ 2.6999998, -2.9      ],
       [ 2.6999998, -2.9      ],
       [ 2.7      , -2.8999999]], dtype=float32)

This is an improvement because LSTM generates a graph with 268 nodes, including While loops.  The above reproducible example has just 1 simple node.  I'm not sure where exactly the discrepancy creeps in; maybe in the function evalGemm, i.e.,
void Eigen::TensorContractionEvaluatorBase<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const> const, Eigen::ThreadPoolDevice> >::evalGemm<true, true, false, 0>(float*) const

but that's just a guess.

Due the numerical instability, I don't think there is anything we can address here (the diff is smaller than the normal limit when we do the tests), I am going to close this bug.

I had hoped to learn where the instability arises.  I assume we see an artifact of optimization, trading accuracy for speed.  And as the error is within your normal tolerance, the result we see must be deemed "correct".  Therefore closing this issue is appropriate.


rgc documentation

test for pull request labeler

:x: Build NanoPyx 366 failed (commit https://github.com/HenriquesLab/NanoPyx/commit/98eec737c7 by @inesmcunha)

:x: Build NanoPyx 369 failed (commit https://github.com/HenriquesLab/NanoPyx/commit/98eec737c7 by @inesmcunha)


fix compressing_blob_test.go

Some test failed.
>go test
.\compressing_blob_test.go:32: undefined: MemoryRandomAccessStoreFactory
.\compressing_blob_test.go:96: undefined: MemoryRandomAccessStoreFactory
FAIL    github.com/fluent/fluentd-forwarder [build failed]

The https://github.com/fluent/fluentd-forwarder/commit/ca0963d1c46bbcefa15374049e04f0ba025d84ec change using the MemoryRandomAccessStoreFactory in go-ioextras.
But test isn't change, so test failed.
I fix it.

thx!


feat: $sendPoll , $getPoll, $getPollVotes , PERM: SEND_POLLS

Bumps discord.js to 14.15.1
Adds:

Polls

SEND_POLLS (permission)
$sendPoll
$endPoll
$getPollVotes

Refactor

$createBanner (use client.user.setBanner())

All three of those need oauth to work properly, no point in having them anymore.

All three of those need oauth to work properly, no point in having them anymore.

which ones?

All three of those need oauth to work properly, no point in having them anymore.

which ones?

removeappperms, addappperms, setappperms
all require oauth

Still need more information on the auth?


Exports viewless state provider

To address Issue #26

I was able to import the viewless state provider on my testing.  LGTM


Fix map/apply by adding serialization for builtins

Add serialization for builtins. Addresses the tests in PR #1493

added json_dumps and object_hook for python builtin functions
data_loads is not consistent for all serializations. For fastmsgpack, it uses pandas to decode to a Series; while other methods decode to a list.

One issue I have with this idea is that users can create a string and then apply exec or eval to it and send that to the server to execute. I think sending functions might only be valid for the pickle backend where the server maintainers have decided security is not a concern.

As a general rule for the blaze server I would prefer if we assumed limited trust for any client that is connected. I think this is important because the blaze server box will have the credentials to access any of your data or have the data available directly. There is also the concern that the box is a shared resource and people will misuse the box because they do not know better. For example, a non-malicious denial of service could come from a data scientist using np.savez on a very large array. I think that this security concern is enough to warrant implementing this with a whitelist. This would allow server administrators to decide which functions are allowed or which are disallowed. I think that we could possibly have some sentinel value that says, "all are allowed" but I would strongly suggest not making this the default. I also don't think that we should manage a whitelist ourselves because then blaze is in the business of doing security audits.

One issue I have with this idea is that users can create a string and then apply exec or eval to it and send that to the server to execute. I think sending functions might only be valid for the pickle backend where the server maintainers have decided security is not a concern.

@llllllllll
It's not implemented with an eval. It'll only look for the functions in the pandas, numpy or builtin namespace. But we might still need to limit this further as you suggest - will investigate further.
Could you please label this PR as Work In Progress?

Providing access to the builtin namespace is providing exec and eval. For example:
In [7]: bz.data(["print('hello from blaze!')"]).map(eval, '?string')
Out[7]: hello from blaze!

0  None

Providing access to the builtin namespace is providing exec and eval. For example:

I understand - thanks for the example. Will add a check for it.

Thanks for the input @llllllllll, it's good to have your usecase factored in as part of this work.
What about the following, to more clearly separate the locked-down server config from the trusted-client server config (all names are changeable):

We create a separate json_dumps_trusted multipledispatch that is a superset of json_dumps with implementations that include whitelisted builtins, numpy, and pandas functions.
We create a object_hook_trusted multipledispatch to go with json_dumps_trusted.
We create json_trusted and msgpack_trusted SerializationFormat instances that use these *_trusted serializers / deserializers.
We ensure that these *_trusted serialization formats are not included in the default formats argument to Server.
The only way for a user to have *_trusted serialization formats is via explicitly instantiating Server() for now -- we don't provide commandline arguments to enable them.

@llllllllll overall, our target user is someone in an environment where using IPython notebook kernels with IPython notebook is acceptable, i.e., within a trusted environment, behind a firewall, etc.

That seems to satisfy both of our use cases and protects users who don't know the difference. On a somewhat related note, which serialization formats have you been using for production?

which serialization formats have you been using for production?

Currently, json, but we might use msgpack in the future.

I would highly reccomend the fast_msgpack format for use with numpy and pandas, this was a really big performance improvement for us.

Closing -- this PR was absorbed into #1504.


Explore doesn't include private but accessible repos

Currently, there is no easy way to find a repo by searching, when there are many private repos in organisations. In our setup we heavily use Organisations and Teams where 99% of the Repos are Private, and none of the Private repos in an Organisation show up under Explore, either browsing or searching.
It would be really useful to include all repos that a User has access to, under Explore.
By extension, the API /api/v1/repos/search should also return all accessible Repos, including Private repos in an Organisation that the User has access to via a Team.

My Explore -> Repositories is also empty, all repos on my server are set to private. Nothing shows up, doesn't matter if I'm logged in  as admin, or not, got access to the organization, repo owner, member, etc..

I can have at look at this

Which version of Gogs are you running ? I can't reproduce this :sweat:

<IP_ADDRESS>2

Mine is <IP_ADDRESS>3.
If I go into Settings for a repo owned by an Organisation, and change Visibility to Not Private, it shows up in Explore. Changing it back to Private, and it no longer shows in Explore. I am in a Team with access to the Repo - and I'm also Admin.
I've also just tested with Repos owned by myself, and the same occurs. Private, and it's not shown in Explore. Not Private, and it is shown.

I've now upgraded to <IP_ADDRESS>6 (using the .deb packages from packager.io), and Explore is still not showing Private-but-allowed repos.

Nvm, I can reproduce it. Working on it :D

any progress on this? @bkcsoft

@Unknwon would you still be willing to examine a pull request for this issue?  I noticed you assigned it to yourself on 7/2/2016 but I was curious and wanted to see if I could fix this issue locally.

@master-d yes you can try, it is somewhat a complicated query.
Just keep in mind there is no guarantee that any proposed PR will be merged.

no worries I just wanted to check to see if you were actively working on it and the "Contributing-code" wiki had mentioned that one should ask before contributing.  I'll try to familiarize myself with the database and go from there.

I made a pull request for this issue.  This is my first one on github so I hope I didn't screw it up too badly
https://github.com/gogits/gogs/pull/4273

Patch has pushed to fix this issue, please test on develop branch or https://try.gogs.io.

I built 96ae0129ef4db2bb5c7fc3bdcbc128700f116ac1 on the develop branch and unfortunately when logged in do not see my private repos.
To be clear, the account I am logged in with is an administrator who is explicitly listed as a contributor with write access to private repo. Still the private repo is not visible on /explore/repos.

@hochhaus currently, you need search, private repos are not listed with empty query string.

@Unknwon Thanks. I just tested and can confirm that searching does show the private repos in my build at https://github.com/gogits/gogs/commit/96ae0129ef4db2bb5c7fc3bdcbc128700f116ac1.

@Unknwon Ijust noticed that if you do a search for a repo when not logged in a 500 is returned.
https://try.gogs.io/explore/repos?q=test (make sure that you are not logged in)

@hochhaus thanks, fixed.

Thanks. Confirmed that your fix works as intended.

sorry about that.  Can I ask what caused the 500 error when performing a search while not logged in?  I get the same behavior on my local branch.  Is it attempting to set the int64 "UserID" variable to a null value which isn't allowed?

@master-d

Is it attempting to set the int64 "UserID" variable to a null value which isn't allowed?

Yes, you can look at my fix patch, is very simple fix, just always need to keep in mind.

Also should the search for empty string return all repositories the user has access to? I should be able to add that in without too much problem

Yes, should be very similar in terms of SQL, you can try that.

Ok.  I noticed a search occurs immediately when the "Explore" page is loaded and it seems to be returning results for GetRecentUpdatedRepositories in repo.go on page load.  Should that remain the same or should I load all of the accessible repositories on page load and run the same query if the user enters an empty string in the search box?

@master-d You should use the same function but include same functionality as search (include private but accessible repo).

PR #4289 should handle empty string searching on page load and manually

Patch has pushed to fix this issue, please test on develop branch or https://try.gogs.io.

@Unknwon Thanks. I tested from the developbranch and I can confirm that this patch works as intended (both for authenticated and unauthenticated users).


get schemas a little smarter and include definitions

Coverage remained the same at 88.843% when pulling f07d8189a09e60228405d119f5558226ed372ccf on gacord23:master into ebe15783b63ea8c3cb4eca2d56b2fe2991b0dcc0 on SciSpike:master.


major cleanup.

@yunong @micahr
I think this is 'complete' now from my perspective, so would appreciate some feedback. The Error constructor signatures have been unified, and I have a restify branch integrated with all unit tests working.
Pretty much everything is backwards compatible, except for one change. Errors sent to res.send() render with a code value that is basically the name of the error minus the word 'Error',  i.e., if you call:
res.send(new errors.BadDigestError('indigestion));
// response renders as:
{
    code: 'BadDigest',
    message: 'indigestion'
}

There's an 'InternalError' (different from InternalServerError), which is used in the uncaught exception scenarios. When those errors are rendered out, you get:
foo.bar();  // bad code, server will blow up
// response renders as :
{
    code: 'Internal',
    message: ''
}

Which can seem weird, because 'Internal' doesn't have a whole lot of meaning. There was some logic in the old impl which would strip/add the 'Error' suffix selectively for the code property depending on if it was InternalError or not, but this seems too one off case. I'd prefer we just rename 'Internal' to something else like 'InternalUncaught' if we need a better name, and/or to differentiate from InternalServerError.

Looks great, thanks for all the work. Taking a closer look now -- but please use make instead of gulp.

Unfortunately this isn't part of the pull request, hence there's no where in code for me to comment; I don't think being able to override the http status code in an error is a good idea. If someone wants to send a different error code, they should create a new error, not override the status code.

This is a little bit hard to read through, since not all of the code is part of this PR. I would say 1) remove gulp and use make, and 2) merge and then I'll take a look at the codebase as a whole.
But overall this looks quite good, so kudos!

Yeah, I was just thinking the same, I will just merge everything down and we can start discussion around the entire codebase.
I'm maintaining backcompat with existing API today, so overridable status codes exist for both HttpErrors and RestErrors. Would be happy to make that a breaking change for HttpErrors, but I think RestErrors should still be configurable.

BTW, I'm going to merge down the code as is so we can look at it whole. I'll work on the make stuff in a separate PR.


[elasticsearch] Revisit readinessProbe

We should revisit the readinessProbe - see https://github.com/elastic/cloud-on-k8s/issues/2248 for what ECK does and why. The TL;DR version is just to call /, but note that there are some subtleties with HTTP response codes and the ES version.

Originally posted by @pugnascotia in https://github.com/elastic/elasticsearch/issues/53426#issuecomment-604547622

Helm Chart readiness probe
We are using a script which run /_cluster/health?timeout=0s endpoint to check that Elasticsearch is started, then /_cluster/health?wait_for_status=green&timeout=1s endpoint to check that Elasticsearch is cluster is OK:
https://github.com/elastic/helm-charts/blob/a0e8d77c6a292636ef771e78f24b04b6ea1158bb/elasticsearch/templates/statefulset.yaml#L215-L227
Note that informations for these checks are retrieved from masters as local query parameter is not set (https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params).
The desired behaviour here is that if the data nodes are unable to talk to their master nodes for whatever reason, then the data nodes will become Unready and therefore be removed from the Service load-balancer until the master nodes are available again (quoting @fatmcgav from https://github.com/elastic/helm-charts/pull/380#issue-343356795).
ECK readiness probe
ECK is using a different approach which check / endpoint to only ensure that Elasticsearch node is started:
https://github.com/elastic/cloud-on-k8s/blob/71a3725335596acdb6b7a13c917f9018d9953a6f/pkg/controller/elasticsearch/nodespec/readiness_probe.go#L67-L77
The intention is to check only the single node independently from overall cluster health/cluster membership to know whether it is principally ready to enter into operation (https://github.com/elastic/cloud-on-k8s/issues/2248#issuecomment-564644499) to avoid issue during rolling upgrade where all nodes loose their ready state and are deleted while master nodes are rolled (more detail in (https://github.com/elastic/cloud-on-k8s/issues/1748#issue-495227831).
Note that from what I know a failed readiness probe should only remove the pod from service so no traffic is sent to it untill readiness probe is successfull again, it shouldn't kill the pod (unless ECK operator force killing pods not ready).

Closed by #586


Cable Length under "Cables" concats length with the length_type id instead of name

Environment

Python version:  3.6
NetBox version:  2.5

Steps to Reproduce

Create a cable connection with length and length_type specified (Ex. 6 "Inches"
View connection in Devices->Cables

Expected Behavior
Cable length defined as 6 Inches
Observed Behavior
Cable length is defined as 62000

Working on a PR for this issue now.


add lein support

This is the best reference I could find: https://github.com/technomancy/leiningen/issues/1820


Can't cook sea turtles

"+cook sea turtle" results in being told it's not a valid item

This was fixed by andre, and will be released in the next update.


Implement, Test, and Verify functionality with v1,v2,v3

Implement, Test, and Verify functionality with v1,v2,v3

:tada: This issue has been resolved in version 1.2.0-develop.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add augsynth tests

Codecov Report
Base: 87.37% // Head: 95.20% // Increases project coverage by +7.83% :tada:

Coverage data is based on head (aab7107) compared to base (c890d99).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #10      +/-   ##
==========================================
+ Coverage   87.37%   95.20%   +7.83%
==========================================
  Files          12       13       +1
  Lines         610      626      +16
==========================================
+ Hits          533      596      +63
+ Misses         77       30      -47

Impacted Files
Coverage Δ

pysyncon/augsynth.py
98.41% <100.00%> (+74.60%)
:arrow_up:

pysyncon/base.py
75.90% <100.00%> (+0.29%)
:arrow_up:

tests/test_augsynth_basque.py
100.00% <100.00%> (ø)

tests/test_synth_basque.py
100.00% <100.00%> (ø)

tests/test_synth_germany.py
100.00% <100.00%> (ø)

tests/test_synth_texas.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Add kubectl-plugin lint to pre-commit

Why are these changes needed?
Adds lint check for kubectl-plugin to pre-commit steps.

Related issue number

Checks

[ ] I've made sure the tests are passing.
Testing Strategy

[ ] Unit tests
[ ] Manual tests
[ ] This PR is not tested :(

cc @MortalHappiness would you mind reviewing this PR? Thanks!

Hi @MortalHappiness please take another look! Thanks!


Remove duplicate calls to bypass-local-dns.yml

common.yml should call verify-agent-os.yml, to verify OS in any pipeline using common.yml
bypass-local-dns.yml is now called from verify-agent-os.yml
Removed duplicate calls to verify-agent-os.yml in pipelines already calling common.yml

/azp run js - template - tests

/azp run js - smoke-tests

/azp run js - template - tests

/azp run js - smoke-tests


fix: Mitigate breaking changes to VPA upgrad v2.x

What does this PR do?
🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.

Motivation

Resolves #288

More

[x] Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
[x] Yes, I ran pre-commit run -a with this PR

For Moderators

[ ] E2E Test successfully complete before merge?

Additional Notes

how has this been validated, are there any schema changes for the chart?

how has this been validated, are there any schema changes for the chart?

the breaking changes introduced will affect existing implementation only during when the intent is to migrate seamlessly without any outages
the new version of the chart introduces new set of default values -
The certificate creation process was changed from using OpenSSL to kube-webhook-certgen to simplify the process. It still uses the same configuration keys (.Values.admissionController.certGen), which makes it impossible to reuse the values from a previous install.
You can mitigate this change by setting the correct image for the upgrade:
helm upgrade <release name> fairwinds-stable/vpa --version 2.0.0 --reuse-values \
  --set "admissionController.certGen.image.repository=registry.k8s.io/ingress-nginx/kube-webhook-certgen" \
  --set "admissionController.certGen.image.tag=v20230312-helm-chart-4.5.2-28-g66a760794"

these new values should be installed when it is installed fresh unless you are trying to do a in place replacement which should be done as shown above .
Furthermore, Previously, the webhook creation was handled by the admission controller itself. This had the downside that Helm is not in control of the resource and therefore required the cleanupOnDelete job.
This version disables the self-registration by the admission controller and creates the MutatingWebhookconfiguration using Helm.
You can either:

Migrate the MutatingWebhookconfiguration by:

adding the label app.kubernetes.io/managed-by: Helm
adding the annotation meta.helm.sh/release-name: <release name>
adding the annotation meta.helm.sh/release-namespace: <release namespace>

delete the configuration and it will be recreated by Helm

or keep the configuration as it is and Helm will ignore it. Execute the tests, to make sure everything works.

Also, the cleanupOnDelete configuration is obsolete.


Io bug fixes

To validate this pull request against other sbt modules, please visit this link.

The scripted tests succeeded on my vm after I published these changes locally.


custom babylon parser support

.babelrc
{
  "parserOpts": {
    "parser": "custom-fork-of-babylon-on-npm-here"
  }
}

if i use a custom babylon parser in babel, is there a similar option that i can parse eslint too?
module.exports = {
  root: true,
  parser: 'babel-eslint',
  parseOptions: {
    // how to pass in "custom-fork-of-babylon-on-npm-here"
  }
};

you would have to replace parser: babel-eslint with something else


Very long term tracking

The docs doesn't make it very clear :
Is it possible to track position (send http requests in background) even is the app is closed for a long time (for some days to some months), even if the devices reboots ?
Does-it depends of the accuracy or the frequency ? Does-it depends on the OS (android or iOS).
Thanks !

Definitely yes on Android (options stopOnTerminate: false and startOnBoot: true).
On iOS there is plan to have it implemented too.

Very interested to have it on iOS. I want to build an app that track users that are one the move (like travelling).
Sure for the inbuilt http posting ! (which work great with my fix ;) )
How can I help ? What do you need to achieve this (time, resources, other) ?

Basically time. But you can help with testing once it's implemented.


Add commandline_app flag for Apps without a GUI.                                                                              On Mac, this will create an App bundle which simply opens a Terminal and runs your code.                                                                              I believe that on Windows, there's nothing special to be done; I think it will Just Work like this by default.

This fixes Issue #16.

[ ] Verify this on windows before submitting.

Codecov Report

Merging #17 into master will decrease coverage by 2.36%.
The diff coverage is 11.11%.

@@            Coverage Diff             @@
##           master      #17      +/-   ##
==========================================
- Coverage   54.48%   52.12%   -2.37%
==========================================
  Files           5        6       +1
  Lines         156      165       +9
==========================================
+ Hits           85       86       +1
- Misses         71       79       +8

Impacted Files
Coverage Δ

src/mac_commandline_app.jl
0% <0%> (ø)

src/BuildApp.jl
86.86% <50%> (-0.77%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7c6150...6e54c80. Read the comment docs.

Pull Request Test Coverage Report for Build 81

1 of 9 (11.11%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-2.4%) to 52.121%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/BuildApp.jl
1
2
50.0%

src/mac_commandline_app.jl
0
7
0.0%

Totals

Change from base Build 79:
-2.4%

Covered Lines:
86

Relevant Lines:
165

💛  - Coveralls


chore(Condensed): shorten some names

As discussed on #15321

bors d+

Thanks!
bors merge

Sorry, I messed up merging master... I opened #17140 instead because I'm bad at git. Could I have a d+/merge on that instead?

Let's try again
bors merge

No deprecations?

This is code that was initially added only a few days ago so I think it's fine not to add deprecations


feat:support ttl function for list

user can use ExpireList to set ttl for list

@xuyukeviki tks your pr，good job！


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target_eb2013d3-4637-4e4e-afa0-d5a15a951861/pull/1 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target_eb2013d3-4637-4e4e-afa0-d5a15a951861.git

Docs Build status updates of commit a7d4b94:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-UserAccountBuildPermissionInsufficientError] Owner v-alji no longer has Read permission of the repository https://github.com/opstest2/docs-archive-test-source. It might be due to recent GitHub changes on SSO. Please ask the owner v-alji to logon to Docs Portal https://ops.microsoft.com once again. If it is not the case, please ask anyone of Admin permission of the repo to assign a new owner via Docs Portal https://ops.microsoft.com

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Improvements to testing suite

Since testing.h a header file, I guess it would make a little more sense to me if it were in base or something.
Furthermore, it would be nice if the compare_relative and associated methods returned a bool that one could pass to the main program and know whether a test failed. This would be helpful again for the automation of testing, where we want to just run the test script and know right away whether everything passed.
Finally it would be nice to add a pass() method to the rootLogger so that we can avoid to type CoutColors::green all the time.

One could also include in testing.h a bool evaluateTestResult() method that replaces code like
    if(lerror) {
        rootLogger.error("At least one test failed!");
        return -1;
    } else {
        rootLogger.info("All tests " ,  CoutColors::green ,  "passed!" ,  CoutColors::reset);
    }

that one finds all over the place. I guess then the pass() method might not be necessary. Then one could end a program with something like
return evaluateTestResult(lerror);

Also right now not every test is run with multiGPU. For some tests this is not necessary, but e.g. the RHMC is not tested with multiGPU in the script at the moment.

Another thought: It would be nice if we were a bit more consistent with our naming scheme in CMakeLists. For example, maybe we can ask that all test executables start with test or something. I keep forgetting how executables are named and have to keep looking into the CMakeList to figure it out...
Or is there some other more convenient way to address this?

In response to the last point about looking in the CMakeLists, this was fixed by adding a script that allows make autocomplete.

OK here is what I will try to do at some point:

[ ] use the methods in testing.h more in the existing tests
[ ] say something in the documentation about testing.h
[ ] implement that evaluateTestResult() thingy
[ ] add the pass() thing to rootlogger?
[ ] have the compare... methods return bools?


Still colliding after removing the body from the world

Hello guys,
I'm working on a personal platform game using this library and wanted to know what is the best way to remove a body from the world.
Let me explain my use case, I'm creating two squares, let's say the first one is the player and the second one is a reward. I want to remove the reward after the player collides. That's easy using Matter.World.remove method because disappear from the game but the player still collides when going to the reward position again and again.
The following code shows how I build the items and check if player and reward collide:
            let view = Matter.Composite.create();
            let box = Matter.Bodies.rectangle(400, 400, 600, 20, { isStatic: true });
            let player = Matter.Bodies.rectangle(400, 300, 50, 20);
            let reward = Matter.Bodies.rectangle(650, 300, 30, 30);
            Matter.Composite.add(view, [box, player, reward]);
            Matter.World.add(engine.world, view);

            Matter.Events.on(engine, 'afterUpdate', function() {
                let collisions = Matter.Query.collides(player, [reward]);
                if(collisions.length > 0) {
                    console.log('keep colliding'); // shows the message even after the reward item has been removed
                    Matter.Composite.remove(view, reward);
                }
            });

Is it a better way to remove items from Composite or World items? Is there something that I'm missing and to be updated in some way?
Regards

I don't think Matter.Query.collides checks if the body is in the world, that could be the problem. If so an easy fix would be to set a flag on it after you remove it and check for that before the collision.
That said I'd recommend looking at the engine collisionStart event which will only include collisions for bodies in the world.

I tried both solutions and worked fine. Personally, I think the proper one is to use the collisionStart event, seems to be more elegant isn't it?
Thanks @liabru


[bug] Compiling DataNode(galaxyengine) needs gcc 9 or later

phenonmenon
When I followed the source installation guide to compile galaxyengine, I found gcc cannot support -fuse-ld=lld option, the error messages of CMakeFiles/CMakeError.log looks like:
Linking C executable cmTC_6284a
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_6284a.dir/link.txt --verbose=1
/usr/bin/gcc  -fno-omit-frame-pointer  -DC_LD_LLD_RESULT    CMakeFiles/cmTC_6284a.dir/src.c.o  -o cmTC_6284a -fuse-ld=lld
gcc: error: unrecognized command line option '-fuse-ld=lld'; did you mean '-fuse-ld=bfd'?
MakeFiles/cmTC_6284a.dir/build.make:97: recipe for target 'cmTC_6284a' failed
gmake[1]: *** [cmTC_6284a] Error 1

analysis
The help of gcc(/usr/bin/gcc -v --help 2>&1 | grep -E "fuse-ld") shows that is -fuse-ld=lld is not a avaiable option. Then, I checked the GNU manuals, I found which version of gcc you recommended in the installation guide DO NOT support -fuse-ld=lld in your cmake receipe in total.

gcc 7.4 options for linking: no -fuse-ld=lld
gcc 7.5 options for linking: no -fuse-ld=lld
gcc 8.4 options for linking: no -fuse-ld=lld
gcc 8.5 options for linking: no -fuse-ld=lld
gcc 9.1 options for linking: support -fuse-ld=lld

According to the release of gcc, it is obvious that the neccessary gcc version is 9.1 or later .
suggestion
Update the source installation guide GCC 7 with GCC 9.
environment
galaxyengine main/4cc03124f8530c03c948d2b9d91624ca196e8f22
openSUSE Leap 42.3
gcc 7.4.0

Duplicated with https://github.com/ApsaraDB/galaxyengine/issues/11. Close this pls.


Edge: ice-lite support

Description
this adds ice-lite support. Used to work with @ggarber's SFU and still does with mine.
Hard to test without an SFU (or other server) unfortunately.

LGTM


How do you calculate the accuracy of the tokens?

I read that the accuracy of the tokens is about 77%, so how to calculate or any script?

Given a table image, if all structure tokens for this image are all classified correctly, it is correct, or it is labeled as wrong. Suppose you have  10000 table images, 8000 are recognized correctly, then the accuracy is 80%.


Prateek

Fixed all Sections URLS
All redirect working correctly now

Hi @prateekmohanty63  Few points from my side

Can you please mention the issue related to this PR
From next time, please try to change a few files in 1 PR. It is really tough to review so many files at once. And most of the changes you made are redundant, which we are about to change (have a look #163 ), though we appreciate your efforts 😊😊.

LGTM @arpit456jain

Hi @prateekmohanty63 Few points from my side

Can you please mention the issue related to this PR
From next time, please try to change a few files in 1 PR. It is really tough to review so many files at once. And most of the changes you made are redundant, which we are about to change (have a look [Style] Add Template Inheritance #165 ), though we appreciate your efforts 😊😊.

Yeah sure I will ensure that I write about the issue , next time
Actually when I ran the project in my local server I saw these redirecting urls not working , so I though of taking the issue up
By the way Thanks for your advice regarding best practices of sending pr :)  . @KKhushhalR2405

@prateekmohanty63 ur issue was to correct only logo path
#147
but u have changed all the urls without informing us
some was working on this issue #166
@KKhushhalR2405

@arpit456jain  I have only corrected logo paths from all files, not changed any other urls.

because all files logo redirect was not working properly

@prateekmohanty63 for that we are working on template inheritance  for now i a, closing this PR
when the template inheritance is done u can check urls if any of the url is wrong u can work on it


Upgrade reactor and use QueueWrapper to support reactor-based instrumentation

There was a new feature added to reactor-core that makes it possible to improve the reactor-based instrumentation by wrapping the queues used by reactor: https://github.com/reactor/reactor-core/pull/2566

Forgive me, what is the benefit? Is it performance or it allows some additional functionality?

@tony-clarke-amdocs in theory, it should cover more cases (queues are at the core of everything that Reactor does).
Also (again, in theory) it should be more performant.
Once integrated (see #1858), we can run Sleuth's benchmarks and compare the performance with the onEachOperator approach.


Create JavaScript loader block in base template

Apps (such as owning-a-home) will likely have their own template that overrides the base template's javascript block (see example). The problem with this is if we want to use the JS loader that's in the base template it will need to be re-created in each child template. This PR creates a javascript_loader block that adds a javascript loader function to the page before the javascript block, so that child templates can choose to only override the javascript block and utilize the JS loader in the parent template.
Additions

Creates javascript_loader block that contains a JS loader function.

Testing

Blog page should still show common.js, email-signup.js, and filterable-list-controls.js in the JS network tab in dev tools.

The only issue is that you always have to call super because the common.js stuff is in the block.

Also, if you remove the function then the s variable has global scope.

@sebworks Okay, I re-added the function. It's in use in https://github.com/cfpb/cfgov-refresh/pull/3818

Ship it

Everyone blame @Scotchester 's shoddy review if this doesn't work 😜


MSUnmerged: Wrong input filters applied when reading from MongoDB

Impact of the bug
MSUnmeged
Describe the bug
Not exactly a bug since we are already running quite safe with the current code, which already has a workaround for this.
When reading from MongoDB we cannot apply filters of the sort:
mongoProjection = {
    "_id": False,
    "name": True,
    ...
    "files": False
}

Because we are getting this result:
In [3]: from WMCore.MicroService.MSUnmerged.MSUnmergedRSE import MSUnmergedRSE

In [4]: rse = MSUnmergedRSE('T2_US_Wisconsin')

In [5]: rse.readRSEFromMongoDB(msUnmergedColl, useProjection=True)
---------------------------------------------------------------------------
OperationFailure                          Traceback (most recent call last)
<ipython-input-5-52e7cdc3e352> in <module>
----> 1 rse.readRSEFromMongoDB(msUnmergedColl, useProjection=True)

~/WMCoreDev.d/WMCore/src/python/WMCore/MicroService/MSUnmerged/MSUnmergedRSE.py in readRSEFromMongoDB(self, collection, useProjection)
     99         if useProjection:
    100             mongoProjection = self.buildMongoProjection()
--> 101             mongoRecord = collection.find_one(self.mongoFilter, projection=mongoProjection)
    102         else:
    103             mongoRecord = collection.find_one(self.mongoFilter)

...

~/WMCoreDev.d/WMCore.venv3/lib64/python3.6/site-packages/pymongo/helpers.py in _check_command_response(response, msg, allowable_errors, parse_write_concern_error)
    157
    158             msg = msg or "%s"
--> 159             raise OperationFailure(msg % errmsg, code, response)
    160
    161

OperationFailure: Cannot do exclusion on field files in inclusion projection

And the reason is because we are hitting some MongoDB functionality constraints here. It simply does not allow to have exclusion fields inside inclusion filters. In those cases the only field we are allowed to exclude is _id, which is basically what we need to exclude for sure. While if we want to skip some of the rest, they simply must not be present into the filter. Here is some discussion on the topic from the MongoDb comunity with some more  details [1]
[1]
https://www.mongodb.com/community/forums/t/projection-does-not-allow-exclusion-inclusion-together/31756
How to reproduce it
Just execute the steps from the code snippet above, trying to read from MongoDB with a complex inclusion filter which is having more than a single exclusion at the _id field.
Expected behavior
Create the mongoDB filter properly so it could be applied at read and write operations with equivalent result without failing any of them.
Additional context and error message
None

@todor-ivanov to provide the full picture, can you please link the PR that injected this bug as well (to the initial description).
And what is not clear to me as well is whether this is a bug in production? Or is it only in testbed? Or both? Thanks

Hi @amaltaro Thank you for taking a look. I really do not remember the PR introducing it. I think it should have been one of the initial MongoDB related ones. And this is not exactly a bug currently reviling itself neither in Prod nor in PreProd, since we never hit it inside the code because we do not query read with filters [1],  while for the write operations we go through the fields one at a time [2].  But it is good to be fixed anyway, because we may spare some CPU cycles and also to avoid hitting it by chance in the future. I happened to discover it while writing the unit tests yesterday. I remember having to deal with that for write operations and we also have it mentioned in our discussions, but somehow we missed the read with filters operation.
[1]
https://github.com/dmwm/WMCore/blob/d728fc69e712e1e0ba50c7247065badf41eb56c9/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L828
[2]
https://github.com/dmwm/WMCore/blob/d728fc69e712e1e0ba50c7247065badf41eb56c9/src/python/WMCore/MicroService/MSUnmerged/MSUnmergedRSE.py#L144

And the fix should be relatively simple. It may go away with a single iteration at the top in the buildMongoProjection method while creating the filter:
https://github.com/dmwm/WMCore/blob/d728fc69e712e1e0ba50c7247065badf41eb56c9/src/python/WMCore/MicroService/MSUnmerged/MSUnmergedRSE.py#L71
I can work on that later today.