🐛 [firebase_crashlytics] Errors reported with recordError have custom keys mixed up

Bug report
Describe the bug
When errors are reported with recordError close to each other, their custom keys get mixed up.
                  WidgetsFlutterBinding.ensureInitialized();
                  await Firebase.initializeApp();
                  FlutterError.onError = FirebaseCrashlytics.instance.recordFlutterFatalError;
                  PlatformDispatcher.instance.onError = (final error, final stack) {
                      FirebaseCrashlytics.instance.recordError(error, stack, fatal: true);
                      return true;
                   };
                  final firebase = FirebaseCrashlytics.instance;

                  await firebase.setCustomKey('log category', 'error');
                  await firebase.setCustomKey('log level', 1);
                  await firebase.setCustomKey('logger name', 'LoginPresenter');
                  await firebase.recordError(null, StackTrace.current, reason: 'Error without exception');

                  await firebase.setCustomKey('log category', 'error');
                  await firebase.setCustomKey('log level', 1);
                  await firebase.setCustomKey('logger name', 'SignupPresenter');
                  await firebase.recordError(Exception('Error'), StackTrace.current, reason: null);

                  await firebase.setCustomKey('log category', 'warn');
                  await firebase.setCustomKey('log level', 2);
                  await firebase.setCustomKey('logger name', 'LoginState');
                  await firebase.recordError(null, StackTrace.current, reason: 'Warn without exception');

                  await firebase.setCustomKey('log category', 'warn');
                  await firebase.setCustomKey('log level', 2);
                  await firebase.setCustomKey('logger name', 'SignupState');
                  await firebase.recordError(Exception('Exception'), StackTrace.current, reason: null);

Steps to reproduce
Steps to reproduce the behavior:

Run the code above in main.dart file
Wait till crash report is sent
Check the Crashlytics console

I explicitly have set the reason to null when the error is related to SignupState, but the reason appears as 'Warn without exception' in the console

In the first call to recordError, I have set the Exception argument to null and the reason to Error without exception. In the second call to recordError, the exception is passed and the reason is set to null. However, the report is exactly the same. That is, the logger_name, flutter_error_exception, and flutter_error_reason keys values have not changed:

Expected behavior
Each report should correspond to the keys that are set before calling recordError

Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
Doctor summary (to see all details, run flutter doctor -v):
[!] Flutter (Channel stable, 3.7.1, on macOS 13.1 22C65 darwin-arm64, locale en-BR)
    ! Warning: `dart` on your path resolves to /opt/homebrew/Cellar/dart/2.16.2/libexec/bin/dart, which is not inside your current Flutter SDK checkout at
      /Users/jose.silva/development/sdks/flutter. Consider adding /Users/jose.silva/development/sdks/flutter/bin to the front of your path.
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0-rc2)
[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.1)
[✓] IntelliJ IDEA Community Edition (version 2022.2.4)
[✓] VS Code (version 1.75.1)
[✓] Connected device (3 available)                                                                                HTTP Host availability check is taking a long time...[✓] HTTP Host Availability

! Doctor found issues in 1 category.

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
Dart SDK 2.19.1
Flutter SDK 3.7.1
client_mobile 0.1.0+2

dependencies:
- carousel_slider 4.1.1 [flutter]
- commons_base 1.3.2 [flutter]
- commons_conversion 1.3.2 [flutter]
- commons_time 1.3.2 [flutter commons_base commons_conversion time_machine]
- commons_validation 1.3.2 [flutter commons_base]
- dio 4.0.6 [http_parser path]
- easy_localization 3.0.1 [flutter shared_preferences intl args path easy_logger flutter_localizations]
- firebase_analytics 10.1.0 [firebase_analytics_platform_interface firebase_analytics_web firebase_core firebase_core_platform_interface flutter]
- firebase_core 2.7.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_crashlytics 3.0.15 [firebase_core firebase_core_platform_interface firebase_crashlytics_platform_interface flutter stack_trace]
- firebase_messaging 14.2.1 [firebase_core firebase_core_platform_interface firebase_messaging_platform_interface firebase_messaging_web flutter meta]
- flutter 0.0.0 [characters collection js material_color_utilities meta vector_math sky_engine]
- flutter_bloc 8.1.1 [flutter bloc provider]
- flutter_secure_storage 5.1.0 [flutter flutter_secure_storage_linux flutter_secure_storage_macos flutter_secure_storage_platform_interface flutter_secure_storage_web flutter_secure_storage_windows meta]
- intl 0.17.0 [clock path]
- json_annotation 4.7.0 [meta]
- lib_widgets_flutter 2.4.0 [flutter intl]
- logging 1.0.2
- path 1.8.2
- sqflite 2.0.3+1 [flutter sqflite_common path]
- timezone 0.9.1 [path]

dev dependencies:
- bloc_test 9.1.0 [bloc diff_match_patch meta mocktail test]
- build_runner 2.2.0 [args async analyzer build build_config build_daemon build_resolvers build_runner_core code_builder collection crypto dart_style frontend_server_client glob graphs http_multi_server io js logging meta mime package_config path pool pub_semver pubspec_parse shelf shelf_web_socket stack_trace stream_transform timing watcher web_socket_channel yaml]
- flutter_lints 2.0.1 [lints]
- flutter_test 0.0.0 [flutter test_api path fake_async clock stack_trace vector_math async boolean_selector characters collection js matcher material_color_utilities meta source_span stream_channel string_scanner term_glyph]
- json_serializable 6.5.4 [analyzer async build build_config collection json_annotation meta path pub_semver pubspec_parse source_gen source_helper]
- test 1.22.0 [analyzer async boolean_selector collection coverage http_multi_server io js node_preamble package_config path pool shelf shelf_packages_handler shelf_static shelf_web_socket source_span stack_trace stream_channel typed_data web_socket_channel webkit_inspection_protocol yaml test_api test_core]

transitive dependencies:
- _fe_analyzer_shared 47.0.0 [meta]
- _flutterfire_internals 1.0.16 [collection firebase_core firebase_core_platform_interface flutter meta]
- analyzer 4.7.0 [_fe_analyzer_shared collection convert crypto glob meta package_config path pub_semver source_span watcher yaml]
- args 2.3.2
- async 2.10.0 [collection meta]
- bloc 8.1.0 [meta]
- boolean_selector 2.1.1 [source_span string_scanner]
- build 2.3.1 [analyzer async convert crypto glob logging meta path]
- build_config 1.1.1 [checked_yaml json_annotation path pubspec_parse yaml]
- build_daemon 3.1.0 [built_collection built_value http_multi_server logging path pool shelf shelf_web_socket stream_transform watcher web_socket_channel]
- build_resolvers 2.0.10 [analyzer async build crypto graphs logging path package_config pool pub_semver stream_transform yaml]
- build_runner_core 7.2.7 [async build build_config build_resolvers collection convert crypto glob graphs json_annotation logging meta path package_config pool timing watcher yaml]
- built_collection 5.1.1
- built_value 8.4.3 [built_collection collection fixnum meta]
- characters 1.2.1
- checked_yaml 2.0.2 [json_annotation source_span yaml]
- clock 1.1.1
- code_builder 4.4.0 [built_collection built_value collection matcher meta]
- collection 1.17.0
- convert 3.1.1 [typed_data]
- coverage 1.6.2 [args logging package_config path source_maps stack_trace vm_service]
- crypto 3.0.2 [typed_data]
- dart_style 2.2.4 [analyzer args path pub_semver source_span]
- diff_match_patch 0.4.1
- easy_logger 0.0.2 [flutter]
- fake_async 1.3.1 [clock collection]
- ffi 2.0.1
- file 6.1.4 [meta path]
- firebase_analytics_platform_interface 3.3.17 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface]
- firebase_analytics_web 0.5.1+8 [_flutterfire_internals firebase_analytics_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins js]
- firebase_core_platform_interface 4.5.3 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.2.1 [firebase_core_platform_interface flutter flutter_web_plugins js meta]
- firebase_crashlytics_platform_interface 3.3.15 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_messaging_platform_interface 4.2.10 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface]
- firebase_messaging_web 3.2.11 [_flutterfire_internals firebase_core firebase_core_web firebase_messaging_platform_interface flutter flutter_web_plugins js meta]
- fixnum 1.1.0
- flutter_localizations 0.0.0 [flutter intl characters clock collection js material_color_utilities meta path vector_math]
- flutter_secure_storage_linux 1.1.2 [flutter flutter_secure_storage_platform_interface]
- flutter_secure_storage_macos 1.1.2 [flutter flutter_secure_storage_platform_interface]
- flutter_secure_storage_platform_interface 1.0.1 [flutter plugin_platform_interface]
- flutter_secure_storage_web 1.1.1 [flutter flutter_secure_storage_platform_interface flutter_web_plugins js]
- flutter_secure_storage_windows 1.1.3 [flutter flutter_secure_storage_platform_interface]
- flutter_web_plugins 0.0.0 [flutter js characters collection material_color_utilities meta vector_math]
- frontend_server_client 2.1.3 [async path]
- glob 2.1.1 [async collection file path string_scanner]
- graphs 2.2.0 [collection]
- http_multi_server 3.2.1 [async]
- http_parser 4.0.2 [collection source_span string_scanner typed_data]
- io 1.0.4 [meta path string_scanner]
- js 0.6.5 [meta]
- lints 2.0.1
- matcher 0.12.13 [meta stack_trace]
- material_color_utilities 0.2.0
- meta 1.8.0
- mime 1.0.4
- mocktail 0.3.0 [collection matcher test]
- nested 1.0.0 [flutter]
- node_preamble 2.0.1
- package_config 2.1.0 [path]
- path_provider_linux 2.1.7 [ffi flutter path path_provider_platform_interface xdg_directories]
- path_provider_platform_interface 2.0.5 [flutter platform plugin_platform_interface]
- path_provider_windows 2.1.3 [ffi flutter path path_provider_platform_interface win32]
- platform 3.1.0
- plugin_platform_interface 2.1.3 [meta]
- pool 1.5.1 [async stack_trace]
- process 4.2.4 [file path platform]
- provider 6.0.5 [collection flutter nested]
- pub_semver 2.1.3 [collection meta]
- pubspec_parse 1.2.1 [checked_yaml collection json_annotation pub_semver yaml]
- shared_preferences 2.0.17 [flutter shared_preferences_android shared_preferences_foundation shared_preferences_linux shared_preferences_platform_interface shared_preferences_web shared_preferences_windows]
- shared_preferences_android 2.0.15 [flutter shared_preferences_platform_interface]
- shared_preferences_foundation 2.1.3 [flutter shared_preferences_platform_interface]
- shared_preferences_linux 2.1.3 [file flutter path path_provider_linux path_provider_platform_interface shared_preferences_platform_interface]
- shared_preferences_platform_interface 2.1.0 [flutter plugin_platform_interface]
- shared_preferences_web 2.0.4 [flutter flutter_web_plugins shared_preferences_platform_interface]
- shared_preferences_windows 2.1.3 [file flutter path path_provider_platform_interface path_provider_windows shared_preferences_platform_interface]
- shelf 1.4.0 [async collection http_parser path stack_trace stream_channel]
- shelf_packages_handler 3.0.1 [path shelf shelf_static]
- shelf_static 1.1.1 [convert http_parser mime path shelf]
- shelf_web_socket 1.0.3 [shelf stream_channel web_socket_channel]
- sky_engine 0.0.99
- source_gen 1.2.6 [analyzer async build dart_style glob meta path source_span yaml]
- source_helper 1.3.3 [analyzer collection source_gen]
- source_map_stack_trace 2.1.1 [path source_maps stack_trace]
- source_maps 0.10.11 [source_span]
- source_span 1.9.1 [collection path term_glyph]
- sqflite_common 2.4.2+2 [synchronized path meta]
- stack_trace 1.11.0 [path]
- stream_channel 2.1.1 [async]
- stream_transform 2.1.0
- string_scanner 1.2.0 [source_span]
- synchronized 3.0.1
- term_glyph 1.2.1
- test_api 0.4.16 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph matcher]
- test_core 0.4.20 [analyzer async args boolean_selector collection coverage frontend_server_client glob io meta package_config path pool source_map_stack_trace source_maps source_span stack_trace stream_channel vm_service yaml matcher test_api]
- time_machine 0.9.17 [meta collection]
- timing 1.0.1 [json_annotation]
- typed_data 1.3.1 [collection]
- vector_math 2.1.4
- vm_service 9.4.0
- watcher 1.0.2 [async path]
- web_socket_channel 2.3.0 [async crypto stream_channel]
- webkit_inspection_protocol 1.2.0 [logging]
- win32 3.1.3 [ffi]
- xdg_directories 0.2.0+3 [meta path process]
- yaml 3.1.1 [collection source_span string_scanner]

Thanks for the report. Was able to replicate using code sample provided.
An old similar issue for reference: https://github.com/firebase/flutterfire/issues/3421

@darshankawar any news on this? :)

Hi @jhenriquedsilva, I think it may be happening because custom keys are global and associated with an exception only when crashes are sent to crashlytics. Could you try to add await firebase.sendUnsentReports(); after each exception recorded.
Anyway, it seems directly related to the native SDK since we are just a thin wrapper around these functions.

@Lyokone I have tried to do what you have recommended, but the call to await firebase.sendUnsentReports(); does not solve the problem. I think it is not related to the custom keys, but rather to the reason. When the reason is set to null, the library completely ignores that and uses the last non-null reason of the previous calls. Therefore, the reason is not being cleaned correctly(I think that's the bug).

This behaviour is not a result of FlutterFire, but the way it is processed by the underlying native SDK. Interestingly, there has been movement by the Firebase android SDK to allow custom keys for specific exceptions which you can follow here: https://github.com/firebase/firebase-android-sdk/issues/3551

Hey @jhenriquedsilva - the corresponding issue in the android SDK has been fixed: https://github.com/firebase/firebase-android-sdk/issues/6341
We can close this out as android BOM v33.5.0 has been merged pending release 👍


A python module?

I was wondering. Would it be possible to make this package a python module, so it would be called with python -m colored_traceback anything.py?
That would make colored-traceback absolutely not intrusive, and be easier to test.
What do you think?

Good idea! I merged your pr (with some tweaks.)


Unknown error: (400, u'Bad Request') with Vivotek FE8172

I've created an OnvifCamera object and I can access the properties of DeviceManagement but I can get profiles of Media Service, I always get Unknown error: (400, u'Bad Request').
Here is my code:
mycam = ONVIFCamera('<IP_ADDRESS>', 80, 'root', '', 'C:\onvif\wsdl\')
media = mycam.create_media_service()
profiles = mycam.media.GetProfiles()
And what I get:
profiles = mycam.media.GetProfiles()
File "onvif\client.py", line 31, in wrapped
raise ONVIFError(err)
onvif.exceptions.ONVIFError: Unknown error: (400, u'Bad Request')
Any suggestion?
I checked the time on the camera and it is correct.  Also, I tested the code on a PTZ camera and it works.

may be the login info error
check the info and login in the web page

u can use the onvif test tools
if it can't work it's the other problems.


Conflict with evil-multiedit

Hi, I am using your amazing package from Doom Emacs and I found a conflict between tree-sitter-hl-mode and evil-multiedit.
To reproduce:

Select a few occurrences of a symbol with evil-multiedit-match-symbol-and-next, one occurrence in a string and one occurrence in a variable name
Try to edit them at once (for instance append some characters at the end) and it will not work.

I work with python and both packages are essential to my workflow! Please let me know if I can do anything to help you debug!

I use multiple cursors and iedit along with tree-sitter and I don't experience this issue

Can you provide more details for to reproduce/debug the issue? For example:

A minimal file with the issue.
Steps to trigger the issue, using that file.
What is the expected behavior? What did you observe instead?

Thank you for the reply.
I think the issue might lay within evil-multiedit, I brought  up the issue in their repo: https://github.com/hlissner/evil-multiedit/issues/40 with snippets to reproduce in a Doom emacs setup.

I reproduced the error.. I have highlighted all (3) occurrences with evil-multiedit-match-all, tried to change them all at once with evil-multiedit--append-line but only 2 out of 3 successfully changed. The debugger outputted this stacktrace:
Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  tree-sitter--after-change(3193 3193 1)
  run-hook-with-args(tree-sitter--after-change 3193 3193 1)
  iedit-update-occurrences-2(#<overlay from 704 to 723 in main_total_forecasts.py> t 723 723 1)
  iedit-update-occurrences(#<overlay from 704 to 723 in main_total_forecasts.py> t 723 723 1)
  delete-char(-1)
  (cond ((and op cl (string= op cl) (and (string= (char-to-string (or (char-before) 0)) op) (setq open-len (length op))) (and (string= (char-to-string (or (char-after) 0)) cl) (setq close-len (length cl)))) (delete-char (- open-len)) (delete-char close-len)) ((and (not indent-tabs-mode) (> tab-width 1) (not (bolp)) (not (doom-point-in-string-p)) (save-excursion (>= (- (skip-chars-backward " \11")) tab-width))) (let ((movement (% (current-column) tab-width))) (if (= movement 0) (progn (setq movement tab-width))) (delete-char (- movement))) (if (memq (char-before) (list 10 32)) nil (insert " "))) ((delete-char -1)))
  (let* ((context (condition-case nil (progn (sp-get-thing)) (error nil))) (op (plist-get context :op)) (cl (plist-get context :cl)) open-len close-len) (cond ((and op cl (string= op cl) (and (string= (char-to-string (or ... 0)) op) (setq open-len (length op))) (and (string= (char-to-string (or ... 0)) cl) (setq close-len (length cl)))) (delete-char (- open-len)) (delete-char close-len)) ((and (not indent-tabs-mode) (> tab-width 1) (not (bolp)) (not (doom-point-in-string-p)) (save-excursion (>= (- (skip-chars-backward " \11")) tab-width))) (let ((movement (% (current-column) tab-width))) (if (= movement 0) (progn (setq movement tab-width))) (delete-char (- movement))) (if (memq (char-before) (list 10 32)) nil (insert " "))) ((delete-char -1))))
  doom/backward-delete-whitespace-to-column()
  (cond ((and end beg (= end (+ beg (length op) (length cl)))) (delete-char (- (length op)))) ((doom-surrounded-p pair 'inline 'balanced) (delete-char -1 killflag) (delete-char 1) (if (= (point) (+ (length cl) beg)) (progn (sp-backward-delete-char 1) (sp-insert-pair op)))) ((and (bolp) (doom-surrounded-p pair nil 'balanced)) (delete-region beg end) (sp-insert-pair op) t) ((run-hook-with-args-until-success 'doom-delete-backward-functions)) ((doom/backward-delete-whitespace-to-column)))
  (let* ((pair (condition-case nil (progn (sp-get-thing)) (error nil))) (op (plist-get pair :op)) (cl (plist-get pair :cl)) (beg (plist-get pair :beg)) (end (plist-get pair :end))) (cond ((and end beg (= end (+ beg (length op) (length cl)))) (delete-char (- (length op)))) ((doom-surrounded-p pair 'inline 'balanced) (delete-char -1 killflag) (delete-char 1) (if (= (point) (+ (length cl) beg)) (progn (sp-backward-delete-char 1) (sp-insert-pair op)))) ((and (bolp) (doom-surrounded-p pair nil 'balanced)) (delete-region beg end) (sp-insert-pair op) t) ((run-hook-with-args-until-success 'doom-delete-backward-functions)) ((doom/backward-delete-whitespace-to-column))))
  (cond ((or (not t) (not (and (boundp 'smartparens-mode) smartparens-mode)) (and (memq (char-before) (list 32 9)) (save-excursion (and (/= (skip-chars-backward " \11" ...) 0) (bolp))))) (doom/backward-delete-whitespace-to-column)) ((let* ((pair (condition-case nil (progn (sp-get-thing)) (error nil))) (op (plist-get pair :op)) (cl (plist-get pair :cl)) (beg (plist-get pair :beg)) (end (plist-get pair :end))) (cond ((and end beg (= end (+ beg ... ...))) (delete-char (- (length op)))) ((doom-surrounded-p pair 'inline 'balanced) (delete-char -1 killflag) (delete-char 1) (if (= (point) (+ ... beg)) (progn (sp-backward-delete-char 1) (sp-insert-pair op)))) ((and (bolp) (doom-surrounded-p pair nil 'balanced)) (delete-region beg end) (sp-insert-pair op) t) ((run-hook-with-args-until-success 'doom-delete-backward-functions)) ((doom/backward-delete-whitespace-to-column))))))
  (cond ((and (use-region-p) delete-active-region (= n 1)) (if (eq delete-active-region 'kill) (kill-region (region-beginning) (region-end) 'region) (funcall region-extract-function 'delete-only))) ((null (or (null overwrite-mode) (<= n 0) (memq (char-before) '(9 10)) (eobp) (eq (char-after) 10))) (let ((ocol (current-column))) (delete-char (- n) killflag) (save-excursion (insert-char 32 (- ocol (current-column)) nil)))) ((= n 1) (cond ((or (not t) (not (and (boundp ...) smartparens-mode)) (and (memq (char-before) (list 32 9)) (save-excursion (and ... ...)))) (doom/backward-delete-whitespace-to-column)) ((let* ((pair (condition-case nil ... ...)) (op (plist-get pair :op)) (cl (plist-get pair :cl)) (beg (plist-get pair :beg)) (end (plist-get pair :end))) (cond ((and end beg ...) (delete-char ...)) ((doom-surrounded-p pair ... ...) (delete-char -1 killflag) (delete-char 1) (if ... ...)) ((and ... ...) (delete-region beg end) (sp-insert-pair op) t) ((run-hook-with-args-until-success ...)) ((doom/backward-delete-whitespace-to-column))))))) ((delete-char (- n) killflag)))
  +default--delete-backward-char-a(1 nil)
  apply(+default--delete-backward-char-a (1 nil))
  delete-backward-char(1 nil)
  #<subr funcall-interactively>(delete-backward-char 1 nil)
  apply(#<subr funcall-interactively> (delete-backward-char 1 nil))
  funcall-interactively(delete-backward-char 1 nil)
  #<subr call-interactively>(delete-backward-char nil nil)
  apply(#<subr call-interactively> (delete-backward-char nil nil))
  explain-pause--wrap-call-interactively(#<subr call-interactively> delete-backward-char nil nil)
  apply(explain-pause--wrap-call-interactively #<subr call-interactively> (delete-backward-char nil nil))
  call-interactively(delete-backward-char nil nil)
  evil-delete-backward-char-and-join(1)
  #<subr funcall-interactively>(evil-delete-backward-char-and-join 1)
  apply(#<subr funcall-interactively> (evil-delete-backward-char-and-join 1))
  funcall-interactively(evil-delete-backward-char-and-join 1)
  #<subr call-interactively>(evil-delete-backward-char-and-join nil nil)
  apply(#<subr call-interactively> (evil-delete-backward-char-and-join nil nil))
  explain-pause--wrap-call-interactively(#<subr call-interactively> evil-delete-backward-char-and-join nil nil)
  apply(explain-pause--wrap-call-interactively #<subr call-interactively> (evil-delete-backward-char-and-join nil nil))
  call-interactively(evil-delete-backward-char-and-join nil nil)
  command-execute(evil-delete-backward-char-and-join)

I use iedit(which is depended on by evil-multiedit) along with tree-sitter, and days ago(2020-11-17) I had the save/similar problem. So at that time I added some hook functions to iedit's hooks to disable tree-sitter-mode when using iedit.
A moment ago I removed these hook functions and made a test again, iedit worked without any errors, and I found that iedit was updated several times after 2020-11-17 and fixed some issues, so maybe the issue has been fixed at iedit side.
I'm using tree-sitter 0.12.1 and iedit 20201125.342, both were installed from melpa, and I'm using emacs's native-comp branch.
@lccambiaghi Maybe you can update iedit (if you're not using the latest version) and make another test?

Hi @zbelial thank you for your comment! Unfortunately DOOM Emacs has pinned iedit to an old version. It is good to know that would fix the problem. I am going to investigate the side effects of unpinning it. I guess this issue is not really emacs-tree-sitter specific then.

closing this since it is a issue with iedit

I took a brief look at iedit. It uses inhibit-modification-hooks and calls the change hooks on its own. Doing that right is tricky.
@lccambiaghi You can try changing the pinned version of iedit in your local Doom installation to see whether the issue persists. If it does, please reopen this.


Bug in handling partial data

Hello @ggrandes,
I tried out your nice module apache22-modules/mod_myfixip.c but it's not working as expected in my environment. Test against Nginx works fine with my AWS ELB ProxyProtocol setup.

After longer digging I found out that the apr_bucket_read() gives back a wrong length.
Every time only 11 chars instead of 46/47.
Thats only the part "PROXY TCPv4 SRCIP DSTIP SRCPORT DSTPORT ":

414     // Read first bucket
415     apr_bucket_read(e, &str, &length, APR_BLOCK_READ);
416 #ifdef DEBUG
417         ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, MODULE_NAME "::helocon_filter_in DEBUG: header='%s', read length=%d, real length=%d", str, length, strlen(str));
418 #endif

there is still no successfull connection possible - I guess that there could be still a buffer problem open which breaks the SSL handshake. Do you have perhaps an idea what could happened?
I am using Debian Wheezy with apache2.2 2.2.22-13+deb7u3 / now 2.2.22-13+deb7u4..

Bests

Currently I have no idea, but... I will investigate this.
You have any tcpdump/pcap with problematic frames?
Thanks.

I have reproduced the problem, I hope find a solution shortly.

Fixed [#1] [commit be8f0f00ac27c47ef6f3dc92e02809a609cd36d0]

The problem was to assume that the data would arrive together in only one bucket.
From PROXY protocol:

Both formats are designed to fit in the smallest TCP segment that any TCP/IP
host is required to support (576 - 40 = 536 bytes). This ensures that the whole
header will always be delivered at once when the socket buffers are still empty
at the beginning of a connection. The sender must always ensure that the header
is sent at once, so that the transport layer maintains atomicity along the path
to the receiver.

Big THX.., now PROXY function works with version 0.6 nice:
<IP_ADDRESS> - - [17/Jan/2015:16:41:00 +0100] "GET / HTTP/1.1" 200 2004 "-" "lwp-request/6.03 libwww-perl/6.04"
<IP_ADDRESS> - - [17/Jan/2015:16:41:41 +0100] "GET / HTTP/1.1" 200 2017 "-" "fetch libfetch/2.0"
```
I would prepare then the connections to switch to the nicer PROXY protocol because it supports SSL SNI and SPDY usage ;)

(btw: an improved old/debian stable available mod_rpaf to use also Proxy IP areas, IPv6  by backported partly from mod-remote-ip functions of 2.4 is here: https://github.com/gnif/mod_rpaf )

Fixed really in [c35ca38466f867eca35d1b9c3ecc718e5f0f3142]

I think that I don't fully understand... What do you want to do?

a small "bug" for Debugging: readbytes is formatted as %lld not %11d:
/files/mod_myfixip.c.orig  2015-01-17 16:36:23.216287915 +0100
+++ salt/base/apache2/files/mod_myfixip.c       2015-01-23 12:01:36.045701072 +0100
@@ -381,7 +381,7 @@
     apr_bucket *e = NULL;

 #ifdef DEBUG
-    ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, MODULE_NAME "::helocon_filter_in IP Connection from: %s to port=%d readbytes=%lld (1)", _CLIENT_IP, c->local_addr->port, readbytes);
+    ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, MODULE_NAME "::helocon_filter_in IP Connection from: %s to port=%d readbytes=%11d (1)", _CLIENT_IP, c->local_addr->port, readbytes);
 #endif

     // Fail quickly if the connection has already been aborted.
@@ -498,7 +498,7 @@
                 // Refill buckets
                 if ((e == APR_BRIGADE_SENTINEL(b)) && APR_BRIGADE_EMPTY(b)) {
 #ifdef DEBUG
-                    ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, MODULE_NAME "::helocon_filter_in DEBUG: refill buffer size=%lld (1)", readbytes);
+                    ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, MODULE_NAME "::helocon_filter_in DEBUG: refill buffer size=%11d (1)", readbytes);
 #endif
                     ap_get_brigade(f->next, b, mode, block, readbytes);
                     e = APR_BRIGADE_FIRST(b);
@@ -570,7 +570,7 @@
         // Refill buckets
         if (APR_BRIGADE_EMPTY(b)) {
 #ifdef DEBUG
-            ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, MODULE_NAME "::helocon_filter_in DEBUG: refill buffer size=%lld (2)", readbytes);
+            ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, MODULE_NAME "::helocon_filter_in DEBUG: refill buffer size=%11d (2)", readbytes);
 #endif
             ap_get_brigade(f->next, b, mode, block, readbytes);
             e = APR_BRIGADE_FIRST(b);

readbytes is apr_off_t
from /usr/include/apr-1.0/apr.h:
    typedef  off64_t           apr_off_t;
    #define APR_OFF_T_FMT APR_INT64_T_FMT
    #define APR_INT64_T_FMT "lld"
    #define APR_OFF_T_FMT "lld"

So, I used "lld", but yes, this is not 100% correct.
Better I use defined format string. Fixed [99674b46dfde53bb05d78d5cddf051946bb7f8fa]

you can tell me about these overflows?

@Reiner030 the latest version (1.2) [050abf80e96a81f2e64104ba50ae854ad1248f97] fix all problems. check this out :-)

Hi @ggrandes , patch looks nice/that there are the right function calls found...
Sadly I have no access to the setup anymore but I would inform the new devs that they can now update/check the AWS ELB in proxy mode.
But I can try/test it on a new setup to verify it - just checked that same fix is also made in 2.4 module ..  Thanx ;)

:+1:


Simplecrawler for subpages

In this pull request a new option will be added: ---baseurlor -b.
When you enable this option, an additional fetch condition will be added which always checks if the parsedUrl from simplecrawler matches the given . This gives the user he opportunity, to create a sitemap.xml for subpages, like all URLs beginning with http://www.example.com/foo

Have you tested this? As I can read from the simplecrawler docs this.crawler.initialPath does not restrict further requests from accessing /.

I have tested it before committing. this.crawler.initialPathis the first URL which is called. Just setting this option does not prevent simplecrawler to crawl URLs above this depth. That's why I have added another addFetchConditioncallback to prevent this behaviour.

I don't understand why the build for node 0.12 has failed while on my repository with the same code it works. See https://travis-ci.org/DennisBecker/node-sitemap-generator-cli/builds/120875515

I reran it, as far as I can see it was just a timeout, nothing to worry about. Thanks, merged!

I will add a log output for pages not matching the base URL.
On huge websites it looks like the crawler won't do anything but in fact it is just removing non-matching URLs


[mixcloud] Failed to parse JSON

[x] I've verified and I assure that I'm running youtube-dl 2017.08.09

[x] At least skimmed through the README, most notably the FAQ and BUGS sections

[x] Searched the bugtracker for similar issues including closed ones

[x] Bug report (encountered problems with youtube-dl)

[ ] Site support request (request for adding support for a new site)

[ ] Feature request (request for a new functionality)

[ ] Question

[ ] Other

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--extract-audio', '--audio-format=mp3', 'https://www.mixcloud.com/zedsdead/zeds-dead-diplo-friends-mix-2016/', '--verbose']

[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2017.08.09
[debug] Python version 3.4.4 - Windows-8.1-6.3.9600
[debug] exe versions: ffmpeg 2.8.4, ffprobe 3.3.2
[debug] Proxy map: {}
[mixcloud] zedsdead-zeds-dead-diplo-friends-mix-2016: Downloading webpage
ERROR: zedsdead-zeds-dead-diplo-friends-mix-2016: Failed to parse JSON  (caused by ValueError('Expecting value: line 1 column 1 (char 0)',)); please rep
ort this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with th
e --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpf7t6lw0k\build\youtube_dl\extractor\common.py", line 676, in _parse_json
  File "C:\Python\Python34\lib\json\__init__.py", line 318, in loads
  File "C:\Python\Python34\lib\json\decoder.py", line 343, in decode
  File "C:\Python\Python34\lib\json\decoder.py", line 361, in raw_decode
ValueError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpf7t6lw0k\build\youtube_dl\extractor\common.py", line 676, in _parse_json
  File "C:\Python\Python34\lib\json\__init__.py", line 318, in loads
  File "C:\Python\Python34\lib\json\decoder.py", line 343, in decode
  File "C:\Python\Python34\lib\json\decoder.py", line 361, in raw_decode
ValueError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpf7t6lw0k\build\youtube_dl\YoutubeDL.py", line 776, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpf7t6lw0k\build\youtube_dl\extractor\common.py", line 433, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpf7t6lw0k\build\youtube_dl\extractor\mixcloud.py", line 89, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpf7t6lw0k\build\youtube_dl\extractor\mixcloud.py", line 69, in _decrypt_play_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpf7t6lw0k\build\youtube_dl\extractor\common.py", line 680, in _parse_json
youtube_dl.utils.ExtractorError: zedsdead-zeds-dead-diplo-friends-mix-2016: Failed to parse JSON  (caused by ValueError('Expecting value: line 1 column
1 (char 0)',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure
to call youtube-dl with the --verbose flag and include its complete output.

Certain Mixcloud shows are failing to download with a JSON parsing error.  In this case the example URL is https://www.mixcloud.com/zedsdead/zeds-dead-diplo-friends-mix-2016

You are using an outdated version of youtube-dl. Read this FAQ entry if you have troubles updating.

Well it was current when I filed it and it continues to fail in 8.13


Replaced options with config in the generate function within the document

I updated the documentation to match the latest implementation.

Thanks for contributing!


Among Us art needs removing

If we're allowing licensed works under the guise of fair use then it should be discussed between maintainers again considering we already came to an agreement on this and purged existing works.

Related to #11115

Nice to see that my latest contribution to the project is possible among us themed copyright violation.

https://www.innersloth.com/fan-creation-policy/
this is the policy for use of the among us IP. as far as i can tell, there's no issue with having game assets using the IP so long as they are not intentionally vulgar, attempting to claim validity, or being used in marketing. I'm not a legal expert, but nothing in the policy states anything wrong with the way in which the assets are currently being used.

https://www.innersloth.com/fan-creation-policy/
this is the policy for use of the among us IP. as far as i can tell, there's no issue with having game assets using the IP so long as they are not intentionally vulgar, attempting to claim validity, or being used in marketing. I'm not a legal expert, but nothing in the policy states anything wrong with the way in which the assets are currently being used.

The copyright for that painting in particular needs to link the fan creation policy. Can you make a PR?


Installing tree-sitter breaks evil motion rebinds

Hi. I have this in my init.el. Without
(straight-use-package 'tree-sitter)
(straight-use-package 'tree-sitter-langs)
(require 'tree-sitter)
(require 'tree-sitter-langs)
(add-hook 'rust-mode-hook #'tree-sitter-mode)

jkl; in dired and help buffers work as they're expected to with the corresponding remappings. But once I uncomment these lines (like in init.el above), the keybindings revert back to hjkl.
Output of C-h, k, k pressed in dired buffer with those lines commented out:

Same, but with those lines uncommented:

I'm very new to Emacs and probably don't understand something, but I'd appreciate some help on it. Thanks.

This looks like an bug with key definition order in evil/general.
You'd get the same issue by replacing the tree-sitter init code with  (require 'dired). Putting that after the key redefinitions works.

I'm very new to Emacs and probably don't understand something, but I'd appreciate some help on it. Thanks.

You can try debugging evil/general key definition order by logging calls to define-key. For example, add this to the top of your init.el:
(define-advice define-key (:before (_keymap key def &rest _) who-defines-k)
  (when (equal key "k")
    (message "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
    (message "k is getting redefined into %s" def)
    ;; Print the backtrace.
    (mapbacktrace (lambda (_evald func args _flags)
                    ;; The bug seems to depend on lib load order, so log more for loads.
                    (if (memq func '(load require))
                        (message "  --  %s  :  %s" func args)
                      (message "  --  %s" func))))))

This looks like an bug with key definition order in evil/general.

Actually it's been working the same way back when I was not using general.el.

You'd get the same issue by replacing the tree-sitter init code with (require 'dired). Putting that after the key redefinitions works.

Yeah, this:
(general-define-key
 :states '(motion, normal, visual, operator)
 "j" 'evil-backward-char
 "k" 'evil-next-line
 "l" 'evil-previous-line
 ";" 'evil-forward-char
 )

(require 'tree-sitter)
(require 'tree-sitter-langs)
(add-hook 'rust-mode-hook #'tree-sitter-mode)
(global-tree-sitter-mode)
(add-hook 'tree-sitter-after-on-hook #'tree-sitter-hl-mode)
;(require 'tree-sitter-hl)

actually worked fine. At least, for dired and and help.


Add distributed-closure.

Are you able to support binary-<IP_ADDRESS>? Stackage uses the version that ships with GHC.

Yep. distributed-closure <IP_ADDRESS> pushed.


[REQUEST] Rebind Ctl+W to close a tab

Currently if you press CTL+W it closes the entire window vs closing the currently open tab.
The expected behavior I believe would be when having multiple tabs it would close the tabs. In the break out windows it would close the window.

This was already the case on macOS, but I made some changes to make the behavior consistent across platforms. Ctrl/Cmd+W is now "close tab" and Ctrl/Cmd+Shift+W is now "close window." Ctrl/Cmd+W will also close any popup windows. macOS also has Cmd+Q to quit the app as before.


While GRAPH is working fine for me, --diagram SEQUENCE is not

Nice tool.
I could not get the sequence (mulefd --diagram SEQUENCE ....) generated:
mulefd --diagram SEQUENCE oracle-main.xml --out oracle-main --target SEQ
2020-12-15 14:29:44,641 INFO  [c.j.m.a.Application] - Mule Flow Diagrams - 0.7.4, Started at 15-Dec-2020 14:29:44.634
2020-12-15 14:29:44,647 INFO  [c.j.m.DiagramRenderer] - Reading source file oracle-main.xml
2020-12-15 14:29:44,676 INFO  [c.j.m.a.Application] - Finished at 15-Dec-2020 14:29:44.676

Thank you @msaeidi  for trying it out.
As of now, only graph diagram is supported. Hopefully sequence will be supported soon. Please feel free to add your thoughts on what would you look for if SEQ is supported. What do you expect to include or exclude in the SEQ diagram.


⚠ SceneTime has degraded performance

In 259ac40, SceneTime ($ST) experienced degraded performance:

HTTP code: 200
Response time: 7594 ms

Resolved: SceneTime performance has improved in a6d09bc after 5 minutes.


[SPARK-41398][SQL] Relax constraints on Storage-Partitioned Join when partition keys after runtime filtering do not match

What changes were proposed in this pull request?

This PR relaxes the current constraint of Storage-Partitioned Join which requires that the partition keys after runtime filtering to be exact the same as the partition keys before the filtering.
Why are the changes needed?

At the moment, Spark requires that when Storage-Partitioned Join is used together with runtime filtering, the partition keys before and after the filtering shall exact match. If not, a SparkException is thrown.
However, this is not strictly necessary in the case where the partition keys after the filtering is a subset of the original keys. In this scenario, we can use empty partitions for those missing keys in the latter.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Modified an existing test case to match the change.

cc @cloud-fan @dongjoon-hyun @viirya @aokolnychyi @huaxingao

Also, cc @wangyum , @MaxGekk , @yaooqinn , @c21

Merged to master for Apache Spark 3.4.0.


[mock] requestUri not working as per docs

realized when investigating: https://stackoverflow.com/q/75953196/143475
as per documentation:

Everything on the right side of the "base URL". This will include everything, including query string parameters if present. For example if the request URL was http://foo/bar?baz=ban the value of requestUri will be /bar?baz=ban.

but right now looks like the ? and the query string are missing

1.4.1 released


Be able to specify target os and target cpu

For cross-compiling android via target_os="android" target_cpu="arm"
Or iOS.

Pull request https://github.com/vsimon/webrtcbuilds/pull/27 has a first cut at this.


fix: remove space character in line comment rule

修复 FlinkSQL 和 Generic SQL 不支持双横线注释的问题，原语法注释需要空格符，例如：
合法：
-- comment content

缺少空格不合法：
--comment content

Fixed #42
变更

修改 FlinkSQL，Generic SQL 语法，移除注释开始标记中的空格符
增加注释单元测试
重新编译语法文件

@wewoor Warning! the automatic builder contains errors!
When I added PGsql support I had to fix those manually, so your last release is buggy!

I have to check it!

Hey @salvoravida, I have validated the PGSQL but found no errors. can you provide more detailed info?


Un-pad result on return

I've finished updating the implementation to use the latest versions of the dependencies. I've also updated the implementation to make better use of OffsetArrays.jl. There is still one missing issue before we can update the test suite and submit a new release. What is the current idiomatic way of un-padding an image that was padded with padarray?
img = zeros(10,10)
pad = padarray(img, Pad(:symmetric, (3,3), (3,3)))

# works but is not officially supported?
padarray(pad, Pad(:symmetric, (-3,-3), (-3,-3)))

cc: @johnnychen94 @timholy

If padarray returns a lazy-evaluated PadArray, then using parent would be more efficient and applicable.
Since it now returns an OffsetArray; I would use pad[1:end-3, 1:end-3] or @view pad[1:end-3, 1:end-3].

Thank you @johnnychen94 , I did pad[1:10,1:10] to return the original image. More generally in N dimensions:
start  = CartesianIndex(ntuple(i->1, ndims(img)))
finish = CartesianIndex(size(img))
pad[start:finish]


Document for cross-compile for low-end devices

I followed the guidance in README to install T-MAC on a S905D3 development board equipped with 4GB RAM, but the process seemed to be lengthy and difficult. During the installation of TVM(automatically executed in "pip install . -e"), a lot of source files need to be compiled, which would take more than 2 hours. Some source files are complex which made the compiler consume more RAM than my device can provide so the compiling processes were killed.
Are there any ways to deploy llama.cpp powered by T-MAC on an embedded device(Linux/Android), without the need to install huge components like python3 and TVM on that device?

The STEP.0,1,2,3 can all be done in host device. Then, you can cross-compile llama.cpp for your target device. Check #12 for an example on how to deploy onto Android.
The cross-compile processes are currently undocumented and not fully tested yet. We will soon add the support to run_pipeline.py.

I have added commits to cross-compile for Android. I'm not familiar with S905D3. However, you can take it as an example on how to cross compile for another device.
You can test if it works for Android and open a new issue if you encounter any problems. I will close this issue for now.


Duplicate Delivery

Just found a duplicate delivery.
We can probably prevent this by doing a select on deliverable_id, user_id before insert

Latest commit fixes this


Upgrade to version 3.2

Checklist

[X] Used a personal fork of the feedstock to propose changes
[X] ~Bumped the build number (if the version is unchanged)~
[X] Reset the build number to 0 (if the version changed)
[X] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

@aradi Thank you for the update - Are you interested in joining me in maintaining this package?


Add note about appgw.environment

appgw.environment is a required parameter for deploying into Sovereign Clouds, but it is not documented on this page. I had to go into the AGIC source code repo and find a troubleshooting doc that identified it. Adding it to the values list and adding a note so that it gets attention.

Learn Build status updates of commit fa68d1d:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/application-gateway/ingress-controller-install-new.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@greg-lindsay

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


CrossShardOptimizer must be used for model training on TPUs

Running the example on a Colab TPU results in the following error:
File "main.py", line 256, in <module>
    main(args)
  File "main.py", line 230, in main
    estimator.train(input_fn=partial(input_fn, global_step=current_step, eval=False), max_steps=next_checkpoint)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3130, in train
    rendezvous.raise_errors()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
    six.reraise(typ, value, traceback)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3125, in train
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1175, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1204, in _train_model_default
    self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2962, in _call_model_fn
    config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1163, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3386, in _model_fn
    _validate_tpu_training_graph(ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3817, in _validate_tpu_training_graph
    'CrossShardOptimizer must be used for model training on TPUs.')
ValueError: CrossShardOptimizer must be used for model training on TPUs.

Ok, currently my guess is main.py is using from tensorflow.python.tpu import tpu_config, tpu_estimator to create and estimator, which in turn imports tensorflow_estimator, which apparently hasn't been updated in years and has no idea how to deal with CrossShardOptimizers and CrossReplicaSum

Ok, currently my guess is main.py is using from tensorflow.python.tpu import tpu_config, tpu_estimator to create and estimator, which in turn imports tensorflow_estimator, which apparently hasn't been updated in years and has no idea how to deal with CrossShardOptimizers and CrossReplicaSum

I had this same issue (after trying to update tensorflow version) and after some amount of work, realized that "Source" cell set tensorflow type to 1.x
Changed this to 2.x and it seems to work like a charm

@samriddhishree
Interesting, since last time I've tried not downgrading tensorflow to 1.15, I ran into the wrong mesh_shape

Pull request I opened w/ my changes, which worked on colab: https://github.com/EleutherAI/gpt-neo/pull/130
Try it out and see if it works for you :)

@samriddhishree
Ok, if I use just your create_tfrecords patch, it still fails.
But if I use it and your optimizers patch, a different error occurs:

TypeError: CrossShardOptimizer only works with tf.training.Optimizer and not Optimizer_v2. If you are using TPUStrategy, OptimizerV2 will sum gradients across replicas.If you are using TPUEstimator, you may instead sum your gradients with: grads = [tf.compat.v1.tpu.cross_replica_sum(g) for g in grads]. If you want to average your gradients, rescale your loss with: loss /= global_batch_size

Given the circumstances, that might be considered progress, I think :-)

did you try updating %tensorflow to 2.x in GPTNeo_example_notebook.ipynb?
I didn't utilize CrossShardOptimizer in my code -- I started with base Colab notebook, and only made those 2 additional changes on the pull request

Why did you remove //args.processes) from the line files = split_list(files, len(files) // args.processes) (context)?   Without this you aren't getting any benefit out of having multiple CPUs, you're just running on each of them independently. It should also cause data duplication, unless I am missing something.

So, as I understand it, we can't just wrap a version 2 Optimizer, which mesh-tensorflow returns, in a CrossShardOptimizer (as discussed here).
Perhaps we need to follow their suggestions and sum our gradients?

I think if you use MTF under the hood, you wouldn't really need to run a CrossShardOptimizer as MTF already does that given a layout and `mesh.
For example, the SIMDMeshImpl in MTF on TPU would lower the reduce_sum` to a mesh specific allreduce, and that translates to proper TPU CrossReplicaSum.
CrossShardOptimizer performs CrossReplicaSum under the hood, and for this case, MTF should get you covered as long as you pass in proper mesh and layout.
So for what you've done above, you probably want to remove CrossShardOptimizer and simply run with tensorflow 2.x?

@samriddhishree
Yep, that as well. That is, I
* switched to ` %tensorflow 2.x`

* removed the `args.processes`

* added the `CrossShardOptimizer` wrapper

Perhaps I should try switching to tensorflow 2.x only.
ETA: Ok, if I only switch to tensorflow 2.x, the CrossShardOptimizer error appears. If I switch to tensorflow and use the CrossShardOptimizer wrapper patch, the above error appears, mentioning TPUStrategies and TPUEstimators

The CrossShardOptimizer should be 100% unnecessary if you're using TF 2.x for the reasons @hthu describes. I am extremely willing to believe that our Colab file is out of date, but I am highly skeptical of the CrossShardOptimizer wrapper.
All patches should be based in TF 2.x. TF 1.x is going away and that all code that only runs in TF 1.x will break in the near future.

@hthu,
I've just re-checked and

my notebook is based on the example notebook
I've just switched to tensorflow 2.x, leaving other parts of the code as they are

This results in the CrossShardOptimizer error.
Just to clarify, I didn't write this code, I'm just trying to use it & make it work.
So, I think that MTF is being used under the hood, but somehow -- due to it being an old version of mtf or mtf itself lacking the defintions necessary or something from the mtf source code being redefined here -- it doesn't seem to produce an optimizer ready to be run on a TPU.

What mesh_shape are you using?
You might want at least 2x2 to start with.

I'm using all:8

Do you have the latest stacktrace?
You probably shouldn't see any TpuEstimator error message or you're still in 1.x realm?

2021-02-15 11:04:26.763273: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-02-15 11:04:26.763406: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Current step 0
Downloading: 100% 1.04M/1.04M [00:00<00:00, 5.93MB/s]
Downloading: 100% 456k/456k [00:00<00:00, 3.22MB/s]
Downloading: 100% 1.36M/1.36M [00:00<00:00, 7.61MB/s]
Saving config to /content/drive/MyDrive/GPT2_colab_trial
2021-02-15 11:04:30.923993: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-15 11:04:30.949992: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-15 11:04:31.006147: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2021-02-15 11:04:31.006204: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (17c0abb61b15): /proc/driver/nvidia/version does not exist
2021-02-15 11:04:31.226784: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
Done!
params = defaultdict(<function fetch_model_params.<locals>.<lambda> at 0x7ff1a9057598>, {'n_head': 32, 'n_vocab': 50260, 'embed_dropout': 0, 'lr': 0.0002, 'lr_decay': 'cosine', 'warmup_steps': 3000, 'beta1': 0.9, 'beta2': 0.95, 'epsilon': 1e-08, 'opt_name': 'adam', 'weight_decay': 0.1, 'train_batch_size': 256, 'attn_dropout': 0, 'train_steps': 572300, 'eval_steps': 0, 'predict_steps': 1, 'res_dropout': 0, 'eval_batch_size': 64, 'predict_batch_size': 1, 'iterations': 100, 'n_embd': 2048, 'datasets': ['nihexporter'], 'model': 'GPT', 'model_path': '/content/drive/MyDrive/GPT2_colab_trial', 'n_ctx': 2048, 'n_layer': 24, 'scale_by_depth': True, 'scale_by_in': False, 'attention_types': ['global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global'], 'mesh_shape': 'all:8', 'layout': 'intermediate_expanded:x,heads:x,vocab:x,memory_length:y,embd:y', 'activation_function': 'gelu', 'recompute_grad': True, 'gradient_clipping': 1.0, 'tokens_per_mb_per_replica': 2048, 'padding_id': 50257, 'eos_id': 50256, 'dataset_configs': {'nihexporter': {'path': '/content/drive/MyDrive/openwebtext_tokenized/*.tfrecords', 'eval_path': '', 'n_vocab': 50256, 'tokenizer_is_pretrained': True, 'tokenizer_path': 'gpt2', 'eos_id': 50256, 'padding_id': 50257}}, 'mlm_training': False, 'causal': True, 'num_cores': 8, 'auto_layout': False, 'auto_layout_and_mesh_shape': False, 'use_tpu': True, 'gpu_ids': ['device:GPU:0'], 'steps_per_checkpoint': 500, 'predict': False, 'export': False, 'sampling_use_entmax': False, 'moe_layers': None, 'slow_sampling': False})
Using config: {'_model_dir': '/content/drive/MyDrive/GPT2_colab_trial', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
cluster_def {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "<IP_ADDRESS>:8470"
    }
  }
}
isolate_session_state: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({'worker': ['<IP_ADDRESS>:8470']}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://<IP_ADDRESS>:8470', '_evaluation_master': 'grpc://<IP_ADDRESS>:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=100, num_shards=8, num_cores_per_replica=1, per_host_input_for_training=4, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1, experimental_allow_per_host_v2_parallel_get_next=False, experimental_feed_hook=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu.tpu_cluster_resolver.TPUClusterResolver object at 0x7ff1a8ff1240>}
_TPUContext: eval_on_tpu True
Querying Tensorflow master (grpc://<IP_ADDRESS>:8470) for TPU system metadata.
2021-02-15 11:04:31.237247: W tensorflow/core/distributed_runtime/rpc/grpc_session.cc:373] GrpcSession::ListDevices will initialize the session with an empty graph and other defaults because the session has not yet been created.
Initializing TPU system (master: grpc://<IP_ADDRESS>:8470) to fetch topology for model parallelism. This might take a while.
Found TPU system:
*** Num TPU Cores: 8
*** Num TPU Workers: 1
*** Num TPU Cores Per Worker: 8
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, -5884039716630823473)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU,<PHONE_NUMBER>4, -3837800863408535284)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU,<PHONE_NUMBER>4,<PHONE_NUMBER>800097538)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU,<PHONE_NUMBER>4, -1421254968311746768)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU,<PHONE_NUMBER>4, -5123767212799155974)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU,<PHONE_NUMBER>4,<PHONE_NUMBER>082208868)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU,<PHONE_NUMBER>4,<PHONE_NUMBER>607135313)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU,<PHONE_NUMBER>4, -4494021334020959541)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU,<PHONE_NUMBER>4,<PHONE_NUMBER>275849123)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM,<PHONE_NUMBER>,<PHONE_NUMBER>324654925)
*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU,<PHONE_NUMBER>4, -3269809841973767338)
From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
Calling model_fn.
WARNING:root:Changing batch size with sequential_input() will result in some data being skipped or repeated. Please ensure your batch size stays constant throughout training.
num_cores_per_replica: 1
computation_shape: [1, 1, 1, 1]
num_replicas: 8
device_assignment.topology.device_coordinates: [[[0 0 0 0]
  [0 0 0 1]
  [1 0 0 0]
  [1 0 0 1]
  [0 1 0 0]
  [0 1 0 1]
  [1 1 0 0]
  [1 1 0 1]]]
device_assignment.core_assignment: [[[0 0 0 0]]

 [[0 0 0 1]]

 [[1 0 0 0]]

 [[1 0 0 1]]

 [[0 1 0 0]]

 [[0 1 0 1]]

 [[1 1 0 0]]

 [[1 1 0 1]]]
2021-02-15 11:04:43.111549: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
device_list = ['/job:worker/task:0/device:CPU:0']
SimdMeshImpl ignoring devices ['', '', '', '', '', '', '', '']
SimdMeshImpl init: Shape[all=8] LayoutRules{('heads', 'x'), ('embd', 'y'), ('intermediate_expanded', 'x'), ('vocab', 'x'), ('memory_length', 'y')}
Device Assignment: <tensorflow.python.tpu.device_assignment.DeviceAssignment object at 0x7ff17ebca240>
serialize_num_microbatches: tokens_per_microbatch_per_replica=2048 batch_dim=Dimension(name='batch', size=256) sequence_length={'inputs': 2048, 'labels': 2048} batch_per_replica=256 num_microbatches=256

N TRAINABLE VARS:
1,315,581,952

ALL DIM NAMES:
intermediate_expanded
heads
embed_sequence
embd
vocab

### lots and lots
## and lots of model details

From /content/GPTNeo/utils.py:284: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
tf_update_ops: [<tf.Operation 'assign_948/group_deps_154' type=NoOp>, <tf.Operation 'assign_949/group_deps_308' type=NoOp>, <tf.Variable 'AssignAddVariableOp' shape=() dtype=int64>]
Create CheckpointSaverHook.
Bypassing TPUEstimator hook
training_loop marked as finished
Reraising captured error
Traceback (most recent call last):
  File "main.py", line 256, in <module>
    main(args)
  File "main.py", line 230, in main
    estimator.train(input_fn=partial(input_fn, global_step=current_step, eval=False), max_steps=next_checkpoint)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3130, in train
    rendezvous.raise_errors()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
    six.reraise(typ, value, traceback)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3125, in train
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1175, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1204, in _train_model_default
    self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2962, in _call_model_fn
    config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1163, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3386, in _model_fn
    _validate_tpu_training_graph(ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3817, in _validate_tpu_training_graph
    'CrossShardOptimizer must be used for model training on TPUs.')
ValueError: CrossShardOptimizer must be used for model training on TPUs.

That's what happens if I try using tensorflow 2.x in a colab notebook.
TPUEstimator hook gets bypassed for some reason I didn't really look into.

I got this error and fixed it by reducing train_batch_size to 8

Hi! Thanks for the feedback. We pushed some updates including a totally revamped notebook, give it a try and let me know if you're still getting this error.


How can download/upload with progress number?

Hi man,
Your library is so great. It save my time more. So, how can use this library to download/upload with progress number?
Thanks man

Thanks for you like it.
The bad news is it not support download/upload progress currently. :[
The god news is it will support soon. :]
Recently busy, I will add these features as soon as possible.

Thanks man, hope it support soon :D


Save only live cells, change concept

Add itertools as dependency for cartesian_product iterator adaptor
Replace explicit CellState type with plain boolean value
Store only live cells coordinates as generation
Store number of generations as age of the World
Remove get_sibling_coords, decide_fate, is_alive and set_state World methods
Implement seed function for initial population of the World
Implement evolve function in more functional style
Show number of live cells and age of the World
Allocate buffer with capacity enough to keep cli representation of the world
Improve lookup performance using HashSet instead of Vec
Store neighbours map to eliminate allocation
Bring back decide_fate method with slightly different semantic

i like it


Targeting PR#12145 to release

Targeting PR #12145 to release
ping @abose

LGTM Merging. Thanks @swmitra .


Add Exclusion Names for Rule file-name-casing

PR checklist

[X] Addresses an existing issue: fixes #4050
[X] New feature, bugfix, or enhancement

[X] Includes tests

[ ] Documentation update

Overview of change:

Add exclusion option for one or more files
Add test files

Is there anything you'd like reviewers to focus on?
N/A

@rwaskiewicz Could you maybe solve the merge conflict (or give me write access to your fork, so I could do it)?

Hey @rwaskiewicz, just a heads up - the file-name-casing rule had some pretty merge-conflict-y changes in the interim, including allowing an options object to apply a different casing preference per file name regexp matcher. That makes solving #4050 a bit easier! Added an "ignored" option to exclude file names.

...and we're merged! Sorry it took so long, and thanks to @rwaskiewicz for getting this started! The "ignored" option will be available in the next TSLint release.


[Website component bug] Do / Don't component has added padding

Need to remove forced padding on Do / Don't component so that it is optional / not true by default:

please also make top bar 3px tall (it is 2px now)

See example of forced padding causing an issue on the Color overview page:

Here is what's happening with text-based do/don't component:


Find a way to detect wrong string interpolation

It would be helpful to the user if we could detect at compile time this subtle problem:
sourceArn = s"${api.executionArn}/*"

instead of runtime error:
Diagnostics:
  aws:lambda:Permission (feedLambdaPermission):
    error: aws:lambda/permission:Permission resource 'feedLambdaPermission' has a problem: "source_arn" (besom.internal.Output@6da60311/*) is an invalid ARN: arn: invalid prefix. Examine values at 'feedLambdaPermission.sourceArn'.

And recommend user to use our interpolator:
sourceArn = p"${api.executionArn}/*"

This is a very common problem in all other SDKs, so this is a nice-to-have.
This might not be possible thou.

It's a solved problem (but we don't have docs so I don't blame you): there's a compiler plugin that crashes compile on any s"" interpolation that has an Output passed to it. We were just talking about that with @KacperFKorban who wrote the plugin because there's #108 and we have a problem there regarding which version should be added automatically to besom user program. Once #108 will be done this plugin will be added to user's program (not in sbt set up manually - we can only add it in sbt templates) automagically. I guess this can be closed as #108 tracks the progress on auto-include of the plugin.

We should probably document the need for compiler plugin before closing this bug, esp. for non-sbt users, WDYT?

I added a page about the compiler plugin here: https://github.com/VirtusLab/besom/pull/127/files#diff-255f8c5e2df43e2808fbac358cf75dcc97993ee8c73080d1b7cd3d64c47ae0ac
And also a note about using the interpolator here: https://github.com/VirtusLab/besom/pull/127/files#diff-e70b16e12d73c3e8a843e82e841da4447af28c91b4711fb6ee839dc4ad4607aeR17-R23

Great, thank you.
Note to self: make sure we have this compiler plugin in our templates!


Use editable-text with $sce.trustAsHtml?

Hello, thank you for this awesome package.  I'm trying to use the editable-text directive with html data.  If I inject the data using {{ data.html }}, it displays it as plain text instead of html (obviously).  But when I inject the data as

, the data is no longer editable, meaning that the editable dialog pops up, but the changes are not written to the $scope.
This is how I'm trying to use the directive:

And this is the original data in my controller:
$scope.user = {name: $sce.trustAsHtml("my
name")};

Is there any way around this?  I tried tinkering with the source but couldn't get it to behave correctly.

This is not an actual issue with the project. Questions such as this are better suited to be posted on sites like stackoverflow.com, using their Q/A format. Thanks.


How to download intersections data as geojson?

Hello, I have imported the json data and the area data with intersections are showing. I would like to export all that processed data as geojson or shp file.
Please help me on that. Thank you and waiting for your reply

Definitely. Any of these options sound better?

drag a .osm file that you clip yourself to the browser, do the conversion there
use the browser app, but be able to drag over a .geojson file with a boundary, and make a very large query to Overpass
draw the boundary polygon in the browser app directly
compile a CLI tool (requiring you to install Rust) and run it on a .geojson polygon, totally out of the browser

We could add a geojson export to the A/B Street UI, where you've already successfully imported a large area (I'm assuming). But I'd prefer to move away from cramming more features in there, especially when we have this new split-out tool that's more focused.
Also curious about your use case for the exported shapes. Is a polygon per road and intersection OK? What kind of properties/metadata about the objects are you interested in?

My 10 cents: from a reproducible research perspective the CLI tool option sounds good. All of the others sound good too and not sure what is best but big :+1: to the CLI option.

Thank you @dabreegster really needed this.
Can you please show me an example in detail, how I can implement a CLI tool. (I apologize for asking, as I'm new at programming)?  Although I tried dragging  an.osm file into the OpenStreetMap web tool, it didn't really help!
how can I implement any of  your suggested methods to export the geometry Geojson for a very large queries.
If you can let me know in detail that would be great.
My main objective is to obtain those intersection geometries for very large boundary extents.
Thank you.

Sorry for not being clear, I was asking which option sounded easiest as an end user, so I could choose what to implement. I decided to shove this into A/B Street for now, since it already has a nice experience for importing large areas. I'll cut a release before Friday and post instructions here how to use the tool.
I'll leave this issue open, because I think handling larger imports in the osm2streets web tool would be good too.

@GopiJogu, the workaround is:

Download the new version, https://github.com/a-b-street/abstreet/releases/tag/v0.3.29
Import your area
In the main view, press Ctrl+S to enable dev mode (bottom right corner)
Press Ctrl+D to enter debug mode
Scroll down the panel and click export geometry to geojson
There won't be any confirmation popup. But there should be a file called map_geometry.json that appears in the folder with A/B Street

Hello @dabreegster
Thanks for releasing new version early.
I've tried with a new version, After I import the new area into OpenStreetMap map viewer,
When I click Ctrl+S or CTRL+D, it shows nothing.
I'm sorry for troubling you, but can you help me out with this issue.
Please see the loom recording for better understanding of my issue.
Thanks again.
https://www.loom.com/share/71c81f1a6c894598b346b204424132ec

Ah, you need to run game.exe instead. On the title screen you'll click "Traffic sim sandbox" and then the Ctrl+S part

And you don't need to import again, all of the tools see the same cities. In the "select a district" panel instead of "Import a new city", scroll all the way down and choose "zz"

I got it right now.
Thanks @dabreegster  for clear explanation. :)

Hello,
I am trying to import few areas, but I see this error (Please see attached screenshot)
Can you please help me on this one?
Thanks.

Send the geojson area please so I can reproduce this issue
On Fri, Aug 19, 2022, 10:10 AM GopiJogu @.***> wrote:

[image: Failure]
https://user-images.githubusercontent.com/110944253/185573984-2a85c567-d25b-4206-b38b-a31bd4389aa5.jpg
Hello,
I am trying to import few areas, but I see this error (Please see attached
screenshot)
Can you please help me on this one?
Thanks.
—
Reply to this email directly, view it on GitHub
https://github.com/a-b-street/osm2streets/issues/57#issuecomment-1220381025,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAMWLF5AS62CQDTEWK4ROQTVZ46P5ANCNFSM56B4B55Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hello, Please see the below area:
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"MINX":<PHONE_NUMBER>3000008,
"MINY": -33.88317376899994,
"MAXX":<PHONE_NUMBER>9000005,
"MAXY": -33.61228139899998,
"CNTX":<PHONE_NUMBER>6000006,
"CNTY": -33.74772758399996,
"AREA": 0.0637835751950892,
"PERIM": 1.012699259999863,
"HEIGHT": 0.2708923699999559,
"WIDTH": 0.23545725999997558
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
150.82349939000005,
-33.74430356376206
],
[
150.58804213000008,
-33.74380365450516
],
[
150.58804213000008,
-33.61228139899998
],
[
150.82349939000005,
-33.61228139899998
],
[
150.82349939000005,
-33.74430356376206
]
]
]
}
}
]
}

Hi @dabreegster
Is there any update on this please?
Thank you.

Hi
Sure. I understand.
Thank you. :)

@GopiJogu, https://github.com/a-b-street/abstreet/releases/tag/v0.3.32 should fix this. By the way, the area you sent generally doesn't have lanes tagged, so even big motorways show up as one lane by default. Depending what you want to do there, contributing some tagging to OSM would be useful!

Thank you @dabreegster it would be very useful for me.
Is it possible for me to modify the width of every road at once?
If I can provide a custom width based on the classification of the highway?

@GopiJogu, this is controlled by https://github.com/a-b-street/osm2streets/blob/3fd448555a9153cfb417fe92df68d741e76055a6/street_network/src/types.rs#L354 right now. It's based on each lane, not the entire road. Ultimately I'd imagine to guess a reasonable default lane width would depend on the locale (suburban US vs middle of London) and OSM highway classification.
Sometimes you might know the entire road width from another source. If you can detangle exactly what that width captures (between the curb? Sidewalks and grass verges/), another needed algorithm would pick reasonable width values for all of the lanes so that the sum matches this value.
What exactly do you want to modify? And how willing are you to jump into the code?

No actionable followup here. We're still working towards locale-dependent lane width defaults.


[Issue/Question] Unable to edit or replace deployment when trying to run two parallel GitHub Actions pipelines

When running azd provision in a GitHub Actions pipeline, it frequently runs into a race condition where if the pipeline was triggered more than once, for example if two changes were pushed, ARM will complain that there's an already existing deployment taking place.
Is there some guidance on what's the best way to approach this? This seems very common.
ERROR: deployment failed: error deploying infrastructure: failed deploying: starting deployment to subscription: PUT https://management.azure.com/subscriptions/***/providers/Microsoft.Resources/deployments/***
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: DeploymentActive
--------------------------------------------------------------------------------
{
  "error": {
    "code": "DeploymentActive",
    "message": "Unable to edit or replace deployment '***': previous deployment from '4/6/2023 11:54:53 PM' is still active (expiration time is '4/13/2023 11:54:50 PM'). Please see https://aka.ms/arm-deploy-resources for usage details."
  }
}
--------------------------------------------------------------------------------

This could be due to a upstream GitHub issue discussed here #1767

@sabbour , you can try a queue strategy like:
https://github.com/marketplace/actions/github-workflows-queue
or
https://github.com/marketplace/actions/workflow-queue
It would make flows to run one after the other instead of on demand.
@rajeshkamal5050  @ellismg  @savannahostrowski , let me know if you want to update our samples using this strategy.
The other strategy is to make any newest workflow to cancel the previous one

@vhvb1989 not exactly, I think this is more around giving some thought into the assumptions. Particularly:

Is there an assumption that only one person is working on the project, hence there will not be multiple code commits that are competing to deploy the changes?
Since azd provision and azd deploy are not doing differential deployments and essentially they're packaging the whole thing, is it really safe to assume that they can be used in a pipeline to deploy the changes to the same environment without running into a race condition?
Should we revisit the purpose of azd (including in docs and examples) so that this is more of a deploy to a "developer environment" per developer versus being a tool to deploy to production?
What if we think about azd in deployment pipelines as a way to create pull request specific e2e testing environments? Can we create examples for that?

I'm just thinking out loud :)

@sabbour , you can try a queue strategy like: https://github.com/marketplace/actions/github-workflows-queue or https://github.com/marketplace/actions/workflow-queue
It would make flows to run one after the other instead of on demand.
@rajeshkamal5050 @ellismg @savannahostrowski , let me know if you want to update our samples using this strategy.
The other strategy is to make any newest workflow to cancel the previous one

The queue approach is a good way to start.


delete duplicate config about resize windows

RT

The reason why I left them in mappings.lua is for the situation that smart-splits.nvim removed should also has the mappings to resize window.


samples issue when not using unsigned byte

http://www.html5gamedevs.com/topic/34517-aa-in-webgl20/?tab=comments#comment-198603

Associated with #3435


Bugfixes

Coverage remained the same at 37.82% when pulling 914abafc414cb00838719bc37fccc96d3181fe93 on next into ba92361becb09ca09ef662d9b6210217415c0c16 on master.

Coverage remained the same at 37.82% when pulling 914abafc414cb00838719bc37fccc96d3181fe93 on next into ba92361becb09ca09ef662d9b6210217415c0c16 on master.

Coverage remained the same at 37.82% when pulling 914abafc414cb00838719bc37fccc96d3181fe93 on next into ba92361becb09ca09ef662d9b6210217415c0c16 on master.


The tracker name and project name styling; Inconsistent on search result

Fixed with 6ffb92a07452f3c993dce1971359a56adefefe01
Can someone please verify and close

Looks good.


in firefox not working

Hello   in firefox all buttons do not work
http://prntscr.com/mtwwv7
regards

Hello,
On my version, all buttons work (Firefox Developer Edition - 66.0b13).

It works for me as well.


Remove unused pip import in setup.py

This import doesn't seem to be used, and it breaks when building this package in an environment without pip.

Hi @tjni
Sorry for the long delay, as we are working on a new process of contributions, for streamline the contribution, and makes everyone's PR easy to merge in.
master is currently the release branch, could you update your PR, and targeting it to the dev branch ?
and you don't need to update the CHANGELOG as well,  so we can get this in ASAP,  Thanks


Replay doesn't apply Choice Variables

Cookiecutter version: 1.5.0
Template project url: private repo, but I can make an example if it helps
Python version: 3.6.0
Operating System: MacOS 10.12.3

Description:
I'm trying to apply a replay file to a cookiecutter template that uses a Choice Variable. The value of the Choice Variable is correctly recorded in the replay file after templating, but the choice variable value is ignored if passed in via extra_content.
What I've run:
I use a custom script utilizing the API rather than the CLI. I can provide an example if it helps (copy and paste won't add much context)

Sorry, my mistake -- I misunderstood how choice variables are applied (thought it would remember the index, instead changes the order). Sorry to bother you, thanks for the great tool!

@mjlyons Did you manage to override a choice variable using extra_context? I'm also using the API rather than the CLI, and I have a use case in which I want to populate the list of choices dynamically.

@jmdacruz Sorry about the absurdly late reply, lost access to my personal development machine and had to get another. Yes this does work for me, the thing that tripped me up is that cookiecutter changes the order of choices (to always make the default the first, I believe). Is this working for you now?

No worries @mjlyons! Yes, this is now working for me too.


Homebrew  can't find package

When installing via homebrew, I am getting the following error:
$ brew install jawshooah/pyenv/pyenv-default-packages
==> Installing pyenv-default-packages from jawshooah/homebrew-pyenv
==> Downloading https://github.com/jawshooah/pyenv-default-packages/archive/0.3.0.tar.gz
######################################################################## 100.0%
curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "pyenv-default-packages"
Download failed: https://github.com/jawshooah/pyenv-default-packages/archive/0.3.0.tar.gz

Sorry about that, I deleted and recreated the repo to remove the fork relationship with rbenv-default-gems, but forgot to push the tags back up with the commits. Should be working now!


Requests by IP in timeline?

Thanks for this tool, just have started playing with it. Sorry to disturb and I guess I have overseen it, but I can't find my personal most important table:
Requests in a timeline, which shows sortable IP, Username, Country, Browser, OS and HTTP Status Code columns. Maybe also RX and TX per request and when I click on the IP, it should show Referring site like it's the User-Agent in the "Visitor Hostnames and IPs" table.
If enabled in the logs, the requested virtual host should be also displayed in a sortable column.
Thank you :)

Hello, #117 will let you do this easily. I'm working on it as we speak and should be coming as soon as I'm on the last stages of it. Stay tuned.

Is #117 still planned?

No worries, it's definitely in the works and looking good. There's a bunch of behind-the-scenes stuff happening before we can roll it out. Just making sure everything runs smoothly. Keep an eye out for updates!


Feature/attendable event overengineered

Proposed Changes
This branch contains the first attempt of making attendable events have their own models, it is somewhat overengineered, so enter at your own risk:)
Types of Changes

[ ] New feature
[ ] Bug fix
[ ] Enhancement/optimization
[ ] Refactor
[ ] Style
[ ] Build/dependencies
[ ] Other

Issue(s) fixed or closed by this PR

Checklist

[ ] I have added tests to cover my changes (for features/bugs)
[ ] I am pleased with the readability of the code (if not, state where you need input)
[ ] I have tested the changes and verified that they work and do not break anything (to the best of my ability)

No longer needed, closing


[opencv4] OpenJPEG feature

[x] Changes comply with the maintainer guide
[x] SHA512s are updated for each updated download
[x] The "supports" clause reflects platforms that may be fixed by this new version
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

@microsoft-github-policy-service agree

Feature openjpeg is tested successfully in the following triplet:
x86-windows
x64-windows
x64-windows-static


Improve the output when a promise was expected to be fulfilled but was rejected

When a promise is rejected which you are expecting to be resolved, the output should report what it was rejected with. Currently I have to go and find the test and change it from thing.should.be.fulfilledWith(expected) to thing.should.be.rejectedWith(expected) so that it tells me why it was rejected.
Assuming that the function rejects with { message: 'Cannot read property \'startDate\' of undefined' }
Currently:
AssertionError: expected [Promise] to be fulfilled

Desired:
AssertionError: expected [Promise] to be fulfilled but was rejected with { message: 'Cannot read property \'startDate\' of undefined' }

Yes, make sense :+1:

Pls, try 8.3.0

That is super awesome! Thanks for turning that round so quickly


how to compile plip for ubuntu 64?

It says in the installation guid of vtorcs-color:
Requires plib 1.8.5, FreeGLUT or GLUT, be aware to compile plib with -fPIC on AMD64 if you run a 64 bit version of Linux. Be aware that maybe just 1.8.5 works.
I'm still new to this kind of stuff, so could you please tell how to do that?

Copied from http://www.berniw.org/tutorials/robot/tutorial.html
Download and Unpack Plib
Download plib-1.8.3.tar.gz from here. TORCS has been tested against 1.8.3, for other versions we guarantee for nothing! Stay as root and do
cd /usr/src
mkdir torcs
cd torcs
tar xfvz /path_to_downloaded_files/plib-1.8.3.tar.gz
cd plib-1.8.3
Compiling and Installing Plib
You are still root and in /usr/src/torcs/plib-1.8.3, if you run a 64 bit version of Linux export the following variables:
export CFLAGS="-fPIC"
export CPPFLAGS=$CFLAGS
export CXXFLAGS=$CFLAGS
Now for all platforms run:
./configure
make
make install
Just to play safe clear the above defined variables:
export CFLAGS=
export CPPFLAGS=
export CXXFLAGS=
If something fails you need to resolve it. In case the configure script complained about something you can find some additional information in the file config.log. The cause for problems are usually missing header files or libraries, wrong versions or multiple versions installed.


Add consolidated metadata to rechunked zarr

What is the issue?
When opening a zarr with xarray, it really helps to have what is called "consolidated metadata". This is a single file (.zmetadata) at the root of the zarr, which combines all the information of all the various .zarray and .zattrs files inside the zarr. While this file is not required to exist in a valid zarr (nor in fact for the zarr to be opened by xarray), it does greatly speed up the process, especially when the data is being read from a remote location.
Sadly, Rechunker currently does not create such a consolidated metadata file for the resulting rechunked zarr. There is also (as far as I have been able to find, at least) no way to enable such behavior via an option parameter.
What would solve the issue?
One of the following feature requests would be able to resolve this issue:

Add an optional boolean parameter to the rechunk() function which allows to user to specify that a consolidated metadata file should be created. For backwards compatibility, if that is desired, this parameter would default to False. Potential parameter names could be consolidated, mirroring the parameter name in xarray, write_consolidated to be more explicit that it only impacts writing, or consolidate_metadata to mirror the function in zarr
Automatically detect the existence of a consolidated metadata file (.zmetadata) in the source zarr, and create one (or not) in the result zarr accordingly.
The combination of options 1 and 2. The parameter could default to a str value of "auto", resulting in the behavior as described in (2), or be given a boolean value by the user to override this behavior.

I look forward to hearing what people think about this feature request, and to know if others would also like to see this feature added!

Thanks for this suggestion @Metamess!
Creating consolidated metadata after rechunking is done is a one-line operation, e.g.
zarr. consolidate_metadata(target_store)

(https://zarr.readthedocs.io/en/stable/tutorial.html#consolidating-metadata)
This can be run on the target store after the rechunking is complete. Would that meet your needs?

Hey @rabernat , thanks for the reply!
I did already know that you can manually create the consolidated metadata like this afterwards, and it is in fact what I am currently using to work around this issue! But it is a step that I would expect to be possible as part of the rechunk operation. In advocating for this feature, I consider the following:

While the consolidated metadata is not part of the zarr specifications, and thus not strictly required for a "valid" zarr, it has seemingly become cornerstone feature. I think it can be safely assumed that a vast amount of zarr users use xarray to work with their data (in fact, even rechunker's tutorial relies on xarray!) and it is clear that xarray has a strong preference for the use of consolidated metadata files. Add to that the fact that Zarr-Python itself contains the functionality to create such files, and this signals to me that the use of consolidated metadata is sufficiently widespread that it would make sense for the rechunker package to support its use.

Indeed, adding the extra zarr.consolidate_metadata(target_store) line at every location the rechunk function is called is technically an option, as would be wrapping the two lines in a function of your own to assist with that. But since I think this use case (wanting consolidated metadata) will occur for so many users, and perhaps even in the majority of cases where rechunker is used, it would seem to make sense for the rechunker package to offer this functionality.

Additionally, consider the case where the source zarr has consolidated metadata. The lack of this file in the result is technically an undocumented side effect (since on top of rechunking the data, this file is also "lost", i.e. not recreated). It breaks the (in my opinion reasonable) assumption of equivalence between the source and rechunked zarr. (I consider the rechunked zarr to not be functionally equivalent, as opening it will now require many reads, and xarray will now print the warnings associated with opening a zarr without consolidated metadata).

Lastly, when zarr.consolidate_metadata(target_store) is called it triggers multiple reads on the result zarr. When the zarr is on a remote machine, this is slow and inefficient (that is after all the entire reason for the consolidated metedata even exists). It seems like it would be way more optimal for the consolidated metadata to be gathered from the various zarr arrays as they are created by the rechunk operation. This is an implementation detail however, and is separate from the previous arguments in favor of the feature in general.

What do you think? Is it worthwhile as a feature in rechunker?


Convert all our scripts to PowerShell

The problem
We have to maintain two versions of some script (.sh and .cmd). This extra maintenance. We also use PowerShell scripts in other cases. So, we need to maintain three types of scripts. PowerShell works on all platforms so let's use that.
The proposed solution
Convert the .sh and .cmd scripts with powershell. Or remove them if possible.
Additional context
Files involved:

Scripts\test-image-copier.cmd
Scripts\InstallDependencies.cmd
Scripts\NugetRestore.cmd
docfx\build-site.cmd
Scripts\Mac\buildpack.sh
Scripts\Mac\cleanup.sh
docfx\build-site.sh

This was done as part of this PR #2551


"New test" with different number of users

It seems likes a bug to me, but please correct me if I'm wrong. It seems that after clicking on "new test" and giving lower amount of users to be created, Locust still uses some of the users created on first run.
Describe the bug
Example:
When i start locust and create (total) 5 users with 5 users per sec, I get correct amount of users/requests. That is correct. Then i click Stop (in the UI).
After clicking on "New test" and specifying lower amount of users(total 1 and 1 per sec), Locust still starts 4 users in total and execute tasks - but in console i can see that 1 user was spawned
Code:
`
import logging
import time
import uuid
from locust import task, HttpUser, SequentialTaskSet, constant_pacing, task, events
from locust.exception import StopUser
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("locust")
class UserTasks(SequentialTaskSet):
wait_time = constant_pacing(1)
@task
def test_api_one(self):
    logger.info(f"Task1: {self.parent.user_id}")
    time.sleep(1)

@task
def test_api_two(self):
    logger.info(f"Task2: {self.parent.user_id}")
    time.sleep(1)

@task
def test_api_three(self):
    self.interrupt(False)

class WebsiteUser(HttpUser):
host = "http://local"
wait_time = constant_pacing(5)
tasks = [UserTasks]
def on_start(self):
    self.user_id = str(uuid.uuid4())
    logger.info(f"START USER: {self.user_id}")

def on_stop(self):`

Expected behavior
I would expect that if i run new test with lower from 5 to 1 users, Locust will just create 1 user and run only that one.
You can see it on the screenshot:
Environment

OS: docker with python 3.9
Python version: 3.9
Locust 2.9.0
locust -f temp.py

Wow, that looks crazy. Can you update to latest and see if it makes a difference?

Hi.
I got the same results on latest (v.2.10.1)

Any difference if you remove test_api_three?

Very strange. I’ll take a look.

Getting the exact same behavior. Additionally, all the users from the previous test are spawned instantaneously and don't respect the chosen "Spawn rate".

Damn, I dropped the ball here. Will pick it up some time next week if nobody else has the time.

I implemented a failing test case in https://github.com/locustio/locust/pull/2152/commits/dfa3074b7a1614cb287266e2bd9b0a5e65f7cc15


Add definitions for MacRuby

Would it be possible to add definitions for MacRuby (either from git or tarball)?

I'd welcome a patch for this, but I'm afraid I don't have the time to work on it myself. Closing the issue for now, but feel free to open a pull request.