RegExp hang

Dunno why but this hangs: test.dart.txt
That is, pattern.allMatches(input).toList() for a version of pattern that is a huge absurd machine-generated RegExp and a version of input that contains a Burmese dictionary.

It's probably the RegExp.
A very quick look at the beginning shows (prettified):
^(
  (?:[-;@#<!.\\"/* ]*
     (?:REM[-;@#<!.\\"/* ]*)?
     [-;@#<!.\"/*]+
  )?)
  ( *)
  (?:BSD 2

That alone can spend quadratic time matching a line of -----------------------------------------------------------.
The first [- ...]* starts by matching everything. Then the "REM" doesn't occur, but that's OK, it's inside a ?.
Then the second [-....]+ matches nothing. No space. Next character is not B, so we backtrack.
That goes back to the first [-...]* which drops back by one -, then the second [-...]+ matches that - and we continue not finding B. This repeats until we have failed to find B N-squared times for N -s.
The continuation is actually
( *)
  (?:BSD 2-Clause License \(http://www.opensource.org/licenses/bsd-license.php\))? *

so the BSD-part itself is also optional and the  * surrounding it can do the same quadratic matching of repeated spaces (which is probably less likely to happen in practice).
I'm guessing that if we continue deciphering the regexp we'll finde more polynomial backtracking issues.
I gave up understanding the rest of the RegExp, though. I'd rather see the code generating it, it's likely to be more readable (RegExp, like Perl, are write-only code :wink:).
Without a textual description of what a RegExp is intended to do, it's pretty much impossible to understand it, or check that it does what it should. Definitely not self-documenting.
I'd recommend just not using a RegExp for something of this magnitude. They're not magical speed-demons that always beat properly written code, and for complex things, they're also nigh impossible to check for correctness.

Looking at it more, it seems the RegExp tries to recognize the 2-clause BSD license in a comment, no matter what comment syntax is used, or how it's line-broken, or how many spaces are put between words.
(Which prompts the question "Why?".)
It almost works on this input, which contains a 3-clause BSD license text. That means it probably matches up to the second clause, then fails.
Then it tries one gazillion other ways to backtrack and match again (probably mostly by rearranging how the whitespace is matched), which it can because the RegExp is ambiguous in several ways, and every time it still fails at the third clause and tries another backtracking.
That's exactly the kind of situation where RegExps perform horribly.
It'll probably complete eventually, if not in this universe's life time, then the next.
(Cutting away the Burmese dictionary and header comment doesn't make any visible difference, just the 3-clause BSD license by itself is enough.)
I'd probably just do something primitive like replacing every non-releveant character sequence with spaces:
void main() {
  var normalized = input.replaceAll(normWordsRE, " ");
  var matches = normalized.contains(bsd2Words);
}

final normWordsRE = RegExp(r'(?:^ *(?:REM\b|/+)|[^a-zA-Z\d.,/:;()"])+');

const bsd2Words =
    'Redistribution and use in source and binary forms, with or without modification, '
    'are permitted provided that the following conditions are met: '
    'Redistributions of source code must retain the above copyright notice, this '
    'list of conditions and the following disclaimer. '
    'Redistributions in binary form must reproduce the above copyright notice, this '
    'list of conditions and the following disclaimer in the documentation and/or '
    'other materials provided with the distribution. '
    'THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND '
    'ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED '
    'WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE '
    'DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR '
    'ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES '
    '(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; '
    'LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON '
    'ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT '
    '(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS '
    'SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.';

It's not precise, but it does require the correct word in the correct order with no other letters or digits mixed in between them.

To be clear I totally agree that this is ridiculous code. It's from the Flutter engine license script (see https://github.com/flutter/engine/blob/main/tools/licenses/lib/patterns.dart and files in that directory; IIRC the precise pattern in this test is from https://github.com/flutter/engine/blob/main/tools/licenses/lib/patterns.dart#L1352) and is absolutely the wrong way to do this. The approach has not scaled. At some point I want to change the code to first tokenize the source files and then run the patterns on them that way.
That said, this seems like a weird pathological case. This regular expression (with [\r\n]+ changed to \n+ in one place) is in production today, and is compared against thousands of files regularly. I think the precise input file is not being tested against it today, but it's not that different from other files that are. It's disturbing to see such a drastic difference in performance.

(If it helps, you can think of this as a security report highlighting a potential DOS attack. :-) )

Using user-provided RegExps on user-provided input (or any input, really) should generally be considered a DOS risk.
Having a complex RegExp applied to arbitrary user input is also a risk. The RegExp should be vetted and tested carefully before being released to the world.
It's not something we can really fix while maintaining the ECMAScript RegExp behavior. The backtracking behavior, and order, is embedded in the specification. There might be some situations where a RegExp compiler can determine that if something fails once, there is no need to go back and try other approaches, but it's not generally solvable (it is equivalent to rewriting the RegExp into another one which matches the same thing, and RegExp equivalence in general is NP-space complete. And that's for real regular expressions, not the expanded ones with back-references which makes the Dart RegExp not even context free).
All in all, some RegExps being able to spend inordinate amounts of time on some inputs is a known problem with no known practical solution, so this is "working as designed".


{CI} Bump pyyaml to latest version 6.0.1

Related command

Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

CI

Knack doesn't pin PyYAML: https://github.com/microsoft/knack/blob/11f2454b8e4d977797057231d0850e7935b26fa6/setup.py#L17
DEPENDENCIES = [
    ...
    'pyyaml',

So I assume Azure CLI should do the same to avoid frequent security alerts triggered by PyYAML.


Plugin don't get Token in iOS 9.*

I'm testing the plugin in version 2.3.1-rc on Iphone version 9.3 and I don't receive the token. OnTokenRefresh not firing. Any idea?

This just happens on iOS 9?  iOS 10+ working properly for you?

iOS 10+ works perfectly.

I'm also facing the same issue. Wasted 1 whole day working with iOS 9.
After reading this post I tried with iOS10 and its working. Thank you for posting this issue.

Please, SomeOne, suggest the solution once you find it.

Hi! same problem here, my iPad Mini 2 and iOS 9.3.5 is not registering a token. My other device with iOS 11.3.1 works fine. I can't try iOS 10 right now.

@rubenmv I'm having the same issue

iPhone 5S iOS 10.3.2
Have you found anything which solves that?

@bMiquelin no sorry, I haven't had the time to look into it :(

I finally made it work!
I've downgraded to these versions:
https://github.com/CrossGeeks/FirebasePushNotificationPlugin/issues/51#issuecomment-341190583
Those errors appeared:
2018-06-15 12:11:51.965 BasicApp.iOS[2117] <Error> [Firebase/Core][I-COR000012] Could not locate configuration file: 'GoogleService-Info.plist'.
2018-06-15 12:11:51.970 BasicApp.iOS[2117] <Error> [Firebase/Core][I-COR000005] No app has been configured yet.
2018-06-15 12:11:55.450 BasicApp.iOS[2117] <Error> [Firebase/Core][I-COR000003] The default Firebase app has not yet been configured. Add `[FIRApp configure];` (`FirebaseApp.configure()` in Swift) to your application initialization. Read more: https://goo.gl/ctyzm8.
2018-06-15 12:11:55.451 BasicApp.iOS[2117] <Error> [Firebase/Core][I-COR000003] The default Firebase app has not yet been configured. Add `[FIRApp configure];` (`FirebaseApp.configure()` in Swift) to your application initialization. Read more: https://goo.gl/ctyzm8.
2018-06-15 12:12:03.293 BasicApp.iOS[2117] <Error> [Firebase/Core][I-COR000003] The default Firebase app has not yet been configured. Add `[FIRApp configure];` (`FirebaseApp.configure()` in Swift) to your application initialization. Read more: https://goo.gl/ctyzm8.

I found out  that my GoogleService-Info.plist Build Action was set to "None", then changed it into "BundleResource". Had to clean & delete bin/obj, and it worked!
Now i'm gonna check why I can only send messages to iOS via firebase console, and not by postman.
Following POST works only for Android:
{
  "to": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "data": {
    "type": "status",
    "title": "Test",
    "body": null,
	"icon": "https://distriqt.github.io/ANE-PushNotifications/images/icon.png",
    "color": "#9DC14C",
  }
}

@bMiquelin Thank you Bruno, I'll try to downgrade next week.
About your message for iOS, try this:
{
  "to": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "notification": {
    "title": "Test",
    "body": null,
    "sound": "default",
    "content_available": true
},
  "data": {
    "type": "status",
    "icon": "https://distriqt.github.io/ANE-PushNotifications/images/icon.png",
    "color": "#9DC14C",
  }
}

Got it working on iOS10 by changing the GoogleService-Info.plist build-action, not the versions. (Updated version isn't showing anything on the console to warn me about that)
I'm still stuck trying to make it work on an iPhone 4S iOS 9.3.5 device. Even using [2.3.1-rc], [2.3.0], [2.0.1-alpha], etc. Token does not trigger.
About the JSON: Thanks, splitting "notification" and "data" was needed, "content_available" was tricky with --dash or __underscore. iOS doesn't allow the body to be null.

Should be fixed on this beta version:
https://www.nuget.org/packages/Plugin.FirebasePushNotification/2.3.5-beta
Will do an official version release next month


feat(examples): add usb-serial example

Description
This adds a usb serial example.
It is basically a port of one of Embassy's examples.
I tested that it works, with Linux as host, on nrf52840dk, rpi-pico and two stm32 (which are not in main yet).
Issues/PRs references

Open Questions

Change checklist

[x] I have cleaned up my commit history and squashed fixup commits.
[x] I have followed the Coding Conventions.
[x] I have performed a self-review of my own code.
[x] I have made corresponding changes to the documentation.

gives me 8000 bauds.

I digged a bit, IMO that can just be ignored.
upstream code says so. Also, the baud rate can only be set from the host.


fix: Resolve send button issue

Description
This PR has the purpose to solve the small shift of the send button in the safari browser, the following steps were performed:

I exported the send svg as a react component (IconSend), and imported this component for use in App.js. I also added the components/ and icons/ folders to help with overall organization
Added CSS classes, where appropriate, to solve the reported issue and give the desired visual effect

Mainly the changes were made to footer section.

Extra:  Also fixed the following lint warnings

Added alternative text alt={"Logo"} to image of logo.
Added rel="noreferrer" to header links, as recommended with links that open a new tab in the browser.
Change a element and use the button element instead.
Fix typo class to className
Add z-index css property to improve the appearance of the scroll bar in Mozilla browser (see image below)
Removed unnecessary CSS styles that had no impact

Related Issue
resolves #9
Motivation and Context
The intention to apply this change is solve the issue and improve the code
How Has This Been Tested?
After making the changes I ran yarn run start and confirmed that everything appeared as expected
Screenshots (if appropriate):

Thanks for your feedback. Changes done!

wow, incredible! tyvm!

I'm having a small quirk here on chrome, this area I painted red is an extra margin that is not clickable as button! maybe remove it? idk if it would break safarai again tho hahahah


[Request for External] Page 344 "Navigate" - Protected variables

Hello, we need the following currently global variable/-s to be set as protected. Can you please check this? Thank you!
protected var
        SalesShptHeader: Record "Sales Shipment Header";
        SalesInvHeader: Record "Sales Invoice Header";
        ReturnRcptHeader: Record "Return Receipt Header";
        SalesCrMemoHeader: Record "Sales Cr.Memo Header";

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 44906.


Importing JS folders with index.jsx file in them requires /index in module specifier

TypeScript Version:  2.3.1-insiders.20170420
Code
See repo here:
https://github.com/mohsen1/ts-2.3-module-resolution-issue
Importing a folder with absolute path when rootDir is set is not working without extra /index at the end.
I have to have import with an extra /index at the end to make import work in baz/index.js

tsc output:
 tsc --traceResolution
======== Resolving module 'foo' from '/Users/mazimi/Desktop/ts23-issue/src/index.ts'. ========
Module resolution kind is not specified, using 'Classic'.
File '/Users/mazimi/Desktop/ts23-issue/src/foo.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/foo.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/foo.d.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/foo.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/foo.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/foo.d.ts' does not exist.
File '/Users/mazimi/Desktop/foo.ts' does not exist.
File '/Users/mazimi/Desktop/foo.tsx' does not exist.
File '/Users/mazimi/Desktop/foo.d.ts' does not exist.
File '/Users/mazimi/foo.ts' does not exist.
File '/Users/mazimi/foo.tsx' does not exist.
File '/Users/mazimi/foo.d.ts' does not exist.
File '/Users/foo.ts' does not exist.
File '/Users/foo.tsx' does not exist.
File '/Users/foo.d.ts' does not exist.
File '/foo.ts' does not exist.
File '/foo.tsx' does not exist.
File '/foo.d.ts' does not exist.
Directory '/Users/mazimi/Desktop/ts23-issue/src/node_modules' does not exist, skipping all lookups in it.
File<EMAIL_ADDRESS>does not exist.
Directory '/Users/mazimi/Desktop/node_modules/@types' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts23-issue/src/foo.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/foo.jsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/foo.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/foo.jsx' does not exist.
File '/Users/mazimi/Desktop/foo.js' does not exist.
File '/Users/mazimi/Desktop/foo.jsx' does not exist.
File '/Users/mazimi/foo.js' does not exist.
File '/Users/mazimi/foo.jsx' does not exist.
File '/Users/foo.js' does not exist.
File '/Users/foo.jsx' does not exist.
File '/foo.js' does not exist.
File '/foo.jsx' does not exist.
======== Module name 'foo' was not resolved. ========
======== Resolving module 'react' from '/Users/mazimi/Desktop/ts23-issue/src/bar/index.jsx'. ========
Module resolution kind is not specified, using 'Classic'.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/react.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/react.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/react.d.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/react.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/react.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/react.d.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/react.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/react.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/react.d.ts' does not exist.
File '/Users/mazimi/Desktop/react.ts' does not exist.
File '/Users/mazimi/Desktop/react.tsx' does not exist.
File '/Users/mazimi/Desktop/react.d.ts' does not exist.
File '/Users/mazimi/react.ts' does not exist.
File '/Users/mazimi/react.tsx' does not exist.
File '/Users/mazimi/react.d.ts' does not exist.
File '/Users/react.ts' does not exist.
File '/Users/react.tsx' does not exist.
File '/Users/react.d.ts' does not exist.
File '/react.ts' does not exist.
File '/react.tsx' does not exist.
File '/react.d.ts' does not exist.
Directory '/Users/mazimi/Desktop/ts23-issue/src/bar/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts23-issue/src/node_modules' does not exist, skipping all lookups in it.
File<EMAIL_ADDRESS>does not exist.
Found 'package.json' at<EMAIL_ADDRESS>'package.json' does not have a 'typings' field.
'package.json' does not have a 'types' field.
File<EMAIL_ADDRESS>exist - use it as a name resolution result.
======== Module name 'react' was successfully resolved to<EMAIL_ADDRESS>========
======== Resolving module 'baz' from '/Users/mazimi/Desktop/ts23-issue/src/bar/index.jsx'. ========
Module resolution kind is not specified, using 'Classic'.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/baz.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/baz.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/baz.d.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz.d.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/baz.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/baz.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/baz.d.ts' does not exist.
File '/Users/mazimi/Desktop/baz.ts' does not exist.
File '/Users/mazimi/Desktop/baz.tsx' does not exist.
File '/Users/mazimi/Desktop/baz.d.ts' does not exist.
File '/Users/mazimi/baz.ts' does not exist.
File '/Users/mazimi/baz.tsx' does not exist.
File '/Users/mazimi/baz.d.ts' does not exist.
File '/Users/baz.ts' does not exist.
File '/Users/baz.tsx' does not exist.
File '/Users/baz.d.ts' does not exist.
File '/baz.ts' does not exist.
File '/baz.tsx' does not exist.
File '/baz.d.ts' does not exist.
Directory '/Users/mazimi/Desktop/ts23-issue/src/bar/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts23-issue/src/node_modules' does not exist, skipping all lookups in it.
File<EMAIL_ADDRESS>does not exist.
Directory '/Users/mazimi/Desktop/node_modules/@types' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/baz.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar/baz.jsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz.jsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/baz.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/baz.jsx' does not exist.
File '/Users/mazimi/Desktop/baz.js' does not exist.
File '/Users/mazimi/Desktop/baz.jsx' does not exist.
File '/Users/mazimi/baz.js' does not exist.
File '/Users/mazimi/baz.jsx' does not exist.
File '/Users/baz.js' does not exist.
File '/Users/baz.jsx' does not exist.
File '/baz.js' does not exist.
File '/baz.jsx' does not exist.
======== Module name 'baz' was not resolved. ========
======== Resolving module 'baz/inner' from '/Users/mazimi/Desktop/ts23-issue/src/baz/index.jsx'. ========
Module resolution kind is not specified, using 'Classic'.
File '/Users/mazimi/Desktop/ts23-issue/src/baz/inner.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz/inner.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz/inner.d.ts' does not exist.
Directory '/Users/mazimi/Desktop/ts23-issue/src/baz/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts23-issue/src/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/node_modules/@types' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts23-issue/src/baz/inner.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/baz/inner.jsx' does not exist.
======== Module name 'baz/inner' was not resolved. ========
======== Resolving module 'bar' from '/Users/mazimi/Desktop/ts23-issue/src/foo/index.jsx'. ========
Module resolution kind is not specified, using 'Classic'.
File '/Users/mazimi/Desktop/ts23-issue/src/foo/bar.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/foo/bar.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/foo/bar.d.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar.d.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/bar.ts' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/bar.tsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/bar.d.ts' does not exist.
File '/Users/mazimi/Desktop/bar.ts' does not exist.
File '/Users/mazimi/Desktop/bar.tsx' does not exist.
File '/Users/mazimi/Desktop/bar.d.ts' does not exist.
File '/Users/mazimi/bar.ts' does not exist.
File '/Users/mazimi/bar.tsx' does not exist.
File '/Users/mazimi/bar.d.ts' does not exist.
File '/Users/bar.ts' does not exist.
File '/Users/bar.tsx' does not exist.
File '/Users/bar.d.ts' does not exist.
File '/bar.ts' does not exist.
File '/bar.tsx' does not exist.
File '/bar.d.ts' does not exist.
Directory '/Users/mazimi/Desktop/ts23-issue/src/foo/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts23-issue/src/node_modules' does not exist, skipping all lookups in it.
File<EMAIL_ADDRESS>does not exist.
Directory '/Users/mazimi/Desktop/node_modules/@types' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts23-issue/src/foo/bar.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/foo/bar.jsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/src/bar.jsx' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/bar.js' does not exist.
File '/Users/mazimi/Desktop/ts23-issue/bar.jsx' does not exist.
File '/Users/mazimi/Desktop/bar.js' does not exist.
File '/Users/mazimi/Desktop/bar.jsx' does not exist.
File '/Users/mazimi/bar.js' does not exist.
File '/Users/mazimi/bar.jsx' does not exist.
File '/Users/bar.js' does not exist.
File '/Users/bar.jsx' does not exist.
File '/bar.js' does not exist.
File '/bar.jsx' does not exist.
======== Module name 'bar' was not resolved. ========
======== Resolving type reference directive 'node', containing file '/Users/mazimi/Desktop/ts23-issue/__inferred type names__.ts', root directory '/Users/mazimi/Desktop/ts23-issue/src/tsd,/Users/mazimi/Desktop/ts23-issue/node_modules/@types,/Users/mazimi/Desktop/ts23-issue/node_modules/**'. ========
Resolving with primary search path '/Users/mazimi/Desktop/ts23-issue/src/tsd, /Users/mazimi/Desktop/ts23-issue/node_modules/@types, /Users/mazimi/Desktop/ts23-issue/node_modules/**'.
Directory '/Users/mazimi/Desktop/ts23-issue/src/tsd' does not exist, skipping all lookups in it.
Found 'package.json' at<EMAIL_ADDRESS>'package.json' does not have a 'typings' field.
'package.json' does not have a 'types' field.
File<EMAIL_ADDRESS>exist - use it as a name resolution result.
Resolving real path for<EMAIL_ADDRESS>result<EMAIL_ADDRESS>======== Type reference directive 'node' was successfully resolved to<EMAIL_ADDRESS>primary: true. ========
======== Resolving type reference directive 'jest', containing file '/Users/mazimi/Desktop/ts23-issue/__inferred type names__.ts', root directory '/Users/mazimi/Desktop/ts23-issue/src/tsd,/Users/mazimi/Desktop/ts23-issue/node_modules/@types,/Users/mazimi/Desktop/ts23-issue/node_modules/**'. ========
Resolving with primary search path '/Users/mazimi/Desktop/ts23-issue/src/tsd, /Users/mazimi/Desktop/ts23-issue/node_modules/@types, /Users/mazimi/Desktop/ts23-issue/node_modules/**'.
Directory '/Users/mazimi/Desktop/ts23-issue/src/tsd' does not exist, skipping all lookups in it.
Found 'package.json' at<EMAIL_ADDRESS>'package.json' does not have a 'typings' field.
'package.json' does not have a 'types' field.
File<EMAIL_ADDRESS>exist - use it as a name resolution result.
Resolving real path for<EMAIL_ADDRESS>result<EMAIL_ADDRESS>======== Type reference directive 'jest' was successfully resolved to<EMAIL_ADDRESS>primary: true. ========
Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'child_process' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'tls' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'http' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'readline' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'crypto' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>src/bar/index.jsx(3,15): error TS2307: Cannot find module 'baz'.
src/baz/index.jsx(1,19): error TS2307: Cannot find module 'baz/inner'.
src/foo/index.jsx(1,26): error TS2307: Cannot find module 'bar'.
src/index.ts(1,17): error TS2307: Cannot find module 'foo'.

use --moduleResolution node.

I get the same errors with node:

tsc --traceResolution                               1563ms  Wed Apr 26 11:28:11 2017
======== Resolving module 'foo' from '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/index.ts'. ========
Explicitly specified module resolution kind: 'NodeJs'.
Loading module 'foo' from 'node_modules' folder, target file type 'TypeScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/foo.ts' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/foo.tsx' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/foo.d.ts' does not exist.
File<EMAIL_ADDRESS>does not exist.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
Loading module 'foo' from 'node_modules' folder, target file type 'JavaScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/foo.js' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/foo.jsx' does not exist.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
======== Module name 'foo' was not resolved. ========
======== Resolving module 'react' from '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/bar/index.jsx'. ========
Explicitly specified module resolution kind: 'NodeJs'.
Loading module 'react' from 'node_modules' folder, target file type 'TypeScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/bar/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/react.ts' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/react.tsx' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/react.d.ts' does not exist.
File<EMAIL_ADDRESS>does not exist.
Found 'package.json' at<EMAIL_ADDRESS>'package.json' does not have a 'typings' field.
'package.json' does not have a 'types' field.
File<EMAIL_ADDRESS>exist - use it as a name resolution result.
Resolving real path for<EMAIL_ADDRESS>result<EMAIL_ADDRESS>======== Module name 'react' was successfully resolved to<EMAIL_ADDRESS>========
======== Resolving module 'baz' from '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/bar/index.jsx'. ========
Explicitly specified module resolution kind: 'NodeJs'.
Loading module 'baz' from 'node_modules' folder, target file type 'TypeScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/bar/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/baz.ts' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/baz.tsx' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/baz.d.ts' does not exist.
File<EMAIL_ADDRESS>does not exist.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
Loading module 'baz' from 'node_modules' folder, target file type 'JavaScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/bar/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/baz.js' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/baz.jsx' does not exist.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
======== Module name 'baz' was not resolved. ========
======== Resolving module 'baz/inner' from '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/baz/index.jsx'. ========
Explicitly specified module resolution kind: 'NodeJs'.
Loading module 'baz/inner' from 'node_modules' folder, target file type 'TypeScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/baz/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
Loading module 'baz/inner' from 'node_modules' folder, target file type 'JavaScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/baz/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
======== Module name 'baz/inner' was not resolved. ========
======== Resolving module 'bar' from '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/foo/index.jsx'. ========
Explicitly specified module resolution kind: 'NodeJs'.
Loading module 'bar' from 'node_modules' folder, target file type 'TypeScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/foo/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/bar.ts' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/bar.tsx' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/bar.d.ts' does not exist.
File<EMAIL_ADDRESS>does not exist.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
Loading module 'bar' from 'node_modules' folder, target file type 'JavaScript'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/foo/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/node_modules' does not exist, skipping all lookups in it.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/bar.js' does not exist.
File '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/bar.jsx' does not exist.
Directory '/Users/mazimi/Desktop/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/mazimi/node_modules' does not exist, skipping all lookups in it.
Directory '/Users/node_modules' does not exist, skipping all lookups in it.
Directory '/node_modules' does not exist, skipping all lookups in it.
======== Module name 'bar' was not resolved. ========
======== Resolving type reference directive 'node', containing file '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/__inferred type names__.ts', root directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/tsd,/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/@types,/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/**'. ========
Resolving with primary search path '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/tsd, /Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/@types, /Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/**'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/tsd' does not exist, skipping all lookups in it.
Found 'package.json' at<EMAIL_ADDRESS>'package.json' does not have a 'typings' field.
'package.json' does not have a 'types' field.
File<EMAIL_ADDRESS>exist - use it as a name resolution result.
Resolving real path for<EMAIL_ADDRESS>result<EMAIL_ADDRESS>======== Type reference directive 'node' was successfully resolved to<EMAIL_ADDRESS>primary: true. ========
======== Resolving type reference directive 'jest', containing file '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/__inferred type names__.ts', root directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/tsd,/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/@types,/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/**'. ========
Resolving with primary search path '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/tsd, /Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/@types, /Users/mazimi/Desktop/ts-2.3-module-resolution-issue/node_modules/**'.
Directory '/Users/mazimi/Desktop/ts-2.3-module-resolution-issue/src/tsd' does not exist, skipping all lookups in it.
Found 'package.json' at<EMAIL_ADDRESS>'package.json' does not have a 'typings' field.
'package.json' does not have a 'types' field.
File<EMAIL_ADDRESS>exist - use it as a name resolution result.
Resolving real path for<EMAIL_ADDRESS>result<EMAIL_ADDRESS>======== Type reference directive 'jest' was successfully resolved to<EMAIL_ADDRESS>primary: true. ========
Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'child_process' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'tls' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'http' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'readline' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'crypto' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'stream' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'net' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>Module 'events' was resolved as locally declared ambient module in file<EMAIL_ADDRESS>src/bar/index.jsx(3,15): error TS2307: Cannot find module 'baz'.
src/baz/index.jsx(1,19): error TS2307: Cannot find module 'baz/inner'.
src/foo/index.jsx(1,26): error TS2307: Cannot find module 'bar'.
src/index.ts(1,17): error TS2307: Cannot find module 'foo'.

OK, sorry did not look well enough the first time.. so either, this is a node runtime environment and then you should use relative path to import, e.g. import Bar, {other} from '../bar';, or this is a web environment where the loader has a concept of a "baseUrl", if this the case, then specify "baseUrl": "./src" along with the moduleResolution.

Nice! Setting the baseUrl fixed it! Thank you for your help!


Can pulse_entries table provide an aggregated count column instead of duplicate rows Cache

Hi :wave:
This is sort of related to my other question, but I'm aware that we have sampling and the ability to group things together, but I'd love it if, for example, rather than creating N number of rows in the pulse_entries table, just have a record_count column, and increment the column for that row if another item comes in matching the same key_hash column.
This way, we'd significantly reduce the number of rows built up in the pulse_entries table. What are your thoughts?

Hi @sts-ryan-holton,
The pulse_aggregates table already does this and is the primary source of data for the dashboard.
The pulse_entries table needs to keep individual rows with their specific dates as it's only used to fill in the gap where a bucket from pulse_aggregates falls partially outside the window.
For example, if we take the 1-hour view, then there are buckets in pulse_aggregates for each minute for the past hour:
09:20:00 - 09:20:59
09:21:00 - 09:21:59
09:22:00 - 09:22:59
...
10:18:00 - 10:18:59
10:19:00 - 10:19:59
10:20:00 - 10:20:59

Given the above example, imagine the time is currently 10:20:30. That means we want to query data from 09:20:30 - 10:20:30. We can't include the 09:20:00 - 09:20:59 bucket because it contains data from outside the window, so we use the pulse_entries table just to fill in the data from 09:20:30 - 09:20:59 (we call this the "tail") and the remaining data comes from pulse_aggregates. The 10:20:00 - 10:20:59 bucket can be included because it's only partially full at that point (i.e. it only contains data from 10:20:00 - 10:20:30).
On the 1-hour view, the tail size can be anywhere from 1 second to 59 seconds, depending on the current time. If we zoom out to the 7-day view, the bucket sizes are 2 hours and 48 minutes each (7 days divided by 60), so the size of the tail can be anywhere from 1 second to 2 hours, 47 minutes, and 59 seconds - depending on the current time. It will always be less than a 60th of the data needed to generate stats for the current period as everything else comes from the pre-aggregated data in pulse_aggregates.
If the pulse_entries data were collapsed, we couldn't query for the very specific window of the "tail", which changes every second. It can become a very large table, but only a very small portion is ever needed at any given point in time.
I guess records from the same second could be collapsed, but I don't know how worthwhile that would be.
I will be explaining more about how this all works at Laracon EU in just over a week if you happen to be around!

@jessarcher Thanks for the comprehensive detailed explanation, it's an interesting thought. In my application, which processes thousands of records (unrelated to Pulse), I have gone with the approach of creating three tables, a minuely table, an hourly table and a daily table. In each table, I provide a logged_at datetime column whereby the minutely table aggregates all of the same data by a particular uniqueness for a particular minute, similar approach to the hourly and daily.
The note here, is that I truncate the minutely table beyond a day, the hourly table beyond a few days, and never/rarely truncate the daily table. This way I'm able to provide users with a minute by minute trend, as I am hourly and daily.
The benefit here is that when zooming out to larger periods, I have the option to use the hourly or daily.
Each table is all handled by one event listener, and it effectively uses the firstOrCreate method to either create or return the record if it exists, and then uses the increment method to increment a column, and this approach has worked quite well on millions of entries.
What are your thoughts on this?


Name subscriptions based on the provider node region

Using MiqRegionRemote to determine the remote region id before creating the subscription.
This will guarantee that the subscription name is unique as we already make that requirement for regions and also makes the subscription name more user friendly should we ever decide to expose it.
@Fryguy @gtanzillo

:+1:


Large network "std::bad_alloc" error

I have tried the deepwalk on a large network (nearly 20M nodes) with a machine with 400G Ram and 4GPUs. Here is an error message:
terminate called after throwing an instance of 'std::bad_alloc' terminate called recursively terminate called recursively   what():  std::bad_alloc terminate called recursively
Is that due to I set the augmentation step too large so the CPU memory is not large enough?

It's the sample pools that cause the memory overflow. The auto value of episode_size will use a very large sample pool to maximize speed, regardless of memory cost. For 400GB Ram, I guess an episode size around 5k might work (the auto value may result in about 10k), if you use the default batch size.
We're planning to add adaptive CPU memory mechanism, but it takes non-trivial effort.

Actually I am using the episode_size as 3500, the batch_size as 100000. Maybe it is still too large?

It's not so large. This is just about 100GB for sample pools, if num_partition is 4.
What dimension do you use for embeddings?

The dimension number is 128. I set num_partition as auto. Should I change the partition number manually?

Not necessary. auto is fine. Could you show me the log of hyperparameters by graphvite?

The old log is covered. Will upload the new log soon.

Just run with episode_size=500. It begins to run normally. Used about 250G memory. I can attach a ss of current log:

Maybe my estimation isn't right. I just estimate that increasing 500 to 3500 will cause additional 77GB memory, which is still available. Anyway, you can continue with your current settings. We may check that in the future, if it really makes an issue.
Using 500 for such scale has perceptible influence on the speed. You may set positive_reuse to 10, and then it equals to the speed of 5000. The performance may hurt a little bit, but the original DeepWalk just does like that, you know.

An additional tip: Since the graph is very dense, the augmentation step should be something like 1 or 2.

Hello,
I'm also facing this issue with node2vec. My network has #vertex: 4119272, #edge: 94873549 and I'm running with 4 GPU cards and 360GB in CPU memory. I used episode_size = 100, batch_size= 100000:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Graph<uint32>
------------------ Graph -------------------
#vertex: 4119272, #edge: 94873549
as undirected: yes, normalization: no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[time] GraphApplication.load: 72.6117 s
[time] GraphApplication.build: 4.31637 s
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
GraphSolver<128, float32, uint32>
----------------- Resource -----------------
#worker: 4, #sampler: 44, #partition: 4
tied weights: no, episode size: 100
gpu memory limit: 32 GiB
gpu memory cost: 1.07 GiB
----------------- Sampling -----------------
augmentation step: 10, p: 1, q: 1
random walk length: 40
random walk batch size: 100
#negative: 1, negative sample exponent: 0.75
----------------- Training -----------------
model: node2vec
optimizer: SGD
learning rate: 0.025, lr schedule: linear
weight decay: 0.005
#epoch: 2000, batch size: 100000
resume: no
positive reuse: 1, negative weight: 5
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 1: 1731210 Aborted

I'm now trying with batch size = 10000. Is this because node2vec is more memory-consuming compared to DeepWalk?

Hello,
I'm also facing this issue with node2vec. My network has #vertex: 4119272, #edge: 94873549 and I'm running with 4 GPU cards and 360GB in CPU memory. I used episode_size = 100, batch_size= 100000:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Graph<uint32>
------------------ Graph -------------------
#vertex: 4119272, #edge: 94873549
as undirected: yes, normalization: no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[time] GraphApplication.load: 72.6117 s
[time] GraphApplication.build: 4.31637 s
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
GraphSolver<128, float32, uint32>
----------------- Resource -----------------
#worker: 4, #sampler: 44, #partition: 4
tied weights: no, episode size: 100
gpu memory limit: 32 GiB
gpu memory cost: 1.07 GiB
----------------- Sampling -----------------
augmentation step: 10, p: 1, q: 1
random walk length: 40
random walk batch size: 100
#negative: 1, negative sample exponent: 0.75
----------------- Training -----------------
model: node2vec
optimizer: SGD
learning rate: 0.025, lr schedule: linear
weight decay: 0.005
#epoch: 2000, batch size: 100000
resume: no
positive reuse: 1, negative weight: 5
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 1: 1731210 Aborted

I'm now trying with batch size = 10000. Is this because node2vec is more memory-consuming compared to DeepWalk?

@mginabluebox Yes. DeepWalk and LINE scales linearly w.r.t. |E|, but node2vec scales at least linearly w.r.t. |E|^2/|V| in the case of d-regular graphs, and can be worse if the degree distribution is skew.
I suggest trying DeepWalk or LINE instead of node2vec. The formers are more robust in terms of default hyperparameters. Node2vec won't bring significant gain unless you perform an exhausitive search of p and q on your dataset.

@mginabluebox Yes. DeepWalk and LINE scales linearly w.r.t. |E|, but node2vec scales at least linearly w.r.t. |E|^2/|V| in the case of d-regular graphs, and can be worse if the degree distribution is skew.
I suggest trying DeepWalk or LINE instead of node2vec. The formers are more robust in terms of default hyperparameters. Node2vec won't bring significant gain unless you perform an exhausitive search of p and q on your dataset.


Installing on 64-bit Ubuntu 18.04.1 LTS?

Is it possible to install on 64-bit Ubuntu 18.04.1 LTS?
I went through every step, but I get stuck on this step:
./scripts/configure.sh -l -F -D
The output says:

CMake Error at cmake/FindGlog.cmake:79 (MESSAGE):
Failed to find glog - Could not find glog include directory, set
GLOG_INCLUDE_DIR to directory containing glog/logging.h
Call Stack (most recent call first):
cmake/FindGlog.cmake:113 (GLOG_REPORT_NOT_FOUND)
CMakeLists.txt:162 (find_package)
-- Configuring incomplete, errors occurred!

Does the installation process get affected because I recently installed ROS (Melodic Morenia)?
Any help is appreciated. Thanks

It appears that you don't have all libraries required to install the software. If you have the glog library already, try deleting it and re-downloading the library and make sure that the library is in a directory containing glog/logging.h. If it doesn't contain that, you may have to rename at least one directory or even repackage the library to work with the software. However, since the code required for the library would already be in the package, it should be easy to repackage if required.

We currently only support ROS kinetic.


feat: offer a way to run without docker

This will avoid to pull and build a docker image on eahc usage since requirements tools are present on Github Actions runner by default.
This will also avoid rate limiting issues while pulling base image on aws.ecr.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Tested on my private organization works well

@liamg I'll glad to known what is still required to help merging this PR 🙏

Hey @jBouyoud - I'm afraid I'm no longer a maintainer of this repo - I would advise you to reach out to @AnaisUrlichs or on the Aqua Slack to get this merged. Good luck! 👍

Hey @AnaisUrlichs , As I already said, I'll be glad to help merging this PR that could help us to use Aqua product in our organization. Currently we are facing a lot of Rate limiting issues when pulling aws.ecr  ; and It seems to me that this PR doesn't introduce extra maintenance.
Met me known If I can do anything else.
Thanks

Hi @jBouyoud, thank you for the contribution! I will be taking over this repo, please give me some time to review the PR.

Hi @giorod3, let me know if I can help you in any ways

@jBouyoud sorry for the delay, I just reviewed the PR. I will merge as soon as receive access. I assume you also need a new release?

Hi! Any chance to merge and release this?

Sounds like a dead project 😢

Hi @jBouyoud really sorry about this -- I will post an update next week


move to production

WHAT DOES THIS PR DO?

[x] https://github.com/airqo-platform/AirQo-api/pull/1510
[ ] Analytics Microservice Changes
[ ] Airflow data pipeline changes

Hi @NoahNsimbe , please review the changes in this PR before we move them to production. cc: @BenjaminSsempala


All achievement marked as locked on opening game

Once you open a game, all its achievements are marked as locked, even if they were unlocked. This has started happening since the last update. I'm still determining if this is the right way to reproduce this, but that is what happened to me. Only the games that I played had all their achievements marked as locked.
So I only played 3 out of 17 of my installed games, and the achievements of only these 3 games were all marked as locked(0/X, with the unlock date gone).

Hello, how do you reproduce this? I have the same exact issue. Do you have achievements unlocked in the other 14 games?

Like I said, I played the games after the last update and stumbled on this bug. Only the games I played got affected.

Hello again:
I have reproduced the issue on my side and also know how to get the achievements to be properly displayed: I have to perform the following steps, if this helps @Lacro59 to find out where the bug might be:

Erase the json that contains the achievement data of the gaming in Extension Data folder in %appdata%/Playnite. I assume what this does is making the extension having to download all the achievement metadata back again.
Close and reopen Playnite
Update SuccessStory data manually by right clicking the game and go to the corresponding option just after playnite opens up. I have to do it while playnite is importing at startup the new games to the library. Else, it won't work.
This sometimes works and sometimes doesn't, don't know what is going on...

please check -> https://www.reddit.com/r/playnite/comments/1fx19be/success_story_not_tracking_epic_games_progression/
For me issue was solved once I changed achievement privacy to public in EGS settings.

Wow, that totally did it, I didn't know that this option existed on the epic store webpage.
Is this mentioned somewhere in Success Story documentation? If not, it should be added


Big checkboxes

This adds the big checkboxes to Capital Framework from the work done in https://github.com/cfpb/cfgov-refresh/pull/2292.
A lot of design decisions were made in the PR above and I did a little work improving/simplifying how we use variable names.
⚠ I noticed that our labels are not up-to-spec. We don't have a task for labels and it's out of scope to handle here. I haven't adjusted those at all and will ask the Dada team when we should tackle this.
Testing
In this folder
cd cf-forms
npm link

Pull down this PR: https://github.com/jimmynotjim/capital-framework-sandbox/pull/18 and run npm link cf-forms then npm start
You should be able to navigate to the form section to see the new buttons. Check the boxes to see what they look like when active.
Screenshots
👉 https://github.com/jimmynotjim/capital-framework-sandbox/pull/18

I'm done reviewing - One comment that needs a commit, one question and one suggested change.

Thanks, I've made some changes based on your reviews in 1849bdb

👍

AndConf inspired bigfoot gif


[Question] How closely related should be the reference fasta

Hi all,
Thank you for working on read-it-and-keep. I was wondering what should be my reference genome to map the reads onto? Could it be a previous assemble from nontreated reads, or should it be a reference genome closer to my species that resides already in a database?
Thanks in advance.

Oh thanks


Support Webhook API

Request to support the webhook API for event-based updates rather than polling devices.
TBD.

Happy to help test this, since I'm familiar with doing it with Home Assistant.  Thanks

I see this support is now in a closed beta. Do you know how to access the closed beta?

This is now live.  See the announcement here.


Integer to Option conversion does not work properly when there are blank option values

When assigning an Integer to an Option field, the conversion does not work properly when there are blank option values. We receive the following compilation error:
severity: 'Error'
message: 'Constant value '11' is outside of the valid ordinal range for this option type'
at: '34364,42'
source: 'AL'
Example:
MyRecord.Industry := 11;
field definition:
field(2;Industry;Option)
{
CaptionML=ENU='Industry',
FRC='Secteur',
ENC='Industry';
OptionCaptionML=ENU='" ",,,"Direct Marketing (MOTO)","Food/Restaurant","Grocery Store/Supermarket",,,,,Retail,"E-Commerce"';
OptionMembers=" ",,,"Direct Marketing (MOTO)","Food/Restaurant","Grocery Store/Supermarket",,,,,Retail,"E-Commerce";
}
Option 11 should be E-Comerce

@jcooper01 Try taking the 5 to see if that turns into e-comerce. If so, MS uses the option to remove empty entries from the array on string.Split().
Then this -> ,, becomes an empty entry in a string[] that is removed.

I think you're probably right, but we have literally thousands of lines of code that refer to options as their integer values. It should behave as in C/AL.

I've never been very fond of keeping options free. But I understand what you mean. We also have the problem with old code.
I would be in favour of supporting this at the moment but to issue a compiler warning, in the future this habit should no longer be possible.

We have certain places where we use the fact that the integer values line up (like when communicating with external services, and the service needs an integer but we represent it in NAV as human-readable text).


Pavan

hi, i have added two classes robot and autoit and also updated pom.xml please verify and merge.

kk tq


Create dom-range.json

https://developer.mozilla.org/en-US/docs/Web/API/Range lists a detailed feature compatibility table. That is to be integrated in the JSON at a later time.

Thanks!


What is the proper way to handle a subscription end due to billable removal?

Hi!
I have a little problem as I mentioned in a title.
Currently, I have code like this:
$billable->subscription()->cancel();
$billable->delete();

The issue arises when the application receives a webhook from Lemon Squeezy (subscription_cancelled, subscription_updated) at which point $billable is already removed. This triggers an error:
LemonSqueezy\Laravel\Events\SubscriptionCancelled::__construct(): Argument #1 ($billable) must be of type Illuminate\Database\Eloquent\Model, null given
How can I properly handle the end or removal of the subscription?

Thanks @pdziewa. I've sent in a PR here to address this: https://github.com/lmsqueezy/laravel/pull/91


Implement support for workspace metadata

Closes #24.

I've had to do a couple of force-pushes to fix some minor issues; the code can be further transformed in this style to reduce the ephemeral Vecs usage - but I'm not sure it's this PR is a good place to do it.
Maybe we should change the code back (omit the last commit), merge this one as it was, and create a new PR with a refactor?

Sounds good. Let do it like you said


SSL handshake timed out and channel closed exceptions

The simplified code
import os
from elasticsearch import AsyncElasticsearch
from fastapi import FastAPI, HTTPException, Depends, status
import secrets
from fastapi.security import HTTPBasic, HTTPBasicCredentials
import uvicorn

application = FastAPI()
security = HTTPBasic()

es_client = AsyncElasticsearch([os.getenv('ES_URI')] , maxsize=10000)
es_index = os.getenv('ES_INDEX')
es_doc_type = os.getenv('ES_DOC_TYPE')

def authorize(credentials: HTTPBasicCredentials = Depends(security)):
    correct_username = secrets.compare_digest(credentials.username, os.getenv("BASIC_AUTH_USERNAME"))
    correct_password = secrets.compare_digest(credentials.password, os.getenv("BASIC_AUTH_PASSWORD"))
    if not (correct_username and correct_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect username or password",
            headers={"WWW-Authenticate": "Basic"},
        )
    return True

query = {
    'query': {
        'function_score': {
            'query': {
                         'bool': ...
                     }
            },
        'score_mode': 'sum', 'boost_mode': 'sum'
    },
    'size': 6,
    '_source': [...]
}

@application.get("/api/v1/smart-search", dependencies=[Depends(authorize)])
async def search(size:int=12, model:str=None, carSize:int=None):
    ...
    es_res = await es_client.search(index=es_index, doc_type=es_doc_type, body=  query, max_concurrent_shard_requests=5000)
    return es_res

if __name__ == "__main__":
    uvicorn.run(application, host="<IP_ADDRESS>", port=5000)

Description
The above code is simplified and creates a dynamic ES query.  Executes the ES query using AsyncElasticsearch. The result is parsed and returned. I tried multiple configuration for amongst others:

maxsize=10000 in AsyncElasticsearch client definition
max_concurrent_shard_requests in search
etc.

The application runs in docker. This is the dockerfile CMD:
CMD gunicorn --do-handshake-on-connect --worker-class uvicorn.workers.UvicornWorker --worker-connections 1000 --workers 3 --bind <IP_ADDRESS>:5000 api
I tried running with:

uvicorn, gunicorn with gevent and gunicorn with uvicorn worker as worker class.
Different number of workers, worker-connections, etc.
Without and with do-handshake-on-connect

All this runs fine for:

Local setup (Run API local in docker with local ES instance) with as many concurrent users as wanted
API running in Kubernetes environment with (2 to 5 pods) and AWS c5.xlarge.elasticsearch ES cluster up to 60 concurrent users

The API starts failing when running in Kubernetes environment with (2 to 5 pods) and AWS c5.xlarge.elasticsearch ES cluster starting for about 70 concurrent users or more. Load generated on API (pods) is not too high and load on ES is low.
Then 70% of the calls fails with the mainly following errors:

SSLException: handshake timed out (59%)
ClosedChannelException (36%)

I don't have more information. Because this is all that is by Gatling load testing tool.
Environment
requirements.txt
elasticsearch[async]>=7.12.0
starlette==0.13.6
elastic-apm==5.8.1
fastapi==0.60.1
uvicorn==0.11.8
gunicorn==20.1.0

Python version:

Version: 3.6
FROM python:3.6-alpine

All the low level networking stuff handled by either uvicorn or gunicorn (Layer 4), FastAPI runs on Layer 7 and doesn't knows anything about the layers below including things like SSL, TLS, Sockets, so this is not a FastAPI problem, you'll have a better chance to get an answer in those repositories.

Ok, thanks. I have reposted this in the uvicorn repository.

Thanks for the help here @ycd ! 👏  🙇
As this wouldn't be related to FastAPI itself, I'm gonna close this one now.

Sorry for the long delay! 🙈  I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.


RFC: Refactor functions into Expressions namespaces and functions on the Expr directly

One thing I disliked a lot about pyspark was the Functions. Since most of if not all DataFusion.functions take in an expression and return an expression, we could rework this in the Expr namespaces, this should help finding the right expressions much easier, some examples:

col().dt.current_date()
col().list.ndims()

I am happy to start working on this and creating a full list of how the Expr and Expr.namespaces can look like

My primary concerns about doing this are:

API churn rate. If we keep changing our interface too rapidly between versions it will turn people off the project. So if we did this I'd probably support defining them on the expr but also having the functions available - one to call the other, almost as an alias.
Familiarity. Like it or not, many if not most, of the people who will be coming to the project will be coming from a pyspark/pandas/polars background. The layout we currently have will be more familiar in some places which I think will lead to greater adoption.
Diverging too much from the datafusion core approach. I think some level of difference is expected since rust and python do operate very differently. However if we diverge too much, it makes maintenance harder and harder to onboard new contributors. This is a smaller concern.

That being said, I have found that some of the functions I expected to be on Expr were in functions and I do find the interface to be somewhat non-intuitive. My proposal would be to handle these on a case by case basis of what should be where.

My primary concerns about doing this are:

API churn rate. If we keep changing our interface too rapidly between versions it will turn people off the project. So if we did this I'd probably support defining them on the expr but also having the functions available - one to call the other, almost as an alias.

I think this can work but with the goal to deprecate the functions after X releases/months. One thing that is problematic for example with PySpark codebases is that there is no consistency due to the aliases. I don't think constant change is an issue, unless you don't document well how to change.

Familiarity. Like it or not, many if not most, of the people who will be coming to the project will be coming from a pyspark/pandas/polars background. The layout we currently have will be more familiar in some places which I think will lead to greater adoption.

In some way this might speak more to PySpark users the current API, but arguably and I think many will agree if you come from polars and pandas the current API isn't close to their familiarity.

Diverging too much from the datafusion core approach. I think some level of difference is expected since rust and python do operate very differently. However if we diverge too much, it makes maintenance harder and harder to onboard new contributors. This is a smaller concern.

That being said, I have found that some of the functions I expected to be on Expr were in functions and I do find the interface to be somewhat non-intuitive. My proposal would be to handle these on a case by case basis of what should be where.

I can make a draft of mapping Functions -> Expr, so we can get a full picture on how this will look like


Config flow opt

optimizing config flow
Already fixed:

[x] Multistep configuration

Wow, cool! I will check it out in detail!


Begin adding typing

This begins to add static type hints, and adds a mypy check.
For the moment, I have left on allow_untyped_defs=true ... and typing can be added incrementally

Codecov Report

Merging #5 (9404233) into main (114d7e2) will decrease coverage by 0.68%.
The diff coverage is 82.66%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main       #5      +/-   ##
==========================================
- Coverage   82.59%   81.92%   -0.68%
==========================================
  Files          23       23
  Lines         408      437      +29
==========================================
+ Hits          337      358      +21
- Misses         71       79       +8

Impacted Files
Coverage Δ

motile/utils.py
0.00% <ø> (ø)

motile/costs/edge_distance.py
47.36% <57.14%> (+4.51%)
:arrow_up:

motile/constraints/constraint.py
77.77% <71.42%> (-2.23%)
:arrow_down:

motile/solver.py
92.30% <76.47%> (-4.25%)
:arrow_down:

motile/variables/variable.py
62.85% <76.92%> (-0.48%)
:arrow_down:

motile/costs/costs.py
77.77% <83.33%> (-2.23%)
:arrow_down:

motile/constraints/__init__.py
100.00% <100.00%> (ø)

motile/costs/__init__.py
100.00% <100.00%> (ø)

motile/track_graph.py
51.28% <100.00%> (+1.93%)
:arrow_up:

motile/variables/__init__.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add --needed to pacman command

By default, pacman reinstalls any package specified that is already installed. The --needed flag says "if this package is already installed, just ignore it". From the man page:
       --needed
           Do not reinstall the targets that are already up-to-date.

I quickly glanced through the other distributions listed, and I couldn't immediately find any others that have the same behavior. However, I haven't used most of them, so I'm not sure.

Every other package manager ive seen doesn't reinstall already installed packages. Pacman is just strange like that!

I personally like this default of pacman, since installing should at the very least hit all the install triggers unless you explicitly want to skip it.
For the sake of brevity I wouldn't add this flag to the readme personally.

Yeah I feel like Archlinux users should probably know about this flag and decide themselves if they want to use it. Adding it wouldn't be bad, but it's shorter without.

Thanks for the PR!
I don't know much about Arch and conventions so I'm deferring to others in this case. It seems like general consensus is not to add this. Feel free to reopen if you feel strongly otherwise.


stdlib: readback fn captures stdout but still echoes out stderr from the shell call to our program's stderr

Should probably redirect subcommand's stderr to /dev/null or something.

This is intentional. Mostly when I call out to other programs (ie. gcc), I want to echo stderr to give the user an impression when something goes wrong. Belongs with https://github.com/Neat-Lang/neat/issues/59 imo: having a configurable process class lets us just close stderr.


[Feature Request]: Display Number of Likes and Dislikes on Post Card

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
Currently, the post card does not show the number of likes or dislikes each post has received. To enhance user engagement and provide feedback on post popularity and sentiment, we should display the number of likes and dislikes on each post card.
Use Case
The viewer can view the number of likes and dislikes on the post card as well
Benefits
The viewer can view the number of likes and dislikes on the post card as well
Add ScreenShots
No response
Priority
Medium
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[X] I have starred the repository

actually, I am planning this to be integrated with the reaction functionality


issues with existing tables

We see that "something" has happened with the tables (in the studio) - e.g. on the crude oil page, with many rows
https://www.equinor.com/energy/crude-oil-assays
We now can't see any info on the row content, and will have to count rows to find the correct one - which is unnecessary time consuming when editing (and also annoying)
All rows are called Untitled and have 0 table cells.
Private Zenhub Image

@fernandolucchesi - the look of the component looks good in editor - but the links on added links and download file has lost the underline:
Private Zenhub Image
See e.g. crude oil table - where links to download files are underlined:
Private Zenhub Image

@fernandolucchesi - the look of the component looks good in editor - but the links on added links and download file has lost the underline:  See e.g. crude oil table - where links to download files are underlined:

good catch, this bug was actually introduced on #1369  @millianapia


[Discussion] Storing md5's in the .import files causes issues with build systems and VCS's.

Currently, the file's md5 is stored along with the import settings in <filename>.import
Examples:
{snip}
[deps]
source_file="res://icon.png"
source_md5="40cd50be8aff465444de5539bc5f8ccf"        <---- MD5

[params]
compress/mode=0                              <----- Import settings
compress/lossy_quality=0.7
compress/hdr_mode=0
{/snip}

Now, many file contain timestamps, such as .dae files:
{snip}
<asset>
    <created>2018-02-28T08:58:31</created>
    <modified>2018-02-28T08:58:31</modified>
</asset>
{/snip}

Storing the md5 as part of the .import file means that if you have a build system that generates assets (eg converts blends into dae files), then every time the build system is run, the output has a different md5, and so every different developer ends up with their version control system notifying them that the .import files have changed.
This problem stems from the fact that the .import files need to be version controlled because they contain import-relevant data, but the md5 of the file frequently changes irrespective of the import settings.

I'm happy enough to put in some development time to fix this, however the mechanism of storing the md5's will need to be decided.
I see two options (there may be more):

Put the md5's in the .import folder with the other auto-generated data (not checked into the VCS)
Create a new file <filename>.md5 for each resource (which can be ignored by the VCS)

Which mechanism would be better? Are there any that I haven't thought of?

These really should be put in a folder that can be ignored by the version control system. These are very useful to have, but are really only relevant to individual machines.
Also, the default Line Ending installation settings for git on Windows, macOS, and Linux are different. So, if you're developing on multiple machines it starts to pointlessly update these files (for dae files in my case) since they'll have LF on one machine and CRLF on another machine, thus changing the hash. If these could be ignored per machine this issue would go away.
So, what happens for me is:
Linux checks out dae files with LF and thus creates HashA on import.
Then I go over to Windows which defaults to CRLF, so it checks out the dae files with CRLF creating HashB.
This can be resolved with a line ending file in the repo, but someone who doesn't know that is going to thing Godot is to blame, not their version control system setup.
I really think these should be stored somewhere like .import/hashes or something like that, or maybe even just make another folder like "res://.local/hashes".
As a side effect, this folder could also be used by plugins that need to have per project, per machine specific settings.

I agree that this is necessary, I had argued for that before 3.0 and failed to convince reduz, but now that more users have experience with this workflow, I think we can give it another try ;)

This is a very specific and corner use case that does not make it worth
changing the entire import system and making it more complex.
We could add a tag or option to import files to disable auto import, or
control how this proces happens better, so this is not a problem.
On Mar 1, 2018 10:16, "Rémi Verschelde"<EMAIL_ADDRESS>wrote:

I agree that this is necessary, I had argued for that before 3.0 and
failed to convince reduz, but now that more users have experience with this
workflow, I think we can give it another try ;)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/17103#issuecomment-369588288,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF-Z28r_BoT0Rb61_hIAOX8DImrnLy_oks5tZ_S9gaJpZM4SWoeR
.

@reduz I would assume that having your project in a VCS is not a "corner use case." And currently, the .import files are difficult to version control because they contain information that is needed (import settings), but they also contain information that is too volatile and results in frequent merge conflicts (md5's). I would not consider disabling auto-import as a solution to this problem - it may make it occur less frequently, but it does nothing to prevent collisions between supposedly unchanged import configurations.
If you have any suggestions about other methods of keeping these pieces of data separate, then I'm completely happy to implement it - but it is a problem that has already caused some grief over the two weeks I (as part of a 3-man team) have been working with godot.

I would assume that having your project in a VCS is not a "corner use case."

Of course not, but you specifically stated that your problem is:

Storing the md5 as part of the .import file means that if you have a build system that generates assets (eg converts blends into dae files), then every time the build system is run, the output has a different md5

Which does seem like a corner case to me. I don't see why the import system would not want to re-import this file, no matter the use case. It seems to me the only solution is disabling automatic re-import.

@reduz

I don't see why the import system would not want to re-import this file, no matter the use case. It seems to me the only solution is disabling automatic re-import.

I think that's not the problem. The problem is that the .import file will change and since those are also in the VCS, you generate a meaningless diff. If the MD5 is stored elsewhere, the asset will be re-imported anyway, but it won't generate any diff in the VCS.

I think this more an issue in 3D games (where you have artists working in separate programs with a defined export process) than in 2D games (where the artists work with the art more directly).
The question comes down to: What things should be version controlled?

On one extreme, you could version control the .import folder, and tell your VCS to ignore the asset files, but then you need to transfer the asset files somehow.
Or you can track the asset files and let Godot handle the import on each developers computer.
With a build pipeline, you take it one step further, and generate the assets. In our case, we version control the .blend files. We don't have the assets themselves as part of our VCS.

Following the "only track source files" thought, the md5 is itself a generated value, and thus shouldn't be VC'd. Unfortunately, the md5 (generated) is currently stored in a file along with other data that is not generated, and thus needs to be tracked.
This stems from godot assuming that the md5 is a good approximation of global asset state, and that a change to the asset files md5 indicates a change to the asset itself. However, due to differences in operating system (as in NathanWardens issue), or timestamps (as in the build system issue), this assumption can be invalid.
However, the md5 is a good approximation of local asset state. It can be used to identify if the file has changed for a specific user, but not if it has changed globally for all people. Changing the md5 from a global to a local state is as simple as removing it from the VC system.

The problem is that the .import file will change and since those are also in the VCS, you generate a meaningless diff. If the MD5 is stored elsewhere, the asset will be re-imported anyway, but it won't generate any diff in the VCS.

I see, so the solution would be to store, besides the .import file, just the text with the hash files in the .import folder?
That probably does make sense.

@reduz I think that's a good idea so we don't clutter the main project directories with more files.

@sdfgeoff, @NathanWarden apologies, I misunderstood your use case, It definitely makes sense to move this to .import folder.
I can work on it when I have a bit of free time, otherwise if anyone wants to give it a try, code is in editor/editor_file_system.cpp

@reduz: I already have a PR open for a trivial implementation (filename.import.md5), but I agree with everyone here that a better place is the .import folder.
I'll give it a shot at moving it to the .import folder tonight, but may will likely end up coming to you for advice on IRC.

I agree that reworking #17138 to actually keep the files in the .import/ folder would be best.

#17138 was merged, and I've been using a branch with that PR in it for the past while, and it has filled my needs. So this issue is resolved.


Support for MP Metrics 2.0

Resolves #833 (MP Metrics 2.0) and #939 (verify new metrics with FT).
These changes add support for MP metrics 2.0, including a Helidon bundle for MP 3.0.
The following new modules support the new version of metrics:
metrics2
    metrics2
    prometheus2
microprofile
   metrics2

Their original counterparts support the previous metrics version 1.1, as before:
metrics
   metrics
   prometheus
microprofile
   metrics

This PR updates examples, tests, and TCK drivers to use the newer metrics version with these exceptions:

MP static content example
MP security example
MP IDCS security example
zipkin integration test

Because MP Metrics 2.0 is incompatible with 1.1, Helidon allows developers with existing metrics-based apps to use the latest Helidon release with the older MP metrics version by depending on the helidon-microprofile-2.2 bundle.
Helidon SE apps that use metrics can continue to depend on the metrics/metrics artifact to use the older metrics implementation. Or, to use the newer SE metrics implementation, they can depend on metrics2/metrics2.
Helidon contains the following internal metrics clients:
   fault tolerance
   grpc metrics
These have been changed so they can work with either version of metrics.
To accomplish this a new metrics abstraction layer exists in the new module common/metrics, a separate module because the abstraction layer is used by more than one module. Therefore its key contents have to be public which is unfortunate. This abstraction layer is not intended for developers using Helidon so its public classes and interfaces have Internal in their names to discourage developers from using them.

BTW, the wercker/build failure is due to wercker timing out, not due to errors in the build.

Currently the build error is because of failing TCK tests of metrics 2.0

The TCK failures are known. I've been working on some things and have changed the title to WIP to reflect that.

Removed WIP from the title. TCKs passed locally for me so they should in the pipeline now also.

Just to clarify, the TCKs were passing, then an issue was found and some fixes cause the TCKs to fail temporarily. After Tim's latest commit all TCKs are passing again like at the time the PR was submitted.


fileio: Replace assert with tprintf() and exit(1)

Assertions are good for programming errors, but not for wrong user input.
The new code no longer needs File::ReadFileToStringOrDie, so remove that
method.
Signed-off-by: Stefan Weil<EMAIL_ADDRESS>

thanks.


3.2 New csproj style with backport of Godot.NET.Sdk

There are a lot of build issues coming from old style projects. At this point fixing every single one of those would require adding patch after patch to the project file, which is a considerable amount work and makes the csproj even more bloated than it already is.
As such I decided this effort would be better spent back-porting the Sdk style support that's already available in 4.0-dev to the 3.2 branch.
This will prevent many issues, but it will also introduce other benefits, among them:

While target framework stays as .NET Framework v4.7.2, it can be changed to .NET Standard 2.0 or greater if desired.
It makes it much easier to add future patches. They are added to Godot.NET.Sdk and the only change required in Godot code is to update the Sdk version to use.
Default Godot define constants are also backported, which fixes IDE issues with the preprocessor.

There are a few differences in the changes applied during patching of the csproj compared to 4.0 with the purpose of preventing breaking builds:

TargetFramework stays net472 both for new projects and when importing old ones. It can be manually changed to netstandard 2.0+ if desired though.

The following features are enabled by default for new projects. Enabling them in imported projects may result in errors that must be fixed manually:

EnableDefaultCompileItems is disabled as it can result in undesired C# source files being included. Existing include items are kept. As long as EnableDefaultCompileItems remains disabled, Godot will continue taking care of adding and removing C# files to the csproj.
GenerateAssemblyInfo is disabled as it guarantees a build error because of conflicts between the existing AssemblyInfo.cs and the auto-generated one.
Deterministic is disabled because it doesn't like wildcards in the assembly version (1.0.*) that was in the old AssemblyInfo.cs.

Of importance:
This is a breaking change. A great effort was put in avoiding build errors after upgrading a project, but there may still be exceptions.
This also breaks forward compatibility. Projects opened with Godot 3.2.3 won't work out of the box with older Godot versions. This was already the case with changes introduced in 3.2.2.
Albeit C# support in 3.2.x was still labeled as alpha, we've been trying to treat it as stable for some time. Still the amount of problems this change solves justifies it, but no more changes that break project compatibility are to be introduced from now on (at least for 3.x).
Fixes #30970
Fixes #41241

The 'Microsoft.Build' upgrade is needed in order to support reading and editing project files that use Sdks (which this PR introduces) as well as other new features.
A common example of issues with new features is having to specify a PackageReference version with a child element rather than the attribute. This is no longer the case now.
Fixes #15386

Thanks!

TargetFramework stays net472 both for new projects and when importing old ones. It can be manually changed to netstandard 2.0+ if desired though.

For .NET newbies like me, the corresponding moniker is netstandard2.1 as per https://docs.microsoft.com/en-us/dotnet/standard/frameworks
Tested locally on Windows with a pre-3.2.3 RC 5 build and it seems to work fine on the Dodge the Creeps C# demo. I was able to use C# 8.0 features like readonly struct members.

The new .csproj seems to enforce the use of the editor to create and move scripts. Is there a way to still be able to do all of this through the IDE without using the godot editor at any point ?

@8nouk The new Sdk does not impose any of this. The project either includes the scripts automatically if the EnableDefaultCompileItems property is not disabled. Otherwise Godot takes care of adding the scripts to the csproj.


time input tests fix

Summary
Currently, if the time input has no time entered, per "+" key press it fills the time with current time +1 second. If "-" pressed -1 second. The wdio tests for that functionality are checking if the input time is in 2 seconds range from expected value. They are tied to current time in seconds and are failing in 80% of cases, as the difference in seconds are more than 2 seconds for seconds increment and decrement.
This is a test PR to determine the scope needed to fix time-input tests failures. In this PR it checks only if the seconds value is a valid second value between 0 and 59 seconds, not that the value is in a specific range.
Testing

This change was tested using:

[x] WDIO
[ ] Jest
[ ] Visual testing (please attach a screenshot or recording)
[ ] Other (please describe below)
[ ] No tests are needed

Reviews
In addition to engineering reviews, this PR needs:

[ ] UX review
[ ] Accessibility review
[ ] Functional review

Additional Details

This PR resolves:
UXPLATFORM-10295

Thank you for contributing to Terra.
@cerner/terra

All checks passed from the first attempt. Re-running checks to ensure it wasn't accidental.

Re-running tests passed as well. I guess that means that a fix to seconds increment/decrement testing will fix the tests failure.


Support for tokensregex/semgrex/tregex

This CoreNLP interface is great for annotation but doesn't support tokensregex/semgrex/tregex like some non-official interfaces. Could support for tokensregex/semgrex/tregex be added?

I made a PR addressing this issue: #4
My implementation is simple: the CoreNLPClient object has new methods (tokensregex(), semgrex(), tregex()) which simply make POST requests to the regex-related endpoints exposed by the CoreNLP server and return the results as dicts with string keys.
I wonder if the code is up to par and I'm certainly open to suggestions/edits. One note is that the new regex methods do not fit in with the style of CoreNLPClient.annotate() as they do not return protobuf objects.

@arunchaganty what do you think?

I think these are very useful enhancements. The added functions can serve as low-level APIs. Beyond that, I think we can also implement a few Annotator, such as TokensRegexAnnotator, so that people can easily integrate these by specifying the annotator list.

@yuhaozhang @dan-zheng also made a PR #4 . Can you take a look at that and merge it if you think it's sufficient?

merged. thanks @dan-zheng !

Ok, I've merged in this feature. A couple of comments when integrating: it's important to check that the server is alive (calling ensure_alive) and turns out that you need to cold start the server to prevent tokensregex/semgrex from timing out. Anyways, thanks for the commit!


Use correct index when mapping element styles.

Styles from elements weren't being applied correctly - this seemed to be the cause.

Aah, gotcha! Thank you!


Nginx Ingress Controller - Allow scoping by selectors

By my reading of the code (which could be wrong) the nginx-ingress-controller only allows you to "scope" your ingress objects by namespace.
I have a case where I'd like to run 2 nginx-ingress-controllers in the same namespace, each with different configuration. But within that single namespace each nginx-ingress-controller will "see" and use all ingress objects.
Does it make sense to allow for the label/selector pattern on ingress objects and nginx-ingress-controller deployments? This is something I'm willing to work on, but it's probably best someone verify my assumptions are correct here.

@bretthoerner using --watch-namespace in the controller you can limit the limit the scope of Ingress objects to one namespace. With that flag you can run 2 controller in the same namespace with different configurations

I tried that, but it seems to use the namespace of the Ingress object as the namespace where services are expected to be.
In other words, running default/nginx-ingress-controller1 that's watching ns1 for ingress objects... then ns1/ingress-object1 that uses vhost to point to service service1 resulted in an error that ns1/service1 didn't exist. I tried using default/service1 but it says it has to be a valid DNS name.
If that made any sense, and is considered a bug, then I can try working on fixing that.

@bretthoerner service1 must exists in the same namespace than the Ingress rule

OK, so that's the cause for my original ticket. Do you think a label/selector use-case might make sense for some people?
For example, say I'd like to break down my namespaces into production and development (which makes the most sense for DNS, e.g. the simple name svc will point to svc.production in production and svc.development in development).
But now within production I want an Ingress endpoint for public services, and another for "internal" services that are not yet inside Kubernetes (likely the case for most companies). So that public internet users can hit my.website.com and non-Kubernetes services inside of AWS can hit my.internal.service (but the public internet can't). Am I doing it wrong? :)

Do you think a label/selector use-case might make sense for some people?

yes it makes sense.
That being said I don't know if this feature should be implemented in the Ingress controller because is not "generic" or maybe not supported in some platform.
ping @bprashanth

There is somewhat of a design double-standard here, because when using '--watch-namespace' the nginx controller still lets you target cross-namespace Services for TCP/UDP, but not cross-namespace Services for HTTP/HTTPS. This is because Ingress resources require a service name in the same namespace as the Ingress, but the configMaps used for TCP/UDP allows you to specify a Services using a 'namespace/service' syntax. (I gather it is planned for TCP/UDP proxy to be rolled into Ingress resources rather than use a configMap eventually, so you might lose this cross-namespace capability for TCP/UDP down the line.)
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-configmap-example
data:
  9000: "default/example-go:8080"

I suggest that the controller implementation could have configuration to optionally target label selector(s). Different controllers can then target different combinations of Ingress resources. An the Ingress resource (for all supported protocols) would still have to be in the same namespace as their Services.
The '--watch-namespace' option would limit the scope of label selectors (if any) to specified namespace (or ideally namespaces)
With this approach, the cross-namespace capability remains an implementation behavior of this nginx controller, and the Ingress resources don't break the namespace separation Ingress/Service resources.
This is I think want @bretthoerner wants, and I suggest he can have it by extending the implementation of the nginx controller to support label selectors for Ingress resources, rather than any change to the way Ingress resources select Services.

We discussed cross namespace ingress on https://github.com/kubernetes/kubernetes/issues/17088
Till now the granularity of Ingress has been at the namespace boundry. If you want to target services across namespaces you can use the serviceloadbalancer: https://github.com/kubernetes/contrib/tree/master/service-loadbalancer. I think allowing cross namespace ingress will make isolation harder. Or rather, it makes enforcing the most common type of isolation (on a ns boundry) require an extra admission controller of some sort. I'm open to ideas, please comment on  https://github.com/kubernetes/kubernetes/issues/17088 for changes to how ingress behaves.

For configurations with a number of related namespaces the current system requires either having the "cross namespace" logic in the upstream load balancer / HA-IP or it requires an upstream ingress point for each namespace, which seems wasteful in many cases.

This issue was moved to kubernetes/ingress#554


Fix/v33 remaining bugs

warn user for overlapping segments which is not currently supported
more general MPR which can be applied on any porotocol; not only on default HP like before

@wayfarer3130 Sure, as discussed privately, we will do it after the release