Gradient not working with tf.math.angle

I tried to create my own loss function that uses tf.math.angle in it. However, when I use this custom loss I get None gradients.
I created a minimum working example (link here).
The expected behavior for this example would be the gradient to be zero as the loss is zero so there is no minimizing it.
Another note is that the loss would be zero for every real number which is what we are going to have using Tensorflow.
My motivation is because I work with complex numbers as you can see here. So this loss function will make sense on the complex domain as not every tensor will have the same angle.
This issue is related to this.

@Saduf2019
I  was able to replicate the issue reported here. Please find the gist.Thanks

Hi @Saduf2019, @NEGU93:
I tried out the code in @UsharaniPagadala's gist--it seems like we're computing tf.math.angle of tf.float32 instances, which requires that an array_ops.where operation be done (and it's not useful because the results are always zero anyway). This disconnects the gradient, resulting in None grads.
If you replace this line:
angle_diff = tf.math.angle(y_true) - tf.math.angle(y_pred)
with this (which we don't want, but for illustration):
angle_diff = tf.math.angle(tf.complex(y_true, 0.)) - tf.math.angle(tf.complex(y_true, 0))
we get tensor gradients.
For complex number applications, I assume we have to explicitly declare the dtype of the input so that we don't run into this problem? I'm a new contributor, so let me know if I have this right. Thanks

@MaanasArora  I tried the change you suggested in the gist but still have the same error... Can you provide an example?

@NEGU93 With this code?
def mean_squared_angle_err(y_true, y_pred):
    angle_diff = tf.math.angle(y_true) - tf.math.angle(y_pred)
    angle_diff = ((angle_diff + np.pi) % (2*np.pi)) - np.pi # [-pi:pi]
    return tf.math.reduce_mean(angle_diff**2)

Sorry, this 'worked' for me but will not return valid losses, I just provided it for illustration.

I am not sure I understand, you said that if I replaced with the line angle_diff = tf.math.angle(tf.complex(y_true, 0.)) - tf.math.angle(tf.complex(y_true, 0)) we should have gradients but I didn't. Is there something else that should be changed for it to work? Why is it not working for me?
Regards,

Sorry, I edited a few typos in the code above. Could you try again?

If it's the point after the zero I corrected that when I tried it not to have the error about the int32. Still, I keep having the None gradients.

Here is the gist with the changes. You can see the gradient is still None.

Did you replace the second y_true with y_pred?

I just meant that the gradients become non-None python values--if you want not NAN numpy values you can replace the 0s by 1s. The angle for a complex number with a zero imaginary component is NAN. This is not a real solution.

Yes, I changed it to 1 and worked!
The angle with a zero imaginary part is NAN? That is a weird behavior. I wonder the reason for that but I guess that is for another issue.

As for the code in the complex number gist, the model outputs are float32 instances which explains why their output returns nans. I assume the CVNN library might be using floats to construct complex numbers, in which case you will need to deconstruct the complex number and call tf.complex. That's why I suspect the issue lies outside of tensorflow. This illustration will not return valid losses.

Well, that is a definition, however, atan(b/a) = atan(0/a) = 0 and not Nan, the problem would be a zero real part (wich is also bad).
Also, using that definition will have the problem of -a-ib (third quadrant) will have the angle of a+ib (first quadrant) as -a/-b = a/b right?.
I would expect a complex number with a zero imaginary part (a+i0) would be on the real axis, therefore, angle = 0.
Moreover, I tried just computing:
tf.math.angle(tf.complex(tf.ones((1,1)), 0.)) and it does return 0 and not Nan.
The problem was indeed solved as you said but I still fail to understand it.

 I got it now! Thank you very much!
I will close the issue then.

Sorry, opened it again, if I do tf.math.angle(tf.zeros((1, 1), dtype=np.complex64)) it gives me 0 and not Nan, so apparently, TF has chosen to make 0+i0 angle to be zero (sounds ok and actually a good choice), and therefore the issue it is still not explained for me.

Sorry, since I am not a member of this repository, my understanding might not be definitive.
The None gradients resulting from a disconnected graph, caused by float inputs to tf.math.angle, would be an issue because the loss will not compute. What I can gather is that nan numpy values for gradients are acceptable because those gradients are differentials of zero values, and this shouldn't cause an issue when actual complex numbers are inserted.

No need to go to the repository, that is my job once this is cleared. My question is with the original gist.
We have the following remarks:

using angle_diff = tf.math.angle(tf.complex(y_true, 0.)) - tf.math.angle(tf.complex(y_pred, 0.)) gives nan gradients.
Using angle_diff = tf.math.angle(tf.complex(y_true, 1.)) - tf.math.angle(tf.complex(y_pred, 1.)) appears to work.

The reason behind this is still unknown to me. even if y_true and/or y_pred is zero, tensorflow appears to return a 0 angle and not Nan as one can see while calling tf.math.angle(tf.zeros((1, 1), dtype=np.complex64)). So why would I have nan gradients?
Why does 0+i0 break the gradients if tensorflow appears to interpret the angle of 0+i0 to equal 0?

Shall I close this and open another issue asking why the gradient of 0+i0 is nan or we continue the chat in this one? I still don't get why

nan numpy values for gradients are acceptable when those gradients are differentials of zero values, and this shouldn't cause an issue when actual complex numbers are inserted.

@NEGU93
Tf.math angle is configured such that it will return 0 instead of nan, the operation must support gradients which is not possible if the value becomes nan or inf.

Sorry for this. But I still don't get it.

Gradients do not work if operation returns nan of inf
This case is coded to return 0, so no nan nor inf.

Then what is the problem?

@NEGU93
That is what it is supposed to do tf.math.angle must support gradients and since gradients are not defined for inf and nan. Kindly move this to closed status and open an issue on tf discussion group for any further queries as this is not a bug or feature request and there is a larger community there to respond. Thanks!

Ok. I will do that. Thank you!


AppBar shadow does not extend to the edge of the screen (particularly noticeable during back gesture)

On iOS, when swiping back, you can see a small portion that is missing its shadow. It's especially visible during the back gesture action, because of the contrast. Notice how in the upper left of the shadow, right under the blue app bar that is moving off the page, you see an area that is lighter than the rest. The effect is more pronounced on your phone itself, as you're moving the view off the screen.

It actually has nothing to do with the back gesture.  You can see this small missing shadow even when not gesturing, it's just less noticeable.
I'm not quite sure why the shadow doesn't extend to the edge of the screen.
The shadow is also the same on Android, so removing the iOS label.
@chinmaygarde is this a quirk of the shadow we use from Skia?

flutter doctor -v
[✓] Flutter (Channel dev, 1.19.0-3.0.pre, on Mac OS X 10.15.5 19F96, locale
    en-GB)
    • Flutter version 1.19.0-3.0.pre at /Users/tahatesser/Code/flutter_dev
    • Framework revision 6135091de9 (31 hours ago), 2020-06-01 17:17:03 -0700
    • Engine revision e39301f23f
    • Dart version 2.9.0 (build 2.9.0-11.0.dev 6489a0c68d)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/tahatesser/Code/sdk
    • Platform android-29, build-tools 29.0.3
    • ANDROID_HOME = /Users/tahatesser/Code/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.5)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.5, Build version 11E608c
    • CocoaPods version 1.9.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 45.1.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.45.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.11.0

[✓] Connected device (4 available)
    • iPhone 11  • 0EC80516-8F99-4A0C-89ED-2273579862F9 • ios            •
      com.apple.CoreSimulator.SimRuntime.iOS-13-5 (simulator)
    • macOS      • macOS                                • darwin-x64     • Mac
      OS X 10.15.5 19F96
    • Web Server • web-server                           • web-javascript •
      Flutter Tools
    • Chrome     • chrome                               • web-javascript •
      Google Chrome 83.0.4103.61

• No issues found!

flutter doctor -v
[✓] Flutter (Channel stable, 1.22.5, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.22.5 at /Users/tahatesser/Code/flutter_stable
    • Framework revision<PHONE_NUMBER> (4 weeks ago), 2020-12-10 11:54:40 -0800
    • Engine revision ae90085a84
    • Dart version 2.10.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.0

[✓] Connected device (2 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios • iOS 14.3
    • iPhone 12 (mobile)   • 12D8FF8E-2815-436E-9951-B8A6A42E4ACF • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)

! Doctor found issues in 1 category.

[✓] Flutter (Channel master, 1.26.0-2.0.pre.200, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.26.0-2.0.pre.200 at /Users/tahatesser/Code/flutter_master
    • Framework revision d0e9959a46 (11 hours ago), 2021-01-05 22:14:03 -0500
    • Engine revision e115066dcd
    • Dart version 2.12.0 (build 2.12.0-179.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.0

[✓] Connected device (4 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios            • iOS 14.3
    • iPhone 12 (mobile)   • 12D8FF8E-2815-436E-9951-B8A6A42E4ACF • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)      • macos                                • darwin-x64     • macOS 11.1 20C69 darwin-x64
    • Chrome (web)         • chrome                               • web-javascript • Google Chrome 87.0.4280.88

• No issues found!

flutter doctor -v
[✓] Flutter (Channel stable, 1.22.5, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.22.5 at /Users/tahatesser/Code/flutter_stable
    • Framework revision<PHONE_NUMBER> (4 weeks ago), 2020-12-10 11:54:40 -0800
    • Engine revision ae90085a84
    • Dart version 2.10.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.0

[✓] Connected device (2 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios • iOS 14.3
    • iPhone 12 (mobile)   • 12D8FF8E-2815-436E-9951-B8A6A42E4ACF • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)

! Doctor found issues in 1 category.

[✓] Flutter (Channel master, 1.26.0-2.0.pre.200, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.26.0-2.0.pre.200 at /Users/tahatesser/Code/flutter_master
    • Framework revision d0e9959a46 (11 hours ago), 2021-01-05 22:14:03 -0500
    • Engine revision e115066dcd
    • Dart version 2.12.0 (build 2.12.0-179.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.0

[✓] Connected device (4 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios            • iOS 14.3
    • iPhone 12 (mobile)   • 12D8FF8E-2815-436E-9951-B8A6A42E4ACF • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)      • macos                                • darwin-x64     • macOS 11.1 20C69 darwin-x64
    • Chrome (web)         • chrome                               • web-javascript • Google Chrome 87.0.4280.88

• No issues found!

This issue is still reproducible on the latest stable/master channel.

Sample code
import 'package:flutter/material.dart';

void main() {
  runApp(
    MaterialApp(
      title: 'Named Routes Demo',
      // Start the app with the "/" named route. In this case, the app starts
      // on the FirstScreen widget.
      initialRoute: '/',
      routes: {
        // When navigating to the "/" route, build the FirstScreen widget.
        '/': (context) => const FirstScreen(),
        // When navigating to the "/second" route, build the SecondScreen widget.
        '/second': (context) => const SecondScreen(),
      },
    ),
  );
}

class FirstScreen extends StatelessWidget {
  const FirstScreen({super.key});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('First Screen'),
        backgroundColor: Colors.blue,
        elevation: 8.0,
        shadowColor: Colors.black,
      ),
      body: Center(
        child: ElevatedButton(
          // Within the `FirstScreen` widget
          onPressed: () {
            // Navigate to the second screen using a named route.
            Navigator.pushNamed(context, '/second');
          },
          child: const Text('Launch screen'),
        ),
      ),
    );
  }
}

class SecondScreen extends StatelessWidget {
  const SecondScreen({super.key});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('Second Screen'),
        backgroundColor: Colors.blue,
        elevation: 8.0,
        shadowColor: Colors.black,
      ),
      body: Center(
        child: ElevatedButton(
          // Within the SecondScreen widget
          onPressed: () {
            // Navigate back to the first screen by popping the current route
            // off the stack.
            Navigator.pop(context);
          },
          child: const Text('Go back!'),
        ),
      ),
    );
  }
}

Demo on iPhone 7, iOS 15.8
https://github.com/flutter/flutter/assets/104349824/1a4d3a8c-3214-42d3-bf95-70b9679976d8

flutter doctor -v (stable and master)
[✓] Flutter (Channel stable, 3.19.1, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.19.1 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision abb292a07e (29 hours ago), 2024-02-20 14:35:05 -0800
    • Engine revision 04817c99c9
    • Dart version 3.3.0
    • DevTools version 2.31.1

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A507
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Applications/Android Studio Hedgehog.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.86.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.82.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 121.0.6167.184

[✓] Network resources
    • All expected network resources are available.

• No issues found!

[!] Flutter (Channel master, 3.20.0-11.0.pre.106, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.20.0-11.0.pre.106 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 59da873ef0 (4 minutes ago), 2024-02-22 19:16:20 -0800
    • Engine revision 0f3ad23b84
    • Dart version 3.4.0 (build 3.4.0-167.0.dev)
    • DevTools version 2.33.0-dev.11
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A507
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Applications/Android Studio Hedgehog.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.86.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.82.0

[✓] Connected device (4 available)
    • RMX2001 (mobile) • EUYTFEUSQSRGDA6D                         • android-arm64  • Android 11 (API 30)
    • iPhone (mobile)  • d9a94afe2b649fef56ba0bfeb052f0f2a7dae95e • ios            • iOS 15.8 19H370
    • macOS (desktop)  • macos                                    • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)     • chrome                                   • web-javascript • Google Chrome 122.0.6261.57

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.


Dimensional reduction of search string embedding

The choice of dimensionality reduction will affect whether or not we can 'record' how dimensionality reduction is effected on the search string embedding at runtime.
Why is this important?
If we can appropriately display the search string embedding then we can calibrate the visualisation so as to place the point of the search string vector at the origin and translate all other points by the search string vector. Furthermore, we can make the visualisation rotation-invariant (it doesn't matter how we rotate it) by using a 'radar' plot.

Linking to #10 as it may inform our method choice as you mention.

The 2D coordinates of the search string embedding can be calculated by weighted interpolation. Since we have the search string embedding and find the embeddings closest to it, then in the low-D space they should also be close to the corresponding projection of the search string embedding. I have a working implementation of this that needs to be cleaned up. Given a set of high-D points and a reference point (search string) we can estimate the position of the search string in 2-D.
One thing to be aware of though: proximity in a high-dimension space does not translate to proximity in a low-dimension space. This is because a projection to a lower dimension will make points that were far away from one another in the projected dimension appear closer in the lower dimension (e.g., the shadow of your ear will be close to the shadow of your foot even though they are far from one another).


fix: handle sharegpt dataset missing

Description

Fix the first problem in #2030
Motivation and Context

I forgot to consider that type:  can be non-str when setting the deprecation error.
How has this been tested?

Screenshots (if appropriate)
Types of changes

Social Handles (Optional)

@NanoCode012 should probably add a test case for this scenario

Added the test!


Remove the support for old Embulk v0.9 SPIs

This embulk-parser-json:0.11.* is expected to run with Embulk v0.11, no longer with Embulk v0.9. It no longer needs to take care of the old SPI.

Thanks!


API returns incorrect excerpt

Issue Summary
The API is not returning my custom excerpt but is instead returning the first lines of the text (in markdown?).
Steps to Reproduce

Create a post with a custom excerpt.
Retrieve the post via the API.
The JSON value of key "excerpt" does not contain the custom excerpt.

Ghost Version
4.32.3
Node.js Version
n/a - hosted by Ghost
How did you install Ghost?
Hosted by Ghost
Database type
Other
Browser & OS version
No response
Relevant log / error output
My custom excerpt:
> Best Christian Music of 2021, Paradise Lost, Shame Deserved and Undeserved, and more...

Post API response:
{
...
> "excerpt" : "Bible Study: Hebrews Part 7 (2:1–4) — Pay Attention to Jesus\n[https:\/\/www.thegardenweekly.com\/bible-study-hebrews-part-7\/]\nIn Hebrews 1, we saw an introduction (Hebrews 1:1–4) that tells us about a Son\nof God. Then, seven quotations from the Hebrew scriptures form an argument that\nthis Son is greater than the angels. That this Son is distinct from the Father,\nbut nonetheless fully deity.\n\nThe author breaks off his argument to give a short aside and application of his\nargument in Hebrews 1…\n\nKeep",
...
}

Code of Conduct

[X] I agree to be friendly and polite to people in this repository

Hi there @joeljfischer. This is expected behaviour. The custom excerpt is returned in the field custom_excerpt.


Add S3 endpoint support

need for minio S3
https://docs.min.io/docs/how-to-use-aws-sdk-for-go-with-minio-server.html

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Владлен Большаков seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


[5.4] Return the insertId of released jobs

The InteractsWithQueue trait returns the response from $this->job->release($delay); in it's release() and the DatabaseQueue class returns the inserted id from it's release() method.  This allows the inserted id to flow back to a job that manually calls release().
My use case involves wanting to release a job back onto the queue without incrementing it's attempts count.  I would use this to manually query the jobs table and decrement the attempts after calling release().
One issue I see with this proposal is that the Job contract describes the release() response as being void, which is violated by this change.

Can you update the contract return type?


Master nmilovanov add missing examples

Expanded and improved API's examples.
Unit tests for Client left undone for now.

Please fix merge conflicts.

Merged as squashed commit.


Auto-calibration (minimalists demo)

TODO's to close this PR:

[x] Skeleton of both classes
[x] Improvements of both classes
[x] Tests for both classes
[x] Classes API documentation
[x] Documentation of how to build a notebook (inputs/outputs, etc..)

For the REVIEWERS, this PR contains:

Main classes: CalibrationNode and CalibrationController [IMPORTANT]
Tests for both classes [IMPORTANT]
5 equal notebooks needed for the tests. [NO IMPORTANT]
a minor extra change in a data_managment test [NO IMPORTANT]

Explanation of the PR:
This PR consists mainly of:

theCalibrationNode, a class that contains the experiment notebooks, responsible for handling everything related to individual notebooks/experiments.
the CalibrationController, a class that links different instances of the previous one in a graph, which is responsible for controlling the workflow of the total calibration.

A basic initialization of such classes, would look like:
# CREATE NODES :
first = CalibrationNode(
    nb_path="notebooks/first.ipynb",
    in_spec_threshold=4,
    bad_data_threshold=8,
    comparison_model=norm_root_mean_sqrt_error,
    drift_timeout=1800.0,
)
second = CalibrationNode(
    nb_path="notebooks/second.ipynb",
    qubit_index=2,
    in_spec_threshold=2,
    bad_data_threshold=4,
    comparison_model=norm_root_mean_sqrt_error,
    drift_timeout=1.0,
    sweep_interval=personalized_sweep_interval,
    number_of_random_datapoints=5,
)

# NODE MAPPING TO THE GRAPH (key = name in graph, value = node object):
nodes = {first.node_id: first, second.node_id: second}

# GRAPH CREATION:
G = nx.DiGraph()
G.add_edge(second.node_id, first.node_id)

# CREATE CALIBRATION CONTROLLER:
controller = CalibrationController(node_sequence=nodes, calibration_graph=G, runcard=runcard_path)

Once you have everything setup, from CalibrationController you could run:

CalibrationController.run_automatic_calibration()
CalibrationController.maintain()

The calibration workflow, has three levels of methods that manage it:
The highest level method to apply, is run_automatic_calibration() which finds all the final nodes (don’t have others past them), and runs CalibrationController.maintain() on those.
The two mid-level methods are maintain() and diagnose(), the first one goes back to all the nodes dependencies starts, and working from them up, checking their last time executions and their data, searching for problematic cases. In such cases, depending on the severity, maintain() would directly calibrate or call the other mid-level method, diagnose(), which is responsible for working in reverse, from the problematic one, going down to all their dependencies, until finds the root of the problem, then passes such information back to the maintain() method, so it can calibrate accordingly, and finish its job.
And finally, during all this process, the mid-level methods would be calling these low-level methods, that act on the nodes:

calibrate(), which calibrates a notebook
check_status(), which checks time executions and compares them with the drift_timeout, to save time, omiting checking data, for recent executions.
check_data(), which checks data, of a small executions with the previous good ones.

All of these methods above, from CalibrationController, will be calling attributes or methods from CalibrationNode constantly, the most important one, would be:

run_execution(), which is responsible for executing the notebooks, passing the corresponding inputs to them, and then retrieving the outputs from export_calibration_outputs(outputs_dict), which should be called inside a notebook.

A basic notebook experiment would look something like (in a .ipynb):
# ``check_data()`` parameters:
check = False
number_of_random_datapoints = 10

# Sweep interval:
sweep_interval = ...

# Extra parameters for this concrete notebook:
param1=...
param2=...
...

# Set the environment and paths:
platform_path...

# Set the platform:
platform = ql.build_platform(path=platform_path)
platform.connect()
platform.initial_setup()
platform.turn_on_instruments()

# Define circuit
circuit = ...

# Loop over the sweeps executing the platform:
results_list = []
for X in sweep_interval:
    platform.set_parameter(alias=alias, parameter=ql.Parameter.X, value=X)
    result = platform.execute(program=circuit, num_avg=hw_avg, repetition_duration=repetition_duration)
    results_list.append(result.array)

results = np.hstack(results_list)

# Define fit
def fit(xdata, results):
    ...

# Fit data and show results
fitted_values, x_data, y_data, figure = fit(xdata=sweep_interval, results=results)
plt.show()

# Export final data from the notebook to the ``CalibrationController``
from qililab.automatic_calibration.calibration_node import export_calibration_outputs
export_calibration_outputs(
    {
        "check_parameters": {"x": sweep_interval, "y": results},
        "platform_params": [(bus_alias0, param_name0, fitted_values[0]), (bus_alias1, param_name1, fitted_values[1])],
        "fidelities": {"fidelity1": 0.9, "fidelity2": 0.95}  # This key in the dictionary is optional
    }
)

References
The main algorithm where this PR is based on is  given by: https://arxiv.org/abs/1803.03226
Here you have that original Marco PR: https://github.com/qilimanjaro-tech/qililab/pull/478
And in Notion there is more info:

https://www.notion.so/qilimanjaro/Automatic-Calibration-53cef01488db4a44a7ecf0f475e08084
https://www.notion.so/qilimanjaro/Automatic-qubit-calibration-high-ff49b69e241943fc942a6c0a9030ed70

Albert left me some comments on person about the PR, concretely about the lack of a qubit_index, so taking that as review on person I just added, such necessary missing thing!
The change mainly consists of this change on CalibrationNode._path_to_name_and_folder():

together with adding the needed new attribute node.qubit_index, which get passed as an input to the notebooks.
And finally the retrieval of parameters and fidelities, now also have this argument inside them, so CalibrationController methods:

_update_parameters()
get_last_set_parameters()
get_last_fidelities()
also got a very small change to include the qubit now.

I have seen your question very quickyl Juanjo, and there are some really good points, I'll make the changes/answers tomorrow more slowly!
Thanks again for the patience! 🙌

Respect conditionals on unitest, I think those will remain, since we have plenty in all qililab, and are pretty necessary here, if not the test would become huge, having to test each branch in a different test.

Respect Sourcery changes, we will make all we think are convenient, and add the convenient/needed ignores for the others! 👍

We will also review all the comments you wrote about the docstrings, and pass them trough ChatGPT, nice idea, with the hurry we forgot doing it one last time before ending the PR (we did it during the work, but then modified things 🙌 )

And finally respect the prints as said above, we are deciding which to convert into loggers and which not, and into which level, we will do that the last thing in the PR, we believe 🚀

@fedonman nice comments man! This is exactly what we need 🙌
With Isaac and less late in the night we will discuss them again, and apply what you suggested or elaborate further on why not, thanks!

Check how to add the notebook iteration for when one notebook is called multiple times (Isaac in local has an approach already)

Added the node_distinguisher parameter to the Nodes, which is needed for the Controller to work 👍

@AlbertMitjans Just, so you know, we have in mind several UX/UI improvements, which will be tackled, if prioritized, in the FUTURE IDEAS task, in case you wanna peek an eye 🚀
The most annoying one, is having to manually pass a node_distinguisher (int | str), when the same node is called multiple times in the same qubit...

I am approving the PR, because I prefer testing this ASAP rather than looking into small details.

Great, but as said in the other comment, we don't really need to merge it. We can focus on ending the lab stuff, go to the lab and test it, and then finish this PR.
Or we can merge it, and then open a new task with the pending changes, don't really care! 🙌

I am approving the PR, because I prefer testing this ASAP rather than looking into small details.

Great, but as said in the other comment, we don't really need to merge it. We can focus on ending the lab stuff, go to the lab and test it from this branch, and then when it works, finish these PR suggestions and merge it!
Or we can merge it, and then open a new task with the pending changes, don't really care! 🙌

Given that this PR is a completely new feature and doesn't (or at least shouldn't!) interact with the rest of qililab, I prefer to merge it and open a new PR with new fixes if needed.
This way it'll be way clearer what are the new changes.

I am approving the PR, because I prefer testing this ASAP rather than looking into small details.

Great, but as said in the other comment, we don't really need to merge it. We can focus on ending the lab stuff, go to the lab and test it from this branch, and then when it works, finish these PR suggestions and merge it!
Or we can merge it, and then open a new task with the pending changes, don't really care! 🙌

Given that this PR is a completely new feature and doesn't (or at least shouldn't!) interact with the rest of qililab, I prefer to merge it and open a new PR with new fixes if needed.
This way it'll be way clearer what are the new changes.

Yess! it only import thing from the rest of qililab, it doesn't export things back to it! So I totally agree 🚀


error when I run validate_on_lfw.py

Hey, guys. When I trying to run  validate_on_lfw.py, it returns an error
The name 'input:0' refers to a Tensor whick does not exist. The operation, 'input', does not exist in the graph.
How to fix this?

This is  because the tensor name has changed. You can try this
images_placeholder = tf.get_default_graph().get_tensor_by_name("batch_join:0")


add security context for typha

Note this change is dependent on https://github.com/projectcalico/typha/pull/312 - so should be merged after https://github.com/projectcalico/typha/pull/312 is merged.

Deploy preview for calico ready!
Built with commit 8ea4f02d9539bb4dbdbb1b5ceae95f79ffc67e5a
https://deploy-preview-2857--calico.netlify.com

@gauravsinhatigera did you mean to close this?

@spikecurtis - yes, I have created a different PR (https://github.com/projectcalico/calico/pull/2858) for after syncing my fork with upstream.


github Copilot Chat,  It can't work

Copilot Chat Extension Version:
VS Code Version:
OS Version:
Logs:

tips : this account is not been used by any extension
Steps to Reproduce:

GitHub Copilot Individual is active for my account
My currently have an active Copilot Individual subscription.
Get started by installing the extension in my preferred IDE（vs code）
right？

I also can not use my extension in vscode, have you solved your problem?

Hre are the steps: https://code.visualstudio.com/docs/copilot/setup
Can you give more details which step does not work? You can also add a sceenshot.

Here's a screenshot of my vscode, I'm trying to use copilot chat, but nothing happens when I open this extension plugin, and nothing happens when I type something in, no action！

Can you try vscode insiders and the latest pre-release version of Copilot Chat? Also please attach your logs via CMD/CTRL + SHIFT + U -> GitHub Copilot Chat

try this https://github.com/orgs/community/discussions/134876#discussioncomment-10257723

@HongHaiyang
Hello, I have read this post, and I have followed this step, but it still doesn't work. The copilot version can only be used normally if I return it to two months ago.😭

@lramos15
output log
[info] [FetcherService] Using the Helix fetcher.
[info] [gitExtensionService] Initializing Git extension service.
[info] [gitExtensionService] Successfully activated the vscode.git extension.
[info] [gitExtensionService] Enablement state of the vscode.git extension: true.
[info] [gitExtensionService] Successfully registered Git commit message provider.
[info] [auth] Logged in as lucker3900
[info] [extension] RootCertificateReader removed 11 expired certificates
[info] [auth] Got Copilot token for lucker3900
[info] [chat] copilot token chat_enabled: true
[info] [githubTitleAndDescriptionProvider] Successfully activated the GitHub.vscode-pull-request-github extension.
[info] [githubTitleAndDescriptionProvider] Initializing GitHub PR title and description provider provider.
[info] [githubTitleAndDescriptionProvider] Successfully registered GitHub PR title and description provider.

@lramos15
github copilot chat version 0.18.2

@lucker3900 If you're using VS Code insiders you should get the pre-release copilot chat which is
v0.20.2024081501 or greater

@lramos15
copilot chat lastest version is v0.18.2， vs code

Yeah you need insiders as described here


Agenda

In the agenda (UitVlaanderen) are still events from the past.
This is for example the case in Herk-de-Stad

This is done on purpose because the end date is optional i thought


FX: Flaky Azure Tests

This PR should fix Azure flaky tests

@ironman-machine ok

Hello @nicolas2bert
Starting end to end procedure using the following payload:
{"REPO_NAME": "S3", "DEFAULT_BRANCH": "master", "SCALITY_S3_BRANCH": "fix/azuretests"}
Please follow http://ci.ironmann.io/gh/scality/Integration/14135 for CI status.


Configure kops to use containerd

Use --container-runtime containerd. Switching to containerd in order to address docker being deprecated as a runtime.

closing, we'll just follow #12


Changing debug output structure. Fixes #403

When this is committed, we'll need to update Quepid to consume the new data structure ;-)


Packet Options Improvements

Combine packet options into a dropdown instead of a list of buttons
Make packet option buttons its own component for reuse

LGTM


 "Say something:"
    input = Stdin.line!
    Stdout.line! "You said: $(input)"

main = run

Also Fixes #7127


Convert library to .net standard

Thanks for this, I'd like to add GitHub Actions to this repo to publish to NuGet automatically. Once I've done that I'll bump the major version and use this PR to target netstandard2.0

@roederja2 would you like a credit in the README?
https://github.com/taspeotis/DeltaCompressionDotNet#contributors

Up to you. Not sure it's significant enough.


x/tools/go/types/objectpath: frequent use of scope.Names burns significant cycles

What version of Go are you using (go version)?
go1.18-pre10
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
Linux, amd64. Not dependent on OS or architecture, however.
What did you do?
The objectpath.For method relies on frequent calls to scope.Names(), which always performs a sort of all names. For large-scale analysis trees, this can consume a significant portion of cycles (30-50% in my case) and can easily be memoized. This can either be done by objectpath as a consumer of the API, or within the *types.Scope object.
What did you expect to see?
Fewer cycles spent sorting.
What did you see instead?
Lots of cycles spent sorting.

Thanks for filing.
Need to think about this. I'll just note that in the profile linked on that CL, there is also significant amount of time spent in Scope.Lookup. But objectpath just needs to walk all the objects, and doesn't care about sorting. I wonder if we should be considering an entirely new Scope API...  Or maybe that is further evidence that it would be better for us to build a layer in object path that can significantly optimize the calculation of paths. Compare x/tools/go/ast/inspector.

@griesemer @alandonovan what do you think about this change? https://go.dev/cl/383075 appears to be stuck in limbo, and it would be good to make a decision for 1.20.
Being conservative, I am inclined to say that changing scope.Names to return a memoized slice is a breaking change, may increase steady-state memory, and is inconsistent with the rest of the go/types API which goes to lengths to avoid exposing internal data (consider Struct.NumFields, etc.). If the problem is mostly isolated to usage of objectpath, we could expose a batch or memoizing API.
But I don't feel strongly. If it is most expedient to save sorted names on the scope, in reality it is probably harmless.

The actual code review hasn't addressed all the comments (e.g. re: atomicity) required for +2, but perhaps that's just because the author doesn't know if the design is worth pursuing.
I agree that if the problem is really just in objectpath, then that's a safer place for the fix: perhaps objectpath could compute the paths for the entire package more cheaply in a single pass, then serve a large number of queries on that---like the AST inspector. Even if we don't implement all that at once, the same API could take the first step of memoizing scope.Names() across multiple calls, but it seems likely there are even bigger potential wins if we take that approach.

https://go-review.git.corp.google.com/c/tools/+/405354 (WiP for memoizing) was stuck due to not having a clear performance justification yet.

@griesemer and I don't think we want to make this change in go/types, and would prefer improving the objectpath API. Even if we pre-sorted, we'd still want to make a copy and so would lose some of the benefit of memoizing. Moving out of the Go1.20 milestone.

I had an overly simplistic form of objectpath.For which I wrote a while ago, which you can see here: https://github.com/burrowers/garble/blob/98466a1f64dfc9a3b9d4bb49dcdb2b3a5ff5f245/main.go#L1533
I tried to switch to objectpath, as it's clearly more complete and I think it might fix two bugs I've been struggling with. However, re-running go test prompty crashed my computer out of memory :) It of course was due to all the Scope.Names() calls.
In my use case, the tool will be calling objectpath.For for almost every object reachable from a package. I know that's perhaps the worst case scenario, but I don't think I could avoid the majority of those calls. I need to essentially record and retrieve a "fact" about every exported name in a package: whether its name was changed or not.
I don't think I can use objectpath right now, but I'd be more than happy to help review or test any improvement or new API in objectpath itself. Memoization or a batch API are likely to make a big difference.

I'd approve a CL that added new API to the object path that allowed it to memoize lookups, e.g. new Encoder(pkg).For(obj). Internally, the Encoder could assert that obj.pkg = enc.pkg. The first time a For call falls through to the need for Scope.Names ("step 4"), the encoder could call Names once (using sync.Once if concurrency is required) then sort the slice into two parts, a slice of TypeName and one of non-TypeName, and retain these two. Then subsequent calls could skip the Names call, its allocation and sort, all the name-based Lookups and type assertions. Would that significantly improve performance in your case? Care to measure it?

I just ran into this while working on the new incremental scalability work.  I observed that building our xrefs Index takes ~50% as long as type checking (6s vs 12s in my benchmark), largely due to calls to scope.Names.
So we're going to need a new API for gopls to use, anyway. We can experiment with it internally first, but should be sure to file a proposal once we're satisfied.

Another place that burned significant cycles was objectpath.canonicalize, which could also be optimized with a stateful encoder.


SyntaxError: 'import' and 'export' may appear only with 'sourceType: module'

Operating System: Linux
Cypress Version: latest
Browser Version: latest
BUG/Question
###Current behavior:
I'm trying to create e2e tests with Cypress + Cucumber + Typescript + cypress/webpack-preprocessor
When step-definition file don't have imports/exports - everything works well
But when I'm trying to use PageObjects in TS, using export-import statement, then I got an error like 'import' and 'export' may appear only with 'sourceType: module'
I don't have the possibility to change imports and exports to require and module.export because in my current project I have a lot of PO's and I want to save 2 approaches in tests: default approach and BDD approach
###Desired behavior:
Possibilities to use import and export
###How to reproduce:

In this PR you can see how to reproduce the issue https://github.com/ludmilanesvitiy/cypress-cucumber-example/pull/1
In the main repository all works well. https://github.com/ludmilanesvitiy/cypress-cucumber-example

@lgandecki Thank you for the reply. Yes, in my case looks like TS just ignored, but in your example for compiling is used browserify-preprocessor. But I need webpack-preprocessor. Any ideas how to solve? And what could be wrong here https://github.com/ludmilanesvitiy/cypress-cucumber-example/blob/master/cypress/plugins/index.js?

@ludmilanesvitiy Have you found a fix for this? I'm seeing the same error using a similar setup to what you have in your repo.

@drewloomer No, I don't have a fix for this. Maybe @lgandecki could advice) According to his comments in #114 "I (and most people) used typescript with import/export." - he should know the secrets)

@drewloomer @ludmilanesvitiy take a look on #115, it should solve your issue

Hi,
I'm still having the same issue.
in plugins/index.ts I have

We now have an example for typescript working with webpack:
https://github.com/TheBrainFamily/cypress-cucumber-webpack-typescript-example/
and without it:
https://github.com/TheBrainFamily/cypress-cucumber-typescript-example/
I'm closing this thread for now, if you have a specific issue, after veryfing that the examples work, and that your configuration is in line with them, or if you need to somehow configure things differently - please create a new one and I will be glad to take a look.

@lgandecki So you have examples with and without webpack, but where do you have an example with "out-of-the-box support", described here, in your readme section:
https://github.com/TheBrainFamily/cypress-cucumber-preprocessor#with-out-of-the-box-support
Because I don't think I have ever seen this working with tsconfig.json and imports / exports.


【stream bug】When use “ switchVideo ” new m3u8 url，But old m3u8 stream still exists！pls help

new DPlayer use "A" (m3u8) play~
when use :
player.switchVideo({url: B});
"B" is New m3u8 url~
old "A" m3u8 hls stream still exists  .
screenshot :

ps. i check html dom, Confirm that there is only one Dplayer in my page
Window Chrome   and Mac Chrome both have this bug
btw: In mac , safari haven't this bug~
How to fix it ?

Yes, there is indeed this bug, even if you stop the video before switching, it is useless.


Update nginx controller to use endpoints. Remove lua to configure upstreams (WIP)

Using the flag --v=2 it is possible to see in the logs the changes in the configuration when an endpoint is added/removed
I0316 12:24:37.581267       1 utils.go:148] NGINX configuration diff a//etc/nginx/nginx.conf b//etc/nginx/nginx.conf
I0316 12:24:37.581356       1 utils.go:149] --- /tmp/922554809	2016-03-16 12:24:37.000000000 +0000
+++ /tmp/079811012	2016-03-16 12:24:37.000000000 +0000
@@ -235,7 +235,6 @@

     upstream default-echoheadersx {
         least_conn;
-        server <IP_ADDRESS>:5000;
         server <IP_ADDRESS>:5000;

     }
I0316 12:24:37.610073       1 command.go:69] change in configuration detected. Reloading...

Using the same flag but in level 5 --v=5 it will configure nginx to run in debug mode

This also removes the dns requirement

Let me know when you're done experimenting with reloads and I'll resume reviewing

This also also adds support for TCP services using ConfigMaps using the flag
--tcp-services-configmap=default/tcp-configmap-example
example
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-configmap-example
data:
  9000: "default/example-go:8080"

ping @bprashanth

LGTM please squash into fewer commits, and add some sections to the README about how to use configmaps to loadbalance tcp etc

Also please add a section about lua reload vs -s and that you investigated the performance to be roughly equivalent

Just a few nits. It would be really nice if we got the README examples to be more similar to the GCE one so people get a cross platform experience. Ideally we'd just have one subdir of examples and link from both the nginx and gce controller to that.
Please also remove wip when you re-up, and I can lgtm.


Federation Gateway

I understand the OAUTH2 flow with IDSRV4 and we're currently implementing token access and user authentication with open ID connect. I've done the external authentication quick-start and read the docs about external providers.
Aside from listed concepts, nothing solid is telling me how do you set up IDSRV4 to be a federation gateway? I'm still not clear on how to configure the server and a user client.
Much help would be appreciated.

http://docs.identityserver.io/en/release/topics/federation_gateway.html

@TomCJones Thanks for the reply.
Yeah, I read that Tom, I mention that in my post but not as clearly, but I also read this article too. https://www.identityserver.com/articles/identityserver4-ws-federation-and-sharepoint/
I'm just wondering if above link is the way to go - which it sounds like to me.

@TomCJones
Thanks for the reply;   read that article but it's still not clear to me. I'm also reading about this: https://www.identityserver.com/articles/identityserver4-ws-federation-and-sharepoint/
Is this, the above link, the way to go or the complete or the complete opposite direction?

A federation gateway is a IS4 with support for external providers. That's it.

Think of a "federation gateway" as a pattern - not a specific configuration.

@leastprivilege Thanks for the reply. I really, really appreciate it.
I'm coming to find out this is bit of misnomer. Sounds like it can mimic the architecture, more so, and that's it? But if I wanted to go the extra mile then I would need to implement my own ws-federation endpoint, like (how Scott Brady did) in that link I posted, or wait until the team releases an official NUGET plugin (like how they did for IDSRV3). Am I right? Does that sound about right?

The ws-fed endpoint means that IS supports more than one protocol as an IdP -
the fed gateway means that IS itself is a client to more than one IdP.
I know it is confusing ;)

I understand the OAUTH2 flow with IDSRV4 and we're currently implementing token access and user authentication with open ID connect. I've done the external authentication quick-start and read the docs about external providers.
Aside from listed concepts, nothing solid is telling me how do you set up IDSRV4 to be a federation gateway? I'm still not clear on how to configure the server and a user client.
Much help would be appreciated.

were you able to solve the issue ?


Incorrect constraints change detection for multi-tenanted documents

Behavior:
Marten change detection (on startup with apply, db-assert/db-apply) keeps finding constraints configuration mismatch on multi tenant documents even if correct constraint already exists.
alter table public.mt_doc_multitest drop constraint pkey_mt_doc_multitest_tenant_id_id CASCADE;
alter table public.mt_doc_multitest add CONSTRAINT pkey_mt_doc_multitest_tenant_id_id PRIMARY KEY (tenant_id, id);
Reason:
The reason is in the columns order. The above behavior manifests only on tables where column 'tenant_id' is the last column.
If db-apply is done for brand new multi-tenanted document, created table will be generated with tenant_id at first position tenant_id, id, data, mt_last_modified, mt_version, mt_dotnet_type. Change detection in that case works correctly.
Column tenant_id can easily become last, e.g. document tenancy added after schema already generated and table was altered by Marten.
Workaround:
The only workaround is to reorder columns in the doc tables so that tenant_id is the first column (which is quite cumbersome in postgres)
PS: I believe this issue started with Marten 7+

@misha-p Hey, we're going to need a bit more in terms of reproduction steps. How have you made that happen? What was your starting point, and what did you move to? Not sure I understand how you got there

Hi Jeremy, sure, here's the example:

Create document without multi tenancy.

public record TestDoc(Guid Id, string Column);
...
// configure Marten - anything that will persist that document in DB, add index for example:
storeOptions.Schema.For<TestDoc>().Index(x => x.Column);

Run application or db-apply to get TestDoc created in database.
Database table will be mt_doc_multitest (id, data, mt_last_modified, mt_version, mt_dotnet_type)
Add tenancy to TestDoc

storeOptions.Schema.For<TestDoc>().Index(x => x.Column).MultiTenanted();

Run application or db-apply to get TestDoc altered in database.
Database table will be mt_doc_multitest (id, data, mt_last_modified, mt_version, mt_dotnet_type, tenant_id)

At this point Marten will keep seeing the change in CONSTRAINT pkey_mt_doc_testdoc_tenant_id_id
E.g. running dotnet run -- db-assert, will produce:
Database 'Marten' does not match the configuration!
Weasel.Core.Migrations.DatabaseValidationException: Configuration to Schema Validation for Database 'Marten' Failed! These changes detected:

alter table public.mt_doc_testdoc drop constraint pkey_mt_doc_testdoc_tenant_id_id CASCADE;
alter table public.mt_doc_testdoc add CONSTRAINT pkey_mt_doc_testdoc_tenant_id_id PRIMARY KEY (tenant_id, id);

To fix it I should either drop mt_doc_testdoc and do db-apply (not good for production system) or move tenant_id column to first position (e.g. create new table, copy data, drop old table, rename new)

The relevant code in Weasel to detect PK columns on an existing table weren't considering column ordering. It's going to take a Weasel change to get this fixed -- which is done, just pending the next Weasel release


Patch rest.py to avoid deprecated urllib3 usage

urllib3's Response.getheaders() is deprecated. With this patch, we can successfully run python -Werrors -m pytest tests/test_asset.py (turning warnings into errors).
The update_generated.sh script seems to be Linux-only. With some modification to the sed commands (-i '' instead of --in-place), I could get the script to run on my Mac, but the original json_safe patch no longer applies.
I'll look into tuning up that patch so that it will apply before I ask for review.

I have to give up on this for now. The code I'm generating doesn't match the patch.


Changed Twig Link Regex which fixes #149 [Markdown Doesn't Handle words

Hi @rhukster,
here is the PR of bugfix for issue https://github.com/getgrav/grav/issues/149 .

Edge case "fixed" :grinning:

That looks like a fix - works for me.

Looks good to me! Thanks!


How to add ifc model in react-map-gl layer.

Description 📝
I am working on the map-relevant project now. I am using react-map-gl to display the map. Also, I want to display the IFC model with this library. But I can't display the model because the renderer is different.
Also, FragmentIfcLoader is not working with the error log "uiElement isn't initialized".
Please help me to resolve this problem, if anyone has an experience of this similar problem.

Hey @force9493! Hope you're doing well 🙂
We try to keep the issues in the repository to report specific bugs or to propose features. To make questions about the library and get help, I invite you to access the community at https://people.thatopen.com/, it's free and there are many people willing to help others!!
I will close this issue, thank you! 🙏


Converted the project to use Avenir by default

Converted the project to use Avenir by default
Removals

Removed webfont-regular from styles where it's no longer needed
Removed short-desc from templates where it's no longer needed

Changes

Converted full-width-sans to full-width-serif

Testing

Check all pages, everything should be Avenir except for the full width molecule on /document-detail/

Review

@duelj
@ajbush
@anselmbradford
@KimberlyMunoz
@sebworks
@schaferjh

Screenshots
Only serif (Georgia) in use

Todos

Move this over to CF at some point.

Checklist

[ ] Changes are limited to a single goal (no scope creep)
[ ] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the front end playbook
[ ] Passes all existing automated tests
[ ] New functions include new tests
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[ ] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)

http://localhost:8000/sub-pages/privacy-impact-assessments-pias/ has too much of the .webfont-medium().

Fixed the Avenir Medium issue.

Checked all the pages and it looks good to me. :+1:

Looks like We work with … needs Avenir too, when compared to refresh anyway.

:+1:


GPUImage Denoise an Image

Hello guys,
Is there a way to remove the noise of a static image using GPUImage?
Thank you

There are many denoising techniques out there, providing different tradeoffs between image quality and denoising level. You may need to implement your own, but things like the median filter or the bilateral blur have been used in denoising of static images.


long mitotic spindle during anaphase during  anaphase A

whoops, missed
long mitotic spindle during anaphase during  anaphase A
descendant of
FYPO:0004438 | long mitotic spindle during anaphase
PMID:12426374

long mitotic spindle during anaphase A FYPO:0006190
edit file: 28cc581ae071c1054d46c2e7040230cc73223744
release: e31b67096fa25e5ce9a6c35ddd11a0f0ecd435b7


Support optional fields

This is discussed in https://github.com/grpc-ecosystem/protoc-gen-grpc-gateway-ts/issues/21
From the language guide: https://protobuf.dev/programming-guides/proto3/#field-labels

optional: An optional field is in one of two possible states:
the field is set, and contains a value that was explicitly set or parsed from the wire. It will be serialized to the wire.
the field is unset, and will return the default value. It will not be serialized to the wire.
You can check to see if the value was explicitly set.

If I recall, this wasn't originally supported in proto3 but is now being recommended as a way of differentiating between nil and zero values, without having to use the Wrapper types.

Because proto3 optional is implemented in terms of a singleton oneof, this almost works out of the box. The generated type looks like this:
type BaseOptionalFieldsRequest = {
  str?: string
  number?: number
}

export type OptionalFieldsRequest = BaseOptionalFieldsRequest
  & OneOf<{ optStr: string }>
  & OneOf<{ optNumber: number }>

The problems with this are that:

str is already optional, so there's not really a functional difference.
I think it's going to require the gateway to have EmitOptional set on the MarshalOptions

wdyt @seanami ?

The above PR make it work, but doesn't update the typescript definitions. That means a non-optional field that is empty, implies zero-value. But there's no good way to differentiate on the client without knowing the API spec.
An option could be to add a flag to the generator: --strict_types which at least requires an explicit zero value to be specified. This would work with EmitUnpopulated=true, but with EmitUnpopulated=false undefined values could be sent over the wire and parsed into an object with non-null fields.

I added a test case here.
With EmitUnpopulated on:

msg1: {"str":"", "msg":null}
msg2: {"str":"test", "optStr":"test", "msg":{"str":"test", "optStr":"test"}, "optMsg":{"str":"test", "optStr":"test"}}

With EmitUnpopulated off:

msg1: {}
msg2: {"str":"test", "optStr":"test", "msg":{"str":"test", "optStr":"test"}, "optMsg":{"str":"test", "optStr":"test"}}

So with EmitUnpopulated=true I think the types might make sense to be as follows:
message TestMsg {
  string str = 1;                  // string
  optional opt_str = 2;            // string?
  TestSubMsg msg = 3;              // TestSubMsg | null
  optional TestSubMsg opt_msg = 4; // TestSubMsg?
}

And with EmitUnpopulated=false:
message TestMsg {
  string str = 1;                  // string?
  optional opt_str = 2;            // string?
  TestSubMsg msg = 3;              // TestSubMsg?
  optional TestSubMsg opt_msg = 4; // TestSubMsg?
}

In which case, a flag seems like the best bet so that users can align the gRPC Gateway settings with the generated typescript. In the same way as the useProtoNames / use_proto_names is passed.

Thanks for the test case illumination!
I see, so null actually gets used for non-optional fields when they're not set. (And I'm imagining that they'd also get used for optional fields when it's set to the empty/default value? Or would that just be the string instead?)
I guess I needed one more test case for each protojson option, which is "What does a 'explicitly set to empty' optional field look like?" Does it ever use null, or does it just use default values like "" or 0 or etc.?
Depending on that answer, I think your proposal is either basically right, or you might need to add null as a possible type for the optional values when EmitUnpopulated=true.

One thing that's not great about your EmitUnpopulated=true proposal: It's always possible that a field could be missing unexpectedly because the server is running a different version of the proto def. I think that's why the current types always includes undefined as a possible value for every field, because it might just not show up on the wire, and the app code should be resilient to that.
But, I agree that your proposal would be nice, and when using protos in TypeScript I've often found myself being annoyed by having to handle undefined everywhere, especially for outgoing messages, where it's really not necessary.
One idea to make your proposal work would be to add templatized helper code in between receiving the response message from an endpoint and handing it to app code. It would use the proto definition to check for all of the expected/non-optional fields, and if any are missing over the wire, it would add the field with the default value, rather than letting it be undefined. This would ensure that, once app code had a response message, it really DOES conform to the type information without falling back on undefined.

Your point about compatibility is an interesting one, getting concrete it would happen under the following situations:

the server deprecates a field prior to clients being expired
a server is rolled back independently of clients
a client is pushed before the server

I'd expect that a lot of code doesn't handle these cases very gracefully right now :-/
Coercing undefined to zero type would require the proto descriptors (or some abstraction) on the client. Supporting nested messages seems especially tricky.

I have a POC of a parameter that updates the types per the above proposal. Here's your example from above:
type BaseOptionalTestMsg = {
  str: string
  msg: OptionalTestSubMsg | null
}

export type OptionalTestMsg = BaseOptionalTestMsg
  & OneOf<{ optStr: string }>
  & OneOf<{ optMsg: OptionalTestSubMsg }>

type BaseOptionalTestSubMsg = {
  str: string
}

export type OptionalTestSubMsg = BaseOptionalTestSubMsg
  & OneOf<{ optStr: string }>

Here's the same message without the flag specified. Based on the corresponding behavior in the gRPC Gateway you can't tell the difference between msg and optMsg.
type BaseOptionalTestMsg = {
  str?: string
  msg?: OptionalTestSubMsg
}

export type OptionalTestMsg = BaseOptionalTestMsg
  & OneOf<{ optStr: string }>
  & OneOf<{ optMsg: OptionalTestSubMsg }>

type BaseOptionalTestSubMsg = {
  str?: string
}

export type OptionalTestSubMsg = BaseOptionalTestSubMsg
  & OneOf<{ optStr: string }>

This seems like a good incremental step to me, since currently the typescript types aren't accurate if the server is set to use EmitUnpopulated.
Then we could explore ways of mapping

Ok, I may have mispoke above.
Here's the proto message:
message OptionalFieldsMsg {
  string empty_str = 1;
  int32 empty_number = 2;
  OptionalFieldsSubMsg empty_msg = 3;
  optional string empty_opt_str = 4;
  optional int32 empty_opt_number = 5;
  optional OptionalFieldsSubMsg empty_opt_msg = 6;

  string zero_str = 7;
  int32 zero_number = 8;
  OptionalFieldsSubMsg zero_msg = 9;
  optional string zero_opt_str = 10;
  optional int32 zero_opt_number = 11;
  optional OptionalFieldsSubMsg zero_opt_msg = 12;

  string defined_str = 13;
  int32 defined_number = 14;
  OptionalFieldsSubMsg defined_msg = 15;
  optional string defined_opt_str = 16;
  optional int32 defined_opt_number = 17;
  optional OptionalFieldsSubMsg defined_opt_msg = 18;
}

message OptionalFieldsSubMsg {
  string str = 1;
  optional string opt_str = 2;
}

This is the default response.

All empty fields are omitted.
Zero fields are undefined unless they are optional — so the client needs to know they are optional in order to convert to empty.

{
  "zeroMsg": {},
  "zeroOptStr": "",
  "zeroOptNumber": 0,
  "zeroOptMsg": {},
  "definedStr": "hello",
  "definedNumber": 123,
  "definedMsg": {
    "str": "hello",
    "optStr": "hello"
  },
  "definedOptStr": "hello",
  "definedOptNumber": 123,
  "definedOptMsg": {
    "str": "hello",
    "optStr": "hello"
  }
}

type BaseOptionalFieldsMsg = {
  emptyStr?: string
  emptyNumber?: number
  emptyMsg?: OptionalFieldsSubMsg
  zeroStr?: string
  zeroNumber?: number
  zeroMsg?: OptionalFieldsSubMsg
  definedStr?: string
  definedNumber?: number
  definedMsg?: OptionalFieldsSubMsg
}

export type OptionalFieldsMsg = BaseOptionalFieldsMsg
  & OneOf<{ emptyOptStr: string }>
  & OneOf<{ emptyOptNumber: number }>
  & OneOf<{ emptyOptMsg: OptionalFieldsSubMsg }>
  & OneOf<{ zeroOptStr: string }>
  & OneOf<{ zeroOptNumber: number }>
  & OneOf<{ zeroOptMsg: OptionalFieldsSubMsg }>
  & OneOf<{ definedOptStr: string }>
  & OneOf<{ definedOptNumber: number }>
  & OneOf<{ definedOptMsg: OptionalFieldsSubMsg }>

type BaseOptionalFieldsSubMsg = {
  str?: string
}

export type OptionalFieldsSubMsg = BaseOptionalFieldsSubMsg
  & OneOf<{ optStr: string }>

Here is the response with emit_unpopulated enabled:

Only unset optional fields are undefined, which is more explicit IMO.

{
  "emptyStr": "",
  "emptyNumber": 0,
  "emptyMsg": null,
  "zeroStr": "",
  "zeroNumber": 0,
  "zeroMsg": {
    "str": ""
  },
  "zeroOptStr": "",
  "zeroOptNumber": 0,
  "zeroOptMsg": {
    "str": ""
  },
  "definedStr": "hello",
  "definedNumber": 123,
  "definedMsg": {
    "str": "hello",
    "optStr": "hello"
  },
  "definedOptStr": "hello",
  "definedOptNumber": 123,
  "definedOptMsg": {
    "str": "hello",
    "optStr": "hello"
  }
}

lueMsg | null
}

type BaseOptionalFieldsMsg = {
  emptyStr: string
  emptyNumber: number
  emptyMsg: OptionalFieldsSubMsg | null
  zeroStr: string
  zeroNumber: number
  zeroMsg: OptionalFieldsSubMsg | null
  definedStr: string
  definedNumber: number
  definedMsg: OptionalFieldsSubMsg | null
}

export type OptionalFieldsMsg = BaseOptionalFieldsMsg
  & OneOf<{ emptyOptStr: string }>
  & OneOf<{ emptyOptNumber: number }>
  & OneOf<{ emptyOptMsg: OptionalFieldsSubMsg }>
  & OneOf<{ zeroOptStr: string }>
  & OneOf<{ zeroOptNumber: number }>
  & OneOf<{ zeroOptMsg: OptionalFieldsSubMsg }>
  & OneOf<{ definedOptStr: string }>
  & OneOf<{ definedOptNumber: number }>
  & OneOf<{ definedOptMsg: OptionalFieldsSubMsg }>

type BaseOptionalFieldsSubMsg = {
  str: string
}

export type OptionalFieldsSubMsg = BaseOptionalFieldsSubMsg

I'm going to call this good for now. Client-side mapping of undefined to zero types would be a nice followup.

Thanks for all of that test code and examples, that's super helpful. 🙏
One annoying thing about this type of OneOf approach in the TypeScript is that the types of the fields become "baked in" after the initial object is defined. So, for example:
export function test() {
  let m: OptionalFieldsMsg = {
    emptyStr: '',
    emptyNumber: 0,
    emptyMsg: {str: ''},
    zeroStr: '',
    zeroNumber: 0,
    zeroMsg: null,
    definedStr: 'test',
    definedNumber: 4,
    definedMsg: { str: 'test'},
    emptyOptStr: 'test',
  };

  m.emptyOptNumber = 4;
}

You'll notice that emptyOptStr is defined as part of the original object literal, whereas emptyOptNumber is assigned to the field after the initial object is created.
Even though this is fine from a "are the fields all valid?" standpoint, TypeScript still throws an error:

This makes working with these types of objects annoying in practice, because you have to do things like this to assign new values after object creation:
  m = {
    ...m,
    emptyOptNumber: 4,
  };

This creates unnecessary objects and is also more verbose. So not ideal. Also, I think that using object spreads like ...m can mask other type issues, as I'm not certain that all of the members of the spread are checked.
Instead of using the OneOf and Absent helpers in TypeScript, do you think we could make the types something like this?
// emit_unpopulated = false

type BaseOptionalFieldsMsg = {
  emptyStr?: string;
  emptyOptStr?: string;
  emptyNumber?: number;
  emptyOptNumber?: number;
  emptyMsg?: OptionalFieldsSubMsg;
  emptyOptMsg?: OptionalFieldsSubMsg;
  // etc...
}

type BaseOptionalFieldsSubMsg = {
  str?: string;
  optStr?: string;
}

// emit_unpopulated = true

type BaseOptionalFieldsMsg = {
  emptyStr: string;
  emptyOptStr?: string;
  emptyNumber: number;
  emptyOptNumber?: number;
  emptyMsg: OptionalFieldsSubMsg | null;
  emptyOptMsg?: OptionalFieldsSubMsg; // Should this have `| null` added? Seems like it doesn't happen...
  // etc...
}

type BaseOptionalFieldsSubMsg = {
  str: string;
  optStr?: string;
}

This would make the types much easier to work with in the editor, and prevent unnecessary contortions. It's not clear that the OneOf/Absent is really getting us anything additional?


Batch insert into STAGING table...

Batch insert into STAGING table to facilitate downstream-system queue draining on the queue FULL event...
@iizrailevsky @ajain11 @ndoshi @jcwuzoegiver

Coverage decreased (-0.09%) to 75.774% when pulling 974442e60e3ea259595fa70ff7cfe1d2780899b6 on feature/drain-trinity-queue-in-batch into ff65e9a61e0e53a45011af644c54696ad4fe366a on develop.

Coverage increased (+0.03%) to 75.893% when pulling 13008fe6edab536c6bf12b317054e72eba1b8b26 on feature/drain-trinity-queue-in-batch into ff65e9a61e0e53a45011af644c54696ad4fe366a on develop.

Looks good! Thanks for adding the unit tests 👍  :shipit:

:shipit:

:shipit:


WIP:  Add QR-DQN support

Add QR-DQN support, related to this issue and PR.
Description

[x] Add QR-DQN support.
[ ] Confirm the performances in Pong and Breakout are comparable with coach.

Motivation and Context

[ ] I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (update in the documentation)

Checklist:

[ ] I've read the CONTRIBUTION guide (required)
[ ] I have updated the changelog accordingly (required).
[ ] My change requires a change to the documentation.
[ ] I have updated the tests accordingly (required for a bug fix or a new feature).
[ ] I have updated the documentation accordingly.
[ ] I have reformatted the code using make format (required)
[ ] I have checked the codestyle using make check-codestyle and make lint (required)
[ ] I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

@araffin
I have the computational resources available now, so I'm running the training in Pong and Breakout.
I'll let you know the results ;)

I have the computational resources available now, so I'm running the training in Pong and Breakout.

I already started too ;) (but not with the same exploration_fraction, btw, why is it not the default?)

Thanks :)

but not with the same exploration_fraction, btw, why is it not the default?

total_timesteps is 5e7 for Atari experiment in the paper, so exploration_fraction is 0.005 and explore 5e7 * 0.005 = 2.5e5 steps by default. This example trains only 1e7 timesteps, so I set exploration_fraction to 0.025 (1e7 * 0.025 = 2.5e5).

@Miffyli
I mentioned the differences between the evaluation protocols of sb3 and the paper below.
https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/13#issuecomment-742913092
If needed, I'd like to implement an eval callback for atari experiments as in the paper.
(Though, I have no computer resource for that long (about 1 week) evaluation.)

Ah okey, thanks for the pointer :). Good discussions there!

If needed, I'd like to implement an eval callback for atari experiments as in the paper.
(Though, I have no computer resource for that long (about 1 week) evaluation.)

I might be able to pitch in computing resources for this once code is ready (depends on the timing). I'd really like to see as-close-as-possible comparison with original hyperparameters in one or two games, as this integrity is main driving point of the contrib repo ^^.

I might be able to pitch in computing resources for this once code is ready (depends on the timing). I'd really like to see as-close-as-possible comparison with original hyperparameters in one or two games, as this integrity is main driving point of the contrib repo ^^.

Also I agree with your statement, I would say I'm fairly confident we can reproduce the results, especially because QR-DQN and TQC share the most essential part, the huber loss ;) (and because TQC is working very well in my experiments, so even if there is a bug, it works ^^")
I also have results similar to intel coach (even with slightly different hyperparameters) for now.


Feature Request: Add SQS option to SNS event

Description:
SAM offers simplified syntax for SNS -> Lambda and with #261 supports adding subscription filter policies. However, a common use case is SNS -> SQS -> Lambda. SAM offers simplified syntax for SQS -> Lambda, but you have to use standard CloudFormation to setup the SNS -> SQS connection and subscription filter policy, which can be verbose.
SAM could add a lot of value by adding an option to the existing SNS event type to create an SQS queue.
Here's an example that uses simple default settings for the queue:
Type: SNS
Properties:
  Topic: arn:aws:sns:us-east-1:123456789012:my_topic
  # new property that creates an SQS queue between the topic and Lambda function
  # When only the boolean is specified, SAM uses the following defaults:
  # AWS::SQS::Queue logicalId: <function logicalId><SNS event key>Queue
  # AWS::SQS::QueuePolicy logicalId: <function logicalId><SNS event key>QueuePolicy
  # BatchSize: default behavior - don't add to `AWS::Lambda::EventSourceMapping` resource so default value is used.
  # Enabled: default behavior - don't add to `AWS::Lambda::EventSourceMapping` resource so default value is used.
  SqsSubscription: true
  FilterPolicy:
    store:
      - example_corp
    price_usd:
      - numeric:
          - ">="
          - 100

Here's an example that specifies additional options for the queue:
Type: SNS
Properties:
  Topic: arn:aws:sns:us-east-1:123456789012:my_topic
  # new property that creates an SQS queue between the topic and Lambda function
  # Default values can be overridden by passing an object instead of a boolean. Supported keys:
  # QueuePolicyLogicalId: overrides default logicalId naming of the AWS::SQS::QueuePolicy resource.
  # QueueArn: allows the user to specify their own queue instead of having SAM create one on their behalf, allowing them to specify non-default properties on the queue.
  # BatchSize: if specified, add BatchSize property with given value to `AWS::Lambda::EventSourceMapping` resource.
  # Enabled: if specified, add BatchSize property with given value to `AWS::Lambda::EventSourceMapping` resource.
  SqsSubscription:
    QueuePolicyLogicalId: CustomQueuePolicyLogicalId
    QueueArn: !GetAtt MyCustomQueue.Arn
    BatchSize: 5
    Enabled: false
  FilterPolicy:
    store:
      - example_corp
    price_usd:
      - numeric:
          - ">="
          - 100

Implementation Note: When SAM expands this into CloudFormation, the AWS::SQS::QueuePolicy resource should look like this:
  ExampleQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref ExampleQueue
      PolicyDocument:
        Version: '2012-10-17'
        Id: ExampleQueuePolicy
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - sns.amazonaws.com
            Action:
              - sqs:SendMessage
            Resource:
              - !GetAtt ExampleQueue.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: <SNS Topic ARN>

That would be really helpful indeed.

Happy to work with anyone interested in picking this up!

@jlhood I would like to work on this issue.

@Buffer0x7cd Awesome, thank you so much! Before implementing, we need to finalize the SAM spec. The draft I created above just has a simple UseSQS boolean, but we'll want a way to surface some of the options of the SQS event source like BatchSize and Enabled. I'll update the spec in the issue description with exactly what we want to support, then I'll post a comment so you'll know when it's ready to start implementation.

@brettstack @keetonian I've updated the issue description with the proposed SAM spec for this feature. Can you both review?

Do we have precedent for overriding generated logical resource id like QueuePolicyLogicalId? Do we need it?
What's a better name than UseSQS - [Forward|Push|Send]ToSqs? SqsSubscriptions?
Can we allow an array of queues also?

Thanks for the feedback, @brettstack!

Do we have precedent for overriding generated logical resource id like QueuePolicyLogicalId? Do we need it?

This would be something new. Maybe this is a little selfish, but in the Event Fork Pipelines apps we just released today, I'm specifying the Queue and QueuePolicy explicitly. Once this SAM feature is released, I'd love to convert over to using it since it makes the template even easier to read. However, without being able to override the logical id, I don't have a way to use the SAM feature while keeping my apps backwards-compatible. 😞 Allowing overrides of logical ids would allow customers to simplify existing templates as new SAM features are added while maintaining backwards-compatibility. However, I won't fight too hard if you really don't want to add it.

What's a better name than UseSQS

Yeah I'm not loving the key name, but I couldn't think of anything better. None of your suggestions are jumping out to me as obviously better. Some others I can think of: EnableSQS or just SQS. I'm kind of liking just SQS, honestly. SQS: true and SQS: <additional properties> seems concise and readable. What do you think?

Can we allow an array of queues also?

You can already achieve this with multiple Events entries. Accepting a list could make the syntax slightly shorter, but I think it adds complexity since then there'd be two ways of doing this within the same resource.

Given that use case I'm okay with adding the logical resource id name override.
The wording in docs is around "subscribing" an SQS Queue to an SNS Topic, so I'd like to get that word in there to make it clear what it is you achieve by specifying this property. SqsSubscription, SubscribeSqs (variation: replace Sqs with Queue or SqsQueue).
Multiple Event entries wouldn't be as succinct.

I like SqsSubscription

Thanks. Updated it to SqsSubscription in the top-level RFC comment.

@Buffer0x7cd I've updated the top-level issue comment with the finalized spec for this feature, so it should be ready to implement once you're finished with your other PR. Let me know if you have any questions. Thanks!

Closing this issue as v1.15.0 is released

Hi,
Thank you for release this feature!
I implemented only simple boolean SqsSubscription option to SNSEvent at first in PR #1065.
I didn't implement whole of this feature all at once because the code-diff might be huge.
Additional options for specifying the queue by SqsSubscription.QueueArn key has not implemented yet(written in the top-level RFC comment).

Here's an example that specifies additional options for the queue:
...

May I reopen this issue for implementing additional options?

Yes absolutely! Thanks for bringing it to our attention. Reopening..

#1209 brings up a good point regarding this feature. Users often want to configure additional properties on the SQS queue like DLQ and visibility timeout. We should keep that in mind while we implement this feature. We could do by allowing for passthrough SQS properties on the SNS event type, but I think it might be simpler to allow users to reference an existing SQS queue and SAM will setup all the necessary subscriptions and permissions on the given queue.

I'd like the option to pass an existing SQS resource and just let SAM handle the connections between the resources, just like @jlhood mentions above.

Hi, I'm on the way implement the option to pass an existing queue.
Now I'm wondering how to specify the Queue URL of a queue policy.
QueuePolicy resource have to specify both QueueARN and QueueUrl:
    "HogePolicy": {
      "Type": "AWS::SQS::QueuePolicy",
      "Properties": {
        "Queues": [
          "https://sqs.ap-northeast-1.amazonaws.com/<aws_account_id>/<queue_name>"
        ],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Action": "sqs:SendMessage",
              "Resource": "arn:aws:sqs:ap-northeast-1:<aws_account_id>:<queue_name>",
              "Effect": "Allow",
              "Condition": {
                "ArnEquals": {
                  "aws:SourceArn": {
                    "Ref": "TweetTopic"
                  }
                }
              },
              "Principal": "*"
            }
          ]
        }
      }
    }

The finalized spec for this feature written in the top-level issue comment defines that SqsSubscription takes QueueARN but does not take QueueURL as a parameter.
When a QueueARN parameter is string, I can easily convert to QueueURL like this.
But when a QueueARN parameter contains Fn::Ref, Fn::Sub, etc.., it is difficult to convert to QueueURL correctly.
In addition, I think users often want to specify a queue using Fn::Ref.
Questions

Is there any good way to specify the Queue URL of a queue policy?
How about adding QueueUrl parameter into SqsSubscription?

I think this is a good way
This way also make it possible to specify a SQS fips endpoint.

@53ningen Thanks so much for working on this! I agree with your proposal to add a QueueUrl parameter to SqsSubscription.

I added a QueueUrl parameter to SqsSubscription:  https://github.com/awslabs/serverless-application-model/pull/1231
pull request is now ready for review.


Change license to Apache-2.0

Changed with https://github.com/nozel-org/serverbot/commit/cf13c6e24ab26d38cecca2837ef4cd7dbb60c0c1.


Update _index.md

in this commit, I have added Beginner-friendly issues and how to solve them.

@0904-mansi Most of these tutorials are PWD based and we don't need sudo fo such environments.

@0904-mansi Most of these tutorials are PWD based and we don't need sudo fo such environments.

ok sir ☹


[test][scalability] Retry performance test on failure

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?

/test pull-kueue-test-scheduling-perf-main

/test pull-kueue-test-scheduling-perf-main

/test pull-kueue-test-scheduling-perf-main