After NuGet update, XBeeNode.SampleReceived handler doesn't get called

Hi Jeff,
I just used NuGet to update my project and my handlers for XBeeNode.SampleReceived are not getting called. Note: Controller.SampleReceived still works.
I was using VS2015 Community, so I installed VS2017 Community and restored my project from backup. Ran my original code in VS2017 and it still works. Then, from VS2017, I used NuGet to update my project again. Same problem. I was hoping you could look at the following and give me an idea of what is wrong.

My original code that works:
package id="BinarySerializer" version="6.0.3" targetFramework="net452"
package id="XBee" version="4.2-beta" targetFramework="net452"
// .Net 4.5.2
controller = new XBeeController( port, 115200 );
await controller.OpenAsync();
controller.SampleReceived += Controller_SampleReceived;
private void Controller_SampleReceived( object sender, SourcedSampleReceivedEventArgs e ) {
}
xBeeNode.SampleReceived += DayNightDetector_SampleReceived;
private void DayNightDetector_SampleReceived( object sender, SampleReceivedEventArgs e ) {
}

Updated from NuGet
package id="BinarySerializer" version="7.5.6" targetFramework="net461"
package id="XBee" version="5.3.3" targetFramework="net461"
package id="XBee.Core" version="1.4.4" targetFramework="net461"
// .Net 4.6.1
controller = new XBeeController();
await controller.OpenAsync( port, 115200 );
controller.SampleReceived += Controller_SampleReceived;
**** This handler still gets called
private void Controller_SampleReceived( object sender, SourcedSampleReceivedEventArgs e ) {
}
xBeeNode.SampleReceived += DayNightDetector_SampleReceived;
****** This handler never gets called now
private void DayNightDetector_SampleReceived( object sender, SampleReceivedEventArgs e ) {
}
I'm lost. Hope you can show me the path.
Thanks, Paul

Yep, this was a regression.  I just pushed it in 1.5-beta, which is beta because of some other stuff I'm testing.  Once I'm finished with that I'll push 1.5.  Thanks.

Just noticed you pushed an update today. I tried nugget again (after installing the latest update to VS17 Community).
XBeeNode.SampleReceived handlers still not being called. Is this to be expected at this point?
package id="BinarySerializer" version="7.5.6" targetFramework="net461"
package id="XBee" version="5.3.3" targetFramework="net461"
package id="XBee.Core" version="1.5.0" targetFramework="net461"
Thanks, Paul

Hm, I thought I fixed that but let me check on it tomorrow. Maybe I left it out

From: PaulNoto<EMAIL_ADDRESS>Sent: Saturday, August 19, 2017 11:35:03 PM
To: jefffhaynes/XBee
Cc: Jeff Haynes; Comment
Subject: Re: [jefffhaynes/XBee] After NuGet update, XBeeNode.SampleReceived handler doesn't get called (#33)
Just noticed you pushed an update today. I tried nugget again (after installing the latest update to VS17 Community).
XBeeNode.SampleReceived handlers still not being called. Is this to be expected at this point?
package id="BinarySerializer" version="7.5.6" targetFramework="net461"
package id="XBee" version="5.3.3" targetFramework="net461"
package id="XBee.Core" version="1.5.0" targetFramework="net461"
Thanks, Paul
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/jefffhaynes/XBee/issues/33#issuecomment-323561324, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSKR_k7sntIjx5UnlrsWuaYejQN9fkiks5sZ6lngaJpZM4O1fv-.

Thanks Jeff. No rush. My code still runs fine with the old software.

Hm, this seems to work for me using 1.5.  Can you tell me a little more about your node configuration?  Are you using 16 or 64-bit addressing?  The way it's supposed to work is the controller receives all events and dispatches them to any nodes the framework knows about.  The nodes are then responsible for comparing the dispatched event address to their own and deciding if it's an event that originated with that node or not.  The bug was in the address comparison so if something is still wrong maybe there's another issue there.

Just checked my email. Ill look into how I'm actually creating a node. I'll probably end up making a test project to simplify what I'm doing. I'll let you know how that goes.

OK, I am still not sure why your old code works and the latest doesn't. My best guess is that somehow the way I create an XBeeNode is the problem. When I look at the Node with the debugger the Sample Received Handler is set OLD and New. I created a Stripped down Test Project with the NEW code and the debugger shows the Node's SampleReceived handler is set, however, it never calls the handler.
WOW! NuGet really bulks up a .Net 4.6 / 4.7 project with extra Microsoft stuff. The test project is almost 300MB. I wanted to send it to you (I'm not sure how to do that since it's 300MB) so you can see how my Reactive system of initializing the XBee nodes works. Remember, all my nodes send a sample every 1 minute, even the sleeping ones. I will include just the  code from my MainWindow.xaml.cs file, so you can see how my code is working.

Results from using the Debugger to inspect nodes created using your old library and the latest library from NuGet. I can't spot a problem. The handler is set Old and New.
My Test XBee Address
5526146530998065 ULong
13A20040F4F731   Hex String
OLD XBEE LIBRARY
xBeeNode = await controller.GetNodeAsync( new NodeAddress( new LongAddress( ulong.Parse( xli.xBeeAddressString, System.Globalization.NumberStyles.HexNumber ) ) ) );
Address = {0013A20040F4F731, FFFE}
Is16BitDisabled = true
LongAddress = {0013A20040F4F731}
High = 1286656
IsBroadcast = false
IsCoordinator = false
Low =<PHONE_NUMBER>
Value =<PHONE_NUMBER>998065
ShortAddress = {FFFE}
IsBroadcast = false
IsDisabled = true
Value = 65534
Controller = {XBee.XBeeController}
DataReceived = null
HardwareVersion = XBee24C
SampleReceived = {Method = {Void DayNightDetector_SampleReceived(System.Object, XBee.SampleReceivedEventArgs)}}
SensorSampleReceived = null
NEW XBEE LIBRARY
xBeeNode = await controller.GetNodeAsync( new NodeAddress( new LongAddress( ulong.Parse( xli.xBeeAddressString, System.Globalization.NumberStyles.HexNumber ) ) ) );
Address = {0013A20040F4F731, FFFE}
LongAddress = {0013A20040F4F731}
High = 1286656
IsBroadcast = false
IsCoordinator = false
IsDisabled = false
Low =<PHONE_NUMBER>
Value =<PHONE_NUMBER>998065
ShortAddress = {FFFE}
IsBroadcast = false
IsCoordinator = false
IsDisabled = true
Value = 65534
Controller = {XBee.XBeeController}
DataReceived = null
FirmwareVersion = 16479
HardwareVersion = XBee24C
Protocol = ZigBee
SampleReceived = {Method = {Void DayNightDetector_SampleReceived(System.Object, XBee.SampleReceivedEventArgs)}}
SensorSampleReceived = null
_maxPayloadLength = null

Code from my test projects MainWindow.xaml.cs file:
using System;
using System.Collections.Generic;
using System.IO.Ports;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using XBee;
namespace WpfApp1 {
///
/// Interaction logic for MainWindow.xaml
///
public partial class MainWindow : Window {
	private XBeeController controller = null;
	private XBeeNode _xBeeNode = null;

	public MainWindow() {
		InitializeComponent();
		Await_DiscoverController();
	}

	private async void Await_DiscoverController() {
		await DiscoverController();
	}

	//---------------------------------------------------------------------------

	private async Task DiscoverController() {

		// NOTE: Sometimes the controller will take several tries before it is discovered

		var ports = SerialPort.GetPortNames();
		if ( ports.Length == 0 ) {
			//historyLog.Add( new HistoryLog_Item( false, "Can't find XBee Controller on any USB Serial Port", HistoryLog_Item_Types.error ) );
			return;
		}
		//historyLog.Add( new HistoryLog_Item( false, "Initializing XBee Controller", HistoryLog_Item_Types.info ) );
		//historyLog.Add( new HistoryLog_Item( false, string.Format( "Checking serial ports: {0}", string.Join( ", ", ports ) ), HistoryLog_Item_Types.info ) );

		int pass = 1;
		do {
			foreach ( var port in ports ) {
				try {
					controller = new XBeeController();
					await controller.OpenAsync( port, 115200 );
					//historyLog.Add( new HistoryLog_Item( false, string.Format( "Opened {0}", port ), HistoryLog_Item_Types.info ) );
					controller.SampleReceived += Controller_SampleReceived;
					pass = 0; // Finished initializing controller
				}
				catch ( Exception e ) {
					controller?.Dispose();
					controller = null;
					//historyLog.Add( new HistoryLog_Item( false, string.Format( "Pass {0}: Failed to open {1} with {2}", pass, port, e.Message ), HistoryLog_Item_Types.error ) );
					pass++;
				}
				if ( pass >= 30 ) {
					//historyLog.Add( new HistoryLog_Item( false, string.Format( "Can't initialize XBee Controller on USB Serial Port {0}", port ), HistoryLog_Item_Types.error ) );
					return;
				}
			}
		} while ( pass != 0 );

		//historyLog.Add( new HistoryLog_Item( false, "Finished Initializing XBee Controller", HistoryLog_Item_Types.info ) );
	}

	//---------------------------------------------------------------------------

	public async void GetXBeeAsync( string hexAddressString ) {
		try {
			// I think somehow this is the problem.
			_xBeeNode = await controller.GetNodeAsync( new NodeAddress( new LongAddress( ulong.Parse( hexAddressString, System.Globalization.NumberStyles.HexNumber ) ) ) );
		}
		catch {
			return;
		}
	}

	//---------------------------------------------------------------------------

	ulong _test_XBeeNodeAddress_ULong =<PHONE_NUMBER>998065;
	string _test_XBeeNodeAddress_HexString = "13A20040F4F731";
	bool _sampleHandlerSet = false;
	/// <summary>
	/// This is the incomming Method of the Controller XBee, every incomming communication hits this before the data is sent to the actual XBee Node object (if one exists).
	/// </summary>
	/// <param name="sender"></param>
	/// <param name="e"></param>
	private void Controller_SampleReceived( object sender, SourcedSampleReceivedEventArgs e ) {
		ulong nodeAddressULong = e.Address.LongAddress.Value;

		if ( nodeAddressULong == _test_XBeeNodeAddress_ULong ) {

			// Pass #1 Create Node
			if ( _xBeeNode == null ) {
				GetXBeeAsync( _test_XBeeNodeAddress_HexString );
				return;
			}
			// Pass #2 Set Handler
			if ( !_sampleHandlerSet ) {
				_xBeeNode.SampleReceived += Node_SampleReceived;
				_sampleHandlerSet = true;
				return;
			} else {
				// Pass #3 For some reason the debugger won't show me the method, the Node's SampleReceived handler is set to, when I inspect the node on the third pass
				// Pass #4 or more will let me inspect what method the Node's SampleReceived handler is set to. WEIRD!
				XBeeNode node = _xBeeNode;
				return; // Set Break Point Here to inspect fully initialized Node
			}
		}
	}// END Controller_SampleReceived()

	//---------------------------------------------------------------------------
	private int _sampleReceived_Cntr = 0;
	private void Node_SampleReceived( object sender, SampleReceivedEventArgs e ) {
		_sampleReceived_Cntr++; // This Break Point never happens
	}

}

}

Hi Jeff,
I've been looking into why using NuGet to update my project causes my project to get bulked up to 300MB on disk. It is being caused by using NuGet to add XBee.Core to the App.
I'm Using VS2017 Community and I create a basic WPF Blank Window Project.
My base  WPF project is 336 KB on disk
My base WPF project after XBee.Core is 301 MB on disk
Do you know why NuGet is adding all the Updates, that are included below, as part of adding XBee.Core?
 Still planning on looking into this I'm just up to my eyeballs.

From: PaulNoto<EMAIL_ADDRESS>Sent: Tuesday, August 22, 2017 12:23:35 PM
To: jefffhaynes/XBee
Cc: Jeff Haynes; Comment
Subject: Re: [jefffhaynes/XBee] After NuGet update, XBeeNode.SampleReceived handler doesn't get called (#33)
Hi Jeff,
I've been looking into why using NuGet to update my project causes my project to get bulked up to 300MB on disk. It is being caused by using NuGet to add XBee.Core to the App.
I'm Using VS2017 Community and I create a basic WPF Blank Window Project.
My base WPF project is 336 KB on disk
My base WPF project after XBee.Core is 301 MB on disk
Do you know why NuGet is adding all the Updates, that are included below, as part of adding XBee.Core?

Reply to this email directly, view it on GitHubhttps://github.com/jefffhaynes/XBee/issues/33#issuecomment-324079125, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSKR72KIuvqzCkOagLz3s9vR4h2KGwnks5sawCHgaJpZM4O1fv-.

I understand.
Thanks, Paul

FYI, I just installed VS2017 update 15.3.3 and NuGet still adds all that stuff when addiding XBee.Core and bulks the disk footprint up to about 300MB. I feel like I am missing part of the .NET framework Library, because, what is the point of having a library if NuGet is going to add the individual library components to my project?
Anyway, No rush on this. I just keep using your old code and my project runs perfect. I just want to be able to keep up with changes to your library using NuGet. The real show stopper is that Node Sample Received Handlers stop working when I update to your latest release. The 300MB footprint is just a hassle for doing backups and remote deployment over the internet.

Ok, so starting with the sample issue.  The only issue I see is that you are explicitly comparing 64-bit addresses.  It would be better to compare the NodeAddress directly, as this will handle things like 16 vs 64 bit addressing.  In your example, if you are receiving samples with 16-bit addresses, this check will never succeed.  However, assuming your nodes are operating in 64-bit mode then this should work.  I used your code exactly, simply swapping out the test address for one of my devices and everything seems to work.  Once the node is setup, the Node_SampleRecieved breakpoint is hit.
It does sound like something weird is going on with your dev environment.  Maybe try uninstalling and re-installing .net core?  I have run into issues in the past where older versions of .net core were clobbering newer versions.

I'm focusing on the XBee.Core issue first. I did as you suggested and removed .net core and reinstalled it, which also gave me version 2.0. It did not help. I have VS2017 15.3.3 installed on 2 other computers. 1 is WIN 7 the other is Win 10. Both do the same thing. NuGet wants to add all those packages to the project. I'm at a loss. I've looked at everything I can think of, and don't see anything wrong.
All I'm doing is creating a Visual C#; Windows Classic Desktop; .Net Framework 4.6.1; WPF App (.Net Framework) Project. Then I go to Tools; NuGet Package Manager; Manage NuGet Packages for Solution. Then I Browse for xbee, and select XBee.Core. Then I try to install it and it comes up with a window listing all the other packages it is going to install.
I was hoping, when you have time, that you could try this procedure and let me know what happens.
Thanks, Paul.

FYI: I was trying different experiments and discovered that if I use the following process of adding things with NuGet I can reduce the disk footprint from 300MB to 190MB and have far fewer packages added to the project.
So I create a WPF App. Then I use NuGet to add NETStandard.Library, which also adds Microsoft.NETCore.Platforms. Then I add XBee.Core. Then I add XBee.
Now my disk footprint is about 190MB and my packages are:
package id="BinarySerializer" version="7.5.7" targetFramework="net461" />
package id="Microsoft.NETCore.Platforms" version="2.0.0" targetFramework="net461" />
package id="NETStandard.Library" version="2.0.0" targetFramework="net461" />
package id="System.Collections" version="4.3.0" targetFramework="net461" />
package id="System.IO" version="4.3.0" targetFramework="net461" />
package id="System.Linq" version="4.3.0" targetFramework="net461" />
package id="System.Linq.Expressions" version="4.3.0" targetFramework="net461" />
package id="System.Reflection" version="4.3.0" targetFramework="net461" />
package id="System.Reflection.Extensions" version="4.3.0" targetFramework="net461" />
package id="System.Reflection.TypeExtensions" version="4.4.0" targetFramework="net461" />
package id="System.Resources.ResourceManager" version="4.3.0" targetFramework="net461" />
package id="System.Runtime" version="4.3.0" targetFramework="net461" />
package id="System.Runtime.Extensions" version="4.3.0" targetFramework="net461" />
package id="System.Text.Encoding" version="4.3.0" targetFramework="net461" />
package id="System.Threading" version="4.3.0" targetFramework="net461" />
package id="XBee" version="5.3.3" targetFramework="net461" />
package id="XBee.Core" version="1.5.0" targetFramework="net461" />
I still think something is wrong with the configuration, because this seems like a very complex process and I think it is still adding too many extra packages to the project. Hopefully, tonight I will have time to look at the Node_SampleReceived issue.

Yeah, this may be a side-effect that I don't totally understand of targeting both classic and .net standard in one package (it took me a long time just to get that to work).  However, you do understand you don't have to deploy the packages directory, correct?  The actually build is only around 1.5MB.

Started working the Node_SampleReceived issue.
Something has changed, because my simplified test code that I sent to you, started throwing the following Exception when it is initializing the controller:
"There was an error reflecting type 'XBee.Frame'"
So I simplified it even more and this exception happens on my development machine and on my production machine.
Using VS2017 Community 15.3.3 , creating a C# , .Net Framework 4.7 , WPF App
Note: I initially was having same problem with 4.61 and switched to for 4.7, but still had problem.

using System;
using System.Windows;
using XBee;
namespace WpfApp2 {
///
/// Interaction logic for MainWindow.xaml
///
public partial class MainWindow : Window {
private XBeeController controller = null;
	public MainWindow() {
		InitializeComponent();
		controller = new XBeeController();
		Await_DiscoverController();
	}

	private async void Await_DiscoverController() {
		try {
			await controller.OpenAsync( "COM3", 115200 );
		}
		catch ( Exception e ) {
			// Catches Exception: "There was an error reflecting type 'XBee.Frame'"
			string s = e.Message;
		}
	}
}

}

Note: I started with my complicated process of using NuGet on the 4.61 App, so on the 4.7 App I used NuGet to just add XBee and let it add the zillion packages. The problem persists no mater what I do.
  Running the same code works for me so maybe there is a difference in how your device is configured that I'm not accounting for.

I think it would be easier to let you RemoteDesktop into the ASROCK BeeBox that I'm using as my Dev System.
So if you would like to do that, let me know.
Here is what I found on the InnerExceptions.
Looks like the InnerException chain ends in the BinarySerializer with:
InnerException   {"XBee.Frames.ModemStatusData.StackStatusModemStatusData is not a subtype of XBee.Frames.ModemStatusData.ModemStatusExtData"}   System.Exception {System.InvalidOperationException}
CHAIN of InnerExceptions:
InnerException   {"Exception has been thrown by the target of an invocation."}   System.Exception {System.Reflection.TargetInvocationException}
Source   "BinarySerializer"   string
Data   {System.Collections.ListDictionaryInternal}   System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException   {"There was an error reflecting member 'Payload'"}   System.Exception {System.InvalidOperationException}
Source   "mscorlib"  string
Data   {System.Collections.ListDictionaryInternal}   System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException   {"There was an error reflecting member 'Content'"}   System.Exception {System.InvalidOperationException}
Source   "BinarySerializer"   string
Data   {System.Collections.ListDictionaryInternal}   System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException   {"Exception has been thrown by the target of an invocation."}   System.Exception {System.Reflection.TargetInvocationException}
Data   {System.Collections.ListDictionaryInternal}   System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
Source   "mscorlib"   string
InnerException   {"There was an error reflecting member 'StatusData'"}   System.Exception {System.InvalidOperationException}
Source   "BinarySerializer"   string
Data   {System.Collections.ListDictionaryInternal}   System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException   {"Exception has been thrown by the target of an invocation."}   System.Exception {System.Reflection.TargetInvocationException}
Source   "mscorlib"   string
Data   {System.Collections.ListDictionaryInternal}   System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException   {"XBee.Frames.ModemStatusData.StackStatusModemStatusData is not a subtype of XBee.Frames.ModemStatusData.ModemStatusExtData"}   System.Exception {System.InvalidOperationException}
Source   "BinarySerializer"   string
Data   {System.Collections.ListDictionaryInternal}   System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException   null   System.Exception
STACK TRACE:
at BinarySerialization.Graph.TypeGraph.ContainerTypeNode.GenerateChild(Type type)
at BinarySerialization.Graph.GraphGenerator.<>c.b__1_0(Type type)
at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
at BinarySerialization.Graph.GraphGenerator.GenerateGraph(Type valueType)
at BinarySerialization.BinarySerializer.Serialize(Stream stream, Object value, Object context)
at XBee.Core.XBeeControllerBase.d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__751.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at XBee.Core.XBeeControllerBase.<ExecuteAtQueryAsync>d__791.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__72.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__83.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__751.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at XBee.Core.XBeeControllerBase.<ExecuteAtQueryAsync>d__791.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__72.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at WpfApp2.MainWindow.<Await_DiscoverController>d__2.MoveNext() in C:_Test\WpfApp2\WpfApp2\MainWindow.xaml.cs:line 21

I can potentially do that but also it looks like you're getting modem status information, which the library doesn't really support right now. I'll take a look and see if I can fix it or at least stop it from breaking.

From: PaulNoto<EMAIL_ADDRESS>Sent: Sunday, September 10, 2017 9:28:36 AM
To: jefffhaynes/XBee
Cc: Jeff Haynes; Comment
Subject: Re: [jefffhaynes/XBee] After NuGet update, XBeeNode.SampleReceived handler doesn't get called (#33)
I think it would be easier to let you RemoteDesktop into the ASROCK BeeBox that I'm using as my Dev System.
So if you would like to do that, let me know.
Here is what I found on the InnerExceptions.
Looks like the InnerException chain ends in the BinarySerializer with:
InnerException {"XBee.Frames.ModemStatusData.StackStatusModemStatusData is not a subtype of XBee.Frames.ModemStatusData.ModemStatusExtData"} System.Exception {System.InvalidOperationException}
CHAIN of InnerExceptions:
InnerException {"Exception has been thrown by the target of an invocation."} System.Exception {System.Reflection.TargetInvocationException}
Source "BinarySerializer" string
Data {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException {"There was an error reflecting member 'Payload'"} System.Exception {System.InvalidOperationException}
Source "mscorlib" string
Data {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException {"There was an error reflecting member 'Content'"} System.Exception {System.InvalidOperationException}
Source "BinarySerializer" string
Data {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException {"Exception has been thrown by the target of an invocation."} System.Exception {System.Reflection.TargetInvocationException}
Data {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
Source "mscorlib" string
InnerException {"There was an error reflecting member 'StatusData'"} System.Exception {System.InvalidOperationException}
Source "BinarySerializer" string
Data {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException {"Exception has been thrown by the target of an invocation."} System.Exception {System.Reflection.TargetInvocationException}
Source "mscorlib" string
Data {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException {"XBee.Frames.ModemStatusData.StackStatusModemStatusData is not a subtype of XBee.Frames.ModemStatusData.ModemStatusExtData"} System.Exception {System.InvalidOperationException}
Source "BinarySerializer" string
Data {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
InnerException null System.Exception
STACK TRACE:
at BinarySerialization.Graph.TypeGraph.ContainerTypeNode.GenerateChild(Type type)
at BinarySerialization.Graph.GraphGenerator.<>c.b__1_0(Type type)
at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
at BinarySerialization.Graph.GraphGenerator.GenerateGraph(Type valueType)
at BinarySerialization.BinarySerializer.Serialize(Stream stream, Object value, Object context)
at XBee.Core.XBeeControllerBase.d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__751.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at XBee.Core.XBeeControllerBase.d__791.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__72.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__83.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__751.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at XBee.Core.XBeeControllerBase.d__791.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at XBee.Core.XBeeControllerBase.d__72.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at WpfApp2.MainWindow.<Await_DiscoverController>d__2.MoveNext() in C:_Test\WpfApp2\WpfApp2\MainWindow.xaml.cs:line 21
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/jefffhaynes/XBee/issues/33#issuecomment-328342715, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSKR0Z-msfGrmRLvxC1H7owL5IdUM34ks5sg-QEgaJpZM4O1fv-.

I fixed at least the immediate problem you're having in 1.5.1 although I'm not totally sure as to why you would even be receiving that status data.  Give it a try but if this is a red herring or something I can try remoting and taking a closer look.

OK, I'll give it a try and let you know.
Thanks, Paul

Same problem.
{"XBee.Frames.ModemStatusData.JoiningModemStatusData is not a subtype of XBee.Frames.ModemStatusData.ModemStatusExtData"}
Looks like NuGet has all your latest packages installed.   Somehow you're getting a large number of diagnostic frames back.  Sorry, try 1.5.2.

Will do.

OK, The controller is working.
I also updated my simple code to check  Node_SampleReceived. That handler is still not getting called.
I need to check it some more and re-verify that my code is setting the handler and that I think it should be getting called. I'll let you know when I finish.
What really perplexes me is why everything works fine on your system, and I have two different systems that have the same problem. To me, it seems like somehow I have chosen a weird configuration for visual studio and the libraries that my project references.
Anyway, thank you for getting me past the first hurdle. I'll be in touch about the  Node_SampleReceived issue in a little while.

Well I think it could easily be the difference between how our nodes are configured. Unfortunately it's trivial to receive the packet but actually complicated to figure out where it came from in all cases.

From: PaulNoto<EMAIL_ADDRESS>Sent: Sunday, September 10, 2017 12:59:11 PM
To: jefffhaynes/XBee
Cc: Jeff Haynes; Comment
Subject: Re: [jefffhaynes/XBee] After NuGet update, XBeeNode.SampleReceived handler doesn't get called (#33)
OK, The controller is working.
I also updated my simple code to check Node_SampleReceived. That handler is still not getting called.
I need to check it some more and re-verify that my code is setting the handler and that I think it should be getting called. I'll let you know when I finish.
What really perplexes me is why everything works fine on your system, and I have two different systems that have the same problem. To me, it seems like somehow I have chosen a weird configuration for visual studio and the libraries that my project references.
Anyway, thank you for getting me past the first hurdle. I'll be in touch about the Node_SampleReceived issue in a little while.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/jefffhaynes/XBee/issues/33#issuecomment-328356090, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSKR8a8GrIaqSrtHFqSxRLYRow8h37Rks5shBVfgaJpZM4O1fv-.

Continued in #36


Remediation behaviour different (and failing) outside of the SecurityHub aggregation region

Describe the bug
I have deployed the sharr solution to an AWS Organization (under a Control Tower landing zone). The CT home region is us-east-1 and that is where SecurityHub has been set up to aggregate findings from all enabled regions (4 of them, including us-east-1) for all accounts in the org.
While both the manually executed remediations in SecurityHub and the EventBridge rules (once I enabled one of them) work correctly in the us-east-1 region, this behaviour is not replicated to the other 3 regions. In those regions, manually evaluating a non-compliant config rule did not have any effect, nor did the automed config evaluations kick off the workflow (although it did kick off the remediation in us-east-1).
This error was presented in the StepFunction log after the Get Automation Document State step failed:
An error occurred while executing the state 'Get Automation Document State' (entered at the event id #14). The JSONPath '$.Payload.remediationrole' specified for the field 'RemediationRole.$' could not be found in the input

Detailed Analysis
The log message did not provide a useful description of what the problem was, but I turned on debug logging in the Get Automation Document State step which revealed that the lambda failed when looking for an SSM parameter called /Solutions/SO0111/aws-foundational-security-best-practices/1.0.0/status. That parameter does not exist, causing the lambda to exit suggesting that the security standard is not enabled.
In us-east-1, the payload to the step function does not include a ProductFields[StandardsControlArn] entry and therefore the __init__ method of the sechub_findings.py code (lines 85-97) falls though to the default values of security-control and 2.0.0. This means that the _set_standard_version_supported method looks for an SSM parameter called /Solutions/SO0111/security-control/2.0.0/status which does exist, so the step function can continue with its remediation.
It is unclear why the inputs to the step function in different regions are different, but my understanding is that these come from AWS Config itself as the EventBridge rule targets suggest that the inputs are passed through in this way.
The failed JSON payload (input to the step function) looks like this, noting that I have replaced the account numbers, bucket names and account names:
{
    "version": "0",
    "id": "ed036737-8415-89d6-fb0a-487c29565f2f",
    "detail-type": "Security Hub Findings - Custom Action",
    "source": "aws.securityhub",
    "account": "111111111111",
    "time": "2024-10-17T20:47:06Z",
    "region": "us-east-1",
    "resources": [
      "arn:aws:securityhub:us-east-1:111111111111:action/custom/ASRRemediation"
    ],
    "detail": {
      "actionName": "Remediate with ASR",
      "actionDescription": "Submit the finding to AWS Security Hub Automated Response and Remediation",
      "findings": [
        {
          "ProductArn": "arn:aws:securityhub:us-west-1::product/aws/securityhub",
          "Types": [
            "Software and Configuration Checks/Industry and Regulatory Standards/AWS-Foundational-Security-Best-Practices"
          ],
          "Description": "This control checks whether server access logging is enabled for an Amazon S3 general purpose bucket. The control fails if server access logging isn't enabled. When logging is enabled, Amazon S3 delivers access logs for a source bucket to a chosen target bucket. The target bucket must be in the same AWS Region as the source bucket and must not have a default retention period configured. The target logging bucket does not need to have server access logging enabled, and you should suppress findings for this bucket.",
          "Compliance": {
            "Status": "FAILED",
            "SecurityControlId": "S3.9",
            "AssociatedStandards": [
              {
                "StandardsId": "standards/aws-foundational-security-best-practices/v/1.0.0"
              }
            ]
          },
          "ProductName": "Security Hub",
          "FirstObservedAt": "2024-06-11T04:09:18.465Z",
          "CreatedAt": "2024-06-11T04:09:18.465Z",
          "LastObservedAt": "2024-10-17T19:25:01.298Z",
          "CompanyName": "AWS",
          "FindingProviderFields": {
            "Types": [
              "Software and Configuration Checks/Industry and Regulatory Standards/AWS-Foundational-Security-Best-Practices"
            ],
            "Severity": {
              "Normalized": 40,
              "Label": "MEDIUM",
              "Product": 40,
              "Original": "MEDIUM"
            }
          },
          "ProductFields": {
            "StandardsArn": "arn:aws:securityhub:::standards/aws-foundational-security-best-practices/v/1.0.0",
            "StandardsSubscriptionArn": "arn:aws:securityhub:us-west-1:222222222222:subscription/aws-foundational-security-best-practices/v/1.0.0",
            "ControlId": "S3.9",
            "RecommendationUrl": "https://docs.aws.amazon.com/console/securityhub/S3.9/remediation",
            "RelatedAWSResources:0/name": "securityhub-s3-bucket-logging-enabled-7946cfb0",
            "RelatedAWSResources:0/type": "AWS::Config::ConfigRule",
            "StandardsControlArn": "arn:aws:securityhub:us-west-1:222222222222:control/aws-foundational-security-best-practices/v/1.0.0/S3.9",
            "aws/securityhub/ProductName": "Security Hub",
            "aws/securityhub/CompanyName": "AWS",
            "Resources:0/Id": "arn:aws:s3:::my-bucket-us-west-1",
            "aws/securityhub/FindingId": "arn:aws:securityhub:us-west-1::product/aws/securityhub/arn:aws:securityhub:us-west-1:222222222222:subscription/aws-foundational-security-best-practices/v/1.0.0/S3.9/finding/d42abbc7-0625-4766-abf4-3b7fbfe5f49c"
          },
          "Remediation": {
            "Recommendation": {
              "Text": "For information on how to correct this issue, consult the AWS Security Hub controls documentation.",
              "Url": "https://docs.aws.amazon.com/console/securityhub/S3.9/remediation"
            }
          },
          "SchemaVersion": "2018-10-08",
          "GeneratorId": "aws-foundational-security-best-practices/v/1.0.0/S3.9",
          "RecordState": "ACTIVE",
          "Title": "S3.9 S3 general purpose buckets should have server access logging enabled",
          "Workflow": {
            "Status": "NEW"
          },
          "Severity": {
            "Normalized": 40,
            "Label": "MEDIUM",
            "Product": 40,
            "Original": "MEDIUM"
          },
          "UpdatedAt": "2024-10-17T19:24:56.786Z",
          "WorkflowState": "NEW",
          "AwsAccountName": "my-account",
          "AwsAccountId": "222222222222",
          "Region": "us-west-1",
          "Id": "arn:aws:securityhub:us-west-1:222222222222:subscription/aws-foundational-security-best-practices/v/1.0.0/S3.9/finding/d42abbc7-0625-4766-abf4-3b7fbfe5f49c",
          "Resources": [
            {
              "Partition": "aws",
              "Type": "AwsS3Bucket",
              "Details": {
                "AwsS3Bucket": {
                  "OwnerId": "f08f0226c40f258c659b515e87ceb6f81ab581e6dec3dfa32dd1b5f461ae5efa",
                  "CreatedAt": "2024-06-11T04:07:44.000Z",
                  "Name": "my-bucket-us-west-1"
                }
              },
              "Region": "us-west-1",
              "Id": "arn:aws:s3:::my-bucket-us-west-1"
            }
          ],
          "ProcessedAt": "2024-10-17T19:25:04.534Z"
        }
      ]
    }
  }

The successfully executed step function in us-east-1 has the following input. Note again that I have replaced the account names, numbers and the bucket names, however the 2 remediation attempts (in us-west-1 above, and us-east-1 below) are for buckets in the same account, just different regions.
Notice how the ProductFields contains fewer elements than the version in us-west-1. Particularly, the StandardsControlArn field is missing, leading the code to fall though to default values, as I described above.
{
    "version": "0",
    "id": "068fbf03-9dc3-b99f-317f-a3dcb643f208",
    "detail-type": "Security Hub Findings - Custom Action",
    "source": "aws.securityhub",
    "account": "111111111111",
    "time": "2024-10-17T20:47:06Z",
    "region": "us-east-1",
    "resources": [
      "arn:aws:securityhub:us-east-1:111111111111:action/custom/ASRRemediation"
    ],
    "detail": {
      "actionName": "Remediate with ASR",
      "actionDescription": "Submit the finding to AWS Security Hub Automated Response and Remediation",
      "findings": [
        {
          "ProductArn": "arn:aws:securityhub:us-east-1::product/aws/securityhub",
          "Types": [
            "Software and Configuration Checks/Industry and Regulatory Standards"
          ],
          "Description": "This control checks whether server access logging is enabled for an Amazon S3 general purpose bucket. The control fails if server access logging isn't enabled. When logging is enabled, Amazon S3 delivers access logs for a source bucket to a chosen target bucket. The target bucket must be in the same AWS Region as the source bucket and must not have a default retention period configured. The target logging bucket does not need to have server access logging enabled, and you should suppress findings for this bucket.",
          "Compliance": {
            "Status": "FAILED",
            "SecurityControlId": "S3.9",
            "AssociatedStandards": [
              {
                "StandardsId": "standards/aws-foundational-security-best-practices/v/1.0.0"
              }
            ]
          },
          "ProductName": "Security Hub",
          "FirstObservedAt": "2024-05-21T02:17:27.639Z",
          "CreatedAt": "2024-05-21T02:17:27.639Z",
          "LastObservedAt": "2024-10-17T14:23:28.025Z",
          "CompanyName": "AWS",
          "FindingProviderFields": {
            "Types": [
              "Software and Configuration Checks/Industry and Regulatory Standards"
            ],
            "Severity": {
              "Normalized": 40,
              "Label": "MEDIUM",
              "Original": "MEDIUM"
            }
          },
          "ProductFields": {
            "RelatedAWSResources:0/name": "securityhub-s3-bucket-logging-enabled-97d7d3e4",
            "RelatedAWSResources:0/type": "AWS::Config::ConfigRule",
            "aws/securityhub/ProductName": "Security Hub",
            "aws/securityhub/CompanyName": "AWS",
            "Resources:0/Id": "arn:aws:s3:::my-bucket-us-east-1",
            "aws/securityhub/FindingId": "arn:aws:securityhub:us-east-1::product/aws/securityhub/arn:aws:securityhub:us-east-1:222222222222:security-control/S3.9/finding/8b1bb0fe-70dc-409b-9477-2b2f986092af"
          },
          "Remediation": {
            "Recommendation": {
              "Text": "For information on how to correct this issue, consult the AWS Security Hub controls documentation.",
              "Url": "https://docs.aws.amazon.com/console/securityhub/S3.9/remediation"
            }
          },
          "SchemaVersion": "2018-10-08",
          "GeneratorId": "security-control/S3.9",
          "RecordState": "ACTIVE",
          "Title": "S3 general purpose buckets should have server access logging enabled",
          "Workflow": {
            "Status": "NEW"
          },
          "Severity": {
            "Normalized": 40,
            "Label": "MEDIUM",
            "Original": "MEDIUM"
          },
          "UpdatedAt": "2024-10-17T14:23:14.029Z",
          "WorkflowState": "NEW",
          "AwsAccountName": "my-account",
          "AwsAccountId": "222222222222",
          "Region": "us-east-1",
          "Id": "arn:aws:securityhub:us-east-1:222222222222:security-control/S3.9/finding/8b1bb0fe-70dc-409b-9477-2b2f986092af",
          "Resources": [
            {
              "Partition": "aws",
              "Type": "AwsS3Bucket",
              "Details": {
                "AwsS3Bucket": {
                  "OwnerId": "f08f0226c40f258c659b515e87ceb6f81ab581e6dec3dfa32dd1b5f461ae5efa",
                  "CreatedAt": "2024-05-21T02:14:33.000Z",
                  "Name": "my-bucket-us-east-1"
                }
              },
              "Region": "us-east-1",
              "Id": "arn:aws:s3:::my-bucket-us-east-1"
            }
          ],
          "ProcessedAt": "2024-10-17T14:23:31.466Z"
        }
      ]
    }
  }

Expected behavior
Regardless of region, the solution should function in the same manner.
Please complete the following information about the solution:

[ ] Version: v1.0.2
[ ] Deployment: via StackSets using the CfCT. Default parameters largely chosen, particularly only enabling SC and not indivivual standards, as recommended in the documentation
[ ] Only the AWS Foundational Best Practice standard is enabled in SecurityHub itself
[ ] Region: us-east-1 (home), but also deployed to us-west-1, ap-southeast-2 and ap-southeast-4 (the issue occurs in all of them)
[ ] The solution has been modified from the original to deploy a single shared KMS key per region, shared to the Org (#204)
[ ] Changes are available in a zip in #204, however they are not in GitHub
[ ] This is not a service quota issue as the solution will execute in us-east-1; it is just providing different inputs in other regions which then lead to the failures.

Hi Julian, thanks for bringing this to our attention. I will start looking into this and get back to you once I have more information.

Hi Julian,
Can you confirm which playbooks you you have deployed in your ASR Admin & Member stacks (aws-sharr-deploy & aws-sharr-member)? Note that these should be the same. It would be most helpful if you could copy & paste the parameters you have configured for both stacks.
If you have deployed the SC playbook, please also confirm whether or not you have Consolidated control findings enabled in all regions where you are using Security Hub. You can verify this in each region by navigating to the Settings>General page in the Security Hub console. This is a requirement when using the SC playbook, since Security Hub emits different events for these types of findings. It is recommended to use the SC playbook since it consolidates all Security Hub controls and means you only have to select the SC playbook when deploying the solution.
My current hunch is that you may have deployed the SC playbook with Consolidated control findings enabled in us-east-1 but disabled in all other regions. This means that the solution would not correctly handle the Security Hub events in regions other than us-east-1.
Thanks!

Hi Ryan,
Thanks for your response. I investigated on my end and it seems that the consolidated control findings setting was only enabled in us-east-1 (my home region), but not others. Once I enabled the setting in all regions, the inputs to the step function were corrected and the sharr solution was able to auto-remediate in every region.
Thanks!
Julian


Export "to" directory

Feature request

What is the expected behavior?
I can do export to the current directory only without the possibility of global import.
What is motivation or use case for adding/changing the behavior?
Sometimes i need to create small modules which needed only in current directory. This will speed up the ide to find only the right modules. And we will be able to call modules with the same names in the current directory without confusion.
How should this be implemented in your opinion?
I suggest adding the specifically word "to".
E.g.:

I export some var in path /module/helpers.js:

const helpres = {};
export helpres to "./";

In /module/module.js i can import this one:

import helpres from "./helpers"; // got success

in base directory ("/") i can't find this one:

import `helpres` from "./module/helpres"; // error someVar not found

Are you willing to work on this yourself?
unfortunately not

It is not standard js syntax so it it out of scope webpack

There is the exports field in package.json feature (experimental and not implemented in webpack yet) which can be used to restrict access to internal modules.
https://nodejs.org/docs/latest-v13.x/api/esm.html#esm_package_exports


Rename the project to faunanet

We converged on naming the project namespace faunanet. Hence the Sparrow* references in the source code should be removed and replaced.

closed by #48


Update release_notes.md

Adding "Send to All Contacts" update to Marketing Campaigns.

Closes #

Hello @sendgrid-ryan,Thanks again for the PR! We appreciate your contribution and look forward to                                              continued collaboration. Thanks!                                             Team SendGrid DX

@sendgrid-ryan - Thanks again for this!


SCB-317 Update the Readme docs

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
 A more thorough check will be performed on your pull request automatically.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #290 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master    #290      +/-   ##
=========================================
- Coverage   70.43%   70.4%   -0.03%
=========================================
  Files          17      17
  Lines        3413    3413
=========================================
- Hits         2404    2403       -1
  Misses        842     842
- Partials      167     168       +1

Impacted Files
Coverage Δ

server/service/instances.go
71.16% <0%> (-0.17%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 15d9ce8...7383832. Read the comment docs.

Coverage decreased (-0.1%) to 73.967% when pulling 7383832e62f48798e323c2fbc0c188d8f9533549 on asifdxtreme:master into 15d9ce8d6cd1b2a60911ddde6b822b7a2c7b925a on apache:master.


cmd + l not work in iframe

cmd + l  format code not work in iframe

Please use the bug issue template. Thanks!


Correcting invitable synchronization with omniauthable?

Thanks for creating such a great tool with invitable.
I have been using Devise with omniauth (primarily for Facebook login) and now I am seeking to implement invitations, yet not lose the omniauth login functionality.  My goal is to allow a user to accept an invitation, then have the option to register/login using omniauth or the password confirmation fields that already exist.
I know to edit "/views/devise/invitations/edit.html.erb" so that there is a link to the omniauth_authorize_path.  However, how would this authorization retain the checks on the invitation token? With omniauth we would still need to ensure a valid invitation_token and record the invited_by.
Many thanks for your help!
-Dave

I was wondering this same thing. No, simply providing a link is not enough. The invitation token will be lost.
There's no good way to pass parameters via omniauth and have them come back, so the best approach is probably to save the invitation token in the session before you send them off to Facebook, and then look for this token in the session in the omniauth callback.

Thanks, @travisp -- that sounds smart.  Were you able to get it to work?  I'm a bit new to this, so if you have any example code that made this happen for you, I would be most grateful to see it and learn from you.
Many thanks!
-Dave

I did get it to work. I've provided some actual and some modified code. Hope this helps you and others:
https://gist.github.com/2007222
Most of the custom code is in the omniauth callbacks controller (and the exact approach will depend on your implementation of the callbacks controller), but it does require modifying the Devise::InvitationsController as well.
Two small edits to Invitations controller:
InvitationsController/edit -- set invitation_token in the session
InvitationsController/update -- remove the session invitation token if they accept the invitation by filling in the standard sign up form
OmniauthCallbacksController:
If this is a callback for an omniauth identity that's not in our database, and an invitation token exists in the session, then find the User matching that invitation token. Apply the omniauth information to the user model (in this case, basically store the omniauth identity). If the user model is valid and the user was an invited user, call accept_invitation! to save the model and clear the invitation token from the database (this is crucial). Finally, clear the session invitation token.
You can make this invite required either by modifying the callbacks controller to deal with when a user is not found with the invitation token, or by adding a validation to the model.

This is quite old already, but just FYI, you can add params to omniauth stages:
A link generated with:
<%= link_to "Sign in with Google", user_google_oauth2_omniauth_authorize_url(invitation_token: params[:invitation_token] ) %>

Will give you access to the invitation_token in request.env['omniauth.params'] on the callback action.


fix remove PCI device namespace group

Summary

pci does not support namespace group

Some translations need to be moved to harvester i18n file.

PR Checklist
~- Is this a multi-tenancy feature/bug?
- [ ] Yes, the relevant RBAC changes are at:~
~- Do we need to backport changes to the old Rancher UI, such as RKE1?
- [ ] Yes, the relevant PR is at:~
~- Are backend engineers aware of UI changes?
- [ ] Yes, the backend owner is:~
Fixes #

Occurred changes and/or fixed issues

Technical notes summary

Areas or cases that should be tested

Areas which could experience regressions

Screenshot/Video

@mergify backport release-harvester-v1.2


fix: add arrow-buffer to arrow-expression feature

When running cargo-publish the delta_kernel crate failed to compile; needed to add arrow-buffer to arrow-expression feature.
error[E0432]: unresolved import `arrow_buffer`
  --> src/engine/arrow_expression.rs:13:5
   |
13 | use arrow_buffer::OffsetBuffer;
   |     ^^^^^^^^^^^^ use of undeclared crate or module `arrow_buffer`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `delta_kernel` (lib) due to 1 previous error

I suppose to catch something like this we could run cargo package in the CI but that feels clunky..? Anyone know if we can mimic cargo-publish just through the right cargo build incantation?

Created #333 for follow-up!


Multistreaming

Reduces memory usage (kernels.cuh)
Restructures the setup of the code
-- reads root files upfront
--runs over events in parallel CPU threads using openMP
-- each CPU thread takes its own cuda stream and event
--all kernels, memcpy and memsets are Asynchronous. uses cudaStreamSynchronize instread of device sync
number of streams set from command line with -s  flag. Default =1
nBlocks set to 80 in kernels.cuh

Validation plots: https://www.classe.cornell.edu/~mgr85/www/SegmentLinking_keep/PR94/

Timing:

Timing:

is the throughput column something to read or mostly ignore for now? (unified s=1 being the best looks odd; here I'm assuming that as in other places the throughput is the number of events per unit time)
the "effective time" shown on Tuesday was a bit more intuitive. It may be better to add a column for it.

So kernel and kernel short replace the previous times we have look at until now. Loop is the time it takes to run over all the events divided by the number of events. And throughput (i guess inverse throughput) is the effective time = (kernel_short/ kernel ) * loop. So explicit cache with 6 streams works the best giving an effective time of 30 ms per event.

Changed the timing format to be less confusing

Can you elaborate what kernel, kernel (short), loop, effective mean in the timing table?

@tresreid Can you elaborate what kernel and kernel (short) mean in the timing table?

@beiwang2003 Kernel is the sum of all the kernel times (what was previously timing) whereas kernel short(previously timing short) is the sum of the kernel times we care about (it excludes the hit loading and pLS duplicate removal). Kernel short was what we were previously using to report our times. Loop time is the time it takes to run over all the events divided by the number of events. For a single stream should be close to the kernel time but may include some overhead. Effective time (which i just changed from throughput) is kernel_short / kernel * loop. It is the timing we can expect for the part of the code we care about over multiple streams. This is the time we want to use now. For a single stream it should hopefully be close to the kernel short time.

Tested unified and explicit on phi3 on PU200 ttbar 50 events and I seem to get reasonable output from the validation plots. LGTM!


.markdownlint.json configuration - MD044 currently has a destructive bug

I noticed that the .markdownlint.json file in azure-docs-pr defines a handful of terms for rule MD044.
There is a destructive bug in the MD044 implementation when running markdownlint from the command line with the --fix parameter. I've opened an issue here:
https://github.com/igorshubovych/markdownlint-cli/issues/98
Anyone running markdownlint --fix in Azure Docs may find certain content unintentionally removed or replaced due to this bug.
@IEvangelist @nschonni @chbain

/cc @SyntaxC4

@alexbuckgit Is this feedback related to a particular doc? Can you share the documentation link?
Under this repository we want to make sure any issue is linked to a doc so we can help others that may be having the same issue.

The only file I know it directly relates to is .markdownlint.json. Because I don't know the scope or root cause of the issue in markdownlint-cli, it could conceivably apply to any document for which someone runs markdownlint --fix.
I'll keep an eye on the open issue in the markdowlint-cli repo. The only remediation option I see at this point would be to remove the MD044 values from .markdownlint.json, or hope that anyone running that command notices that some content was unintentionally deleted.

@alexbuckgit As mentioned by @AshokPeddakotla-MSFT Could you please share the Azure document link you are referring to help you better on this.

The linked issue was closed https://github.com/igorshubovych/markdownlint-cli/issues/98#issuecomment-646395495

Since your issue is resolved as mentioned in the linked issue,
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

The repo owner asked me to split my original issue because there were two different underlying causes.
The relevant issue is still open:
https://github.com/igorshubovych/markdownlint-cli/issues/100

Due for resolution in the next release.


Wallpaper not changing on Gnome 42.1

After updating to Gnome 42.1 pywal will not update the background image. It runs as normal, and changes terminal colours, but no changes to the background occur.

Pywal doesn't seem to be working after GNOME 42 was released. GNOME 42 provide light/dark themes. If the light theme is set, pywal works as expected. If the dark theme is set, the colors on the terminals will match the wallpaper you chose but it won't change the background. You can test what I just said to confirm this is the case (I actually came here to find a solution for it cause it's also bothering me)


Update and rename content/get-started/quickstart/index.md to เนื้อหา/…

…เริ่มต้น/เริ่มต้นอย่างรวดเร็ว/index.md
soffieprincess02:revert-8795-repo-sync

Why:
Closes [issue link]

What's being changed:

Check off the following:

[x] I have reviewed my changes in staging (look for the latest deployment event in your pull request's timeline, then click View deployment).
[x] For content changes, I have completed the self-review checklist.

Writer impact (This section is for GitHub staff members only):

[ ] This pull request impacts the contribution experience

[ ] I have added the 'writer impact' label
[ ] I have added a description and/or a video demo of the changes below (e.g. a "before and after video")

8871


About learn page files routes

added in starter files for the About & Learn Pages
(About.js, about.css, path= /about & Learn.js, learn.css,  path= /learn)
added in route paths on app.js for each page + some notes on the existing paths

resolved the merge conflicts ✅


Tree Traversal?

I'm not fully sure this is a good idea, or there is a nice way to do it. More a discussion point at the moment. I found myself needing to do a bit of modification of parent, child, or sibling elements. Having me() is still nice, but the moment you need to do any tree traversal from there you lose the sugar methods, or end up with some awkward constructs to get them back (or you just go with vanilla js).
I was thinking the sugar call could automatically sugar some attributes like parentElement nextElementSibling and methods like querySelector to make tree traversal chainable while retaining the sugar methods.
e.g.

me(me().parentElement).classToggle("something")->me().parentElement.classToggle("something")
me(".things", me()).fadeOut()->me().querySelector(".things").fadeOut()

PS: After having written this out, and come up with the examples, I don't mind this syntax and nested me calls as much. The only thing that's weird to me is the use of the name me whenever we provide a selector. It no longer has anything to do with the current location, and is just adding the sugar methods, which makes the me name feel awkward.

Very interesting! Will be considering possibilities for this.

The issue here is: Adding that style would require surreal to edit all existing nodes in the DOM (to add surreal chaining), which I'd like us to avoid.
I'm a huge fan of how non-intrusive yet still intuitive "just feed the element into me()" is.
One can think of the new me() chain as a new context.
I think this comes down to discovering/using new patterns, but feel free to continue the discussion if you feel I'm missing something important!

Yeah, I came to the same conclusion. Adding the me() wrapping is not bad, and after using it some more I sorta like that it's explicit.
That being said, chaining doesn't require editing all the existing nodes in the DOM, it can be done lazily via property descriptors.


Switch from storing settings in config.json to storm.yaml

This isn't quite ready for primetime yet, but I wanted to create the PR as a discussion point.
What currently works is that if we find you have a config.json file, we'll automatically load it and then dump the settings in there into a streamparse dict within storm.yaml. get_config and everything downstream of it has been adapted to handle this extra level of nesting in the dict that get_config returns.
What I have not done yet, but would like to do, is switching our environment-specific settings to match the general storm.yaml setting names (i.e., nimbus.host or nimbus.seeds instead of just nimbus, and topology.workers instead of workers), and then falling back on the storm.yaml settings if you haven't setup an environment at all. This would allow people to reuse what they may already have instead of potentially putting the settings in two places in two different formats.
Since we'll start parsing storm.yaml with this PR, I'm sure there are some other settings that we could probably pull out of it that we don't currently use that might be useful.

Coverage decreased (-0.5%) to 44.679% when pulling 3dc3d1e739756e9d763bdc2d48de84e8167ddc97 on feature/storm_yaml into 98f12db1a04fab1aa74fd39ca35f5cc4bbfa5ac6 on master.

Coverage decreased (-0.5%) to 45.03% when pulling c643a68c261b4d4565d8e11062688bd09bba5e94 on feature/storm_yaml into 18fd2e9f0c2a8f716aad243bff961c2e962e800e on master.

Coverage decreased (-0.5%) to 44.318% when pulling 33a5aecd223ca381666d7bf4d6983b017e16136a on feature/storm_yaml into ddf8a32285ad05519806c6071e6d9c5c73b62888 on master.

Closing as very stale

Closing as very stale

Who's to say I won't just pick this up after 8 years? 😆


OperationOutcome: The requested action is not allowed

I see this for some of the generated files when trying to PUT into the server:
LOADING FILE: ./Org-1067.json...
annotating bundle:
failed to send bundle to server: Method Not Allowed
{'_content': b'{"resourceType":"OperationOutcome","id":"b73e574d-187f-4e83-baf6'
             b'-13f4e1f0f2d3","issue":[{"severity":"error","code":"forbidden","'
             b'diagnostics":"The requested action is not allowed."}]}'

This error code happens when a client tries to perform an HTTP action over HTTPS, if I understand the documentation correctly.
Looking into the fhir server logs, it looks like we're trying to post to an https:// url over port 80, so that seems to fit.
However, looking at the code we aren't doing that explicitly - we're POST'ing to the base root URL.
Interestingly, when I point my web browser at https://prototype-erp-fhir.azurewebsites.net/ I immediately get a file downloaded, which appears to be an XML patient bundle.
<?xml version="1.0" encoding="utf-8"?>
<Bundle xmlns="http://hl7.org/fhir">
    <id value="4a1eb23e-b6c7-460c-b19d-072d644748e5" />
    <meta>
        <lastUpdated value="2020-03-26T20:23:24.5367353+00:00" />
    </meta>
    <type value="searchset" />
    <link>
        <relation value="self" />
        <url value="http://prototype-erp-fhir.azurewebsites.net/" />
    </link>
    <entry>
        <fullUrl value="http://prototype-erp-fhir.azurewebsites.net/Patient/AAN9930" />
        <resource>
            <Patient>
                <id value="AAN9930" />
                <meta>
                    <versionId value="1" />
                    <lastUpdated value="2020-03-23T21:08:07.407+00:00" />
                </meta>
                <name>
                    <use value="official" />
                    <family value="Project-9489" />
                    <given value="Argonaut-9210" />
                </name>
                <gender value="other" />
                <birthDate value="1935-02-11" />
            </Patient>
        </resource>
        <search>
            <mode value="match" />
        </search>
    </entry>
</Bundle>

This is definitely new behavior since yesterday, and I'm at a loss to explain how this happened.

This was just a confusing error message returned by the Azure FHIR service.  Pointing at a HAPI server gave this message:
{
  "resourceType": "OperationOutcome",
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "Invalid request Bundle.type value for transaction: \"collection\""
  } ]
}

This made it clear that the transaction bundle needed to be type 'bundle' instead of 'collection'.


ERROR:  relation "chq_settings" does not exist at character 29

Hello,
I'm following this doc: https://hub.docker.com/r/cachethq/docker/
Step - Development of Cachet using this docker environment.
I already got the repos, did composer install, altered docker-compse.yml to add the volume, configured the API_KEY and now i'm trying to do docker-compose up
When I execute docker-compose up the following error ocurs:
 postgres_1  | ERROR:  relation "chq_settings" does not exist at character 29
Stack trace:

merineira@merini:~/cachet-teste/cachet-docker$ docker-compose up
Starting cachet-docker_postgres_1 ... done
Recreating cachet-docker_cachet_1 ... done
Attaching to cachet-docker_postgres_1, cachet-docker_cachet_1
cachet_1    | Initializing Cachet container ...
postgres_1  |
postgres_1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1  |
postgres_1  | LOG:  database system was shut down at 2020-02-17 17:22:46 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  autovacuum launcher started
postgres_1  | LOG:  database system is ready to accept connections
cachet_1    | Attempting to connect to database ...
cachet_1    |
cachet_1    | Table chq_sessions does not exist! ...
cachet_1    | Initializing Cachet database ...
postgres_1  | ERROR:  relation "chq_settings" does not exist at character 29
postgres_1  | STATEMENT:  select "name", "value" from "chq_settings"
cachet_1    | PHP Fatal error:  Uncaught ErrorException: count(): Parameter must be an array or an object that implements Countable in /var/www/html/vendor/graham-campbell/exceptions/src/ExceptionIdentifier.php:51

I could fix the first error, but the following still happens

I could fix the first error, but the other one still happens

how?


the idea of a default forge.config.js needs to be finished.

Pre-flight checklist

[X] I have read the contribution documentation for this project.
[X] I agree to follow the code of conduct that this project uses.
[X] I have searched the issue tracker for a bug that matches the one I want to file, without success.

Electron Forge version
6.0.0-beta.59
Electron version
v10.1.1
Operating system
fedora 34
Last known working Electron Forge version
? this has always been broken.
Expected behavior
honestly, just let us do :
electron-forge start --config=./forge-config.js
Actual behavior

Steps to reproduce
https://github.com/airtonix/electron-forge-broken-default-config-reproduction

have pnpm 6+
have node 16+
clone https://github.com/airtonix/electron-forge-broken-default-config-reproduction
pnpm install
pnpm app:foo start

Additional information
No response

It seems like based on the input in this bug, you've filed a feature request instead. forge.config.js is a convention rather than a strict filename.
I'm probably not going to implement this. I'd rather have an external config file be named in package.json rather than have an implied default config filename, or have to supply this in the CLI. (I really don't like npm scripts with CLI flags specified if it can be avoided.)


fix encode string as integer

encoder should be strict about the strings that can be encoded as integer.
as strconv.ParseInt() is not accurate for this case, we need to check the string content and pick up
the "pure" uint32 or uint64 numbers.
Also the WriteSetObject() function has a wrong state to fix.
Fix issue https://github.com/HDT3213/rdb/issues/38
Fix issue https://github.com/HDT3213/rdb/issues/40

I will review it later today

Actually, -1 should be encoded as int
<IP_ADDRESS>:6379> set a -1
OK
<IP_ADDRESS>:6379> object encoding a
"int"

Actually, -1 should be encoded as int
<IP_ADDRESS>:6379> set a -1
OK
<IP_ADDRESS>:6379> object encoding a
"int"

I read some rdb libs where decoders read uint. so encoder better only encodes uint too.

For your information, redis use string2ll to determine whether int encoding can be used.

Actually, -1 should be encoded as int
<IP_ADDRESS>:6379> set a -1
OK
<IP_ADDRESS>:6379> object encoding a
"int"

I read some rdb libs where decoders read uint. so encoder better only encodes uint too.

Let me think about it

rdbtools supported signed int32 too. I prefer to follow the  redis official strategy.
https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/rdbtools/parser.py#L494

rdbtools supported signed int32 too. I prefer to follow the redis official strategy.
https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/rdbtools/parser.py#L494

ok. but the encoder and decoder should be modified accordingly. current codes only support uint.

rdbtools supported signed int32 too. I prefer to follow the redis official strategy.
https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/rdbtools/parser.py#L494

ok. but the encoder and decoder should be modified accordingly. current codes only support uint.

Although the code is a bit strange, it does work for negative numbers.😂 I'll sort it out.
https://github.com/HDT3213/rdb/blob/master/core/string.go#L92


Fix preSign signature calculation to honor resource lists.

Currenlty we used to sign all the query parameters, this
is not in accordance with AWS S3 spec. We should only
canonicalize query params which are present in resouce list.
Fixes #825

@vadmeste @krisis waiting on your reviews.


Change npm package request

Describe the proposal
The npm package request has been deprecated since 11 Feb 2020. https://github.com/request/request#deprecated
Find alternatives to it. https://github.com/request/request/issues/3143

This is already fixed in feature/new_quickstarts.  we use Axios and supported packages now.  Close this? @mukundansundar @msfussell


[Feature Request] Dataset component (Experience tab)

Overview
This part of the dataset component should give the users a clear overview of the specific dataset's course evaluations. Currently, the scraping for this is still in progress, so the exact information to be displayed has not be determined yet. Providing placeholder spots for data should be enough to modify later to complete this feature.
Proposal
Implement the design shown in the main card on Search Page (by class).
Supplementary Information
The design can be viewed in the main card on Search Page (by class) in the Figma, as well as labeled extended-view-experience in the Components page

@AbhiramTadepalli should this be closed since course evaluation data is no longer on the horizon?

mhm

Yes


adds PKGBUILD for uploading to aur

first two are required by aur. the third adds and removes links to
binary and updates icon caches
beware of deps, also kindly check pkgnames and etc.

I tried to avoid repackaging, but couldn't find any solutions. the PKGBUILD downloads the .pacman version and repacks it and then installs it.

Thanks a lot @HadiLatifi ,
I'll try to install arch quickly to test it and let you know the result

Just tested it on Monjaro and it works perfectly. As soon as I get it to AUR I'll merge and close this PR.

@HadiLatifi the PR is merged. Thanks again for making these files for me.
The package is published with todoist-electron name since as you noticed todoist and todoist-linux names are taken.


Update virtual realsense class

I forgor to make pull request

Should I include the udp streaming stuff in the same file, or should it go in separate files for those classes


Crash caused by Nuklear

Version
3.3.2
Platform
Windows x64
JDK
OpenJDK Runtime Environment Corretto-<IP_ADDRESS>.1
Module
org.lwjgl.nuklear.Nuklear
Bug description
I experienced mildly frequent crashes on my LWJGL project.
I discovered by reading the hid_pid_err.log that the crashes are coming from Nuklear nnk_convert() method.
Attached is the jvm error log: hs_err_pid11372.log
The code (in my application) that keeps causing the crash is right here:
import static org.lwjgl.nuklear.Nuklear.*;
import org.lwjgl.nuklear.*;
import static org.lwjgl.system.MemoryUtil.*;

{
. . .

            // load draw vertices & elements directly into vertex + element buffer
            ByteBuffer vertices = Objects.requireNonNull(glMapBuffer(GL_ARRAY_BUFFER, GL_WRITE_ONLY, max_vertex_buffer, null));
            ByteBuffer elements = Objects.requireNonNull(glMapBuffer(GL_ELEMENT_ARRAY_BUFFER, GL_WRITE_ONLY, max_element_buffer, null));
            try (MemoryStack stack = stackPush()) {
                // fill convert configuration
                NkConvertConfig config = NkConvertConfig.calloc(stack)
                        .vertex_layout(VERTEX_LAYOUT)
                        .vertex_size(20)
                        .vertex_alignment(4)
                        .tex_null(null_texture)
                        .circle_segment_count(22)
                        .curve_segment_count(22)
                        .arc_segment_count(22)
                        .global_alpha(1.0f)
                        .shape_AA(AA)
                        .line_AA(AA);

                // setup buffers to load vertices and elements
                NkBuffer vbuf = NkBuffer.malloc(stack);
                NkBuffer ebuf = NkBuffer.malloc(stack);

                nk_buffer_init_fixed(vbuf, vertices/*, max_vertex_buffer*/);
                nk_buffer_init_fixed(ebuf, elements/*, max_element_buffer*/);
                nk_convert(ctx, cmds, vbuf, ebuf, config);//<------------ This line is causing keeps causing a JVM crash
            }
            glUnmapBuffer(GL_ELEMENT_ARRAY_BUFFER);
            glUnmapBuffer(GL_ARRAY_BUFFER);

. . .
}

Stacktrace or crash log output
From the his_err_pid.log:

# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

I did that and it still crashes sometimes:
hs_err_pid8720.log
Here is a more complete chunk of code containing the source of the crashes.

public abstract class NKWindow extends BaseWindow {
. . .

    //This prevents the byteBuffers from being garbage collected before the render is done with them, thus preventing a JVM crash
    private ByteBuffer vertices, elements; //Used to fix the crash bug (https://github.com/LWJGL/lwjgl3/issues/986)
    private NkBuffer vbuf, ebuf; //Also stored out of scope to fix the crash bug
    private NkConvertConfig config;//Also stored out of scope to fix the crash bug

    private static final int AA = NK_ANTI_ALIASING_ON;
    private static final int max_vertex_buffer = MAX_VERTEX_BUFFER;
    private static final int max_element_buffer = MAX_ELEMENT_BUFFER;

    public void NKrender() {
        try (MemoryStack stack = stackPush()) {
            enableNuklear();
            // setup program
            glUseProgram(shader.getID());
            glUniform1i(uniform_tex, 0);
            glUniformMatrix4fv(uniform_proj, false, stack.floats(2.0f / getWidth(), 0.0f, 0.0f, 0.0f,
                    0.0f, -2.0f / getHeight(), 0.0f, 0.0f,
                    0.0f, 0.0f, -1.0f, 0.0f,
                    -1.0f, 1.0f, 0.0f, 1.0f
            ));
            GL11.glViewport(0, 0, getDisplay_width(), getDisplay_height());
        }

        {
            // convert from command queue into draw list and draw to screen

            // allocate vertex and element buffer
            glBindVertexArray(vao);
            glBindBuffer(GL_ARRAY_BUFFER, vbo);
            glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, ebo);

            glBufferData(GL_ARRAY_BUFFER, max_vertex_buffer, GL_STREAM_DRAW);
            glBufferData(GL_ELEMENT_ARRAY_BUFFER, max_element_buffer, GL_STREAM_DRAW);

            // load draw vertices & elements directly into vertex + element buffer
            vertices = Objects.requireNonNull(glMapBuffer(GL_ARRAY_BUFFER, GL_WRITE_ONLY, max_vertex_buffer, null));
            elements = Objects.requireNonNull(glMapBuffer(GL_ELEMENT_ARRAY_BUFFER, GL_WRITE_ONLY, max_element_buffer, null));
            try (MemoryStack stack = stackPush()) {
                // fill convert configuration
                config = NkConvertConfig.calloc(stack)
                        .vertex_layout(VERTEX_LAYOUT)
                        .vertex_size(20)
                        .vertex_alignment(4)
                        .tex_null(null_texture)
                        .circle_segment_count(22)
                        .curve_segment_count(22)
                        .arc_segment_count(22)
                        .global_alpha(1.0f)
                        .shape_AA(AA)
                        .line_AA(AA);

                // setup buffers to load vertices and elements
                vbuf = NkBuffer.malloc(stack);//TODO: If we keep getting crashes, maybe this has to be stored out of scope too?
                ebuf = NkBuffer.malloc(stack);

                nk_buffer_init_fixed(vbuf, vertices/*, max_vertex_buffer*/);
                nk_buffer_init_fixed(ebuf, elements/*, max_element_buffer*/);
                nk_convert(ctx, cmds, vbuf, ebuf, config);//<----------------------- This line is causing keeps causing a JVM crash
            }
            glUnmapBuffer(GL_ELEMENT_ARRAY_BUFFER);
            glUnmapBuffer(GL_ARRAY_BUFFER);

            // iterate over and execute each draw command
            float fb_scale_x = (float) getDisplay_width() / (float) getWidth();
            float fb_scale_y = (float) getDisplay_height() / (float) getHeight();

            long offset = NULL;
            for (NkDrawCommand cmd = nk__draw_begin(ctx, cmds); cmd != null; cmd = nk__draw_next(cmd, cmds, ctx)) {
                if (cmd.elem_count() == 0) {
                    continue;
                }
                glBindTexture(GL_TEXTURE_2D, cmd.texture().id());
                glScissor((int) (cmd.clip_rect().x() * fb_scale_x),
                        (int) ((getHeight() - (int) (cmd.clip_rect().y() + cmd.clip_rect().h())) * fb_scale_y),
                        (int) (cmd.clip_rect().w() * fb_scale_x),
                        (int) (cmd.clip_rect().h() * fb_scale_y)
                );
                glDrawElements(GL_TRIANGLES, cmd.elem_count(), GL_UNSIGNED_SHORT, offset);
                offset += cmd.elem_count() * 2;
            }
            nk_clear(ctx);
            nk_buffer_clear(cmds);
        }
        disableNuklear();
    }
. . .
}

Things allocated using MemoryStack are freed when the stack is closed, i.e. at the end of the try (...) {}. Storing them in a variable won't change this. You should allocate them using MemoryUtil or BufferUtils (and properly free them later) if the lifecycle won't fit into a try.

Ok. When should I free the memory? Do i need to deallocate() it? If so where is the safest place to free it up?

When it should be freed depends on what's using it, generally you should try to free it as soon as possible after it is done being used -- read the documentation of each function or the library to determine when that is.
Anything allocated with MemoryUtil must be freed using it, or it will leak. Buffers allocated with BufferUtils will be freed when garbage collected, but this may not happen immediately.
See https://github.com/LWJGL/lwjgl3-wiki/wiki/1.3.-Memory-FAQ.

Hey @zipCoder933,

I did that and it still crashes sometimes:

The vertex & element buffers are unrelated to the problem. Those are stored by the OpenGL driver, the Java-side buffers are gone when you call glUnmapBuffer and you don't need to keep them around.
See the post I linked above please. The crash happens in the NkUserFont::query callback, you need to look at the code that sets up font rendering. There should be a call to stbtt_InitFont somewhere, passing the font data as a buffer. There should be a strong reference to that buffer for as long as you're rendering with the font.

So I followed the advice in the other bug report that you posted (https://github.com/LWJGL/lwjgl3/issues/535#issuecomment-590983833) However the same crash still occurs. I put the fontBuffer in a public static context so that it isnt possible for it to become garbage collected.
Theme.java
class Theme{
    public static ByteBuffer fontBuffer;

    public static void initialize(NkContext context, boolean largerFonts) throws IOException {
        fontBuffer = NKFontUtils.loadFontData("fonts\\Press_Start_2P\\PressStart2P-Regular.ttf", 512 * 1024);

        font_24 = NKFontUtils.TTF_assignToNewTexture(fontBuffer, largerFonts ? 26 : 24);
        font_22 = NKFontUtils.TTF_assignToNewTexture(fontBuffer, largerFonts ? 24 : 22);
        font_18 = NKFontUtils.TTF_assignToNewTexture(fontBuffer, largerFonts ? 20 : 18);
        font_12 = NKFontUtils.TTF_assignToNewTexture(fontBuffer, largerFonts ? 14 : 12);
        font_10 = NKFontUtils.TTF_assignToNewTexture(fontBuffer, largerFonts ? 12 : 10);
        font_9 = NKFontUtils.TTF_assignToNewTexture(fontBuffer, largerFonts ? 10 : 9);
...
  }
}

NKFontUtils.java
The loadFontData method uses the LWJGL demo repository's "ioResourceToByteBuffer()" method
public class NKFontUtils {

    final static int BITMAP_W = 1024;
    final static int BITMAP_H = 1024;

    public static final ArrayList<ByteBuffer> fontBuffers = new ArrayList<>();

    public static ByteBuffer loadFontData(String path) throws IOException {
        ByteBuffer buff = ioResourceToByteBuffer(path, 512 * 1024);
        fontBuffers.add(buff);
        return buff;
    }

    public static NkUserFont TTF_assignToNewTexture(final ByteBuffer fontBytes, int fontHeight) {
        NkUserFont font = NkUserFont.create();

        int fontTexID = glGenTextures();
        TextureUtils.addTexture(fontTexID);

        STBTTFontinfo fontInfo = STBTTFontinfo.create();
        STBTTPackedchar.Buffer cdata = STBTTPackedchar.create(95);

        float scale;
        float descent;

        try (MemoryStack stack = stackPush()) {
            stbtt_InitFont(fontInfo, fontBytes);
            scale = stbtt_ScaleForPixelHeight(fontInfo, fontHeight);

            IntBuffer d = stack.mallocInt(1);
            stbtt_GetFontVMetrics(fontInfo, null, d, null);
            descent = d.get(0) * scale;

            ByteBuffer bitmap = memAlloc(BITMAP_W * BITMAP_H);

            STBTTPackContext pc = STBTTPackContext.malloc(stack);
            stbtt_PackBegin(pc, bitmap, BITMAP_W, BITMAP_H, 0, 1, NULL);
            stbtt_PackSetOversampling(pc, 4, 4);
            stbtt_PackFontRange(pc, fontBytes, 0, fontHeight, 32, cdata);
            stbtt_PackEnd(pc);

            // Convert R8 to RGBA8
            ByteBuffer texture = memAlloc(BITMAP_W * BITMAP_H * 4);
            for (int i = 0; i < bitmap.capacity(); i++) {
                texture.putInt((bitmap.get(i) << 24) | 0x00FFFFFF);
            }
            texture.flip();

            glBindTexture(GL_TEXTURE_2D, fontTexID);
            glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, BITMAP_W, BITMAP_H, 0, GL_RGBA, GL_UNSIGNED_INT_8_8_8_8_REV, texture);
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

            memFree(texture);
            memFree(bitmap);
        }

        font
                .width((handle, h, text, len) -> {
                    float text_width = 0;
                    try (MemoryStack stack = stackPush()) {
                        IntBuffer unicode = stack.mallocInt(1);

                        int glyph_len = nnk_utf_decode(text, memAddress(unicode), len);
                        int text_len = glyph_len;

                        if (glyph_len == 0) {
                            return 0;
                        }

                        IntBuffer advance = stack.mallocInt(1);
                        while (text_len <= len && glyph_len != 0) {
                            if (unicode.get(0) == NK_UTF_INVALID) {
                                break;
                            }

                            /* query currently drawn glyph information */
                            stbtt_GetCodepointHMetrics(fontInfo, unicode.get(0), advance, null);
                            text_width += advance.get(0) * scale;

                            /* offset next glyph */
                            glyph_len = nnk_utf_decode(text + text_len, memAddress(unicode), len - text_len);
                            text_len += glyph_len;
                        }
                    }
                    return text_width;
                })
                .height(fontHeight)
                .query((handle, font_height, glyph, codepoint, next_codepoint) -> {
                    try (MemoryStack stack = stackPush()) {
                        FloatBuffer x = stack.floats(0.0f);
                        FloatBuffer y = stack.floats(0.0f);

                        STBTTAlignedQuad q = STBTTAlignedQuad.malloc(stack);
                        IntBuffer advance = stack.mallocInt(1);

                        stbtt_GetPackedQuad(cdata, BITMAP_W, BITMAP_H, codepoint - 32, x, y, q, false);
                        stbtt_GetCodepointHMetrics(fontInfo, codepoint, advance, null);

                        NkUserFontGlyph ufg = NkUserFontGlyph.create(glyph);

                        ufg.width(q.x1() - q.x0());
                        ufg.height(q.y1() - q.y0());
                        ufg.offset().set(q.x0(), q.y0() + (fontHeight + descent));
                        ufg.xadvance(advance.get(0) * scale);
                        ufg.uv(0).set(q.s0(), q.t0());
                        ufg.uv(1).set(q.s1(), q.t1());
                    }
                })
                .texture(it -> it
                        .id(fontTexID));

        return font;
    }
}

I'm out of ideas, could you prepare an MCVE please?

Here is a working MRE: NuklearMRE.zip
The challenge is that this crash only occurs VERY sparsely in my main application, and so, this makes it very difficult for me to try to reproduce the error in MRE code as well.
The only way I can give you a crash log is if i open this application enough times that Maybe I get lucky and the crash resurfaces??
Anyways, I'm not sure if opening and closing the window many times per second might help?...

It reproduces the crash with two changes:

Comment-out fontBuffers.add(buff); at NkFontUtils.java:56. Setting fontBuffer to null won't do anything if there's another strong reference to the instance.
Add Runtime.getRuntime().gc(); to the main loop. It's very hard to reproduce such a crash without actual memory pressure, the GC will never run.

With these two changes, font rendering breaks as soon as I press the button and the program usually crashes a few seconds later.
If your application still crashes after making sure there's always a strong reference to the font data, then it must be another unrelated issue.

Interesting. The MRE now indeed crashes when the byteBuffer is garbage collected.
The trace in the crash log however isn't coming from nk_convert like my application is. This one is coming from nk_begin:
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 1108  org.lwjgl.stb.STBTruetype.nstbtt_GetCodepointHMetrics(JIJJ)V (0 bytes) @ 0x0000022de6c8c3f6 [0x0000022de6c8c3a0+0x0000000000000056]
J 1236 c2 com.xbuilders.window.nuklear.NKFontUtils.lambda$TTF_assignToNewTexture$0(Lorg/lwjgl/stb/STBTTFontinfo;FJFJI)F (190 bytes) @ 0x0000022de6c99278 [0x0000022de6c99080+0x00000000000001f8]
J 1180 c1 com.xbuilders.window.nuklear.NKFontUtils$$Lambda$63+0x0000000800c308d8.invoke(JFJI)F (18 bytes) @ 0x0000022ddf9d97a4 [0x0000022ddf9d9720+0x0000000000000084]
J 1179 c1 org.lwjgl.nuklear.NkTextWidthCallbackI.callback(JJ)V (62 bytes) @ 0x0000022ddf9da9dc [0x0000022ddf9da020+0x00000000000009bc]
v  ~StubRoutines::call_stub
j  org.lwjgl.nuklear.Nuklear.nnk_begin(JJJI)Z+0
j  org.lwjgl.nuklear.Nuklear.nk_begin(Lorg/lwjgl/nuklear/NkContext;Ljava/lang/CharSequence;Lorg/lwjgl/nuklear/NkRect;I)Z+38
j  com.xbuilders.app.TopMenu.render()V+60

Either way, there are indeed 2 strong references to the font byteBuffer in my application, so the font bytebuffer must not be what is causing the crash.

Specifically because this is crashing in GetPackedQuad, it must be from some memory access in that function. The only pointers used by it are chardata, xpos, ypos, and q. Since you just stack allocated the last three, I think that only chardata could possibly be in an invalid state. However, it isn't freed, as all callbacks are GC roots, so there is a strong reference to it as long as the owning NkUserFont is not freed, which it appears to not be in your MRE at least (held by static field). But, there is a memory access violation that can occur in your code even without any freed memory!
You take codepoint - 32 without checking if codepoint is at least 32. This means you can pass a negative index to GetPackedQuad, which isn't caught by LWJGL's checks. This results in GetPackedQuad accessing below the allocated object, which could occasionally result in an access error. This could be caused by something as simple as the tab or newline character appearing in some text somewhere, as those both have a codepoint below 32.
Try adding a check in your query callback to ensure that no codepoint is below 32 when you call that method. Whether you hard-crash or just return some dummy data in ufg, just make sure not to call the GetPackedQuad method.


chatbot/rag/summarizer: sync READMEs

Sync the descriptions of the three READMEs on running the recipes via Podman Desktop and locally.
@rhatdan PTAL

LGTM


Use of relations for image and sheet file names inside the spreadsheet files

Hi @tanaikech, Thanks for making this library, it helped me at work deal with batch uploading product photos to an accounting system from Excel files.
While using the library, some .xlsx files had issues with the library due to the use of relationship files that map a specific drawing to filenames inside the Excel file. This also applies to linking sheet names to file names inside the Excel file. I've modified the library to work around this issue and get the file / image names from the relationship files.
I was wondering if you are using a build step for the code available here in the repo? If not I can open a pull request with my changes (which I am sure are not great since I made them in a hurry and I am not really familiar with the Excel file specifications and XML in general.)

I've emailed you two sample files.
Here's an example for the sheet file name stored in a relationship file:
xl/workbook.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"
  xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"
  xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
  mc:Ignorable="x15 xr xr6 xr10 xr2"
  xmlns:x15="http://schemas.microsoft.com/office/spreadsheetml/2010/11/main"
  xmlns:xr="http://schemas.microsoft.com/office/spreadsheetml/2014/revision"
  xmlns:xr6="http://schemas.microsoft.com/office/spreadsheetml/2016/revision6"
  xmlns:xr10="http://schemas.microsoft.com/office/spreadsheetml/2016/revision10"
  xmlns:xr2="http://schemas.microsoft.com/office/spreadsheetml/2015/revision2">
  <fileVersion appName="xl" lastEdited="7" lowestEdited="7" rupBuild="23801" />
  <workbookPr />
  ...
  <sheets>
    <sheet name="GLASSWARE" sheetId="3" r:id="rId1" />
  </sheets>
  ...
</workbook>

xl/_rels/workbook.xml.rels
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
  <Relationship Id="rId3" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles" Target="styles.xml" />
  <Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/theme" Target="theme/theme1.xml" />
  <Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet1.xml" />
  <Relationship Id="rId5" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/calcChain" Target="calcChain.xml" />
  <Relationship Id="rId4" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/sharedStrings" Target="sharedStrings.xml" />
</Relationships>

For my use case, I've already made changes to the library to support getting the drawing & sheet file names from these relations.

For example, when you want to correspond each image file to your expected values, how about doing it after the image data was obtained from XLSX data? If this was not your expected direction, I apologize.

Right now without my modifications, the library will throw an error when using the getImages() method or the getSheetByName() depending on the usage of the rels file for image or sheet file names.
I can open a pull request so that you can review the modifications I've made.

Thank you for replying. About Right now without my modifications, the library will throw an error when using the getImages() method or the getSheetByName() depending on the usage of the rels file for image or sheet file names., in this case, I would like to recommend to try to use the library after your XLSX was converted to Spreadsheet.

Thanks for the suggestion. You are right about converting to Google Spreadsheet before using the library, this seems to solve the issue with these relationships.  Feel free to close the issue.