Multiuser Support

Hello! Thank you for your work on the integration, it is great.
I have a suggestion how to improve the integration even more. This is about multiuser support. For example, the users could be distinguished by the measured weight.
You could configure it like this for example:
bodymiscale:
  model_miscale: "181B"
  sensors:
    weight: sensor.weight_aurelien
    impedance: sensor.impedance_aurelien
  users:
    - name: "Username 1"
      height: 176
      born: "1990-04-10"
      gender: "male"
      weight_low: 70
      weight_heigh: 100
    - name: "Username 2"
      height: 170
      born: "1990-01-12"
      gender: "female"
      weight_low: 0
      weight_heigh: 70

Each user would then get his own entity. For this to be possible, the values would probably also have to be persisted in core.restore_state.

Hello, at the beginning I was part in this direction then my wife and I we have a weight too close. So I went with this solution. In addition it keeps the data well after a restart.
https://community.home-assistant.io/t/exemple-xiaomi-miscale-select-the-person-who-weighs-himself-notif-action/299321

A way to segregate the users would be great.

Multiuser would be fine without nodered.
Anyone got this working without node red?
Best would be that dckiller can implement this.
If it is only a thing of changing the conig.yaml, can somebody give examples how to do?

I'll add a Plus one - this seems like the most logical extension of this.

plus 1 from my side as well!

My solution is to take the readings from the scale, pass through Node-RED to check who it is and then send to their own weight/imp sensors. It also solves the retained issue at the same time.

You must create one bodymiscale per user. The component is based on https://www.home-assistant.io/integrations/plant/

Dear @dckiller51,
Thanks a lot for your work! Can you please provide an example of configuration for this?

You must create one bodymiscale per user. The component is based on https://www.home-assistant.io/integrations/plant/

Dear @dckiller51, Thanks a lot for your work! Can you please provide an example of configuration for this?

You must create one bodymiscale per user. The component is based on https://www.home-assistant.io/integrations/plant/

hello, your request should not be here. but here is an example
https://community.home-assistant.io/t/exemple-xiaomi-miscale-select-the-person-who-weighs-himself-notif-action/299321

Hello , if i create one bodymiscale per user. Only one card show statistic but  others are unavailable ,till next messure, then new is showed other are unavailable.
https://ibb.co/mG2Fnrk

Hello , if i create one bodymiscale per user. Only one card show statistic but others are unavailable ,till next messure, then new is showed other are unavailable. https://ibb.co/mG2Fnrk https://ibb.co/YbLhsHG

Hello, Any updates on this? I have the same issue.

plus 1

plus 1

What is your configuration? it is important to have a weight and impedance sensor for each user.

I use templates to create a new weight sensor per person, then add that sensor to the person's bodymiscale instance.
Here's an example that assigns the new scale values to the closest weight sensor:
template:
  - trigger:
      - platform: state
        entity_id:
          - sensor.mi_smart_scale_mass
    sensor:
      - name: "Person 1 Mass"
        unit_of_measurement: "kg"
        state: >
          {% set weight = states('sensor.mi_smart_scale_mass') | float %}
          {% set distance_a = (states('sensor.person_1_mass') | float - weight) | abs %}
          {% set distance_b = (states('sensor.person_2_mass') | float - weight) | abs %}
          {% if distance_a < distance_b %}
            {{ weight }}
          {% else %}
            {{ states('sensor.person_1_mass') }}
          {% endif %}
      - name: "Person 2 Mass"
        unit_of_measurement: "kg"
        state: >
          {% set weight = states('sensor.mi_smart_scale_mass') | float %}
          {% set distance_a = (states('sensor.person_1_mass') | float - weight) | abs %}
          {% set distance_b = (states('sensor.person_2_mass') | float - weight) | abs %}
          {% if distance_b < distance_a %}
            {{ weight }}
          {% else %}
            {{ states('sensor.person_2_mass') }}
          {% endif %}

How about more users, like e.g. 4?

What if I have to distinguish impedance too, based on weight? How can it be guaranteed that the correct value will be set?

@maciek188, Following the same logic, something like this should work for 4 people:
Note: since HA doesn't initially know the values of sensor.person_1_mass, sensor.person_2_mass, sensor.person_3_mass and sensor.person_4_mass, they'll all be zeroes at first. You should set their proper values in Developer Tools.
template:
  - trigger:
      - platform: state
        entity_id:
          - sensor.mi_smart_scale_mass
    sensor:
      - name: "Person 1 Mass"
        unit_of_measurement: "kg"
        state: >
          {% set weight = states('sensor.mi_smart_scale_mass') | float %}
          {% set delta_a = (states('sensor.person_1_mass') | float - weight) | abs %}
          {% set delta_b = (states('sensor.person_2_mass') | float - weight) | abs %}
          {% set delta_c = (states('sensor.person_3_mass') | float - weight) | abs %}
          {% set delta_d = (states('sensor.person_4_mass') | float - weight) | abs %}
          {% if delta_a == [delta_a, delta_b, delta_c, delta_d, 5] | min %}
            {{ weight }}
          {% else %}
            {{ states('sensor.person_1_mass') }}
          {% endif %}
      - name: "Person 2 Mass"
        unit_of_measurement: "kg"
        state: >
          {% set weight = states('sensor.mi_smart_scale_mass') | float %}
          {% set delta_a = (states('sensor.person_1_mass') | float - weight) | abs %}
          {% set delta_b = (states('sensor.person_2_mass') | float - weight) | abs %}
          {% set delta_c = (states('sensor.person_3_mass') | float - weight) | abs %}
          {% set delta_d = (states('sensor.person_4_mass') | float - weight) | abs %}
          {% if delta_b == [delta_a, delta_b, delta_c, delta_d, 5] | min %}
            {{ weight }}
          {% else %}
            {{ states('sensor.person_2_mass') }}
          {% endif %}
      - name: "Person 3 Mass"
        unit_of_measurement: "kg"
        state: >
          {% set weight = states('sensor.mi_smart_scale_mass') | float %}
          {% set delta_a = (states('sensor.person_1_mass') | float - weight) | abs %}
          {% set delta_b = (states('sensor.person_2_mass') | float - weight) | abs %}
          {% set delta_c = (states('sensor.person_3_mass') | float - weight) | abs %}
          {% set delta_d = (states('sensor.person_4_mass') | float - weight) | abs %}
          {% if delta_c == [delta_a, delta_b, delta_c, delta_d, 5] | min %}
            {{ weight }}
          {% else %}
            {{ states('sensor.person_3_mass') }}
          {% endif %}
      - name: "Person 4 Mass"
        unit_of_measurement: "kg"
        state: >
          {% set weight = states('sensor.mi_smart_scale_mass') | float %}
          {% set delta_a = (states('sensor.person_1_mass') | float - weight) | abs %}
          {% set delta_b = (states('sensor.person_2_mass') | float - weight) | abs %}
          {% set delta_c = (states('sensor.person_3_mass') | float - weight) | abs %}
          {% set delta_d = (states('sensor.person_4_mass') | float - weight) | abs %}
          {% if delta_d == [delta_a, delta_b, delta_c, delta_d, 5] | min %}
            {{ weight }}
          {% else %}
            {{ states('sensor.person_4_mass') }}
          {% endif %}

Awesome template
how can we add impedance to that one
I notice that when HA is rebooted the new sensors has to be set in developer tools and I was wondering if there is a way to store the mesurments in permanent variable and we read that in that template ?


FileInfo.ExtractGZipToDirectory Not Working

I was unable to get ExtractGZipToDirectory() working. It just created an empty file when I tried to decompress it. I ended up creating this method which did work:
private static FileInfo DecompressGZipFile(this FileInfo compressedFilePath)
{
	using (var fs = compressedFilePath.OpenRead())
	{
		using (var zipStream = new GZipStream(fs, CompressionMode.Decompress))
		{
			var buffer1 = compressedFilePath.Name.Split('.');
			var buffer2 = buffer1[0] + "." + buffer1[1];
			var filePath = Path.Combine(compressedFilePath.DirectoryName, buffer2);

			using (var fOutStream = new FileStream(filePath, FileMode.Create, FileAccess.Write))
			{
				var tempBytes = new byte[4096];
				int i;
				while ((i = zipStream.Read(tempBytes, 0, tempBytes.Length)) != 0)
				{
					fOutStream.Write(tempBytes, 0, i);
				}
			}

			return new FileInfo(filePath);
		}
	}
}

Hello @adamfisher , @prasadtelkikar ,
Thank you for reporting and your interest.
It has been a long time since the last time we really make a big change to this project. However, I still use it on a daily usage so that's for sure a project I would like to keep up to date.
I will discuss with my team and check about the future of this project and what we could plan for it.
Best Regards,
Jonathan

@adamfisher ,I am new to this repo. I would like to contribute to this project.
Can you please provide me steps to reproduce this issue.

Just try using the ExtractGZipToDirectory() extension method on any zipped file and it fails.

@prasadtelkikar Any movement on this one?


Tighten Up Deployment Process

Off of the discussion with Wonderland
https://docs.google.com/presentation/d/1GNY9MxbyIyHl2XUuCQ6lHmdBH88U_7v1xA5vbb-8Vew/edit?usp=sharing
https://discord.com/channels/992549151134982234/1108087126907682866

Objective

do no keep private keys or mneumonics in configs

Two ways:

Only use encrypted keystores (encrypted json)
Have employer key used only during deployment process, then key has no impact on system

Complexity: 8
See the discussion in Sprint Poker #37
Powered by Parabol

potentially redo all in forge

Will happen as part of Chimera upgrade.


fix coverage for typescript (karma-remap-coverage)

I tested this setup with a few classes and it seems to give correct coverage reports 😃

Thanks !
There's still an issue with code typescript coverage due to the  regexp lying in test/unit/index.ts that returns no modules to require
So currently your typescript code coverage only includes source files that are required by your tests, but i'll fix this asap.

Another problem : it displays after running unit tests the following message
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): SyntaxError: Unrecognized report type of "lcov".

any idea ?

I guess lcov is not supported by karma-remap-coverage and remap-istanbul: https://github.com/SitePen/remap-istanbul/blob/684fb0e88b3673a1563c61b59cf3091f54c1ddae/tests/unit/lib/writeReport.js#L18
So I'd suggest to use only lcononly and html (by the way: the folder for html is currently pointing to ./test/unit/coverage/lcov-report.
By the way: nice to see that all sources are included now, nice fix! 👍

Yes that's what i've done with latest commit : https://github.com/Toilal/vue-webpack-template/commit/f506b513d9cdd43c42c1aa619ebeffa0be3fd9ce
Thank you too for help and engagement :)

You are welcome :)
By the way: did you advertise this project on https://github.com/vuejs/vue-cli/issues/263 ? Others might also like it!

Done thanks :)
I a near future, we could ask for this repository to become an official template hosted at https://github.com/vuejs-templates


HvEptAddPageHook failing

This is an issue that I've experienced every time I load the hypervisor. I'm using 'dsefix' to disable driver signature enforcement, and using the Windows 'sc' service manager to start the driver. @autisticlittleguy mentioned that he had the same problem in issue #2 .
I've added some more debug prints and found out that the failure in HvEptGetPml1Entry lies here:
	if(!PML1) 	{ 		HvUtilLogError("HvEptGetPml1Entry: !PML1\n"); 		return NULL; 	}
This seems to indicate that the OsPhysicalToVirtual function is failing for some reason. I haven't made any other modifications to the source besides adding debug prints.

Do you know if this is a problem only on 2004?

I only have a test environment setup for 2004, so I am unsure. I assumed it had something to do with the Windows version, because it seems like the hypervisor has been extensively tested on other versions and works fine.

I've printed PML2Pointer->PageFrameNumber, and that seems to be valid each time, so the issue is in the call to MmGetVirtualForPhysical itself. I've done a bit of research, and I don't see any obvious changes to this function in Windows 2004...

Very strange. Yeah, I can't say I have ever seen this issue myself, but my
current hypothesis is something to do with 2004. This could also be a
problem with a newer version of vmware as well. To rule that out, could you
potentially test it on an 1809 iso? You don't need to do dsefix, just
booting into test signing mode will be sufficient.
On Mon, Oct 5, 2020, 1:03 PM CJ Sculti<EMAIL_ADDRESS>wrote:

I've printed PML2Pointer->PageFrameNumber, and that seems to be valid each
time, so the issue is in the call to MmGetVirtualForPhysical itself. I've
done a bit of research, and I don't see any obvious changes to this
function in Windows 2004...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Gbps/gbhv/issues/18#issuecomment-703762509, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKZAKDMWFRFLMNRM3KEWJDSJH36JANCNFSM4SE3X2UA
.

Right now, I'm not even using a VM, just testing on my main Windows install. If I get time later, I'll setup an 1809 VM and try it out.
@autisticlittleguy are you experiencing this same issue on Windows 2004, or another version?

What is the address of the hook target passed to HvEptAddPageHook?

[DEBUG] TargetFunction fffff8012dc1d2f0 HookFunction fffff8018c0c2060 OrigFunction fffff8018c0ca030 PhysicalAddress from OsVirtualToPhysical: 2e1d000
The last address is the one returned from this call:
	PhysicalAddress = (SIZE_T) OsVirtualToPhysical(VirtualTarget); 	HvUtilLogDebug("TargetFunction %llx HookFunction %llx OrigFunction %llx PhysicalAddress from OsVirtualToPhysical: %llx \n", TargetFunction, HookFunction, OrigFunction, PhysicalAddress);

That seems to be in order as well. I wonder if this is a new security protection that was slid in.

@Gbps @Seegee
My apologies for answering in an untimely manner, i lost my other github account due to using a temp-mail. That being said:
I'm experiencing the same issue on both vmware and a physical machine. The issue seems to present itself randomly, the hook can work on some processors and completely fail on others. The check it fails is SEEMS to ALWAYS be the one that happens after OsPhysicalToVirtual is called. The PageFrameNumber on the PML2Pointer is not null, neither is the PML2Pointer, so that makes me think it could either be a wrong PageFrameNumber, or the OsPhysicalToVirtual function failing. Sometimes though, it all works smoothly.
Windows verison: 2004.
My prints atm:

        /* Translate to the PML1 pointer */
	PML1 = (PEPT_PML1_ENTRY) OsPhysicalToVirtual((PVOID)(PML2Pointer->PageFrameNumber * PAGE_SIZE));

	HvUtilLogDebug("PML2Pointer->PageFrameNumber = %p\n", PML2Pointer->PageFrameNumber);

	if(!PML1)
	{
		HvUtilLogDebug("Couldn't translate physical memory %p to virtual memory %p, aborting...\n", PhysicalAddress, PML1);
		return NULL;
	}

	/* Index into PML1 for that address */
	PML1 = &PML1[ADDRMASK_EPT_PML1_INDEX(PhysicalAddress)];

	HvUtilLogDebug("Translated PML1 for physical address %p: %p\n", PhysicalAddress, PML1);

Hmm... also Windows version 2004.

@autisticlittleboy for me, it happens 100% of the time on all processors. I'm never able to get the hook to properly initialize.

I doubt the PageFrameNumber is incorrect, so the only real unknown is the translation back to virtual failing. That, or the translation to physical was incorrect to begin with.

@Seegee Hmm. I've only tested hooking syscalls so far, try hooking something that's not a syscall, maybe some random function in ntoskrnl and see if it's still failing with the OsPhysicalToVirtual check, if not, maybe the issue is within the EPT configuration. Just throwing some thoughts here since this issue is way weirder than others i encountered.

I wonder if this is related to some of the additional protections placed on MmMapIoSpace where it forbid you from mapping certain areas of physical memory from the API level. It could potentially be that in 2004, there was an edit that affected the ability to translate these mappings freely.

Looking at an old article from Sina Karvandi (https://rayanfam.com/topics/inside-windows-page-frame-number-part2/), it looks like the MiGetPhysicalAddress function has, somewhat, changed. I've never done address translations manually but if the OS is trying to prevent us from getting the physical address this may be a way to get around it.

Well that was back in 2018, which I never encountered this issue or anyone else around that time. Technically, NtCreateFile is in paged memory, which could potentially be paged out at the time of hook. A call to MmProbeAndLockPages on the target virtual address would at least lock them in, which currently the code base does not do.
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-mmprobeandlockpages
It would be very curious, though, to ever see NtCreateFile paged out. This however might be different if you guys are trying to hook a different systemcall that is less used. I guess it's worth a shot to try to lock them prior, this would match up with the seemingly randomness of @autisticlittleboy 's experience

I've seen hyperplatform and its sub projects use MmGetVirtualForPhysical and MmGetPhysicalAddress, so i doubt these functions are at fault here. I'll try locking the pages in to see what happens.

I am trying to hook NtCreateFile. I don’t know too much about Windows
memory management but I assumed that NtCreateFile is called enough to not
have to worry about it being paged out.
On Mon, Oct 5, 2020 at 2:08 PM Gbps<EMAIL_ADDRESS>wrote:

Well that was back in 2018, which I never encountered this issue or anyone
else around that time. Technically, NtCreateFile is in paged memory, which
could potentially be paged out at the time of hook. A call to
MmProbeAndLockPages on the target virtual address would at least lock
them in, which currently the code base does not do.
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-mmprobeandlockpages
It would be very curious, though, to ever see NtCreateFile paged out.
This however might be different if you guys are trying to hook a different
systemcall that is less used. I guess it's worth a shot to try to lock them
prior, this would match up with the seemingly randomness of
@autisticlittleboy https://github.com/autisticlittleboy 's experience
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Gbps/gbhv/issues/18#issuecomment-703797724, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AA26O6SQFABMCRNSF42E5RLSJIDQVANCNFSM4SE3X2UA
.

@Seegee
I think you were right @Gbps
It worked for 4 consecutive system restarts after locking the page for read access:
PMDL NtCreateFileMDL = IoAllocateMdl(PAGE_ALIGN(NtCreateFile), PAGE_SIZE, FALSE, TRUE, NULL);
MmProbeAndLockPages(NtCreateFileMDL, KernelMode, IoReadAccess);
HvEptAddPageHook(ProcessorContext, (PVOID)NtCreateFile, (PVOID)NtCreateFileHook, (PVOID*)&NtCreateFileOriginal);
MmUnlockPages(NtCreateFileMDL);

Of course if you use a DSE bypass technique that maps your driver you won't have SEH and you'll error out horribly if an exception is thrown, but that's fine by me in my project. I do not want anything to run if hooks fail.

It might be a good idea to lock the pages beforehand and only unlock them once the hypervisor is done placing its hooks and virtualizing the system, i'm still restarting and virtualizing the system over and over to check whether it's really fixed or not.

It looks like it failed on the 8th restart for one of the processors:
[*] HvInitializeAllProcessors: Starting.
[*] Total Processor Count: 2
[DEBUG] EPT PML4 is<PHONE_NUMBER>000000
[DEBUG] EPT PML2 is FFFFB378CA8020B8 large page is<PHONE_NUMBER>000000
[DEBUG] PML2Pointer->PageFrameNumber =<PHONE_NUMBER>007B4C
[DEBUG] Couldn't translate physical memory<PHONE_NUMBER>E1D000 to virtual memory<PHONE_NUMBER>000000, aborting...
[!] HvEptAddPageHook: Failed to get PML1 entry for target address.
[*] HvInitializeLogicalProcessor[#0]: Allocated Context [Context = 0xffffe28fbd94c000]
[DEBUG] EPT PML4 is<PHONE_NUMBER>000000
[DEBUG] EPT PML2 is FFFFB378CAC020B8 large page is<PHONE_NUMBER>000000
[DEBUG] PML2Pointer->PageFrameNumber =<PHONE_NUMBER>050039
[DEBUG] Translated PML1 for physical address<PHONE_NUMBER>E1D000: FFFFE28FBDBFE0E8
[DEBUG] OffsetIntoPage: 0x2f0
[DEBUG] Number of bytes of instruction mem: 14
[DEBUG] Trampoline: 0xffffe28fb7902a90
[DEBUG] HookFunction: 0xfffff80644773140
[*] HvInitializeLogicalProcessor[#1]: Allocated Context [Context = 0xffffe28fbda6b000]
[*] VmcsRevisionNumber: 1
[+] HvEptCheckFeatures: All EPT features present.
[DEBUG] EPT: Number of dynamic ranges: 8
[DEBUG] MTRR Range: Base=0x0 End=0xFFFFFFFFF Type=0x6
[DEBUG] MTRR Range: Base=0xC0000000 End=0xFFFFFFFF Type=0x0
[DEBUG] Total MTRR Ranges Committed: 1
[DEBUG] VmxOnRegion[#0]: (V) 0xffffa38080c33000 / (P) 0x7ff5e000 [1]
[DEBUG] VmxOnRegion[#1]: (V) 0xffffa38080c71000 / (P) 0x7fe7e000 [1]
[DEBUG] HvSetupVmcsControlFields: VmError = 0
[DEBUG] HvSetupVmcsControlFields: VmError = 0
[DEBUG] GdtRegister: 0xfffff60c60a37940, Base: 0xffffa380807eafb0, Limit: 0x57
[DEBUG] GdtRegister: 0xfffff60c629ef030, Base: 0xfffff80642264fb0, Limit: 0x57
[DEBUG] VmxLaunchProcessor: VMLAUNCH....
[DEBUG] VmxLaunchProcessor: VMLAUNCH....
[+] HvInitializeAllProcessors: Success.

@Seegee
I think you were right @Gbps
It worked for 4 consecutive system restarts after locking the page for read access:
PMDL NtCreateFileMDL = IoAllocateMdl(PAGE_ALIGN(NtCreateFile), PAGE_SIZE, FALSE, TRUE, NULL);
MmProbeAndLockPages(NtCreateFileMDL, KernelMode, IoReadAccess);
HvEptAddPageHook(ProcessorContext, (PVOID)NtCreateFile, (PVOID)NtCreateFileHook, (PVOID*)&NtCreateFileOriginal);
MmUnlockPages(NtCreateFileMDL);

Of course if you use a DSE bypass technique that maps your driver you won't have SEH and you'll error out horribly if an exception is thrown, but that's fine by me in my project. I do not want anything to run if hooks fail. Also i mean, we do have a subverted processor, either kill patchguard and get SEH by adding your invalid mapped region to the tables or get creative :)

Still fails for me on all 12 processors when locking the page like this.

I appreciate the testing you all our doing. I wish I had a better explanation.

Thinking more about it, it's highly likely this is related to the page splitting code in HvEptSplitLargePage, since that code is responsible for generating the pointer to the PML1 entries that are retrieved by the function in question.
Specifically, the value of PageFrameNumber in the PML1 translation function should always be the same as the one set here:
https://github.com/Gbps/gbhv/blob/master/gbhv/ept.c#L421
Which would end up being wrong if:
(SIZE_T)OsVirtualToPhysical(&NewSplit->PML1[0])

Returned the wrong value. But again, never seen this return the wrong value here.
Worth tracing output to ensure that NewPointer.PageFrameNumber (SplitPage) matches PML2Pointer->PageFrameNumber (GetPML1) and NewPointer is the same virtual address as PML2Pointer.

Traced some more callls. Unsure how to interpret the result here. I need to really give a good read to the manual.
[DEBUG] Page split! NewPointer = 000000006ADF9007
[DEBUG] Page split! NewPointer.PageFrameNumber=<PHONE_NUMBER>06ADF9
[DEBUG] EPT PML4Entry is<PHONE_NUMBER>000000
[DEBUG] EPT PML2Pointer is FFFFBC7DB52020B8 PML2->LargePage =<PHONE_NUMBER>000000
[DEBUG] PML2Pointer->PageFrameNumber =<PHONE_NUMBER>06ADF9
[DEBUG] Couldn't translate physical memory<PHONE_NUMBER>E1D000 to virtual memory<PHONE_NUMBER>000000, aborting...
[!] HvEptAddPageHook: Failed to get PML1 entry for target address.
[*] HvInitializeLogicalProcessor[#0]: Allocated Context [Context = 0xffffe30534725000]
[DEBUG] Page split! NewPointer =<PHONE_NUMBER>BF5007
[DEBUG] Page split! NewPointer.PageFrameNumber=<PHONE_NUMBER>024BF5
[DEBUG] EPT PML4Entry is<PHONE_NUMBER>000000
[DEBUG] EPT PML2Pointer is FFFFBC7DB56020B8 PML2->LargePage =<PHONE_NUMBER>000000
[DEBUG] PML2Pointer->PageFrameNumber =<PHONE_NUMBER>024BF5
[DEBUG] Translated PML1 for physical address<PHONE_NUMBER>E1D000 in virtual is: FFFFE3053365A0E8
[DEBUG] OffsetIntoPage: 0x2f0
[DEBUG] Number of bytes of instruction mem: 14
[DEBUG] Trampoline: 0xffffe3052d962a90
[DEBUG] HookFunction: 0xfffff80428eb3120
[*] HvInitializeLogicalProcessor[#1]: Allocated Context [Context = 0xffffe30534959000]
[*] VmcsRevisionNumber: 1
[+] HvEptCheckFeatures: All EPT features present.
[DEBUG] EPT: Number of dynamic ranges: 8
[DEBUG] MTRR Range: Base=0x0 End=0xFFFFFFFFF Type=0x6
[DEBUG] MTRR Range: Base=0xC0000000 End=0xFFFFFFFF Type=0x0
[DEBUG] Total MTRR Ranges Committed: 1
[DEBUG] VmxOnRegion[#0]: (V) 0xffffac8013cbe000 / (P) 0x7dbb5000 [1]
[DEBUG] VmxOnRegion[#1]: (V) 0xffffac8013d11000 / (P) 0x7dbac000 [1]
[DEBUG] HvSetupVmcsControlFields: VmError = 0
[DEBUG] GdtRegister: 0xffff8108147c2030, Base: 0xfffff80426664fb0, Limit: 0x57
[DEBUG] HvSetupVmcsControlFields: VmError = 0
[DEBUG] GdtRegister: 0xffff810812637940, Base: 0xffffac80137eafb0, Limit: 0x57
[DEBUG] VmxLaunchProcessor: VMLAUNCH....
[DEBUG] VmxLaunchProcessor: VMLAUNCH....
[+] HvInitializeAllProcessors: Success.
[DEBUG] EPT Violation => 0x2E1D2F0
[+] Made Exec

Thanks again, that's very helpful.
Can you give me the value of &NewSplit->PML1[0] and the value of OsVirtualToPhysical(&NewSplit->PML1[0]) at this line for a successful and unsuccessful hook:
https://github.com/Gbps/gbhv/blob/master/gbhv/ept.c#L421
Hopefully that will tell the story.

Alright here's the logs with the PML1 added (dual core vm, Windows v2004, one core fails one succeeds):
[*] HvInitializeAllProcessors: Starting.
[*] Total Processor Count: 2
[DEBUG] Page split! NewPointer =<PHONE_NUMBER>366007
[DEBUG] Page split! NewPointer.PageFrameNumber=<PHONE_NUMBER>023366
[DEBUG] Page split! &NewSplit->PML1[0] = FFFFAE8F2C966000
[DEBUG] Page split! (SIZE_T)OsVirtualToPhysical(&NewSplit->PML1[0]) =<PHONE_NUMBER>366000
[DEBUG] EPT PML4Entry is<PHONE_NUMBER>000000
[DEBUG] EPT PML2 is FFFFD87FB58020B8 PML2->LargePage =<PHONE_NUMBER>000000
[DEBUG] PML2Pointer->PageFrameNumber =<PHONE_NUMBER>023366
[DEBUG] Couldn't translate physical memory<PHONE_NUMBER>E1D000 to virtual memory<PHONE_NUMBER>000000, aborting...
[!] HvEptAddPageHook: Failed to get PML1 entry for target address.
[*] HvInitializeLogicalProcessor[#0]: Allocated Context [Context = 0xffffae8f2e2b2000]

[DEBUG] Page split! NewPointer =<PHONE_NUMBER>751007
[DEBUG] Page split! NewPointer.PageFrameNumber=<PHONE_NUMBER>042751
[DEBUG] Page split! &NewSplit->PML1[0] = FFFFAE8F2E6BF000
[DEBUG] Page split! (SIZE_T)OsVirtualToPhysical(&NewSplit->PML1[0]) =<PHONE_NUMBER>751000
[DEBUG] EPT PML4Entry is<PHONE_NUMBER>000000
[DEBUG] EPT PML2 is FFFFD87FB5C020B8 PML2->LargePage =<PHONE_NUMBER>000000
[DEBUG] PML2Pointer->PageFrameNumber =<PHONE_NUMBER>042751
[DEBUG] Translated PML1 for physical address<PHONE_NUMBER>E1D000 in virtual is: FFFFAE8F2E6BF0E8
[DEBUG] OffsetIntoPage: 0x2f0
[DEBUG] Number of bytes of instruction mem: 14
[DEBUG] Trampoline: 0xffffae8f273e5a90
[DEBUG] HookFunction: 0xfffff8020fff3170
[*] HvInitializeLogicalProcessor[#1]: Allocated Context [Context = 0xffffae8f2e26f000]

[*] VmcsRevisionNumber: 1
[+] HvEptCheckFeatures: All EPT features present.
[DEBUG] EPT: Number of dynamic ranges: 8
[DEBUG] MTRR Range: Base=0x0 End=0xFFFFFFFFF Type=0x6
[DEBUG] MTRR Range: Base=0xC0000000 End=0xFFFFFFFF Type=0x0
[DEBUG] Total MTRR Ranges Committed: 1
[DEBUG] VmxOnRegion[#0]: (V) 0xffffc88050c27000 / (P) 0x7dc4d000 [1]
[DEBUG] VmxOnRegion[#1]: (V) 0xffffc88050c3c000 / (P) 0x7db2b000 [1]
[DEBUG] HvSetupVmcsControlFields: VmError = 0
[DEBUG] HvSetupVmcsControlFields: VmError = 0
[DEBUG] GdtRegister: 0xffffac888c82efb0, Base: 0xffffc880507eafb0, Limit: 0x57
[DEBUG] GdtRegister: 0xffffac888cbe7030, Base: 0xfffff8020d864fb0, Limit: 0x57
[DEBUG] VmxLaunchProcessor: VMLAUNCH....
[DEBUG] VmxLaunchProcessor: VMLAUNCH....
[+] HvInitializeAllProcessors: Success.
[DEBUG] EPT Violation => 0x2E1D2F0
[+] Made Exec

Logging code for reference (OsVirtualToPhysical wasn't printed as it's used (You split it by PAGE_SIZE)):
NewPointer.Flags = 0;
NewPointer.WriteAccess = 1;
NewPointer.ReadAccess = 1;
NewPointer.ExecuteAccess = 1;
NewPointer.PageFrameNumber = (SIZE_T)OsVirtualToPhysical(&NewSplit->PML1[0]) / PAGE_SIZE;

HvUtilLogDebug("Page split! NewPointer = %p\n", NewPointer);
HvUtilLogDebug("Page split! NewPointer.PageFrameNumber= %p\n", NewPointer.PageFrameNumber);
HvUtilLogDebug("Page split! &NewSplit->PML1[0] = %p\n", &NewSplit->PML1[0]);
HvUtilLogDebug("Page split! (SIZE_T)OsVirtualToPhysical(&NewSplit->PML1[0]) = %p\n" (SIZE_T)OsVirtualToPhysical(&NewSplit->PML1[0]));

Well, those values also look completely normal and expected! It just absolutely seems like OsPhysicalToVirtual is just straight up failing.
I'll keep brainstorming...

 Would adding a third parameter to HvEptGetPml1Entry that simply contains the expected virtual address work? We can proceed with the regular calculation via API, if the API returns zero, we use what was passed in the parameter and hope for the best. Janky, stupid solution that definitely shouldn't be used on production software but it could work, since the other processor suceeds just fine. This is definitely one of the dumbest solutions i came up with but i'll try it.

Without figuring out the root cause of the issue, it will cause further bugs later down the line. I'm still looking into why it's happening right now.

 I then tested on a Windows 10 2004 VM, and it failed, as it did on my desktop.

That certainly narrows some stuff down. I'll take a look at it over the
weekend and see if I can figure it out.
 I then tested on a Windows 10 2004 VM, and it
failed, as it did on my desktop.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Gbps/gbhv/issues/18#issuecomment-705788856, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKZAKC7E5BDCDQPNFDDRDTSJYKGLANCNFSM4SE3X2UA
.

I'm not sure how relevant this is but https://github.com/hyperdbg/hyperdbg uses the same method to gather the PML1Entry.  I can't get that vmm to run since it's giving weird issues where vmware simply freezes but maybe you can get it to run @Seegee . If you do, try placing ept hooks with !epthook, documentation to do it should be in the repo. If they work in a project that uses such a similar method to grab the PML1Entry we might be able to get some clues as to why it's not working in this project. If it doesn't work there either, well, that'll be interesting. I'll try getting hyperdbg to work again tomorrow. "load vmm" simply freezes my whole machine rn.

I think I've located the problem, as I got it to repro on my machine... will update with more information if I'm able to fix it.

@autisticlittleboy @Seegee
I've created a new branch which I believe resolves the issue:
https://github.com/Gbps/gbhv/tree/win10-2004
You can either compile it from source, or use the binary under release/. Please let me know if this resolves this bug for you so I can go ahead and merge this into master. As for root cause, below is the comment I added to ept.c talking about it:
	/*
	* Allocate the PML1 entries for the split.
	* NOTE: This would *not* need to use contiguous aligned pages normally, except for a bug which is experienced
	* in Windows 10 v2004 where changes to the nonpaged pool allocator resulted in some page aligned allocations
	* being mapped as 4MB large pages rather than the expected 4KB pages. This causes the following VtoP and PtoV
	* functions to fail, because the Mm APIs are not able to properly translate physical addresses within a large page
	* back to its virtual address due to a null PTE pointer inside the PFN database entry for the large page.
	*
	* From my testing, I was unable to find a way to coerce Mm to split a nonpaged pool large page for me, so the best
	* alternative was to use the contiguous aligned pages allocator because, in my testing, it resulted in only 4KB virtual
	* allocations. This allocator also utilizes nonpaged pool frames, so it is more-or-less the same as the other allocator.
	*/

Worked perfectly on windows v2004 on a laptop that i previously wouldn't have been able to place any hooks on. I'll keep it running overnight to see if there's any stability issues and i'll try loading the hypervisor several times to see if the issue is still present. Thank you very much for taking the time to look into this. @Gbps

Awesome! Glad to hear it.
On Sat, Oct 10, 2020, 10:02 AM autisticlittleboy<EMAIL_ADDRESS>wrote:

Worked perfectly on windows v2004 on a laptop that i previously wouldn't
have been able to place any hooks on. I'll keep it running overnight to see
if there's any stability issues and i'll try loading the hypervisor several
times to see if the issue is still present. Thank you very much for taking
the time to look into this. @Gbps https://github.com/Gbps
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Gbps/gbhv/issues/18#issuecomment-706580222, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKZAKDUFAK4HXWTH3EGII3SKCHTNANCNFSM4SE3X2UA
.

Closing this as it was fixed in #19 .


FAPI-CIBA Backchannel Authentication Request : encrypt/decrypt login_hint

The design document for FAPI-CIBA states that it requires login_hint encryption and decryption.
However, the current CIBA prototype implementation does not support it.
The goal of this issue is to implement on the CIBA prototype implementation login_hint encryption and decryption by following the design document for FAPI-CIBA.

Could take over this one from our team side.

Thank you @DmitryMishchuk

Hello @DmitryMishchuk ,
FYI, the User Resolver described in page 50 of the CIBA Prototype Implementation Practical Guide might be helpful.

@tnorimat It states that login_hint should be encrypted/decrypted  using client secret with AES128/192/256bit
But client_secret is an UUID it is 36 bytes long, so it is not good as a key for encryption/decryption.

Hello @DmitryMishchuk ,
IMO, client_secret is generated by UUID.randomUUID().toString() in keycloak when creating a client and according to oracle's javadoc, UUIDv4 is generated by UUID.randomUUID().toString() using a cryptographically strong pseudo random number generator. According to RFC 4122, UUID is 128 bits long.
Considering these points, binary representation of client_secret might be 128bit. It might be OK to use it for AES 128bit key.
I'm not sure what I've mentioned just above is right or not. Could you review it?

Yes you're absolutely right. Thank you.

merged. https://github.com/tnorimat/keycloak/pull/5


Limited list of enpoint protection solutions

Why are there only a few solutions outside of Microsoft included? Will other EDR solutions be added? For example, CrowdStrike, SentinelOne, CyberReason, Cylance, etc.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 9443deec-cad8-89cc-9938-77ebe66a0dd9
Version Independent ID: 1c0030c2-0552-d50b-0d2b-5127d0d8f701
Content: Assessment checks for endpoint detection and response solutions - Microsoft Defender for Cloud
Content Source: articles/defender-for-cloud/endpoint-protection-recommendations-technical.md
Service: defender-for-cloud
GitHub Login: @dcurwin
Microsoft Alias: dacurwin

@brussrus
Thanks for your feedback! We will investigate and update as appropriate.

@brussrus The ones listed are the only ones currently supported. For any request for future supportability please create a support ticket.

#please-close

@brussrus
you create a support ticket since the support team will be able to respond much more quickly and have a conversation with you to figure out what could be going on.


shadow map prefiltering for VSM

Very excited about the incoming VSM changes. thanks for adding it!!
I believe shadow maps prefiltering work with VSM but not PCF.  Can we have some support for that so we can apply a blur directly on the shadow map for really soft shadow?
or any pointers for where is the best place to add a temp workaround would be much appreciated. looks like shadow map render pass is currently outside of frame graph, wanted to minimize engine change to avoid future merge headaches.
thanks again

The whole point of adding VSM is to do prefiltering. We're working on this.


fix: pin go version

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Requirements:

[ ] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] tested upgrade from previous version

Special notes for your reviewer:
Release note:
none

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall first build on amaheshwari/goversion at 61.355%

Totals

Change from base Build<PHONE_NUMBER>:
61.4%

Covered Lines:
2653

Relevant Lines:
4324

💛  - Coveralls


[OpenAI-Assistant] Added vector store feature

Added support for vector store, vector store with file and a batch of files operations.

CreateVectorStoreFileRequest

No worries. I am able to record the new tests' json file and works fine in playback mode now.
Will address the unresolve feebacks soon


Add ability to use custom cluster login commands

Accepts arguments and config values (and env variables) for setting
custom cluster login commands.  Defaults to $SHELL for the shell,
ocm backplane login for the login command, and
/usr/bin/gnome-terminal for the terminal.
Fixes #3
Fixes #6
Signed-off-by: Chris Collins<EMAIL_ADDRESS>

/lgtm


Q: Support for client credential and password grants

It seems that currently this library concentrates on supporting OAuth2 authorization code grants.
Are there any plans to support also client credential and password grants?
For the time being, would you recommend to extend the OpenIDConnectFilter to support also other type of OAuth2ProtectedResourceDetails implementations than currently limiting it to OpenIDAuthorizationCodeResourceDetails. Any hints very appreciated, thanks in advance!

Hi @nikos thanks for the question.
FusionAuth does not currently support the client credentials grant, this will likely be coming this summer with a suite of IoT features.
We could look into adding the Password grant to this library, or if you'd like to submit a PR that would be great as well.
I'm not a Spring expert at all...   so without digging back into the code your guess is as good as mine. If I have some cycles to take a look and see if I can stub it out I'll report back.
If you're looking to use the password or credentials grant, does this mean you'll be using it for something other than a normal web login workflow?

Hi,
I recently heard that spring started including an OpenID Connect feature and I haven't had the time to look into it. When I get the chance I will update our example code to use the latest spring features.

I upvoted this issue because i encountered the exact same problem. The filter given in this library is only useful for the Authorization Code grant, by analyzing the token in response of the redirectUri after a succesful login.
I made a custom TokenFilter that i register in the security chain, which check the validity of the token (with the public certificate), and then extract the informations needed in the JWT to create an authenticated user, with authorities extracted from the 'roles' key in the claims.
If i have time, i'll submit a PR of my work to help people, but for the moment the class is not generic at all and only fits to my specific needs...

@tyduptyler13

I recently heard that spring started including an OpenID Connect feature and I haven't had the time to look into it. When I get the chance I will update our example code to use the latest spring features.

Hi Typler, are you refering to the changes in Spring Security 5.1.x ?
https://github.com/spring-projects/spring-security/wiki/OAuth-2.0-Features-Matrix
Might be quite an impact worth a new branch since there seem many changes compared to Spring Security OAuth 2.x

@damienherve  my current approach is to  make use of Spring's "standard" class  when it comes to the password grants flow: org.springframework.security.oauth2.client.token.grant.password.ResourceOwnerPasswordResourceDetails and add userInfoUri for accessing OpenID metadata.
OpenIDResourceDetails
package io.fusionauth.patch.oauth;

import org.springframework.security.oauth2.client.resource.OAuth2ProtectedResourceDetails;

public interface OpenIDResourceDetails extends OAuth2ProtectedResourceDetails {

    String getUserInfoUri();

    void setUserInfoUri(String userInfoUri);

}

OpenIDPasswordResourceDetails
package io.fusionauth.patch.oauth;

import org.springframework.security.oauth2.client.token.grant.password.ResourceOwnerPasswordResourceDetails;

public class OpenIDPasswordResourceDetails extends ResourceOwnerPasswordResourceDetails implements OpenIDResourceDetails {

    private String userInfoUri;

    public String getUserInfoUri() {
        return userInfoUri;
    }

    public void setUserInfoUri(String userInfoUri) {
        this.userInfoUri = userInfoUri;
    }

}

OpenIDConnectFilter
package io.fusionauth.patch.oauth;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.oauth2.client.OAuth2RestOperations;
import org.springframework.security.oauth2.common.OAuth2AccessToken;
import org.springframework.security.oauth2.common.exceptions.OAuth2Exception;
import org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter;
import org.springframework.web.client.RestTemplate;

import java.io.IOException;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import io.fusionauth.security.FusionAuthUserDetails;

// Patches io.fusionauth.security.OpenIDConnectFilter
public class OpenIDConnectFilter extends AbstractAuthenticationProcessingFilter {

    private final OpenIDResourceDetails openIDResourceDetails;
    private final OAuth2RestOperations restTemplate;

    public OpenIDConnectFilter(String defaultFilterProcessesUrl, OpenIDResourceDetails openIDResourceDetails, OAuth2RestOperations restTemplate) {
        super(defaultFilterProcessesUrl);
        this.openIDResourceDetails = openIDResourceDetails;
        this.restTemplate = restTemplate;
        setAuthenticationManager(new NoopAuthenticationManager());
    }

    @Override
    public Authentication attemptAuthentication(HttpServletRequest request, HttpServletResponse response) throws AuthenticationException {
        OAuth2AccessToken accessToken;
        try {
            accessToken = restTemplate.getAccessToken();
        } catch (final OAuth2Exception e) {
            throw new BadCredentialsException("Could not obtain access token", e);
        }
        try {
            FusionAuthUserDetails user = new FusionAuthUserDetails(getUserInfo(accessToken), accessToken);
            return new UsernamePasswordAuthenticationToken(user, null, user.getAuthorities());
        } catch (Exception e) {
            throw new BadCredentialsException("Failed to validate the token", e);
        }
    }

    private JsonNode getUserInfo(OAuth2AccessToken accessToken) throws IOException {
        HttpHeaders headers = new HttpHeaders();
        headers.set("Authorization", "Bearer " + accessToken.getValue());

        HttpEntity<String> httpEntity = new HttpEntity<>(headers);
        ResponseEntity<String> response = new RestTemplate().exchange(openIDResourceDetails.getUserInfoUri(), HttpMethod.GET, httpEntity, String.class);
        if (response.getStatusCode() == HttpStatus.OK) {
            return new ObjectMapper().readTree(response.getBody());
        }

        throw new BadCredentialsException("Failed to request user details from the UserInfo API. " +
                "Status code [" + response.getStatusCodeValue() + "] Message [" + response.getBody() + "]");
    }

    private static class NoopAuthenticationManager implements AuthenticationManager {
        @Override
        public Authentication authenticate(Authentication authentication) throws AuthenticationException {
            throw new UnsupportedOperationException("No authentication should be done with this AuthenticationManager");
        }
    }
}

I started working on a new example, based on how its going I think this repo will be deprecated in favor of just using spring security 5.
https://github.com/FusionAuth/fusionauth-spring-security-example/tree/SpringSecurity5

After further experimentation with Spring Security 5.1 it turns out that with a valid JWT at hand, securing your resources in your Spring Boot application is as straight forward as:
@Configuration
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class SecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    public void configure(WebSecurity web) {
        web.ignoring().antMatchers("/resources/**", "/error");
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
                .csrf().disable()
                .authorizeRequests()
                .mvcMatchers("/products/**").hasAuthority("MYROLE")
                .and()
                  .oauth2ResourceServer()
                    .jwt()
                      .jwkSetUri("http://localhost:9011/.well-known/jwks.json")
                      .jwtAuthenticationConverter(grantedAuthoritiesExtractor())
        // this disables session creation on Spring Security
        //.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS)
        ;
    }

    private Converter<Jwt, AbstractAuthenticationToken> grantedAuthoritiesExtractor() {
        return new JwtAuthenticationConverter() {
            @Override
            protected Collection<GrantedAuthority> extractAuthorities(Jwt jwt) {
                Collection<String> authorities = (Collection<String>) jwt.getClaims().get("roles");
                return authorities.stream()
                        .map(SimpleGrantedAuthority::new)
                        .collect(Collectors.toList());
            }
        };
    }
}

Does anyone have any news for us non-java devs on how this may be going? Is there a simple method for allowing our customers apps to authenticate with our APIs using client_credentials?

Any update to support client_credentials grant type? thank you

FusionAuth does not currently support the client credentials grant.
See FusionAuth feature request and ensure to upvote it.
https://github.com/FusionAuth/fusionauth-issues/issues/155
If you have a business requirement for this, hit the FusionAuth Contact form and we can provide you a quote with a timeline.
Thanks!


Update browser title from Liunx to Linux

@jaredr80,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

@jaredr80 : Thanks for your contribution to the Azure documentation! The author, @zivr, has been notified to review your proposed change.

Thanks for the feedback. The public doc has already been updated to address this issue. Apologies that the original author did not incorporate these updates earlier.
#please-close


Also >= <= and <> was not being parsed correctly.
Should be good now.

Coverage increased (+0.02%) to 92.499% when pulling 177e68a887a19871e1908a3c0cfce693e4ac79b8 on terrycain:fix_1261 into f3623e3cd3845411a2ff31a71b83f412e86f6a41 on spulec:master.

Hold off on this probs can fix #1263 here too

Coverage decreased (-0.003%) to 92.477% when pulling 78526d623fbe91708721c46ca6dbbcb4867839fe on terrycain:fix_1261 into f3623e3cd3845411a2ff31a71b83f412e86f6a41 on spulec:master.

Ok, I think that'll do it. Though I'm sure the community will find more bugs now that FilterExpressions are no longer ignored :wink:, but if they do I'll fix them when they come up

Great work, @terrycain !

This is released as Moto 1.1.23

Don't think this is 100% yet.  Seems be some issues with AND / OR.  For example: Attr('epoch_ts').lt(cutoff) & Attr('fanout_ts').not_exists()  results in  Bad Filter Expression: Malformed filter expression

@batkuip Yeah there's going to be some issues with it, raise an issue with that expression and I'll look into it


Dropbear doesn't seem to support dss keys anymore.

It seems that dropbear has dropped support for dss keys, probably for security reasons.
Here
https://bugs.archlinux.org/task/60523
it is mentioned that dropbear recently has improved its security.
Error output:
When running mkinitcpio with the dropbear hook, i get following error:
  -> Running build hook: [dropbear]
Generating dss host key for dropbear ...
Unknown key type 'dss'
Usage: dropbearkey -t <type> -f <filename> [-s bits]
-t type	Type of key to generate. One of:
		rsa
		ecdsa
-f filename    Use filename for the secret key.
               ~/.ssh/id_dropbear is recommended for client keys.
-s bits	Key size in bits, should be a multiple of 8 (optional)
           ECDSA has sizes 256 384 521
-y		Just print the publickey and fingerprint for the
		private key in <filename>.

Suggested fix: remove the dss key generation entirely.

PS:
This error described in the above linked archlinux bug report also happend:
  -> Running build hook: [dropbear]
Error: Unrecognised key type
Error reading key from '/etc/ssh/ssh_host_rsa_key'
Error: Unrecognised key type
Error reading key from '/etc/ssh/ssh_host_dsa_key'
Error: Unrecognised key type
Error reading key from '/etc/ssh/ssh_host_ecdsa_key'

As described in the bug report, re-creating the openssh host keys with the -m PEM option fixed this. But this is probably not a bug in mkinitcpio-dropbear.

I ran into both issues as well. Thanks for the link to the dropbear bug report!

I don't think this is a bug on this hook. But, if there's a need to change it, please let me know. I have not been using this hook, but he's now part of archlinux's [community] repo.

I don't think this is a bug on this hook. But, if there's a need to change it, please let me know. I have not been using this hook, but he's now part of archlinux's [community] repo.

In the current state, the hook fails as dropbear cannot generate a dss key, does not complete all steps, so you cannot ssh afterwards.
If you remove dss from the line 47 in dropbear_install
for keytype in rsa dss ecdsa ; do
with
 for keytype in rsa ecdsa ; do
it works as expected, and you can ssh afterwards
I don't think that removing dss from dropbear is a bug, so I would believe the script should be updated with this simple change.
Regards,

@JeanLucJFr
Indeed the hook itself needs to not check for dss keys anymore. If you are able, can you create a PR for this?

I sure can try :)

@JeanLucJFr
Never mind, please see #9

The modification in the PR #9 is indeed cleaner.
However, unless I'm mistaken, there was no merge and the modification didn't make it to the arch repository.
I understand this package is not used very much, so I'm adding a note in the wiki until this is fixed.

Yes, I need to merge the outstanding PR's to it. I have not used this hook in a while, but I'll try to merge this later and make sure it's working.

I understand this package is not used very much

But there are people using it :)
Thanks for efforts anyway, was to lazy to fix this yet... I just created dss keys for dropbear, I think.

I understand this package is not used very much

But there are people using it :)
Thanks for efforts anyway, was to lazy to fix this yet... I just created dss keys for dropbear, I think.

Yes, me amongst others :)
It's a polite way of pointing out that the bug was not fixed since 6 monthes but that's understandable...
Anyway, the Arch wiki now points to this thread in dm-crypt specialties, so there is a way for people to quickly have remote unlock up&running with dropbear.

Ever since I've switched to tinyssh, I have been using that, hence the reason for not testing this hook properly anymore. I have now set a VM for this purpose and I should soon be able to merge and test the PR's

I don't think you need to test this PR in a VM.
The change is minor and straightforward, it just removes a variable and an if-branch.
I have tested it multiple times.

Hi,
I tested the change in a VM and all appears well. I was able to boot, ssh and unlock remotely with the suggested change. Please can this hook be released so that others can benefit from your good work :-)

Any movement on this?

This issue seems to be fixed in PR #9, it would be nice with a new release though as some the Arch community package could then get the source from a more stable source.

@grazzolini did you manage to set up a VM to test this? If so, could you just make a release which would then make it to downstream users? Thanks!

I may be willing to jump in on this if you're open to another maintainer. I do use this on half a dozen systems still and am an Arch TU to ease that end of things too. I wouldn't be able to do anything serious until mid-December though as I am traveling internationally and won't want to test on remote systems without access to fix them ;-)

Since this repository seems to be abandoned, I started a new project and have already rewritten some of grazzolini's hooks, including dropbear.
My version generates/expects rsa, ecdsa and ed25519 keys by default, but this can also be configured via a config file.
In case anyone's interested: https://gitlab.com/allddd/mkinitcpio-extras
The package is also in the AUR.


Could not construct chart because [Couldn't find values to plot.]

I get the error in the title when passing the following input to the chart program
0       1
2       463008

@jamesk this actually works on scatter plots and line charts. You probably tried either a pie chart or a bar chart.
The reason it doesn't render a chart is because those two are expecting at least one string column, given that pie and bar charts always have one "discrete dimension" (as opposed to "continuous"), and these two floats are by default interpreted as continuous.
e.g. On a poll results pie chart with data:
Yes	0.46
No	0.54

Yes/No is the "discrete dimension"
Given:
t -> time
f -> float
s -> string

A proposed sensible default (in the case there are no s columns) could be:

If there is a t column and also an f column, use the tcolumn as the discrete dimension
If there are 2 f columns, use the 1st f column as the discrete dimension (your case)

I'll start implementing it. Let me know what you think.

@jamesk could you go get -u github.com/marianogappa/chart and try again? It should work now.
In addition, bar and pie charts should now support time columns and additional sensible defaults.

Cool, working now, thanks!

I'll add a comment when I release the fix. It'll be a patch version bump.


Fixed Gradient.remove_point not allowing fewer than two points.

This PR addresses #35863. This may be a bug since the editor allows setting gradient texture to have only one point, but remove_point() does not.
Bugsquad edit: Fixes #35863

Thanks!

Cherry-picked for 3.2.4.


The @var annotation contains a non existent class "self"

        $registry = (new ClassRegistry())
            ->addClassDirectory($this->projectDir . '/var/classes/DataObject')
            ->load();

The directory contains files from PIMcore, and among others scanned class Pimcore\Model\DataObject\AbstractObject with property with annotation:
    /**
     * @internal
     *
     * @var self|null
     */
    protected $o_parent;

Should be there self keyword resolved automatically into Pimcore\Model\DataObject\AbstractObject class?

Do i understand correctly, that you have a class that extends Pimcore\Model\DataObject\AbstractObject and AbstractObject has this $o_parent prop that is not resolved properly?

It would help if i would know about the structure (can be a dummy example), but probably properties with self should be ignored, since this can lead to endless schema recursion :smile:

Sorry, I can't reproduce it now with changes by https://github.com/php-kafka/php-avro-schema-generator/issues/38.
So, sorry for the noise.

All good, no worries :v: this error is only present in 2.x since in 3.x parsing is done differently, probably that's why.
You still gave very good input and new use cases, which i appreciate  a lot :v:


Possible to fork one version > 128TB?

Seems like this libmdbx has a good use case for a lot of things.

Seems like this libmdbx has a good use case for a lot of things.

I didn't understand the question.
Nonetheless, PLEASE read documentation, at least the three chapters:

Overall
Intoduction
Getting started


Fatal error after generating types on development

Hello,
Sometimes, when I update the graphql schema, I face this issue where my development server crashes just after model recreation.
I don't know if it is related to reloading schema or not.
I'm using node 16.16.0 on Windows and 'yarn 1.22.19', here is my package.json
  "devDependencies": {
    "@fortawesome/fontawesome-free": "^6.1.2",
    "@nuxtjs/strapi": "^1.5.0",
    "flag-icons": "^6.6.4",
    "nuxt": "3.0.0-rc.6",
    "primeflex": "^3.1.0",
    "primeicons": "^5.0.0",
    "primevue": "^3.15.0"
  },
  "dependencies": {
    "graphql": "^16.5.0",
    "nuxt-graphql-client": "^0.1.9"
  }

Stack trace
√ [GraphQL Client]: Generation completed in 4421ms                                        nuxt-graphql-client 11:09:32
FATAL ERROR: v8::FromJust Maybe value is Nothing.
 1: 00007FF7230B7C4F v8::internal::CodeObjectRegistry::~CodeObjectRegistry+114207
 2: 00007FF723045EC6 DSA_meth_get_flags+65542
 3: 00007FF723046D7D node::OnFatalError+301
 4: 00007FF7239660D5 v8::V8::FromJustIsNothing+53
 5: 00007FF72302EEB3 v8::base::CPU::has_sse+31507
 6: 00007FF72311ABC7 uv_timer_stop+1207
 7: 00007FF72311715B uv_async_send+331
 8: 00007FF7231168EC uv_loop_init+1292
 9: 00007FF723116A8A uv_run+202
10: 00007FF7230E58F5 node::SpinEventLoop+309
11: 00007FF722F7E520 v8::internal::interpreter::BytecodeLabel::bind+35712
12: 00007FF722F79B78 v8::internal::interpreter::BytecodeLabel::bind+16856
13: 00007FF72310715D uv_poll_stop+557
14: 00007FF723F20740 v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+146416
15: 00007FFBB5977034 BaseThreadInitThunk+20
16: 00007FFBB5C62651 RtlUserThreadStart+33
error Command failed with exit code 134.

@PrinceEntree8 Thank you for raising this issue, I haven't encountered this as yet so I'll have to run some tests to determine when exactly as well as how frequently this occurs to get a better understanding of the underlying cause here.

It seems to be a know Nuxt v3 problem nuxt/framework#837


Yii Log FileTarget except does not work

What steps will reproduce the problem?
In config,
'log' => [
            'traceLevel' => YII_DEBUG ? 3 : 0,
            'targets' => [
                [
                    'class' => 'yii\log\FileTarget',
                    'levels' => ['error', 'warning', 'info'],
                    'maxFileSize' => 102400, //1024000KB - 100MB
                    'except' => [
                        'email',
                        'yii\db\*'
                    ]
                ],
            ]
        ]

What is the expected result?
Expected the db queries not to be logged in app.log file
What do you get instead?
2018-03-15 15:26:54 [::1][163][-][info][yii\db\Command::query] SELECT * FROM auth_assignment WHERE user_id='163'
DB queries still getting logged.
Additional info
However, if I set the file target to write the logs elsewhere (other than app.log)
[
                    'class' => 'yii\log\FileTarget',
                    'levels' => ['error', 'warning', 'info'],
                    'maxFileSize' => 102400, //1024000KB - 100MB
                    'logFile' =><EMAIL_ADDRESS>                    'except' => [
                        'email',
                        'yii\db\*'
                    ]
                ],

It will work. DB queries will not be logged in app2.log

Q
A

Yii version
<IP_ADDRESS>

PHP version
7.1.5

Operating system
Windows 10

Added test to check the behavior. Failed to reproduce. In case I'm wrong in test – change it to reproduce your problem, please.


[Issue 5887]Fix old version download link error

Fix old version download link error
Fixes #5887
Motivation
Old version download link error
Modifications
Change the download link to https://archive.apache.org/dist/pulsar/

I think this fix seems incorrect. The download link on this page http://pulsar.apache.org/docs/en/2.3.0/standalone/ should be apache-pulsar-2.3.0-bin.tar.gz instead of rpm package download link, so I think the following code should be added and then the downloadPageUrl function should be fixed to support the download of the old version.
     binaryReleaseUrl(`${v}`),
      connectorReleaseUrl(`${v}`),
      // add this function
      offloaderReleaseUrl(`${v}`),
      prestoPulsarReleaseUrl(`${latestVersion}`),
      // Fix this function for support old version
      downloadPageUrl(),
      rpmReleaseUrl(`${v}`, ""),
      rpmReleaseUrl(`${v}`, "-debuginfo"),
      rpmReleaseUrl(`${v}`, "-devel"),
      debReleaseUrl(`${v}`, ""),
      debReleaseUrl(`${v}`, "-dev"),

run cpp tests

run cpp tests

run cpp tests

run cpp tests

run cpp tests

retest this please

retest this please

retest this please

run java8 tests


Implement completion candidate function for resources

This should use the discovery cache rather than, say, kubectl api-resources (though might be fun to do a benchmark to verify that using the cache is vastly more efficient).
For details, see:

https://serverfault.com/questions/1049006/what-does-kubectl-store-in-the-cache
https://jonnylangefeld.com/blog/the-kubernetes-discovery-cache-blessing-and-curse

Completed by #39.


Update our coverage badge

Reason for change

Testing

Codecov Report

Merging #112 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #112   +/-   ##
=======================================
  Coverage   53.03%   53.03%
=======================================
  Files          30       30
  Lines         709      709
  Branches      164      164
=======================================
  Hits          376      376
  Misses        331      331
  Partials        2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 659010d...8ade258. Read the comment docs.

lol I wish


Move unit tests location

Hey @roberthein,
This PR is a bit subjective so it's up to you to decide if it makes sense to you or not (I'm ok with booth decisions 🙂)
I think it makes more sense to move the unit tests away from the example project and put them next to the implementation code.
This PR moves the unit tests to Observable/Tests.
The unit tests can be ran by opening the Package.swift file.
What do you think about this?
Do you think it makes sense?
Thanks 👍

Thanks for merging this so quickly, and by the way thanks for the 2.0.0 release, it's a nice one 🙂

Yes it is. Thanks to you and Eduardo! 👌
(I was thinking the same about merging the tests in one place..) 🙏


The @buf/evmos_evmos.bufbuild_es dependency is broken

https://github.com/evmos/evmosjs/blob/8bfa2febb7c707b8069348ca21979e4b044f70d7/packages/proto/src/proto/evmos/vesting/tx.ts#L1C71-L1C71
This line should be changed to (note v1 is changed to v2)
export * from<EMAIL_ADDRESS>
Because the package @buf/evmos_evmos.bufbuild_es version is upgraded to 1.2.0-20230822152208-9101da12dca9.1, so the file structure is changed.

I might have run into the same issue. I'm getting the following output from a react application on webpack:
Failed to compile.

Attempted import error: 'MsgClawback' is not exported from '../../proto/evmos/vesting/tx.js' (imported as 'MsgClawback').
ERROR in<EMAIL_ADDRESS>3:18-29
export 'MsgClawback' (imported as 'MsgClawback') was not found in '../../proto/evmos/vesting/tx.js' (module has no exports)

ERROR in<EMAIL_ADDRESS>10:10-30
export 'MsgClawback' (imported as 'MsgClawback') was not found in '../../proto/evmos/vesting/tx.js' (module has no exports)

ERROR in<EMAIL_ADDRESS>19:18-49
export 'MsgCreateClawbackVestingAccount' (imported as 'MsgCreateClawbackVestingAccount') was not found in '../../proto/evmos/vesting/tx.js' (module has no exports)

ERROR in<EMAIL_ADDRESS>31:10-50
export 'MsgCreateClawbackVestingAccount' (imported as 'MsgCreateClawbackVestingAccount') was not found in '../../proto/evmos/vesting/tx.js' (module has no exports)

ERROR in<EMAIL_ADDRESS>1:0-71
Module not found: Error: Can't resolve<EMAIL_ADDRESS>in '/Users/mpicco/code/work/wormhole-connect/node_modules/@evmos/proto/dist/proto/evmos/vesting'

webpack compiled with 5 errors

And this is the error logged on the console:

Reverting to version 0.2.17 worked, but it's missing the memo field for IBC transfers, which is critical for this application.


Small refactor of ClipBox

This breaks up the impl blocks a bit so that fewer methods
require the W: Widget<T> bound.
It also includes a few small tweaks to the docs and the order
of method declarations.
This patch makes no functional changes.

@jneem it was being used in another widget where I wanted to be able to have setters that called child_mut, and where the trait bounds on the widget impl were more complex. Definitely not a requirement or anything, just a little help. :)


Vote on new features in Discussions

Hi torchtitanists,
Thank you for your interests in torchtitan!
We created https://github.com/pytorch/torchtitan/discussions/693 for the community to add feature requests and vote on them. We'll try to prioritize on the most requested features. Please share what you'd like to see next!

cc: @samsja @qsh-zh @Amerehei @FilyaGeikyan @thepowerfuldeez @Borda @starstream @leeeizhang @abatilo @mayank31398 @vermouth1992 @SeunghyunSEO @aniltrkkn @jiagaoxiang @casper-hansen @carmocca @daviswer @zigzagcai @jeromeku @Yangyi-Chen @tangjiasheng @zijian-hu @crcrpar @Doraemonzzz @jvwilliams23 @152334H @yzs981130 @MinghaoYan @rlrs @philippguevorguian

cc: @LeoXinhaoLee @mathmax12 @yundai424 @vedantroy @dz1iang @goldhuang @evkogs @0781532 @jaysonfrancis @nighting0le01 @vighneshbirodkar @zyushun @medivh-xp @Yuxin-CV @qiziAI @ad8e @prathameshtd @bkchang @arjvik @jaywonchung @bhuvan777 @ahatamiz @dmammfl @TJ-Solergibert @karan-dalal @gau-nernst @lucasjinreal @qiongerfei @rshah918 @eminorhan @thomasneff @viai957 @pansershrek

@tianyu-l
Hi developers,
Thanks for the great work!
I just have one confusion about the usage of FSDP2 fully_shard.
Does FSDP2 support mixed precision such like torch.float32 and torch.bfloat16 within a FSDPParamGroup?
To put it more clear, in most use cases of training LLM such like Lllama2, the precision of RMSNorm is usually torch.float32, but other components within the DecoderLayer is usually torch.bfloat16. When we want to train the model with the help of FSDP2, we have to wrap RMSNorm seperately since it has a seperate dtype, which will introduce additional all-gather and reduce-scatter.
From the profiling results, we found this approach (warpping RMSNorm seperately) will lead to bad computation-communication overlapping.

@zigzagcai RMSNorm only has activations in fp32, the weights are still bf16.
Also, FSDP2 is quite flexible in having different dtypes for different tensors I believe

cc: @awgu

it should be simple but
Gradient Accumulation
it is very useful for sfting big models.

it should be simple but
Gradient Accumulation
it is very useful for sfting big models.

gradient accumulation is not that worth it with fully shared since you need to all gather the weight at each forward anyway.
Tho yeah could makes sense to still have it

@samsja you can avoid the all-gather/reduce-scatter per microbatch with FSDP2 with hopefully intuitive APIs:
for microbatch_idx, microbatch in enumerate(batch):
    is_last_microbatch = microbatch_idx == num_microbatches - 1
    model.set_requires_gradient_sync(is_last_microbatch)  # only reduce-scatter on last microbatch
    model.set_reshard_after_backward(is_last_microbatch)  # only all-gather on 1st microbatch
    # Run forward/backward
optim.step()
optim.zero_grad()

This will use extra memory since unsharded parameters and gradients are held through forward and backward (roughly equivalent to ZeRO-1).

hmm nice I did not know that set_reshard_after_backward was a thing.
Yeah so grad acc makes sense with zero 1 but not zero 2

@zigzagcai sorry for the delay -- I was out last week.

Does FSDP2 support mixed precision within one warpping module, such like torch.float32 and torch.bfloat16 within a FSDPParamGroup?

This is not well-supported (at least not simply). Part of this is an API design question trading off with performance. E.g., how would you specify which parameters in the parameter group are using fp32 vs. using bf16? (Let me know if you have ideas here.)

To put it more clear, in most use cases of training LLM such like Lllama2, the precision of RMSNorm is usually torch.float32, but other components within the DecoderLayer is usually torch.bfloat16. When we want to train the model with the help of FSDP2, we have to wrap RMSNorm seperately since it has a seperate dtype, which will introduce additional all-gather and reduce-scatter.
From the profiling results, we found this approach (warpping RMSNorm seperately) will lead to bad computation-communication overlapping, especially in the backward pass.

The reason is what FSDP2's default prefetching algorithm is to only allow effectively one in-flight all-gather at a time in backward. This will lead to poor overlapping like you saw when the all-gather sizes are flipping between small and large since we cannot overlap the transformer block all-gather with just the RMSNorm backward for example.
FSDP2 exposes some manual APIs to configure the prefetching that can help here. I will need to find some time to put together an example. Let me see if I can do it tomorrow or later this week. Mainly, you can use set_modules_to_forward_prefetch and set_modules_to_backward_prefetch to overwrite the default prefetching schedule.

Apart from that, there are also some other use cases: dtype of MoE gating layers is required to be torch.float32, but other components in the DecoderLayer is torch.bfloat16. We can also found that seperately warpping MoE.GateLayer would cause bad overlapping of computation-communication.

Do you mean that the MoE router weight must be in fp32? I do want to clarify the use case somewhat (though the feature request is still valid). For the cases I have seen, using bf16 RMSNorm weight and bf16 router weight are sufficient. The computation kernel can upcast intermediates as needed, but that does not mean the weight itself needs to be in fp32.

it should be simple but
Gradient Accumulation
it is very useful for sfting big models.

@aniltrkkn Thanks for the great suggestion! I also agree that gradient accumulation is quite important. I myself implemented one version. Hope it would help a bit.
Line 517: https://github.com/zyushun/Adam-mini/blob/main/examples/llama/train.py

@tianyu-l Thanks for organizing the great discussion. I have one request but I am not sure if we have it now: is there a demo code that transform the saved checkpoint into the format by Huggingface Transformers? That would be quite useful for downstream evaluation or further SFT, RL.

@tianyu-l Thanks for organizing the great discussion! I have one request but I am not sure if we have it now: is there a demo code that transform the saved checkpoint into the format by Huggingface Transformers? That would be quite useful for downstream evaluation or further SFT, RL.

There is a conversion script that should be compatible with torchtitan here: https://github.com/PrimeIntellect-ai/prime/blob/main/scripts/export_dcp.py

@zyushun I agree such a script is desirable but missing.
Alternatively you may try following the "DCP -> torch.save -> HF" flow noted in https://github.com/pytorch/torchtitan/issues/420#issuecomment-2190202398

@zigzagcai do you have a repro of your setup? On 8xH100s Llama3-8B, I see some exposed collectives (e.g. norm reduce-scatter), but it should not be detrimental to training throughput:
diff --git a/torchtitan/parallelisms/parallelize_llama.py b/torchtitan/parallelisms/parallelize_llama.py
index 4d4c60b..95b1871 100644
--- a/torchtitan/parallelisms/parallelize_llama.py
+++ b/torchtitan/parallelisms/parallelize_llama.py
@@ -339,10 +339,16 @@ def apply_fsdp(
     Apply data parallelism to the model. FSDP2 is used here.
     """
     mp_policy = MixedPrecisionPolicy(param_dtype=param_dtype, reduce_dtype=reduce_dtype)
+    fp32_policy = MixedPrecisionPolicy(output_dtype=torch.bfloat16)
     fsdp_config = {"mesh": dp_mesh, "mp_policy": mp_policy}
+    fp32_config = {"mesh": dp_mesh, "mp_policy": fp32_policy}
     if cpu_offload:
         fsdp_config["offload_policy"] = CPUOffloadPolicy()

+    for module_name, module in model.named_modules():
+        if "norm" in module_name:
+            fully_shard(module, **fp32_config)
+
     for layer_id, transformer_block in model.layers.items():
         if pp_enabled:
             # For PP, do not reshard after forward to avoid per-microbatch

add tests/support for  low bit optimizers and Flash attention-3


Can't Delete Images

I can't delete the image I've uploaded to my guide, and I couldn't delete an image I uploaded to the crops.

Note: this is fixed on crops, still need to test it for guides.

Status of this right now

[ ] Images get upload correctly.
[ ] Featured images get removed correctly.
[ ] Previously uploaded stage images don't show on edit guide.
[ ] Action images doesn't work at all (can't upload, can't do nothing).

This would be fixed by #695.


@luc/fix root collection

What is the purpose of this change?

Fixed copying files to estuary root
Fixed listing files without modTime

a couple of minor comments, but otherwise lgtm.


npm run ganache error

Hi, I'm trying to run your script on a DigitalOcean droplet. (Ubuntu 16)
When I run npm run ganache, I get this error.
**> @ ganache /home/naman/all-things-dapp/create-react-dapp/mydapp

ganache-cli

/usr/local/lib/node_modules/ganache-cli/build/cli.node.js:71648
function VM (opts = {}) {
^
SyntaxError: Unexpected token =
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Function.Module.runMain (module.js:442:10)
at startup (node.js:136:18)
at node.js:966:3
npm ERR! Linux 4.4.0-112-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "run" "ganache"
npm ERR! node v4.2.6
npm ERR! npm  v3.5.2
npm ERR! code ELIFECYCLE
npm ERR! @ ganache: ganache-cli
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ ganache script 'ganache-cli'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the  package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     ganache-cli
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /home/naman/all-things-dapp/create-react-dapp/mydapp/npm-debug.log**
Please help me resolve it. Thanks. Btw awesome implementation 💯

It looks like you're using an old node 4.x version which doesn't support the syntax that the program that's tripping up the library.  You probably just need to upgrade it to v8.x. Let me know if that works.

yes it worked! thanks.
one more thing, will it run on AWS or Digital Ocean droplet?
On 18-Feb-2018 03:07, "John McLaughlin"<EMAIL_ADDRESS>wrote:

It looks like you're using an old node 4.x version which doesn't support
the syntax that the program that's tripping up the library. You probably
just need to upgrade it to v8.x. Let me know if that works.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mjhm/hello_world_dapp/issues/4#issuecomment-366473326,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQ3PReUwWiU3qFl2eWHtHnmLVrGhpK-5ks5tV0aogaJpZM4SH3mQ
.

I don't know any reason it shouldn't, everything is pretty generic.  Though I haven't tried.  Of course on AWS (for example) you'd need to set up a Security Group to open up port 8000 to the internet.


Suggestion lists

Dunno where else to post this but I just have a couple suggestions that would be pretty cool to see in a update:

Custom client icon
Pretty sure most people have suggested this before and maybe there is a reason for it not to be added so I won't go on about it for to long but basically just allow an option to change the actual client logo

Custom launcher icon
Similar to custom client icon except with the launcher, the launcher uses the bloxstrap icon when loading a game in the taskbar, it would be pretty cool if we could change that, it would make it more seamless

Multi-clients mod
I can see this not being added but maybe add a mod that allows us to open multiple clients

Choose to use desktop mode
When closing a Roblox client it will usually prompt us are we sure and will send us to the desktop app, would be cool to have an option to toggle this, I haven't actually tested how it works on BloxStrap (I forgor 💀) but I'm pretty sure it automatically removes this pop-up so having an option for it to appear would be interesting if possible

Game specific mods
So certain mods will only load in certain games, could potentially be done by making a folder in the "Modifications" folder and giving it the name of a place ID

I hope you like my ideas :)

I've dismissed idea 1 before as I don't want to modify the game client executable, and idea 2 seems like a really niche thing. Also, I don't make this clear, but you can open the desktop app through the "Play Roblox" shortcut in the start menu. I guess I could add a launch option for it too. Ideas 3 and 5 do seem interesting, though not sure what use idea 5 would have to warrant adding.
If I ever feel like working on Bloxstrap again, I might add some of these. Thank you!

I thought of idea 5 (Game specific mods) when playing RetroStudio and they are trying to simulate the old Roblox clients and a pretty annoying feature of Roblox is they are uncapable of removing the CoreGui (And even if they did it would violate the Roblox TOS) so I thought it would be cool to personally modify my Roblox files to remove the pause menu however having to keep reinstalling and uninstalling a mod when I wanna play a difference game would be annoying which is why I suggested it
Would love if you could add it :)


Use default fonts on UIButtons using json theme files

Is your feature request related to a problem? Please describe.
Yeah, i would like to make a button and change its font by using JSON themes, but i cannot without download the ttf files. I have to precise where to find the file and I don't know where to get them in my OS for default system fonts.
Describe the solution you'd like
Based on the doc's configuration, why not to add a new option to tell pygame-gui to use a sysfont ?
{ 	"name": "segoeui", 	"size": "11", 	"bold": "0", 	"italic": "0", 	"sysfont": "1" }
and nothing more !
Describe alternatives you've considered
Maybe we can already do this by specify the font path in the default operating system or by another solution which i don't know for now
Additional context
I tryied searching for a solution on the web or in the project issues section but i didn't found any solution for now.

Hello!
Currently the library doesn't use the SystemFonts submodule of pygame. It's had a few issues over the years with slow loading times (this could even effect apps where you didn't directly use the system fonts if care was not taken) and a rather sparse font selection which made me not want to include it by default.
I also haven't personally found it useful because:
a) It's not properly portable, different fonts are available on linux, windows & mac.
b) The way I work I tend to stick with a default font while developing until ready for custom font selection then I always grab whatever fonts I want to use from a place like google fonts which is an actually usable font browser. Then I bundle the font files in with the app if I want to make it portable/compile it with PyInstaller.
If somebody wanted to patch in usage of the SystemFont module to the library in the future I wouldn't be opposed - this is an open source project and accepts pull requests, but it is not a priority for me personally.

Thanks you for your very clear and complete response !
I understand more now why we cannot do this, so in my case I consider my issue as 'resolved'
Following your advice, I downloaded a font then I encapsulated it in my project


Filtering out undefined events

This makes an attempt to fix issue #68 .
Filter out the undefined events before passing it to Day component, where it calculates 'N more...' label based on the entries length.

@yusuf-multhan Thank you for submitting PR!

Resolves #68.


Unable to resolve Buffer when bundling kde2d with webpack

Description
Encountered an error when on a fresh yarn create react-app, I did yarn add @stdlib/stats-kde2d and tried to require it
Related Issues
No response
Questions
No.
Demo
No response
Reproduction
1. `yarn create react-app`
2. `yarn add @stdlib/stats-kde2d`
3. Insert `var kde2d = require( '@stdlib/stats-kde2d' );` in `App.js`

Expected Results
I'd be able to compute a `kde` with the function.

Actual Results
Failed to compile.

Module not found: Error: Can't resolve 'buffer' in '/.../kde-test/node_modules/@stdlib/buffer-ctor/lib'
Did you mean './buffer'?
Requests that should resolve in the current directory need to start with './'.
Requests that start with a name are treated as module requests and resolve within module directories (node_modules, /.../kde-test/node_modules).
If changing the source code is not an option there is also a resolve options called 'preferRelative' which tries to resolve these kind of requests in the current directory too.

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "buffer": require.resolve("buffer/") }'
        - install 'buffer'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "buffer": false }
WARNING in [eslint]
src/App.js
  Line 4:5:  'kde2d' is assigned a value but never used  no-unused-vars

ERROR in<EMAIL_ADDRESS>20:11-35
Module not found: Error: Can't resolve 'buffer' in '/.../kde-test/node_modules/@stdlib/buffer-ctor/lib'
Did you mean './buffer'?
Requests that should resolve in the current directory need to start with './'.
Requests that start with a name are treated as module requests and resolve within module directories (node_modules, /.../kde-test/node_modules).
If changing the source code is not an option there is also a resolve options called 'preferRelative' which tries to resolve these kind of requests in the current directory too.

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "buffer": require.resolve("buffer/") }'
        - install 'buffer'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "buffer": false }

webpack compiled with 1 error and 1 warning

### Version

^0.0.7

### Environments

Node.js

### Browser Version

_No response_

### Node.js / npm Version

node 16.14.2, yarn 1.22.18

### Platform

nixos

### Checklist

- [X] Read and understood the [Code of Conduct](https://github.com/stdlib-js/stdlib/blob/develop/CODE_OF_CONDUCT.md).
- [X] Searched for existing issues and pull requests.

Thanks for letting us know about this behavior!
As the error indicates, starting with Webpack version 5, Polyfills for Node.js APIs are not included by default anymore, which causes compilation in this case due to buffer not being available (which is used in our ndarray implementation). We will discuss this internally and post any updates to this issue thread.
In the meantime, please consider installing the buffer package and adding it to your Webpack configuration as indicated in the error message:
'resolve': {
  'fallback': {
    'buffer': require.resolve( 'buffer' )
  }
}

Will close as resolution of this issue resides outside our control. Thanks, @quinn-dougherty, for reaching out and hopefully the above addresses the encountered problem.


load_in_4bit still not failed with fsdp in latest source build

System Info
Python 3.9.18
bitsandbytes             0.43.0.dev0 bitsandbytes # built from source (commit 259ad44110940259b6a6b545b8a8b2a69289bbef)
torch                    2.2.0
transformers             4.37.2
trl                      0.7.11.dev0 # latest source code
accelerate               0.27.2
Reproduction
I can use latest source code build of bitsandbytes to run with fsdp in this example. But When I run dpo_llama2.py in trl, it failed.
steps to reproduce:

get data:

    git clone https://huggingface.co/datasets/lvwerra/stack-exchange-paired

it will take a very long time to download dataset. So I clone it. And change the get_stack_exchange_paired to:
    dataset = load_dataset(
        "/nas/lili/codes/pt/ft/trl/stack-exchange-paired",
        split="train",
        cache_dir=cache_dir,
        data_dir=data_dir,
    )

If you just want datasets to download it to cache, skip this step.

run

accelerate launch --config_file 7b-fsdp.yaml examples/research_projects/stack_llama_2/scripts/dpo_llama2.py \
    --model_name_or_path="/nas/lili/models_hf/Llama-2-7b-hf" \
    --output_dir="dpo" \
    --report_to="tensorboard" \
    --sanity_check=true

accelerate config:
$ cat 7b-fsdp.yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: false
  fsdp_offload_params: false
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

errors:
Traceback (most recent call last):
  File "/nas/lili/codes/pt/ft/trl/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py", line 236, in <module>
    dpo_trainer.train()
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1687, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1228, in prepare
    result = tuple(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1229, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1105, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1387, in prepare_model
    model = FSDP(model, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 477, in __init__
    _auto_wrap(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/_wrap_utils.py", line 101, in _auto_wrap
    _recursive_wrap(**recursive_wrap_kwargs, **root_kwargs)  # type: ignore[arg-type]
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/wrap.py", line 543, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/wrap.py", line 543, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/wrap.py", line 543, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
  [Previous line repeated 2 more times]
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/wrap.py", line 561, in _recursive_wrap
    return _wrap(module, wrapper_cls, **kwargs), nonwrapped_numel
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/wrap.py", line 490, in _wrap
    return wrapper_cls(module, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 503, in __init__
    _init_param_handle_from_module(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/_init_utils.py", line 594, in _init_param_handle_from_module
    _init_param_handle_from_params(state, managed_params, fully_sharded_module)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/_init_utils.py", line 606, in _init_param_handle_from_params
    handle = FlatParamHandle(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/_flat_param.py", line 573, in __init__
    self._init_flat_param_and_metadata(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/_flat_param.py", line 623, in _init_flat_param_and_metadata
    ) = self._validate_tensors_to_flatten(params)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch/distributed/fsdp/_flat_param.py", line 759, in _validate_tensors_to_flatten
    raise ValueError("Cannot flatten integer dtype tensors")
ValueError: Cannot flatten integer dtype tensors

Expected behavior
no exception

Even I'm facing the same issue. I'm using
GPU: A100
torch==2.1.2
accelerate==0.26.1
bitsandbytes==0.42.0.

cc @Titus-von-Koeller I think FSDP is still unstable on main, titus can confirm !
Can you make sure to compile bnb from main following this tutorial: https://huggingface.co/docs/bitsandbytes/main/en/installation

2.1.2

yes. Because When I don't use compiled bnb, I can't even run this example.
I checked it as following steps:

get bnb install path

>>> import bitsandbytes
>>> print(bitsandbytes.__file__)
/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/bitsandbytes/__init__.py
>>>

check so md5

$ md5sum /home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
61a4ba4a75a81e5b295c9d8a2778e05e  /home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda121.so

md5 from my own build:

ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/ft/bitsandbytes$ find . -name '*.so'|xargs md5sum
61a4ba4a75a81e5b295c9d8a2778e05e  ./bitsandbytes/libbitsandbytes_cuda121.so
61a4ba4a75a81e5b295c9d8a2778e05e  ./build/lib.linux-x86_64-cpython-39/bitsandbytes/libbitsandbytes_cuda121.so

cc @Titus-von-Koeller I think FSDP is still unstable on main, titus can confirm ! Can you make sure to compile bnb from main following this tutorial: https://huggingface.co/docs/bitsandbytes/main/en/installation

I am using 259ad44110940259b6a6b545b8a8b2a69289bbef to compile from source.
I only remember using this command:
CUDA_VERSION=121 make cuda12x

Hey all,
FSDP + QLoRA is still not supported. There's active work on it, but currently the only official way to run it is with BNB from source (after the last release) and with this repo as a training script.
Compilation from source is documented here.
Keep in mind however, that there are some ongoing issues we're facing there still. First and foremost some memory issue with QLoRA itself. We'll officially release this and write a blog post about it when it's ready. Just follow me on twitter and you'll know about it.
With that answerdotai repo and compiled from source that FSDP error trace should go away, but as just mentioned there's an excessive memory use somewhere that makes it a bit useless right now, until solved.

a blog post abo

In my case, the situation was reversed. I encountered issues running the program when building from source, but I could start the run by using the "pip install" method.


Easiest way to update to latest version...

If I don't want to accidentally overwrite my database, what is the easiest way to update to the earliest version? If I just drop it in the same directory and run the normal install, will it overwrite and work normally?

Figured it out. I moved the old folder, made sure the data directory and ardb.conf was backed up, then installed new version in same location, moved data directory and ardb.conf back into location and started server back up.


Fix disposing of effects for submeshes

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Snapshot stored with reference name:
refs/pull/15910/merge
Test environment:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/index.html
To test a playground add it to the URL, for example:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/15910/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/15910/merge
https://gui.babylonjs.com/?snapshot=refs/pull/15910/merge
https://nme.babylonjs.com/?snapshot=refs/pull/15910/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/15910/merge#BCU1XR#0

Visualization tests for WebGPU (Experimental)
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/testResults/webgpuplaywright/index.html

WebGL2 visualization test reporter:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/testResults/webgl2playwright/index.html

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Visualization tests for WebGPU (Experimental)
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/testResults/webgpuplaywright/index.html

WebGL2 visualization test reporter:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/testResults/webgl2playwright/index.html

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

WebGL2 visualization test reporter:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/testResults/webgl2playwright/index.html

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Visualization tests for WebGPU (Experimental)
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15910/merge/testResults/webgpuplaywright/index.html