This is a potential solution to help fix the issues with intersection tests for the rotation gizmo. Originally, the code was testing the relationship (dot product) between two direction vectors; the mouse ray and gizmo plane intersection ray direction and the direction vector from the center of the gizmo to the intersection position. The test is to see if the mouse is hitting the front half or the back half of the rotation gizmo sphere. I would expect the test to look for a negative dot product (as a potential hit), since the direction vector from the center of the gizmo to the front half should be facing towards the screen, and the mouse ray faces away from the screen. Instead the test was checking to see if the vectors were both facing away from the screen. This works in the ImGuizmo sample program for some reason. But in my own program this test was failing and causing the back half of the gizmo to be intersected with instead. This might have something to do with left handed vs right handed, but I dont think so. Did not feel like taking the time to look into why the demo was different. The solution here brings everything into view space and compares depths. In the demo, depth in view space is in -Z, while in my program view space depth is in +Z. I suspect this has a lot to do with the issues. So now the code is subtracting the absolute depth values (so it doesn't matter if depth is positive or negative) of the mouse ray and gizmo plane intersection position in view space from the center of the gizmo in view space. If the resulting value is positive, the center of the gizmo is further from the camera than the mouse intersection point, indicating we are in the front half of the gizmo sphere.
This is a potential solution to help fix the issues with intersection tests for the rotation gizmo. Originally, the code was testing the relationship (dot product) between two direction vectors; the mouse ray and gizmo plane intersection ray direction and the direction vector from the center of the gizmo to the intersection position. The test is to see if the mouse is hitting the front half or the back half of the rotation gizmo sphere. I would expect the test to look for a negative dot product (as a potential hit), since the direction vector from the center of the gizmo to the front half should be facing towards the screen, and the mouse ray faces away from the screen. Instead the test was checking to see if the vectors were both facing away from the screen. This works in the ImGuizmo sample program for some reason. But in my own program this test was failing and causing the back half of the gizmo to be intersected with instead. This might have something to do with left handed vs right handed, but I dont think so. Did not feel like taking the time to look into why the demo was different. The solution here brings everything into view space and compares depths. In the demo, depth in view space is in -Z, while in my program view space depth is in +Z. I suspect this has a lot to do with the issues. So now the code is subtracting the absolute depth values (so it doesn't matter if depth is positive or negative) of the mouse ray and gizmo plane intersection position in view space from the center of the gizmo in view space. If the resulting value is positive, the center of the gizmo is further from the camera than the mouse intersection point, indicating we are in the front half of the gizmo sphere.

sherief/ImGuizmo,sherief/ImGuizmo,CedricGuillemet/ImGuizmo,CedricGuillemet/ImGuizmo