Master Version confusing

Prerequisites

[X] Put an X between the brackets on this line if you have done all of the following:

Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description
[Description of the issue]
Steps to Reproduce
git pull in master Branch
you get 1.10.**
change to 1.20 Branch
git pull
allready up to date
you get version 1.20.**
This confuses me a little

[First Step]
[Second Step]
[and so on...]

Expected behavior: [What you expect to happen]
Actual behavior: [What actually happens]
Reproduces how often: [What percentage of the time does it reproduce?]
Versions
You can get this information from copy and pasting the output of atom --version and apm --version from the command line. Also, please include the OS and what version of the OS you're running.
Additional Information
Any additional information, configuration or data that might be necessary to reproduce the issue.

git cloned all the branches when you first cloned, it is expected behavior you can still check the SHA-1 hashes if you feel it wasn't cloned properly.

no, I was confused that the Master Branch has not the actual Version.


Some instructions and tweaks for using user-defined release for Forseti

Rename the variable to "release-version"
Some instructions on how to configure

Adding cla: yes label, as the PR author is a googler at the time of this PR was created.


NXT-163 Display word conversion

https://nicedigital.atlassian.net/browse/NXT-163

Can't currently be tested as guidance web serves the guidance product pages.


Inconsistent results on Jupyer notebook and Intellij IDE: Python

I am trying to compute medcouple using robustats module in python
The results on jupyter note book and IntelliJ Ide do not match.
Result on my jupyter notebook:
from robustats import medcouple
import numpy as np

x = np.array([9325.06, 6206.00, 10000.00, 9569.78])
print(medcouple(x))

-0.6442066420664204

Result on IntelliJ Ide
from robustats import medcouple
import numpy as np

x = np.array([9325.06, 6206.00, 10000.00, 9569.78])
print(medcouple(x))

nan

Please do let me know if I have to change any setting on IDE. I have made sure both are running against the same virtual env

Hi, @Iravata. Thank you for submitting this issue.
Could you please let me know what versions of Python and Numpy you are using?

Hi @FilippoBovo .
I am using the below versions
numpy - 1.21.0
python - 3.9.4

Are the versions the same in IntelliJ and Jupyter?

yes they both are same

Hmm, then it is a bit odd.
Could you try to use simple lists, instead of Numpy arrays?
from robustats import medcouple

x = [9325.06, 6206.00, 10000.00, 9569.78]
print(medcouple(x))

If I use normal list the result matches. I think the issue is with numpy
`from robustats import medcouple
import numpy as np
x = np.array([9325.06, 6206.00, 10000.00, 9569.78])
print(medcouple(x))
x = [9325.06, 6206.00, 10000.00, 9569.78]
print(medcouple(x))`
and the results are.
nan -0.6442066420664204

Ok, then it may be a precision issue, that is, robustats using floats, rather than doubles.
Just in case, could you check the dtype of the Numpy array in Jupter and IntelliJ (x.dtype)?
If the dtypes are different, you may try to define the Numpy array setting an explicit dtype, like the following.
x = np.array([9325.06, 6206.00, 10000.00, 9569.78], dtype=np.float64)

However, I am not sure that this is the problem, as it may also be due to the C compiler that you use to compile the Robustats library.

x = np.array([9325.06, 6206.00, 10000.00, 9569.78], dtype=np.float64)
even this is returning nan

Ok. Could you try to create new virtual environments with Python 3.8 and the latest version of Numpy and try again, using the same virtual environment in Jupyter and IntelliJ?

sure will try that .. and let you know

I tried it on a different machine, with Python 3.8 and numpy - 1.21.0 . I dont see the issue very strange.

ok we can close it. I will get back to you if i get more inofrmation.

Ok. Thanks.


actionmenu: onClick on a menu disables focus on hover

If you hover an item  w/o an onClick, it highlights.  If you, after that, hover an item w/ an onClick, the previous highlight remains and does not move to the hovered item.
From https://pluralsight.slack.com/archives/C70HPSJPP/p1547674116032400?thread_ts=1547673845.031700&cid=C70HPSJPP

We just looked this and couldn't repro.  If it's still a problem for you, @michaelkushlan , if you could point me to where it's happening for you in the app, that might be nice.


xii - 'node_modules/sinon/pkg/sinon.js' would be in serve_files

Chapter 0. Project Setup > xii Include Lo-Dash And jQuery

This was fixed in an earlier revision, thanks.


Add CODEOWNERS file

WHY are these changes introduced?
It's important that when contributing changes to different areas of the codebase, the teams that might be concerned about the changes are added as reviewers.
WHAT is this pull request doing?
I'm adding a CODEOWNERS file to ensure owners of different areas of the codebase are assigned as reviewers based on the changed files on the PR.

TIL: CODEOWNERS is a thing. Really cool!


Environment variable secrets with key vault and firewall/private network

Hi
Good to see environment variable secrets can now be linked to key vault.
Can we enable firewall rules with Allow Azure services bypass? What about key vaults with private endpoints?
Would be good to clarify this in the docs.
Kind regards
Kim

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ba670cf8-5810-79c4-7a2a-101099ffe1d7
Version Independent ID: 69b7e1fb-9f56-8c07-b1d5-8480fe6d8e1a
Content: Use environment variables in solutions - Power Apps
Content Source: powerapps-docs/maker/data-platform/EnvironmentVariables.md
Service: powerapps
Sub-service: dataverse-maker
GitHub Login: @caburk
Microsoft Alias: caburk

Please assign to Shan McArthur. He's the PM for this feature.
Casey Burke
Senior Program Manager, Power Apps ALM |<EMAIL_ADDRESS>From: Bishal Goswami @.>
Sent: Friday, January 28, 2022 2:00 AM
To: MicrosoftDocs/powerapps-docs @.>
Cc: Casey Burke @.>; Assign @.>
Subject: Re: [MicrosoftDocs/powerapps-docs] Environment variable secrets with key vault and firewall/private network (Issue #3018)
Assigned #3018https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fpowerapps-docs%2Fissues%2F3018&data=04|01|caburk%40microsoft.com|78bae598666e46ea30ce08d9e2450355|72f988bf86f141af91ab2d7cd011db47|1|0|637789608314871351|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=96tymB89Z%2FlHMQ65EVFcmToPJoEsN1D05n4fGNKqSTc%3D&reserved=0 to @caburkhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcaburk&data=04|01|caburk%40microsoft.com|78bae598666e46ea30ce08d9e2450355|72f988bf86f141af91ab2d7cd011db47|1|0|637789608314871351|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=Csdf4pya7cagXhyH%2B76x8J1q%2Fumqov5c2%2BM8vm97rQ4%3D&reserved=0.

Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fpowerapps-docs%2Fissues%2F3018%23event-5969520628&data=04|01|caburk%40microsoft.com|78bae598666e46ea30ce08d9e2450355|72f988bf86f141af91ab2d7cd011db47|1|0|637789608314921340|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=LIUXmfqDFwPIq7YPwtmcYUkK59GrvCMsLcZoELFbRxg%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIGKY3CDZMIMBHY27SGORE3UYJSLZANCNFSM5NAI6V3A&data=04|01|caburk%40microsoft.com|78bae598666e46ea30ce08d9e2450355|72f988bf86f141af91ab2d7cd011db47|1|0|637789608314921340|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=mkP0isdUS8tz3jVXmfULSZYOAVqaxm%2FzvQdcwMqUj%2FI%3D&reserved=0.
Triage notifications on the go with GitHub Mobile for iOShttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04|01|caburk%40microsoft.com|78bae598666e46ea30ce08d9e2450355|72f988bf86f141af91ab2d7cd011db47|1|0|637789608314921340|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=FWrP87ttHxgFrW%2BZqt0S1%2Fd8nV7k4zYg61KNvmhMQMU%3D&reserved=0 or Androidhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04|01|caburk%40microsoft.com|78bae598666e46ea30ce08d9e2450355|72f988bf86f141af91ab2d7cd011db47|1|0|637789608314921340|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=EUeA7LI92g2mVQMtT%2FpBFBsiHm780jr30IhdXrPmehk%3D&reserved=0.
You are receiving this because you were assigned.Message ID: @.***>

Hi,
Any update on this? I receive the following error when trying to get the secret environment variable value.

Client address is not authorized and caller is not a trusted service.

Allow trusted Microsoft services to bypass this firewall is set to Yes in Azure KeyVault but it seems Dataverse is not added to these trusted services. Can you tell me if this is planned?
Thanks in advance!

Hi,
Any updates on this topic?
Faced with the same issue as @LarsBauer

I am a little unclear of the overall scenario, but I think it is when the customer creates an azure network and attempts to limit access to their keyvault to their network.  When our platform attempts to access that secured KeyVault, it is blocked by their firewall policies.  The ask seems to be to include our service in the 'trusted microsoft services' list so that we can automatically bypass this when they turn this switch on.  I am not familiar enough with Azure firewall or which team to engage, and I have my doubts they would access this.  I am wondering if there are better ways for customers to configure their allow lists to enable our access to that KeyVault.  If that doesn't exist, then the answer is we don't support KeyVaults protected by private network capabilities.  Carlos' team does not have any work booked over this next semester to invest in our backlog of secrets work.  There are higher priorities for this.  As such, unless someone can point to a better resource, I think the best response is there is no present plan to support this scenario and to use a KeyVault not hidden behind a protected virtual network.
Shan
From: Casey Burke @.>
Sent: Wednesday, May 4, 2022 10:40 AM
To: MicrosoftDocs/powerapps-docs @.>; MicrosoftDocs/powerapps-docs @.>; Shan McArthur @.>; Sameer Chabungbam @.>
Cc: Mention @.>
Subject: RE: [MicrosoftDocs/powerapps-docs] Environment variable secrets with key vault and firewall/private network (Issue #3018)
@Shan @.> @Sameer @.> - can you please help here.
Casey Burke
Senior Program Manager, PowerApps ALM |<EMAIL_ADDRESS>From: Yaroslav Zghurovskyi<EMAIL_ADDRESS>Sent: Wednesday, May 4, 2022 5:29 AM
To: MicrosoftDocs/powerapps-docs<EMAIL_ADDRESS>Cc: Casey Burke<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/powerapps-docs] Environment variable secrets with key vault and firewall/private network (Issue #3018)
Hi,
Any updates on this topic?
Faced with the same issue as @LarsBauerhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FLarsBauer&data=05|01|shmcarth%40microsoft.com|db230f20b5414c38983f08da2df510fe|72f988bf86f141af91ab2d7cd011db47|1|0|637872827820768953|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=53VMaT0EzpHM%2FdfG2JcmJGYA6m7UukPW%2FUB6SQF6WM4%3D&reserved=0

Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fpowerapps-docs%2Fissues%2F3018%23issuecomment-1117255531&data=05|01|shmcarth%40microsoft.com|db230f20b5414c38983f08da2df510fe|72f988bf86f141af91ab2d7cd011db47|1|0|637872827820768953|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=GigFq9bVYiQeKDzFwMVj31eKCjPqIbPwEmOG3ARlAlQ%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIGKY3EDCHUT2MBSRGAGKBTVIJUSRANCNFSM5NAI6V3A&data=05|01|shmcarth%40microsoft.com|db230f20b5414c38983f08da2df510fe|72f988bf86f141af91ab2d7cd011db47|1|0|637872827820768953|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=CF8E6JH9M2jiOFupmUCo%2BsVpz%2BDRLPH9NqmYWS%2BI%2BqA%3D&reserved=0.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Trusted Service could be one option. I see another options which may work.
Add the ability to use service tags in KeyVault Networking like AzureCloud.xxx. (Or even better introduce a new service tag for Dynamics.)

@Shan @.> @Sameer @.> - can you please help here.
Casey Burke
Senior Program Manager, PowerApps ALM |<EMAIL_ADDRESS>From: Yaroslav Zghurovskyi @.>
Sent: Wednesday, May 4, 2022 5:29 AM
To: MicrosoftDocs/powerapps-docs @.>
Cc: Casey Burke @.>; Mention @.>
Subject: Re: [MicrosoftDocs/powerapps-docs] Environment variable secrets with key vault and firewall/private network (Issue #3018)
Hi,
Any updates on this topic?
Faced with the same issue as @LarsBauerhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FLarsBauer&data=05|01|caburk%40microsoft.com|2b0344b5369c4e99433508da2dc9bbdd|72f988bf86f141af91ab2d7cd011db47|1|0|637872641714771029|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=t6ka9MWY25hYZIJwD%2Bl7obQTEcFQS6XiRuxYtxORmkw%3D&reserved=0

Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fpowerapps-docs%2Fissues%2F3018%23issuecomment-1117255531&data=05|01|caburk%40microsoft.com|2b0344b5369c4e99433508da2dc9bbdd|72f988bf86f141af91ab2d7cd011db47|1|0|637872641714771029|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=PcSEOSmvjXOxGDXvhrWEO0V%2F9La5AFtdb%2BbwDJm4L60%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIGKY3EDCHUT2MBSRGAGKBTVIJUSRANCNFSM5NAI6V3A&data=05|01|caburk%40microsoft.com|2b0344b5369c4e99433508da2dc9bbdd|72f988bf86f141af91ab2d7cd011db47|1|0|637872641714771029|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=HTSL3OrjAdsaYTwpiceji6FXsg62t85VITgKRT%2Fs0FA%3D&reserved=0.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

I am facing this struggle with two enterprise customers I am working with. This seems short sided to simply not support this and certainly does not align with Microsofts one cloud ecosystem story. Many enterprises do not want to expose their Key Vaults to the public net. Whitelisting the Power Platform IP addresses is not practical as it is a long list that changes regularly. We should be able to leverage service tags or have Power Platform acknowledged as a "microsoft service".

Any updates on this issue? We are facing the same challenge.


Release 1.2.0rc1

@agjohnson package-json.lock is updated but theme.js is unaffected, so it checks out :100:


Support recovery

This is the tracking issue about recovery progress.  The recovery progress is aparted into two parts:

Segment Recovery: an old leader might crashed, new leader need replicate and ack the previous entries.
Replica Recovery: an replica might lost some entries even the segment is sealed.

Segment Recovery
For segment recovery, there are sets of tests need to pass:

Ack entries iff all entries in previous segment are acked.
Don't replicate new segment if there exists two unsealed segments.
New leader could recovery a segment which fault in recovering.
New leader could recovery a new segment which created in recovering.

Tasks:

[x] #67
[x] #68
[x] #69
[x] #70
[x] #72
[x] #76
[x] #77
[x] #134

Replica Recovery
TODO

Closes this issue since all sub tasks are finished.
The tasks about replica recovery are migrated to engual.


Gert/update feature

Coverage remained the same at 100.0% when pulling 5b859510cedf6746cb55a3942f9ba360db1ac326 on gert/updateFeature into b0411ede020acd54535cd9b05e514105330cc512 on master.


Missing release assets

The download links for at least the linux binaries on the releases page seem to be misbehaving:
https://github.com/roboll/helmfile/releases/download/v0.18.0/helmfile_linux_amd64 ends up with a S3 error.
Is it possible they haven't uploaded properly?

It looks like the last build failed due to a Github error and left some release assets deleted: https://circleci.com/gh/roboll/helmfile/622#tests/containers/0
It should be enough to rerun this build.
For the future I would like to suggest to treat already released versions as immutable and not to release the exact same version multiple times. Instead it would be better to release a new bugfix release.
@roboll @mumoshu
Thanks for working on this great tool, it made using helm much easier!

Would it be possible to get the release rerun or cut a new release with the changes since 0.18.0?

Thanks for the feedback and the notice! Just pushed v0.19.0.

Looks like there's a v0.19.0 tag from 14 days ago? But not a full release?

Sorry I had tagged a wrong commit 😉 It should be good now.


[FEATURE REQUEST] - Marshal and Unmarshal built doc for storing as byte

Is it possible to Marshal the pdf doc to byte and store in database. And later we just unmarshal the bytes to show to user or download?

Hello,
yes, just added an example here : https://github.com/angelodlfrtr/go-invoice-generator/blob/master/examples/bytes/main.go#L107


PageSpeed prevent image loaded by my function in Laravel

I save uploaded image by users into Storage folder with customized path. The image's name will be parsed to know where image will be save in my Storage. Therefore, I write function to return image data when user request image by GET method. It means that image is not get directly. For example, user requests:
https://example.com/image/d3fe6896b9dcbc02031eaf36761009c6.jpg
I get image name (in this case, d3fe6896b9dcbc02031eaf36761009c6.jpg) and parse it to know where I save the image in Storage folder. Before applying PageSpeed, image is loaded normally. However, image is not found (404 error) after I use PageSpeed for optimizing. I don't modify any PageSpeed Filters (use CoreFilters). My ssl.conf in nginx as following:
pagespeed on;

# Needs to exist and be writable by nginx.  Use tmpfs for best performance.
pagespeed FileCachePath /var/ngx_pagespeed_cache;

# Ensure requests for pagespeed optimized resources go to the pagespeed handler
# and no extraneous headers get set.
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
  add_header "" "";
}
location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }

location ~* .(woff|eot|ttf|svg|mp4|webm|jpg|jpeg|png|gif|ico|css|js)$ {
    expires 365d;
}

When debugging, I see that my code for response image can not reach.
//PHP code
//Parse image name and generate full path
//After that, get image data and response to client
$imageFile = File::get($fullPath);
return response($imageFile, 200)->header('Content-type', 'image/jpeg');
Therefore, I think that PageSpeed fetched image link before my code. As a result, image is not found.
How to solve the problem?

Hi -- I am not sure I fully understand the problem.  If you have PHP code manipulating your image links, that should not be affected by PageSpeed because PageSpeed runs after PHP (or at least it should>
However, if you have javascript code that is sensitive to URL syntax, that can indeed be disturbed by PageSpeed's URL manipulations.  In that case, one solution is to use "pagespeed ImagePreserveUrls on" or "OptimizeForBandwidth", so that you get the benefits of image file reduction, without the risk of URL modification.
Note that in that case you will not get the benefit of image inlining or cache extension (though given your PHP you probably don't need cache extension).

The problem come from the code:
location ~* .(woff|eot|ttf|svg|mp4|webm|jpg|jpeg|png|gif|ico|css|js)$ {
expires 365d;
}
When I comment it, image is load normally.

When the code is uncommented, what happens, exactly?


Wrong indentation for comment after nested unbraced control constructs

Description
When nesting multiple control constructs (like if or for) nesting increases with each of them. When there is a comment following that, it gets indented at the level of the last control construct. Which in my opinion is even stranger than if it were indented with the level of the final leaf statement that is not a control construct.
Input and Expected Output
The code looked like this before beautification, and should have looked like this after beautification as well:
if (console.log)
    for (var i = 0; i < 20; ++i)
        if (i % 3)
            console.log(i);
// all done
console.log("done");

Actual Output
The  code actually looked like this after beautification:
if (console.log)
    for (var i = 0; i < 20; ++i)
        if (i % 3)
            console.log(i);
        // all done
console.log("done");

Steps to Reproduce
I first observed this using the<EMAIL_ADDRESS>npm package.
But I can reproduce this on http://jsbeautifier.org/ right now, with default settings.

So many ways comments can go off the rails.

Thanks for the quick fix here, particularly at this time of the year! ❤
Although https://github.com/beautify-web/js-beautify/issues/1090#issuecomment-269631990 suggests that unfortunately this is not the end of the story yet, since the fix here made issues reported there considerably more severe. 😢


Write a JavaScript program to convert binary to octal

Description
Write a JavaScript program to convert binary to octal
Binary is a base-2 number system. It uses only two digits: 0 and 1. The digits are called bits. A bit is the smallest unit of information in a computer. A bit can be either 0 or 1. A bit can be used to represent a single value, such as 0 or 1, or it can be used to represent a single state, such as on or off.
Octal is a base-8 number system. It uses only eight digits: 0, 1, 2, 3, 4, 5, 6, and 7. The digits are called octits. An octit is the smallest unit of information in a computer. An octit can be used to represent a single value, such as 0, 1, 2, 3, 4, 5, 6, or 7, or it can be used to represent a single state, such as on or off.
To convert a binary number to octal, we can use the following algorithm:

Convert the binary number to decimal.
Convert the decimal number to octal.

Input  : 1010
Output : 12

How to contribute

Save the solution in ConvertBinaryToOctal.js file
Add ConvertBinaryToOctal.js file in convert-binary-to-octal folder

I would like to contribute to this. Please assign this issue to me.

!assign


test

Just a test.
Signed-off-by: Roman Mohr<EMAIL_ADDRESS>
What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release notes:

/ok-to-test

/close
everything looks fine.
@cchengleo thanks for the help :)


Can i get any one field value on change

Hello, Dear
my console.log showing : {Id: "17", Name: "HDD", Brand: "WD", Models: "1TB DESKTOP", HASC: "84717020", …}
how to get value var rValue = $('#r-prod').inputpicker('element', 'Brand');
not a fieldText:'Name', or
fieldValue:'Id',  i actually get other value, please help me

Hi @ummar091 :+1:
Thank you very much for the feedback. I have revised the 'element' function, and you can get the specific element through the following method:
$('#r-prod').inputpicker('element', 'value', 'fieldName');

Thank you very much for best compliment.


#1st issue . Waiting_animation in /fancy dir

please review it :)

Looks good, just executable bit is missing, can you do chmod +x fancy/* and push that?

Updated as per instructions. Please review it.

LGTM 🎉, Thanks for your contribution!


bug: override not working

Did you check docs and existing issues?

[X] I have read all the neodev.nvim docs
[X] I have searched the existing issues of neodev.nvim
[X] I have searched the exsiting issues of plugins related to this issue

Neovim version (nvim -v)
0.8.2
Operating system/version
Ubuntu 22.04
Describe the bug
I'm using chezmoi to manage my dotfiles, and as such my init.lua is not in the default location.
I use the override option to setup to enable neodev in the chezmoi folder:
  override = function(root_dir, options)
    vim.pretty_print('ROOT: ' .. root_dir)
    if require('neodev.util').has_file(root_dir, os.getenv('HOME') .. '/.local/share/chezmoi') then
      options.enabled = true
      options.runtime = true
      options.types = true
      options.plugins = true
    end
  end,
}

Since a34a9e7e775f1513466940c31285292b7b8375de that function does not seem to be working anymore (or even called?)
The debug print I have in there is no longer showing up in the messages.
Steps To Reproduce

Have non-default location of init.lua
set the override option

Expected Behavior
Have the override option (or something equivalent) work again :)
Repro
-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")
-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({ "git", "clone", "--filter=blob:none", "https://github.com/folke/lazy.nvim.git", lazypath, })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  "folke/tokyonight.nvim",
  "folke/neodev.nvim",
  -- add any other plugins here
  'neovim/nvim-lspconfig',
}
require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

require('neodev').setup {
  override = function (root_dir, options)
    print('hello')
  end
}

require('lspconfig').sumneko_lua.setup {}

That has_file was an internal function, but I removed that since it's no longer used.
change it to if rootdir:find("chezmoi") then or something similar

I don't think it is calling the override function anymore. In the repro.lua I've added a print which is no longer showing up since that commit.

Closing again. It does work for me.
Use vim.notify("hello") to see the message

Okay, must be something with my machine then. Even in a docker container the notify doesn't show up.

The notify will only come up when lspconfig is configured

I added some prints and found out that in lsp.lua lua_root is nil, which causes #lua_root to error silently.
When changing ... and #lua_root to ... and lua_root and #lua_root it does continue to calling the override function.
Now just need to figure out why lua_root is nil on my setup.

Fixed!

Thanks!

correct. just updated the readme


Contribution on the core package

Can you recommend a good first issue related to the core package for a Rust beginner?
I wanted to play with Rust for a while now, it might be a good occasion 😃

I would suggest:

https://github.com/dust-tt/dust/issues/114
Relatedly, you could also play with adding a command line command to delete a DataSource.
https://github.com/dust-tt/dust/issues/144

#144 is a bit more heavy but quite straightforwrad nonetheless. Happy to take the front-end part of it if you decide to go with it.

Thx going to look at #144

Awesome. Don't hesitate to ping me here or on Discord.


filter shown lists

add filter option
filter:
 include: regex
 exclude: regex

Oh man, I was trying to use the "entity" option to drill down to something line sensor.our_groceries.XXXXXX where XXXXX was the sensor "ID" for the list. This would be a VERY helpful feature!

Also need this, please!

I've put in a PR for these kind of changes.
https://github.com/ljmerza/our-groceries-card/pull/16
It includes:
show_empty as detailed above
show_lists allows you to specify which lists are shown (using the list IDs)

added in v1.6.0 via
hide_empty_lists: true
show_lists:
  - listid


Information about CPU architecture for GitHub hosted runners

Why:
Closes #13794

What's being changed:

As mentioned by the member of GitHub Actions team, I've specified that Linux and Windows GitHub hosted runners have amd64 architecture and macOS runners have Intel x86_64 architecture.
Although the GitHub Actions member has specified that all runners are amd64 I've purposely written architecture as Intel x86_64 for macOS runners because macOS has only two archs : Intel x86_64 & arm64
Check off the following:

[X] I have reviewed my changes in staging.
[X] For content changes, I have completed the self-review checklist.

I feel that simply saying 2-core CPU (x86_64) and 3-core CPU (x86_64) would be better. Both "amd64" and "Intel x86_64" mean the same thing.

@thispsj Thanks so much for opening a PR! I'll get this triaged for review ⚡

@janiceilene No worries!

@ dependably merge

@ dependabot merge

@ dependabot close


Config file not found on running StartSECC (Properties file location './SECCConfig.properties' not found (FileNotFoundException)

I clone the project from Github, set it up in IntelliJ as a project using the PARENT pom.xml. Get all the dependencies. That all runs fine.
But when I try to run StartSECC it tells me this
MiscUtils: Properties file location './SECCConfig.properties' not found (FileNotFoundException).
Then, apparently because it can't find the config file it says
MiscUtils: No entry found in the properties file for property 'network.interface'
I've previously gone through the instructions to make sure that IPv6 is configured on this machine.
Here's a screenshot of the error.

If I run the same code in Eclipse, it tells me that there's no loopback. Stuck either way.
24 hours in to the one hour course. Linux and Win 10, and several IDEs.
H

Problem Actual, for solve IT need change route in file RISE-V2G-master\RISE-V2G-Shared\src\main\java\com\v2gclarity\risev2g\shared\enumerations\GlobalValues.java

Or you can add RISE-V2G-master\RISE-V2G-SECC to PATH.


Black screen after successful model loading ARR

I have the black screen as described in the Troubleshooting guide. However I am NOT receiving any frames from the server in the Unity app. There is nothing in the troubleshooting guide to help with this. I get no errors, and I can find no server side logs to help with the issue.
I checked the codec as described and it is current. I attempted to delete the "temp" libraries and obj folders in the Unity project. In this case it is the sample Quickstart project. I do find a folder called Libraries but not a folder call obj.  I deleted the previous build folder and the Library folder, restarted Unity and rebuilt the project. Same result, black screen.
The Quickstart app did build and operate as described in the documentation.
I used a model from a working Unity project .glb type and uploaded it to my ARR in storage container. I used the ARRT app to initiate the conversion which took just under 10 minutes. After it finished its conversion and still in the ARRT app to load and render. It did load and render.
I then followed the instruction in the Quickstart to use my custom model. I got the SAS token link from Azure portal. I ran the app in Unity using the play button. After it connected it "loaded" the model but never streamed it to Unity player. I then deployed the app to HoloLens 2 to see if it might be my computer. I got the same result, connected, loaded, but not frames received.
I am unable to find any troubleshooting for this type of error.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7101211b-84cb-9b18-1014-98549844008d
Version Independent ID: 62c85d3f-af2b-a2fd-6c20-2ef24dfc4c48
Content: Troubleshoot - Azure Remote Rendering
Content Source: articles/remote-rendering/resources/troubleshoot.md
Service: azure-remote-rendering
GitHub Login: @FlorianBorn71
Microsoft Alias: flborn

Hi, if it renders correctly in ARRT then it means that the model itself is OK. Also the fact that Quickstart did load the model without error indicates that everything went fine on that end.
However one important difference is that ARRT by default re-centers the model so its bounding box is inside the camera frustum. What I guess happened is that the source model has some kind of large translation on the root node so that by default it is out of camera range. To find out, you can check the model's bounding box as written out by the conversion. Specific instructions can be found here:
https://docs.microsoft.com/en-us/azure/remote-rendering/resources/troubleshoot#common-client-side-issues
A simple way to mitigate this specific issue is to export the model with the re-centering option enabled ('recenterToOrigin'): https://docs.microsoft.com/en-us/azure/remote-rendering/how-tos/conversion/configure-model-conversion#geometry-parameters
Does this solve the problem?
Cheers,
Florian

Did check the bounding box as was suggested in the troubleshooting guide. The largest number in the X was 167. This seemed to be in the "good" range compared to the 1000 or 2000 in the guide. However I will attempt a new rendering export.

Just for additional info, your comments seem to indicate that if the object is not in the user view then ARR does not continue to update the object? For instance, the object is in another room out of my sight?

Re: bounding box size. The units are in meters, so if the box is167m off the origin, you would not see it with a default camera far-clip distance of 20m. Unless the bounds are like -167...167, in which case you would be located in the middle of a building and you should see it.
With regard to your second question, can you elaborate what you mean by "does not continue to update an object"? The ARR server does not perform any kind of simulation on the objects, it does only rendering based on what's near the camera. So if an object is clipped because it's outside the camera view frustum, ARR will not render anything and send black video frames.
That said, you can still transform the root object. If you apply the opposite transform of the bounding box center, you can move the whole model dynamically into the origin of the scene, where it should be rendered.

Closing this because the off-center model bounds seem to have caused this. Please reopen if there is anything to follow-up.

#please-close


[stable/concourse] Lost OIDC setting on main team, when web pod restarted

Describe the bug
Lost OIDC setting of main team, when web pod recreated
Version of Helm and Kubernetes:
Helm:
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}

Kubernetes:
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
stable/concourse
version: 5.3.0
What happened:
Lost OIDC setting of main team, when web pod restarted.
What you expected to happen:
Keep the setting, when web pod restarted
How to reproduce it (as minimally and precisely as possible):
I have two teams setting: main and shared
Role setting
main_role.yaml:
roles:
- name: owner
  local:
    users: ["admin"]
  oidc:
    groups: ["/ci/main/owner"]
- name: member
  oidc:
    groups: ["/ci/main/member"]
- name: pipeline-operator
  oidc:
    groups: ["/ci/main/pipeline-operator"]
- name: viewer
  oidc:
    groups: ["/ci/main/viewer"]

shared_role.yaml:
roles:
- name: owner
  local:
    users: ["admin"]
  oidc:
    groups: ["/ci/shared/owner"]
- name: member
  oidc:
    groups: ["/ci/shared/member"]
- name: pipeline-operator
  oidc:
    groups: ["/ci/shared/pipeline-operator"]
- name: viewer
  oidc:
    groups: ["/ci/shared/viewer"]

Set up Role
TARGET=main
fly -t $TARGET set-team -n main -c main_role.yaml
fly -t $TARGET set-team -n shared -c shared_role.yaml

Check teams details
fly -t main teams --details

name/role                  users            groups
main/member                none             oidc:/ci/main/member
main/owner                 local:admin      oidc:/ci/main/owner
main/pipeline-operator     none             oidc:/ci/main/pipeline-operator
main/viewer                none             oidc:/ci/main/viewer
shared/member              none             oidc:/ci/shared/member
shared/owner               local:admin      oidc:/ci/shared/owner
shared/pipeline-operator   none             oidc:/ci/shared/pipeline-operator
shared/viewer              none             oidc:/ci/shared/viewer
Actual Results
After web  pod recreated
fly -t main teams --details

name/role                  users            groups
main/owner                 local:admin      oidc:/ci/main/owner
shared/member              none             oidc:/ci/shared/member
shared/owner               local:admin      oidc:/ci/shared/owner
shared/pipeline-operator   none             oidc:/ci/shared/pipeline-operator
shared/viewer              none             oidc:/ci/shared/viewer
The main team has only owner role setting, other setting are lost.
Anything else we need to know:
This is because in helm  concourse/templates/web-deployment.yaml , the  env  CONCOURSE_MAIN_TEAM_OIDC_GROUP reset the main team oidc setting, when web pod restarted or recreated.
{{- if .Values.concourse.web.auth.mainTeam.oidc.group }}
 - name: CONCOURSE_MAIN_TEAM_OIDC_GROUP
   value: {{ .Values.concourse.web.auth.mainTeam.oidc.group | quote }}
{{- end }}

The main/owner role was define by web web-deployment env. Other addition oidc setting are set up after concourse deployed through fly cli.
If does not set  {{ .Values.concourse.web.auth.mainTeam.oidc.group  }}, the oidc setting of main team will remove completely.

We can mount roles config yaml into pods, and  in {{ .Values.concourse.web.auth.mainTeam.config }} set up the yaml file path.
In values.yaml at {{ .Values.concourse.web.auth.mainTeam }} add
mainTeam:
  config: /concourse-auth/main-roles-config
  mainRolesConfig: |-
    roles:
    - name: owner
      local:
        users: ["admin"]
      oidc:
        groups: ["/ci/main/owner"]
    - name: member
      oidc:
        groups: ["/ci/main/member"]
    - name: pipeline-operator
      oidc:
        groups: ["/ci/main/pipeline-operator"]
    - name: viewer
      oidc:
        groups: ["/ci/main/viewer"]

In web-secrets.yaml add
  {{- if .Values.concourse.web.auth.mainTeam.mainRolesConfig }}
  main-roles-config: {{ .Values.concourse.web.auth.mainTeam.mainRolesConfig | b64enc | quote }}
  {{- end }}

In web-deployment.yaml add
at spec.template.spec.volumes[name: auth-keys] add
{{- if .Values.concourse.web.auth.mainTeam.mainRolesConfig }}
- key: main-roles-config
  path: main-roles-config
{{- end }}

Your can also use {{.Values.web.additionalVolumeMounts}} and  {{.Values.web.additionalVolumes}} mount your roles config yaml.


undefined: stringsutil.SliceTrimSpace

Hi,
I'm running into the following issue when running go get github.com/grokify/swaggman from my project:
$ go get github.com/grokify/swaggman
# github.com/grokify/swaggman/swagger2
../../../../pkg/mod/github.com/grokify/swaggman@v1.0.0/swagger2/count.go:15:9: undefined: stringsutil.SliceTrimSpace

This is a project using Go Modules and Go v1.13.8.
Let me know if I can provide any more info.

Thanks for reporting this.
This should be fixed by updating the following packages:

github.com/grokify/swaggman
github.com/grokify/gotilla

To get your updates I had to pin to the latest commit with:
go get github.com/grokify/swaggman@80df5a79d115c2c5fbe470d3d2c0beaabb3f1e42

I think otherwise it uses the latest release which from I can see doesn't have any dependency management so whenever you to try to install it will use the latest versions of any dependencies hence why this issue happened.
It may be worth doing another release as I don't think currently anyone can install this using go get github.com/grokify/swaggman?

Thanks for the feedback. I've been meaning to spend more time with Go Modules. I've experimented with it but don't understand all the details yet.

Go modules is now supported with 25d9c04690c6b988a9ae8ab89331b30d2ed160c0


Scroll Bar style modified

Scroll bar style has been modified to give a better look to the website

Thanks for Contributing.


🐛 [firebase dynamic links ] cannot open app in ios

firebase_dynamic_links: ^0.7.0
when clicked the dynamic link app ,cannot open the app , it will show a preview page, and didnot open the app

void initFireBaseDynamicLinks() async {
FirebaseDynamicLinks.instance.onLink(
onSuccess: (PendingDynamicLinkData dynamicLink) async {
final Uri deepLink = dynamicLink?.link;
Get.log('deeplink $deepLink');
if (deepLink != null) {
Future.delayed(Duration(seconds: 1), () {
Navigator.push(
Get.context,
MaterialPageRoute(
builder: (context) => WebViewPage(
url: "${ImageConfig.siteHostName}/appBlankPage",
redirectUrl: deepLink.toString())));
});
}
}, onError: (OnLinkErrorException e) async {
print('onLinkError');
});
final PendingDynamicLinkData data = await FirebaseDynamicLinks.instance.getInitialLink();
final Uri deepLink = data?.link;
Get.log('deeplink2 $deepLink');
}

void initFireBaseDynamicLinks() async {
FirebaseDynamicLinks.instance.onLink(
onSuccess: (PendingDynamicLinkData dynamicLink) async {
final Uri deepLink = dynamicLink?.link;
Get.log('deeplink $deepLink');
if (deepLink != null) {
Future.delayed(Duration(seconds: 1), () {
Navigator.push(
Get.context,
MaterialPageRoute(
builder: (context) => WebViewPage(
url: "${ImageConfig.siteHostName}/appBlankPage",
redirectUrl: deepLink.toString())));
});
}
}, onError: (OnLinkErrorException e) async {
print('onLinkError');
});
final PendingDynamicLinkData data = await FirebaseDynamicLinks.instance.getInitialLink();
final Uri deepLink = data?.link;
Get.log('deeplink2 $deepLink');
}

FirebaseDynamicLinks.instance.onLink cannot listen , when click the email

FirebaseDynamicLinks.instance.onLink cannot listen , when click the email

Hi @wangxingxing123654
Could you please provide your flutter doctor -v and flutter pub deps -- --style=compact?
Thank you

Hi @wangxingxing123654
Could you please provide your flutter doctor -v and flutter pub deps -- --style=compact?

Thank you

yes.thank u very much

Hi @wangxingxing123654
Could you please provide your flutter doctor -v and flutter pub deps -- --style=compact?
Thank you

yes.thank u very much

Hi @wangxingxing123654
Could you please provide your flutter doctor -v and flutter pub deps -- --style=compact?
Thank you

yes.thank u very much

but android worked very well

Hi @wangxingxing123654
Could you please provide your flutter doctor -v and flutter pub deps -- --style=compact?
Thank you

yes.thank u very much

but android worked very well

Are you getting the data from the link correctly?
Have you tried running the official example code?

Are you getting the data from the link correctly?
Have you tried running the official example code?


Fix negative tests in Table Tests to close connection when panic

Purpose

$subject

Changes are covered with https://github.com/ballerina-platform/ballerina-lang/pull/12903
Will be merged along with the same.


Update graphql-tools monorepo

I'm getting the error Unable to serve mesh: Unable to merge GraphQL type "Query": asNext is not iterable when there are multiple sources defined in the mesh.yaml.
It seems like when trying to merge directives, it fails because it's undefined (on the node).
I'm still trying to understand exactly how all of this works, is there any context here that would be helpful in finding out what's going on?

Turns out graphql-compose creates nodes with directives as undefined (which is valid). Fix for graphql-tools is here: https://github.com/ardatan/graphql-tools/pull/1671


[SPARK-15962][SQL] Introduce additonal implementation with a dense format for UnsafeArrayData

What changes were proposed in this pull request?
This PR introduces two implementations for UnsafeArrayData. One is an sparse array UnsafeArrayDataSparse that is the original format of UnsafeArrayData. The other is a dense array UnsafeArrayDataDense that is a new format of UnsafeArrayData.
UnsafeArrayDataSparse accepts null value in each entry of an array. However, it increases memory footprint (e.g. 2x for an integer array) due to existence of offsets area. In some cases (e.g. an unsafe array is created from an primitive array by a method fromPrimitiveArray, we know there is no null value in an array. UnsafeArrayDataDense can reduce memory footprint since it does not have offsets area.
Example of memory foot comparision:
1024x1024 elements integer array
Size of baseObject for UnsafeArrayDataSparse: 4 + 1024x1024 + 1024x1024 = 2M bytes
Size of baseObject for UnsafeArrayDataDense: 4 + 4 + 1024x1024 = 1M bytes
How was this patch tested?
Tested by existing unit tests

Test build #60556 has started for PR 13680 at commit 639b32d.

Test build #60556 has finished for PR 13680 at commit 639b32d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60556/
Test FAILed.

Merged build finished. Test FAILed.

Test build #60566 has started for PR 13680 at commit d06d200.

Test build #60566 has finished for PR 13680 at commit d06d200.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60566/
Test FAILed.

Test build #60568 has started for PR 13680 at commit 6c09d72.

Test build #60568 has finished for PR 13680 at commit 6c09d72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60568/
Test PASSed.

Thanks for working on it! One of my concern is: do we really need 2 unsafe array implementations? For the UnsafeArrayDataDense, can we follow unsafe row and introduce a null-bits to make it support null value? Then we can have a unified format: [numElements] [null bits] [offset or primitive values] [values]

Test build #61095 has started for PR 13680 at commit 85f862c.

@cloud-fan thank you for your good comment. I also read previous proposal.
I love to have only single format (or implementation). Since I thought that there are some reasons to keep the old format, I introduced a new dense format.
IMHO, a new unified format should have three properties.

Remove indirect offset (for performance and footprint)
Have capability of presence of nullbit (for generality)
Quickly get information on existence of null value in an array (for performance, in particular, primitive array)

Based on them, how about this single format?
[numElements] [all zero in null bits?] [null bits] [values] [variable length portion]

If we want to reduce memory footprint in the case of primitive array, we can drop [null bits] part if [all zero in null bits?] has a special value.

null bits won't take a lot of memory(1 bit per element), and having the all zero in null bits? flag will slow down elements retrievement: we need an extra if branch to check this flag, and make the format more complex. I'd like to always have the null bits region, cc @davies

It is OK to always keep [null bits]
One question: Is this format to keep fixed space for [values]? I mean if [null bit] is true, the corresponding element in [value] occupy space. It allow us to always access all of the elements by 4 + (null bits size) + (element size) * offset.
If an answer of the above question is yes, I agree with no [all zero in null bits?]. Then,  I will do prototype for this.
cc @hvanhovell

Test build #61095 has finished for PR 13680 at commit 85f862c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61095/
Test PASSed.

@kiszk yea, even the null bits is true, the element still take space at [offset or fixed-length values] region.

Good to hear. I will make an implementation for single format. If I would meet some issues, I will raise them here.

Thanks! Feel free to ask any questions!

@cloud-fan , I have one question about null field. Should we put zero into the corresponding field to position where setNullAt() is called as UnsafeRow does.
If we avoid to put zero, this avoidance affects two properties.

row equality
undetermined value may be included in the array returned by UnsafeArrayData.to<PrimitiveType>Array()

In my current implementation, a width of each element depends on element type (4: Int, 8: Double, etc). Thus, it is hard to do the same approach as it did. Since  UnsafeRow always use 8 bytes per field. Since we want to make data conversion fast between primitive array and unsafe array, we have to keep the type-based element-width (e.g. 4: Int, 8: Double, etc).
What do you think? Should we have to keep the above two property by clearing a field?

One potential performance issue is that we have to always clear all of null bits at UnsafeArrayWriter.initialize(). This is because holder.buffer is reused for each row. If one row has more than one variable length arrays, holder.cursor may have different offset for the same array among different rows. Thus, we have to clear all of null bits.
This overhead would be larger in machine learning use cases that use large arrays.
If we have a flag to show whether there is an area of [null bits] or we have two implementations with [null bits] and without [null bits], we may alleviate performance overhead.
What do you think?
cc: @hvanhovell , @mengxr

@kiszk we should definitely put zero into the corresponding field when set null. It will be a little harder than UnsafeRow, as we need setNullBoolean, setNullInt, etc. but it's still doable.
about clear out all null bits, yea, it's a big overhead for array with small element like boolean array, but I'm not sure this worth 2 different implementations, cc @rxin

@cloud-fan , for the first issue, we are on the same page. Your proposal is what I am thinking about as possible solutions. I will do that.
For the second issue, it seems to be design choice between

introduce one conditional branch in isNullAt() in one implementation
have two implementations without conditional branch at isNullAt()

having 2 implementations is also kind of a branch: the virtual function call need to be dispatched between these 2 implementations, while the only one implementation can be marked as final and doesn't have this overhead.

I see. I assumed that virtual call will be devirtualized by declaring final method and by optimistically propagating type information in the JIT compiler. Would it be better to add a flag like containNull ?

I don't have a strong preference here, each choice has its advantage and weakness:

alway have the null bits region: faster element access(read the null bits and then read the element), slower writing(always need to clear out the null bit region for primitive array)
only have null bits region if containsNull flag is true: slower element access(read flag then read null bits then read the element), faster writing(check the containsNull flag and don't need to clear out null bits if flag is false)

cc @davies

I am kind in favor of the single implementation for a couple of reasons:

Declaring methods final is not a magic bullet. If you are invokingisNull or get* on the common ancestor it is still a virtual call. If you are only using two classes the JVM is clever enough to add a branch in the code and call one of the two methods depending of the type (it is probably also clever enough to inline the methods). We could get around this if we can use the concrete type during code generation (which should be doable).
Nulling out the null region is IMO not really a problem. JVMs nowadays are very good at optimizing array fill like operations. See this post for some more information: http://psy-lob-saw.blogspot.com/2015/04/on-arraysfill-intrinsics-superword-and.html
On our side code generation should be clever enough not to do null checks for non-nullable fields (is it?). So we are not incurring useless branch here.

It would help to have a number of performance benchmarks.

We should do this more holistically, i.e. thinking about what we want to do with primitive arrays for machine learning and how to handle everything end to end. Let's not rush an implementation change just for the low level data structure.

Test build #61218 has started for PR 13680 at commit 517aa72.

Test build #61218 has finished for PR 13680 at commit 517aa72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61218/
Test FAILed.

Test build #61222 has started for PR 13680 at commit bf12e72.

Test build #61222 has finished for PR 13680 at commit bf12e72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61222/
Test FAILed.

Test build #61233 has started for PR 13680 at commit 682c397.

Test build #61233 has finished for PR 13680 at commit 682c397.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61233/
Test FAILed.

Test build #61235 has started for PR 13680 at commit 687be4d.

Test build #61235 has finished for PR 13680 at commit 687be4d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61235/
Test FAILed.

Test build #61236 has started for PR 13680 at commit 138810b.

Test build #61236 has finished for PR 13680 at commit 138810b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61236/
Test PASSed.

Test build #61239 has started for PR 13680 at commit 500e978.

Test build #61239 has finished for PR 13680 at commit 500e978.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61239/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan and @hvanhovell thank you for your comments. Based on your comments, I implemented UnsafeArrayData by using one implementation with explicit clearing null bits by Arrays.fill().
I would appreciate it if you review this PR.

@rxin thank you for your comment. As you said, holistic view is important. This PR is not only for machine learning.
This PR has another use case for improving projection of an array in any program by https://github.com/apache/spark/pull/13911. This PR can also reduce memory footprint of an array representation and DataFrame.cache()/Dataset.cache() in any program.
What do you think?

looks pretty good, could you add a benchmark in this PR? an example is https://github.com/apache/spark/pull/12640/files#diff-b118a818177121a108fa92d0354871a6R26

Test build #61362 has started for PR 13680 at commit 9c113aa.

Test build #61362 has finished for PR 13680 at commit 9c113aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61362/
Test PASSed.

Test build #61377 has started for PR 13680 at commit 8b1cbbb.

Test build #61377 has finished for PR 13680 at commit 8b1cbbb.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class UnsafeArrayDataBenchmark extends BenchmarkBase

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61377/
Test FAILed.

Test build #61379 has started for PR 13680 at commit 45d2ec2.

Test build #61379 has finished for PR 13680 at commit 45d2ec2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61379/
Test PASSed.

@cloud-fan , could you please review this again?
I added benchmark programs and their results, and addressed your review comments.

can we also rerun UDTSerializationBenchmark? I think it should be faster now.

Test build #61620 has started for PR 13680 at commit 6a8a97f.

Test build #61620 has finished for PR 13680 at commit 6a8a97f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61620/
Test PASSed.

Test build #61660 has started for PR 13680 at commit 7576c19.

Test build #61660 has finished for PR 13680 at commit 7576c19.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61660/
Test FAILed.

Test build #61671 has started for PR 13680 at commit 2cf96b4.

Test build #61671 has finished for PR 13680 at commit 2cf96b4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61671/
Test FAILed.

Merged build finished. Test FAILed.

Test build #61673 has started for PR 13680 at commit 243252a.

Test build #61673 has finished for PR 13680 at commit 243252a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61673/
Test PASSed.

Merged build finished. Test PASSed.

One more thought about the format: UnsafeRow use 8 bytes to store offset and length for variable-length type, this is because UnsafeRow is word-aligned, so we can't calculate the element size by subtracting 2 adjacent offsets, but need an extra length information. This is also why in unsafe row writer we call zeroOutPaddingBytes before we write variable-length field.
However, UnsafeArrayData doesn't have this requirement, as it's already a field of row and itself will be word-aligned. So we can:

only use 4 bytes to store offset for variable-length type element, and calculate the length by subtracting 2 adjacent offsets.
still use 8 bytes to store offset and length, and follow unsafe row writer to zeroOutPaddingBytes before write.

Personally I prefer option 1 as it's more compact.

Option 1 can work for this array: UnsafeDataArray: ...[integer] [offset] [offset] [float]. This is because 2 offsets are adjacent.
Can option 1 work for this UnsafeDataArray: ...[integer] [offset] [float] [offset]? How do we subtract 2 offsets that are not adjacent?

hmmm, looks like we are not in the same page... How could 2 offsets not adjacent? We only keep offsets in the value or offset region, and put them one by one.

Sorry. This is my misunderstanding. I was confused between UnsafeRow and UnsafeArrayData. UnsafeArrayData keeps only one type in an instance.
[integer] [offset] [float] [offset] is invalid. [integer] [integer] [integer] or [offset] [offset] [offset] is valid. Option 1 could work.

Test build #61731 has started for PR 13680 at commit 1771ba3.

Implemented 4-byte offset instead of 8-byte length&offset

Test build #61731 has finished for PR 13680 at commit 1771ba3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61731/
Test FAILed.

@cloud-fan Option 1 did not work for some cases. If UnsafeArrayWriter.write() is not called for all of elements, e.g. only some of elements are written. For example, this code writes only some of UTF8 elements.
Is it better to take option 2?

how about [offset1] [offset2] [not written, use offset3] [offset3]? Then we are still able to calculate the length by subtracting adjacent offsets.

I think that it is not easy to put [not written, use offset3] with good performance. I am thinking about two cases.
In case 1, my assumptions are

Do not initialize [offset area] before writing offsets for performance
Order of writing elements may not be ascending

Here, the following two steps are executed.

writ e offset0
write offset3
At step 2, it is not easy to determine which [not written ] fields should be filled by [ use offset3]. This is because we cannot assume any values in [not written], and then hard to recognize [offset0] has been written.

offset: 0             1             2             3             4
init :  [not written] [not written] [not written] [not written] [not written]
step1:  [offset0]     [not written] [not written] [not written] [not written]
step2:  [offset0]     [not written] [not written] [offset3]     [not written]

In case 2, my assumptions are

Initialize [offset area] by zero before writing offsets. But, it may lead to performance issue.
Order of writing elements may not be ascending

Here, the following two steps are executed.

writ e offset4 and fill all of predecessor fields, which have not been written, by using [use offset4]
write offset2and fill all of predecessor fields, which have not been written, by using [use offset2]
This approach always check and fill all of predecessor fields until a field, which have been written, is found.

offset: 0             1             2             3             4
init :  [zero       ] [zero       ] [zero       ] [zero       ] [zero       ]
step1:  [use offset4] [use offset4] [use offset4] [use offset4] [offset4]
step2:  [use offset2] [use offset2] [offset2]     [use offset4] [offset4]

What do you think?

The old unsafe array format also store offsets only, can we follow it to handle this case? BTW, when you say not written, do you mean writing null?

Ah, you are right. Exactly, this code writes  UTF8 elements into only some fields, and write null into other fields in ascending order. E.g. (4-bit nullbits)[0010] [offset0] [offset1] [0] [offset3].
We could change it into (4-bit nullbits)[0010] [offset0] [offset1] [-offset3] [offset3]. But, it fills non-zero into a field that have null. Is it acceptable? If this property is acceptable, I will do that.

(4-bit nullbits)[0110] [offset0] [-offset3] [-offset3] [offset3]

Why negative the offset3? The old format negative it to indicate nullability, but we have null bits now.

it fills non-zero into a field that have null

This is fine. We zero out the null field instead of leaving some random bytes there because we wanna make sure equality is deterministic. The new rule(using next offset for null element) is also deterministic.

To use negative is for ease of debugging. We could use positive value, too. Is the positive value preferable to you?

This is fine. We zero out the null field instead of leaving some random bytes there because we wanna make sure equality is deterministic. The new rule(using next offset for null element) is also deterministic.

Sure, that makes sense. I have implemented it in my environment already. When we decide to use positive or negative value, I will push it soon.

I think using negative has some performance penalty, we need to call math.abs when calculating the element length. What do you think?

Test build #61839 has started for PR 13680 at commit 65fca21.

Test build #61839 has finished for PR 13680 at commit 65fca21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61839/
Test PASSed.

Test build #61917 has started for PR 13680 at commit e6da751.

Test build #61917 has finished for PR 13680 at commit e6da751.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61917/
Test FAILed.

LGTM except some minor comments, it's pretty close! One easy-to-ignore comment: https://github.com/apache/spark/pull/13680/files#r69849567

Test build #61972 has started for PR 13680 at commit 04d2e4b.

the code LGTM. @kiszk can you improve the PR description to explain more about the new format? And can you comment on the benchmark result in the benchmark file with benchmark result before your PR? Then it's easier for other reviewers to compare the results.
This is a large path, I'm asking more people to review it. Thanks for working on it!

Test build #61972 has finished for PR 13680 at commit 04d2e4b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61972/
Test FAILed.

retest this please

Test build #61977 has started for PR 13680 at commit 04d2e4b.

@cloud-fan , thank you for a lot of great review comments. This PR looks much better than the first commit.
I updated the description of this PR. Is it fine with you?

Test build #61977 has finished for PR 13680 at commit 04d2e4b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61977/
Test PASSed.

yea LGTM. And please make sure the benchmark results are up to date, as the code has changed a lot of times.

I double-checked benchmark results, and corrected results in Write UnsafeArrayData. Now, all of them are up to date.

@cloud-fan How about status of other reviewers?

We need inputs from @davies , but unfortunately he is on vacation now...

Thank you for letting us know the status. I did not know @davies is on vacation.

retest this please

cc @davies can you review this? thanks!

Test build #63144 has started for PR 13680 at commit 04d2e4b.

Test build #63144 has finished for PR 13680 at commit 04d2e4b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63144/
Test PASSed.

Merged build finished. Test PASSed.

Test build #63301 has started for PR 13680 at commit 7b4e819.

Test build #63301 has finished for PR 13680 at commit 7b4e819.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63301/
Test FAILed.

Test build #63318 has started for PR 13680 at commit 87aca80.

Test build #63318 has finished for PR 13680 at commit 87aca80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63318/
Test PASSed.

Test build #63467 has started for PR 13680 at commit d9a269d.

Test build #63467 has finished for PR 13680 at commit d9a269d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63467/
Test PASSed.

@davies could you please review this again since I addressed comments?

@cloud-fan @davies Are there additional things for me to do to merge this PR?

see https://github.com/apache/spark/pull/13680#discussion_r74798477
We'd like to change the format to:
[numElements] [null bits] [offset or primitive values] [values]
in the [offset or primitive values], we store the primitive values directly, or store length and offset for variable length element, like UnsafeRow
in the [values], we always align each element to 8 bytes.

@cloud-fan I see. I think that we have already done 1. I will address 2. Am I correct?

in the [offset or primitive values], we store the primitive values directly, or store length and offset for variable length element, like UnsafeRow
in the [values], we always align each element to 8 bytes

for 1, currently we only store offsets for var-length element, we should store offset and length as the element content will be word-aligned.

You are right. I am addressing both.

Test build #64006 has started for PR 13680 at commit cd28879.

Test build #64006 has finished for PR 13680 at commit cd28879.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64006/
Test FAILed.

Test build #64121 has started for PR 13680 at commit f418f9c.

Test build #64121 has finished for PR 13680 at commit f418f9c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64121/
Test FAILed.

I am investigating reasons of these failures, but I have not found a root cause yet.

Test build #65156 has started for PR 13680 at commit 813cea6.

Test build #65156 has finished for PR 13680 at commit 813cea6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65156/
Test PASSed.

@cloud-fan Could you please review this since all tests are passed.

ping @cloud-fan

@cloud-fan would it be possible to review this? I think that I implemented your suggestions.

retest this please

Test build #65643 has started for PR 13680 at commit 813cea6.

Sorry for the delay, it's pretty close! I left some minor comments, thanks for working on it!
BTW, can you rerun the benchmark and update your PR description? The word alignment may have some impact.

Test build #65643 has finished for PR 13680 at commit 813cea6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65643/
Test PASSed.

Test build #65651 has started for PR 13680 at commit 1333247.

Test build #65651 has finished for PR 13680 at commit 1333247.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65651/
Test PASSed.

Test build #65657 has started for PR 13680 at commit 11d930a.

Test build #65657 has finished for PR 13680 at commit 11d930a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65657/
Test PASSed.

Test build #65676 has started for PR 13680 at commit f19020f.

Test build #65676 has finished for PR 13680 at commit f19020f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65676/
Test PASSed.

When I rerun the benchmark, I got the performance regression only for Write UnsafeArrayData with double. I am investigating reasons.

Test build #65920 has started for PR 13680 at commit 2ef6e3b.

Jenkins, retest this please

Test build #65930 has started for PR 13680 at commit 2ef6e3b.

Test build #65920 has finished for PR 13680 at commit 2ef6e3b.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65920/
Test FAILed.

Test build #65930 has finished for PR 13680 at commit 2ef6e3b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65930/
Test PASSed.

@cloud-fan could you please review this again?
I addressed your comments. After rebasing, the performance issue has been solved.

thanks for your great work! merging to master!


Handle interrupt exceptions consistently in the codebase

Summary
Interrupt exceptions are handled in different ways throughout the codebase, some of which are not following best practice which can lead to bugs or unexpected behavior.
Define a convention for handling them and apply this to the codebase. Ideally add static analysis to ensure the conventions are adhered to.
For example, if an interrupt is logical as part of the operation, propagate the interrupt after possible cleanup steps.
Alternatively, if the method can logically handle the interrupt, handle it within the method and propagate the interrupt flag before returning to the caller.
Urgency
We can avoid unexpected and hard to debug runtime behavior.

Will reconsider it when exception redesign. close it for now, feel free to reopen it and contact us if this is a strong request.


Hope this doesnt happen to you to...

https://github.com/DUOLabs333/photopea/issues/19

Don't think it will, since the creator, not I, gets money from the ad revenue when people use this app.

You get ads?  I dont, I get one of these two......

@PAEz Shoot, I'm not sure... but I certainly didn't write in any adblocking code...

@PAEz Got one


[patch] Adding support to Tesla OpenTelemetry and telemetry events

Pull Request Test Coverage Report for Build 5e6870a537268f1f7b296877ffefe8bab8e9d505-PR-34

16 of 27 (59.26%)  changed or added relevant lines in 5 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.7%) to 84.837%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

lib/connection/executor/client.ex
0
1
0.0%

lib/connection/executor/request_executor.ex
4
8
50.0%

lib/telemetry.ex
10
16
62.5%

Files with Coverage Reduction
New Missed Lines
%

lib/connection/executor/request_executor.ex
1
69.23%

Totals

Change from base Build 44d5fa42478125b8737d5c96832b86f06fc10d5a:
-0.7%

Covered Lines:
649

Relevant Lines:
765

💛  - Coveralls


Feature : Switch back and forth between the previously focused workspace when focusing the current workspace.

Switch back and forth between the previously focused workspace when focusing the current workspace. instead of using :
 alt-tab = 'workspace-back-and-forth'
for example :
If i am on workspace 1, i select alt+3 for workspace 3, if i click again on alt+3, it will take me back to the previous workspace '1'
  toggle_workspace_on_refocus: true

It is already possible. Please read the documentation

Thank you for your response. Can you please send me the link to where i can find this ?
i finished setting up aerospace, and it's the only thing that i'm missing in my config. I didn't find it when looking in the docs


Rasa NLU imbalancing and Features

Rasa NLU version:
Operating system (windows, osx, ...):
Content of model configuration file:

Issue:
How do I set the yaml file so that it runs spacy_scikit while performing balancing. In addition, I also would like to add negation features (e.g. not, never, no) in the pipeline.
also, is it necessary to factor in balancing when running the tensorflow_embedding pipeline?
Thank you for your help

Usually imbalance happens because you have a lot of examples of one intent, where you really only want to train the entities. In the json format, you can simply leave out the intent field and these won't be used to train the intent classifier.
If you want to do anything more sophisticated you'll have to write code (e.g. subclass a classifier), that's not something you can do in config

Hello Thanks for the answer,
Regarding the balancing issue (for intent classification), is there a rule of thumb when it comes to finding the optimal distribution for each of the categories?
Would increasing the number of records of one category reduce the accuracy of another?
thank you

that's a very broad ML question :)
please post it in the forum - there will be many users who can share their experience


Update to three >= r91

Without these changes recent versions of three cause raycasting to spew infinite warnings to the console. This patch makes it compatible with everything north of r91. (Prior to that is now broken because of this renaming.)

Resolves #14.

https://rawgit.com/mquander/threejs-fast-raycast/three-update/example/bundle/boundsTree.html
Looks good! Thanks for updating it!


freerdp server peer handle custom channel error.

I send a msg "Hello server!" from rdp client custom virtual channel to freerdp server,but obviously the server did not process this message correctly.
static BOOL WTSReceiveChannelData(freerdp_peer* client, UINT16 channelId, const BYTE* data,
                                  size_t size, UINT32 flags, size_t totalSize)
{
	UINT32 i;
	BOOL status = FALSE;
	rdpMcs* mcs = client->context->rdp->mcs;

	for (i = 0; i < mcs->channelCount; i++)
	{
		if (mcs->channels[i].ChannelId == channelId)
		{
			rdpPeerChannel* channel = (rdpPeerChannel*)mcs->channels[i].handle;

			if (channel)
				status = WTSProcessChannelData(channel, channelId, data, size, flags, totalSize);
			break;
		}
	}

	return status;
}

will return FALSE if this channle if a custom channel.

@RyanCode this is an issue tracker not an interactive how to learn programming tutorial

it is fundamental that YOU provide enough information about what you´re doing and what you expect (we are no fortune tellers here)
MS-RDPBCGR is what you need to understand

I do not know what are you taking about.I'am here not for how to learn programming.I doubt if you have read my first sentence, I repeat it here,I send a msg "Hello server!" from rdp client custom virtual channel to freerdp server.But the server was immediately shut down.As for the code I posted, what I want to say is that I got channle null here

well, then tell me what should happen if a client sends random garbage?


Remove bash dependency from scripts in glibc

Known occurances:
* ldd

The issue with ldd has been solved. Currently looking for more occurrences before shipping a new glibc builld.

additional scripts discorved in glibc

xtrace
sotruss

546a49660094cac1edcfd26b26f66b84e2e4a5ff closes this.

New changes were unstable and had to be reverted.

The fix has now been applied without changing to 2.31


Components Annotation Schema

Add reference schema

We can document this in file SCHEMA.md

A small review of the content of schema.md may be done, but most of the descriptions and examples are now provided.


Please support SARIF output format

Please support the SARIF standard as an output format.
You may want to point to the requriements.txt file and the lines where the vulnerable dependencies are specified.

Sounds great, looking forward to it! Is this also why the docs https://docs.safetycli.com/safety-docs/output/sbom-output is empty?

@SafetyQuincyF how can we produce the SARIF output format? I can't find "SARIF" via the safety docs search


Properly reset blend mode when resetting canvas in compatibility renderer

Fixes: https://github.com/godotengine/godot/issues/72488
Fixes: https://github.com/godotengine/godot/issues/72918
When using BLEND_SUB the blend equation is changed, so it needs to be reset before rendering. In 3D it is already reset before rendering happens

Thanks!


[bug]: timeout waiting for funding confirmation: channel still appears in logs

Background
A peer tried to open a channel with my node, but never published the funding transaction. After a while (two weeks?) my lnd experienced a timeout and cancelled the funding flow:
2022-11-28 09:55:47.019 [WRN] FNDG: canceled waiting for funding confirmation, stopping funding flow for ChannelPoint(xxx:1)
2022-11-28 09:55:47.017 [DBG] CNCT: ChannelArbitrator(xxx:1): attempting state step with trigger=chainTrigger from state=StateDefault
2022-11-28 09:55:47.020 [DBG] CNCT: ChannelArbitrator(xxx:1): new block (height=765010) examining active HTLC's
2022-11-28 09:55:47.026 [DBG] CNCT: ChannelArbitrator(xxx:1): checking commit chain actions at height=765010, in_htlc_count=0, out_htlc_count=0
2022-11-28 09:55:47.032 [DBG] CNCT: ChannelArbitrator(xxx:1): no actions for chain trigger, terminating
2022-11-28 09:55:47.032 [DBG] CNCT: ChannelArbitrator(xxx:1): terminating at state=StateDefault
2022-11-28 09:55:47.027 [ERR] FNDG: Unable to advance pending state of ChannelPoint(xxx:1): timeout waiting for funding tx (xxx:1) to confirm

However, the channel still seems to exist and causes disk backups and useless log entries:
2022-11-28 09:56:41.554 [DBG] CNCT: ChannelArbitrator(xxx:1): attempting state step with trigger=chainTrigger from state=StateDefault
2022-11-28 09:56:41.560 [DBG] CNCT: ChannelArbitrator(xxx:1): new block (height=765011) examining active HTLC's
2022-11-28 09:56:41.571 [DBG] CNCT: ChannelArbitrator(xxx:1): checking commit chain actions at height=765011, in_htlc_count=0, out_htlc_count=0
2022-11-28 09:56:41.571 [DBG] CNCT: ChannelArbitrator(xxx:1): no actions for chain trigger, terminating
2022-11-28 09:56:41.571 [DBG] CNCT: ChannelArbitrator(xxx:1): terminating at state=StateDefault
2022-11-28 10:04:32.339 [DBG] CNCT: ChannelArbitrator(xxx:1): attempting state step with trigger=chainTrigger from state=StateDefault
2022-11-28 10:04:32.344 [DBG] CNCT: ChannelArbitrator(xxx:1): new block (height=765012) examining active HTLC's
2022-11-28 10:04:32.348 [DBG] CNCT: ChannelArbitrator(xxx:1): checking commit chain actions at height=765012, in_htlc_count=0, out_htlc_count=0
2022-11-28 10:04:32.352 [DBG] CNCT: ChannelArbitrator(xxx:1): no actions for chain trigger, terminating
2022-11-28 10:04:32.355 [DBG] CNCT: ChannelArbitrator(xxx:1): terminating at state=StateDefault
2022-11-28 10:04:32.440 [WRN] CHBU: Replacing disk backup for ChannelPoint(xxx:1) w/ newer version
2022-11-28 10:04:32.505 [WRN] CHBU: Replacing disk backup for ChannelPoint(xxx:1) w/ newer version
2022-11-28 10:04:32.558 [WRN] CHBU: Replacing disk backup for ChannelPoint(xxx:1) w/ newer version
2022-11-28 10:10:54.199 [DBG] CNCT: ChannelArbitrator(xxx:1): attempting state step with trigger=chainTrigger from state=StateDefault
2022-11-28 10:10:54.205 [DBG] CNCT: ChannelArbitrator(xxx:1): new block (height=765013) examining active HTLC's
2022-11-28 13:17:58.196 [DBG] CNCT: ChannelArbitrator(xxx:1): terminating at state=StateDefault

Your environment

lnd v0.15.5-beta
Linux server 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
bitcoind 0.24

I don't think the channel is removed from the contractcourt when c.CloseChannel is called in the funding manager

I can take this one if no one else is working on this yet.

I don't think anybody is working on this @morehouse

I repro'd the issue locally.
@C-Otto The quick fix to the log spam is to restart LND.
As @Crypt-iQ suspected, after the timeout cancels the funding flow, the channel arbitrator isn't notified to clean up its state.  However, the arbitrator does correctly forget the canceled channel after LND is restarted.
It would be good to fully clean up all the state without a restart.  I'll see what we can do here.


Look for .eslintrc up the tree

Can this please work like any other eslint plugin? Doesn't seem to lookup eslintrc from project.

I retested this and it works for me. The plugin actually uses the eslint API to do this. So if you use a decent eslint version it should work.
@wmira could you provide a test case demonstrating the failed behavior so that I can investigate.

Hi @dbaeumer  initially I thought it wasn't looking it up but it looks like the plugin just does not work. I loaded the same project on Atom and Code and Code wasn't showing eslint errors. Do I need to tell it where to look for eslint? Can it use the global eslint? I really like to try VSCode but not having a linter really is stopping me from trying it out. I'm on Ubuntun 14.04 btw.
Is there a console (sorry new with vscode) I can look at to see any errors in the plugin?
Thanks.

@dbaeumer I just tried this again with 1.2 and eslint works now. Closing this.


(fix Input) choose file text color in dark mode

This pull request fixes the issue with the choose file text color in dark mode. The changes ensure that the text color is correctly displayed in dark mode.
the reason this bug doesn't happen in the shadcn website is because it sets color-scheme:dark on the document;
this fix makes sure this always works even if color scheme isn't set.
Closes #4842

You need to run pnpm run build:registry to change the registry of file content

You need to run pnpm run build:registry to change the registry of file content

done


[Feature Request] Split file deletions

Right now all table related files are deleted in bulk.
Separate deletion of .res, .ini., .pov and .vbs files.

Fixed with 2.20.2


style: remove spacing for tabs

#502
updated the tabs widget as specified in the issue, updated for divider and removed spaces before the tabs, tell me if any further changes required.

can someone please explain these check?

Continuous Integration / lint (pull_request) failed because your commit message has to start with the name of the task you've done, here's a few examples:

docs: import prelude::* in doc examples
chore: only run check pr action on pull_request_target events

List of available tasks here ; I guess changing the name of the PR is not enough!
Continuous Integration / coverage (pull_request) failed because it seems some tests failed

Just to close the loop on this - this PR needs to hold off a minute before some feedback on https://github.com/ratatui-org/ratatui/issues/502 is resolved. I want the change, and the change is probably relatively simple but there are a couple of approaches.

What does that mean? PR need to hold for a minute before feedback?

I was noting the reason for not reviewing this yet. It's not because of any problem with the code etc. It's because the issue wasn't yet ready to implement. I was making sure that the conversation in https://github.com/ratatui-org/ratatui/issues/502#issuecomment-1722124429 was clearly communicated (closing the loop).
If this is the alternative that is chosen, I'd be happy to ping you to complete this PR, but I don't want to waste your time with addressing code issues before the design approach is certain.

Ya you are right.

Another person is working on https://github.com/ratatui-org/ratatui/pull/629 which fixes this in a way that works better. Closing this out for now. Thank you for submitting this PR and I do hope you will consider doing so in the future too.


Duplicate definitions with name <..>

I have gotten this error with both 8.1.x and 8.2.x and tried multiple times to delete the registry and re-download/re-compile.
error[E0592]: duplicate definitions with name `inner_add_git_map_entries`
   --> /Users/odysseas/.cargo/registry/src/index.crates.io-6f17d22bba15001f/vergen-8.2.4/src/feature/git/cmd.rs:520:5
    |
520 | /     fn inner_add_git_map_entries(
521 | |         &self,
522 | |         path: Option<PathBuf>,
523 | |         idempotent: bool,
...   |
526 | |         rerun_if_changed: &mut Vec<String>,
527 | |     ) -> Result<()> {
    | |___________________^ duplicate definitions for `inner_add_git_map_entries`
    |

Looks like you don't have the git feature configured correctly.   Check out the docs.   If that doesn't help, send me the snippet of your Config.toml where you are defining the vergen dependency.

@CraZySacX it seems that it's configured correctly. git eixsts in PATH.
Here is my config.toml snippet:
[build-dependencies]
vergen = { version = "8.1.0", default-features = false, features = [
  "build",
  "cargo",
  "git",
  "gitcl",
] }

Hi, any updates?

Still encountering this

I'm going to need much more information (the output from rustc -Vv and cargo -Vv would be a good start).  I cannot reproduce this issue on my mac, linux, or windows environments.  Every feature combination is run out in CI as well on all three environments, on stable, beta, nightly, and 1.68.0 (MSRV).

Could it be because I am using a git dependency that also uses vergen ? Some times cargo glitches with git imports. Let me look into it.

That is likely the cause.  vergen is intended for use with binaries, so I've never seen it included as part of a library.   I'll setup that scenario locally to see if I can reproduce this (and see if it's solvable).

Running into the same issue and I suspect the same cause (attempting to use a git dependency that uses vergen itself).

I wasn't able to reproduce this error with a simple setup I made locally.  However, I was able to reproduce it on the mev project you referenced.
I was able to get the mev binary to build with vergen in a strange manner.  I set the build-dependencies as below
[build-dependencies]
vergen = "8.2.10"

and the build.rs as
use std::error::Error;
use vergen::EmitBuilder;

fn main() -> Result<(), Box<dyn Error>> {
    // Emit the instructions
    EmitBuilder::builder()
        .git_sha(true)
        .build_timestamp()
        .cargo_features()
        .cargo_target_triple()
        .emit()?;
    Ok(())
}

This shouldn't work as there are no features declared on the top-level vergen build dependency, but it's compiles and runs happily.   My hunch is cargo is doing something funky when determining features for a build dependency that is declared across multiple Cargo.toml.   I'll take a look at the cargo source later this week to see how that is happening.

Could it be because I am using a git dependency that also uses vergen ? Some times cargo glitches with git imports. Let me look into it.

@odyslam could you try the workaround I described above (declaring the vergen dependency with either no features turned on, or only the features that differ from the upstream dependency feature set).

I found the cause.  cargo performs a union of all the features declared for a dependency (https://doc.rust-lang.org/cargo/reference/features.html#feature-unification).   The gitcl, git2, and gitoxide features are mutually exclusive right now which is not recommended (https://doc.rust-lang.org/cargo/reference/features.html#mutually-exclusive-features).   In order to fix this properly, I would have to completely re-architect the git feature.   For now, take a look at these inspections and hopefully the your feature set overlap.
In the meantime, I'll try to figure out how to re-architect to avoid this problem.   Going to close this as there is a 'workaround'.   Will be starting work on version 9 to resolve correctly.


Desktop Trading Page: Order Ticket

Design       |         Build

[x]  Update Tab Styling. Headers on desktop are 48px now.

[x]  make Outcome text 14px Medium Weight. Required for more space

[x]  Update input label styling - make mono caps and update colour

[x]  Update type style of denominations (SHARES,ETH,ETH)

[x]  Update type and spacing within this area:

[x]  change type style of button

[x]  check "Place Buy/Sell Order" button color - design looks a lot more vibrant

[x]  Update outcome display/input - see below

-[ ] Update scalar input to better fit the denominations. Text showing the denomination can drop down to 9px. See below

outcome/denomination for scalar markets only make 50% each side

closing


feat: enable lsp FUI-1198

Review Guidance
📓   Related JIRA
PTC-0
🤔   What does this PR do?

Add bullets..

🚀   Where should the reviewer start?
Add file / directory pointers.
📑   How should this be tested?
npx -y @genesislcap/genx@latest init prtest -x --ref YOUR-BRANCH-NAME

✅   Checklist

[ ] I have tested my changes.
[ ] I have added tests for my changes.
[ ] I have updated the project documentation to reflect my changes.

Build failing:
Error:  ERROR in<EMAIL_ADDRESS>1:14-26
[tsl] ERROR in /tmp/testapp/client/node_modules/@genesislcap/foundation-ui/dist/dts/combobox/combobox.template.d.ts(1,15)
      TS1005: ',' expected.

ERROR in<EMAIL_ADDRESS>1:14-26
[tsl] ERROR in /tmp/testapp/client/node_modules/@genesislcap/foundation-ui/dist/dts/stepper-tab/stepper-tab.template.d.ts(1,15)
      TS1005: ',' expected.

ERROR in<EMAIL_ADDRESS>1:14-26
[tsl] ERROR in /tmp/testapp/client/node_modules/@genesislcap/foundation-ui/dist/dts/stepper-tab-panel/stepper-tab-panel.template.d.ts(1,15)
      TS1005: ',' expected.

> Task :client:npmBuild
[success] [webpackbar] Webpack: Compiled with some errors in 51.21s

ERROR in<EMAIL_ADDRESS>1:14-26
[tsl] ERROR in /tmp/testapp/client/node_modules/@genesislcap/foundation-zero/dist/dts/stepper-tab/stepper-tab.template.d.ts(1,15)
      TS1005: ',' expected.

@matteematt I'll have some spare later today, will have a go at this one!

Tested locally:


Implement a simple "invite link" system

I am thinking something as simple as:

there's a "Get App URL" link in the overflow menu of the visual editor
when clicked, VE sends invite link request to the server, receives an invite link and copies it to the clipboard

The way invite link works:

a unique id that last 48 hours-ish, like bb.live/boards/blah.app?invite=
id is associated with the board's app
when app loads, it pings the server with that invite id
server generates a "viewer key", which is different from the board server user key in that it only allows viewing one particular board
app stores viewer key in the local storage
app redirects to bb.live/boards/blah.app url, eating the invite link

@paullewis WDYT?

Okay, a first stab of at the problem landed. Here's how it works:

VE users are automatically "signed into" board server. When they connect or refresh the board server's provider, their key is added to the local storage at the board server's origin.

When signed in, board server users see extra options in the App View of their boards. They can create an invite link, see existing links, and delete old ones (UI pending). They only see these options for their boards. I can't create a share link for Bill's board.

A share link is a unique link that is active for about 4 days. Anyone who follows this link gets access to running boards with remote runner in App View. So, basically, I can share my App View with others and let them play with it without having to enter any keys.

Once the key expires, others can still play with the board, but when they run it, they'll be asked to enter keys. We probably need some UI to tell them that this is because their invite link expired.

Ooh. What if there was also: "request to join the server" link for non-signed in users?