_zsh_highlight_load_highlighters consumes much resources?

when test this plugin, I've notice that this function _zsh_highlight_load_highlighters always on top of function that consumes resources that slow down prompt a bit.
Is there any reason why this function consumes so much resources? even higher than compinit.
for i in $(seq 1 10); do time zsh -i -c exit; done with zmodload zsh/zprof

Clarify, please: Are you commenting about the initial loading stage (the time between opening the terminal emulator and the first prompt appearing) or about the code that runs upon every  keystroke?

What highlighters do you have enabled? What are the version numbers of zsh and z-sy-h? What's the zprof output?

Here's my data:
% head .zshenv .zshrc
==> .zshenv <==
zmodload zsh/zprof

==> .zshrc <==
zprof -c
autoload compinit
compinit
source /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
zprof
%

% repeat 3 >/dev/null ZDOTDIR=$PWD zsh -ic exit
%                     ZDOTDIR=$PWD zsh -ic exit
num  calls                time                       self            name
-----------------------------------------------------------------------------------
 1)    2          10.90     5.45   42.87%     10.90     5.45   42.87%  compaudit
 2)    1          16.94    16.94   66.66%      6.05     6.05   23.79%  compinit
 3)    1           3.97     3.97   15.62%      3.97     3.97   15.62%  _zsh_highlight_bind_widgets
 4)    1           4.09     4.09   16.08%      3.83     3.83   15.05%  _zsh_highlight_load_highlighters
 5)    1           0.36     0.36    1.40%      0.36     0.36    1.40%  is-at-least
 6)    2           0.32     0.16    1.26%      0.32     0.16    1.26%  add-zsh-hook

-----------------------------------------------------------------------------------
⋮

The ratio of compinit to z-sy-h is about 2:1.  Of the time spent in z-sy-h, the widget binding half will likely diminish when the feature/redrawhook branch is merged.  As to the other half, 91% of the work in _zsh_highlight_load_highlighters hides in the single statement . "$highlighter_dir${highlighter}-highlighter.zsh" [confirmed by dropping an anonymous function around that statement], so once we fix #252 that will be addressed too, thanks to zcompile.
I have labelled #252 with the Performance label.  Thanks.
Other than that… Well, I guess we should keep this issue open, and close it once redrawhook is merged and #252 is fixed, even though there'll likely be nothing to do here once those two are merged.
@tmpm697 Did I miss anything?

the time between opening the terminal emulator and the first prompt appearing

Yes, it is.
I have pretty much stable output as this:
num  calls                time                       self            name
-----------------------------------------------------------------------------------
 1)    6          12.45     2.07   37.23%      8.04     1.34   24.03%  (anon)
 2)    2           6.79     3.40   20.31%      6.79     3.40   20.31%  _zsh_highlight_bind_widgets
 3)    1           5.42     5.42   16.20%      5.32     5.32   15.92%  _zsh_highlight_load_highlighters
 4)    1           4.44     4.44   13.29%      4.44     4.44   13.29%  enable-fzf-tab
 5)    1           3.65     3.65   10.92%      3.65     3.65   10.92%  compinit
 6)    6           1.42     0.24    4.24%      1.42     0.24    4.24%  is-at-least
 7)    5           1.26     0.25    3.78%      1.26     0.25    3.78%  add-zsh-hook
 8)    1           1.52     1.52    4.55%      0.49     0.49    1.46%  prompt_powerlevel9k_teardown
 9)   12           0.44     0.04    1.32%      0.44     0.04    1.32%  _fzf_tab_add_default
10)    1           2.12     2.12    6.34%      0.41     0.41    1.21%  prompt_powerlevel9k_setup
11)    1           4.00     4.00   11.96%      0.35     0.35    1.04%  _zcompinit_custom
12)    1           0.77     0.77    2.32%      0.24     0.24    0.72%  _p9k_preinit
13)    1           0.54     0.54    1.60%      0.23     0.23    0.68%  gitstatus_start_p9k_
14)    2           0.13     0.07    0.39%      0.13     0.07    0.39%  _p9k_restore_special_params
15)    1           0.12     0.12    0.35%      0.12     0.12    0.35%  p10k
16)    1           1.63     1.63    4.88%      0.11     0.11    0.33%  _p9k_setup

I do compinit each 24h so it does not took much resource but zsh-syntax-highlighting still. Beside that I can't know (anon) is from zsh-syntax-highlighting or not.
For sure if I disable zsh-syntax-highlighting, something else would took up % but speed would quite good improved.

num  calls                time                       self            name

6          12.45     2.07   37.23%      8.04     1.34   24.03%  (anon)
2           6.79     3.40   20.31%      6.79     3.40   20.31%  _zsh_highlight_bind_widgets
1           5.42     5.42   16.20%      5.32     5.32   15.92%  _zsh_highlight_load_highlighters

_zsh_highlight_bind_widgets was called twice but _zsh_highlight_load_highlighters was called once.
There is no codepath in z-sy-h that calls _zsh_highlight_bind_widgets twice.
I'm guessing that one of your other plugins calls that function too.  You'll have to take those 3.40 seconds of slowdown up with the maintainer of that plugin, whichever it is.

Beside that I can't know (anon) is from zsh-syntax-highlighting or not.

"(anon)" is a cumulative entry for all anonymous functions covered by that zprof output.  To see z-sy-h's contribution, disable z-sy-h entirely and compare the figures with and without z-sy-h.
Also, could you drop a line to<EMAIL_ADDRESS>about making zprof emit file:line coordinates from funcstack for anonymous functions?

For sure if I disable zsh-syntax-highlighting, something else would took up % but speed would quite good improved.

Mind if I ask what kind of environment this is (CPU, disks, and $ZSH_VERSION and $ZSH_PATCHLEVEL)?  5.32 + (6.79/2) seconds — ignoring the double call, which isn't our fault — for loading z-sy-h is quite a lot.  You don't have to answer.

Duplicate of #635.

Note about the issue title: I've changed this issue's title back to the original one and moved the title "Please optimize loading z-sy-h (time to first prompt in a new shell)" to #635.

compare to fast-syntax-hight, I think i'll go for it.

Thanks for the feedback, @tmpm697.
To be clear, the only reason I closed this issue is that there's another issue about the very same matter.  I do not consider this issue resolved yet.  (You may want to subscribe to #635, if you haven't already.)
Regarding z-sy-h, merging redrawhook is a priority.  Once that's out of the way, #252 should be trivial to implement.  At that point z-sy-h should load much faster than it does now.  However, merging redrawhook requires making a change to zsh itself first (https://github.com/zsh-users/zsh/pull/57; we're working on that now.  (See, for example, the long, ongoing discussion on https://github.com/zsh-users/zsh-autosuggestions/issues/529.)
All that said, we'll respect whichever decision you make.  If you end up choosing f-sy-h, then please give my regards to @psprint :-)


Update Meta “authors”

Automatically generated by Netlify CMS

👷 Deploy Preview for agitated-galileo-2a9f9a processing.
🔨 Explore the source changes: 1ba498aced2a0605f36bf4c91d7a8d17f783d467
🔍 Inspect the deploy log: https://app.netlify.com/sites/agitated-galileo-2a9f9a/deploys/61ca2b398b8aeb0008a9c952


build_visit: PAR_COMPILER env var support broken in graphical mode

rmid: 21rmsubmitter: Cyrus Harrisonrmsubmitdate: 04/20/2010 11:42 am
-----------------------REDMINE MIGRATION-----------------------
This ticket was migrated from Redmine. As such, not all
information was able to be captured in the transition. Below is
a complete record of the original redmine ticket.
Ticket number: 96
Status: Pending
Project: VisIt
Tracker: Bug
Priority: Normal
Subject: build_visit: PAR_COMPILER env var support broken in graphical mode
Assigned to: -
Category: -
Target version: -
Author: Cyrus Harrison
Start:
Due date:
% Done: 0%
Estimated time:
Created: 06/21/2010 07:16 pm
Updated:
Likelihood: 3 - Occasional
Severity: 2 - Minor Irritation
Found in version: < 1.12
Impact:
Expected Use:
OS: All
Support Group: Any
Description:
rmid: 21rmsubmitter: Cyrus Harrisonrmsubmitdate: 04/20/2010 11:42 am
Comments:

graphical mode of build_visit is no more


changes from wso2-dev

Closing the PR, since changes are conflicting with the latest changes and there is no changes. Please fix the conflicts and reopen the PR, if you need to add this change


ZipAlign creates corrupted APK for already aligned APK

Title says it all. In case you're wondering why it's necessary, we don't know if an APK is aligned or not in first place. So, we need to do that for both aligned and unaligned APKs.

I am also reiterating the need for #6. Adding tests would greatly reduce issues like this. I feel that most issues I opened here could've been resolved if there were adequate tests.

My guess is that the problem occurs because this line, which may not always be the case.

Well, I need to look into this. But the zipalign library has an option to verify if an APK is zip aligned. Maybe that could be implemented apart from this.
Here's the source code: https://github.com/MuntashirAkon/zipalign-android/blob/master/zipalign/src/main/cpp/zipalign/ZipAlign.cpp#L124-L168
Seems quite easy to implement. I can implement that if you want.

I haven't encountered this myself, but my guess is that it occurs because this line assumes all compressed entries have a data descriptor, which may not always be the case.

Well, I've tried about a dozen APK files, each of them has this issue. So, I believe that most of them do not have a data descriptor. But is there a way to actually detect this using the ZipFile API? I don't see any option to do that. And how it is that it only affects signed APK files, and not unsigned ones?

ZipAlign is borrowed class and I am not full understood it. But my guess is that at line 149 byte[] extra = entry.getExtra(); is always null.
Now I am focusing on new archive management lib archive2 which also solves such issue, until then hopefully someone will find a solution.
@Axelen123
@Kirlif

Well, I've tried about a dozen APK files, each of them has this issue. So, I believe that most of them do not have a data descriptor. But is there a way to actually detect this using the ZipFile API? I don't see any option to do that.

I dont think there is, since the Java Zip library does not expose flags.
REAndroids guess is also plausible.
Pull request #21 uses a different zip implementation, so its possible its not affected by the issue. I havent tested the static methods on ZipAlign much, but if you do test it only to find that its broken, please let me know

Oh @Axelen123 is right, the problem was "flag" is calculated based on assumption that all non-STORED entries have data descriptor and counts 16 bytes. But non-STORED entries will not always have data descriptor.
 It still fails with the following error:
java.util.zip.DataFormatException: invalid stored block lengths
        at java.util.zip.Inflater.inflateBytes(Native Method)
        at java.util.zip.Inflater.inflate(Inflater.java:278)
        at java.util.zip.Inflater.inflate(Inflater.java:299)

Are you testing on android os? Maybe getFlag(ZipEntry) is returning null.

No, hidden API enforcement policies are bypassed. It's something to do with calculating the offsets, I think.

if you are interested, archive2 is ready now for early experiment

Getting NPE for some APKs:
java.lang.NullPointerException: Attempt to invoke virtual method 'long com.reandroid.archive2.block.CentralEntryHeader.getCrc()' on a null object reference
 at com.reandroid.archive2.block.LocalFileHeader.mergeZeroValues(LocalFileHeader.java:32)
 at com.reandroid.archive2.model.LocalFileDirectory.visitLocalFile(LocalFileDirectory.java:63)
 at com.reandroid.archive2.model.LocalFileDirectory.visitLocalFile(LocalFileDirectory.java:48)
 at com.reandroid.archive2.model.LocalFileDirectory.visit(LocalFileDirectory.java:42)
 at com.reandroid.archive2.Archive.<init>(Archive.java:47)
 at com.reandroid.archive2.Archive.<init>(Archive.java:62)

Let me know if you need a sample.

The new API works great except the aforementioned cases. BTW, here's the class I use for verification (it was converted from the ZipAlign library for Android):

ZipAlignVerifier.java
public class ZipAlignVerifier {
    public static final String TAG = ZipAlignVerifier.class.getSimpleName();

    public static final int kPageAlignment = 4096;

    public static boolean verify(File file, int alignment, boolean pageAlignSharedLibs) {
        Archive zipFile;
        boolean foundBad = false;
        Log.d(TAG, String.format(Locale.ROOT, "Verifying alignment of %s (%d)...\n", file, alignment));

        try {
            zipFile = new Archive(file);
        } catch (IOException e) {
            Log.e(TAG, String.format(Locale.ROOT, "Unable to open '%s' for verification\n", file), e);
            return false;
        }
        List<ArchiveEntry> entries = zipFile.getEntryList();
        long lastFileOffset;
        for (ArchiveEntry pEntry : entries) {
            String name = pEntry.getName();
            lastFileOffset = pEntry.getFileOffset();
            if (pEntry.getMethod() == ZipEntry.DEFLATED) {
                Log.d(TAG, String.format(Locale.ROOT, "%8d %s (OK - compressed)\n", lastFileOffset, name));
            } else if (pEntry.isDirectory()) {
                // Directory entries do not need to be aligned.
                Log.d(TAG, String.format(Locale.ROOT, "%8d %s (OK - directory)\n", lastFileOffset, name));
            } else {
                int alignTo = getAlignment(pageAlignSharedLibs, alignment, pEntry);
                if ((lastFileOffset % alignTo) != 0) {
                    Log.w(TAG, String.format(Locale.ROOT, "%8d %s (BAD - %d)\n", lastFileOffset, name, (lastFileOffset % alignTo)));
                    foundBad = true;
                    break;
                } else {
                    Log.d(TAG, String.format(Locale.ROOT, "%8d %s (OK)\n", lastFileOffset, name));
                }
            }
        }

        Log.d(TAG, String.format(Locale.ROOT, "Verification %s\n", foundBad ? "FAILED" : "successful"));

        return !foundBad;
    }

    private static int getAlignment(boolean pageAlignSharedLibs, int defaultAlignment, ZipEntry pEntry) {
        if (!pageAlignSharedLibs) {
            return defaultAlignment;
        }

        if (pEntry.getName().startsWith(".so")) {
            return kPageAlignment;
        }

        return defaultAlignment;
    }
}

Thanks I was hopping someone to test it on android os, how is the performance relative to old API ?

Let me know if you need a sample.

Yes please

Yes please

app-release.apk.zip

app-release.apk.zip

Is this compressed by archive2 ?

Is this compressed by archive2 ?

No, this is just an example APK file which archive2 fails to open.

All working fine with ZipAlignVerifier. So far, I've only tested reading the archive files. I don't know about writing though.

Here's the method I used for aligning an APK file:
    public static void align(File input, File output) throws IOException {
        File dir = output.getParentFile();
        if (dir != null && !dir.exists()) {
            dir.mkdirs();
        }
        Archive archive = new Archive(input);
        try (ApkWriter apkWriter = new ApkWriter(output, archive.mapEntrySource().values())) {
            apkWriter.write();
        }
        if (!ZipAlignVerifier.verify(output, 4, true)) {
            throw new IOException("Could not verify aligned APK file.");
        }
    }

Before aligning the file, the verifier outputs the following error:
      39 .DS_Store (OK - compressed)
     480 AndroidManifest.xml (OK - compressed)
    2976 classes.dex (OK - compressed)
 1485993 lib/.DS_Store (OK - compressed)
 1486449 lib/arm64-v8a/libcmph.so (BAD - 3697)
Verification FAILED

After, running aligning the file, I still get a failure:
      39 .DS_Store (OK - compressed)
     480 AndroidManifest.xml (OK - compressed)
    2976 classes.dex (OK - compressed)
 1485993 lib/.DS_Store (OK - compressed)
 1486902 lib/arm64-v8a/libcmph.so (BAD - 54)
Verification FAILED

Conclusion
Alignment in archive2 is not working.

Thanks for your excellent testing report!
As on this commit @Axelen123 suggested that all native libraries must have alignment of 4096
You can control each file alignment as:
    public static void align(File input, File output) throws IOException {
        File dir = output.getParentFile();
        if (dir != null && !dir.exists()) {
            dir.mkdirs();
        }
        Archive archive = new Archive(input);
        ApkWriter apkWriter = new ApkWriter(output, archive.mapEntrySource().values());

        ZipAligner zipAligner = new ZipAligner();
        zipAligner.setDefaultAlignment(4);
        zipAligner.clearFileAlignment();

        apkWriter.write();
        if (!ZipAlignVerifier.verify(output, 4, true)) {
            throw new IOException("Could not verify aligned APK file.");
        }
    }

The aligner for archive2 should work properly in #31.

Getting this error:
java.lang.ArrayIndexOutOfBoundsException: src.length=102400 srcPos=0 dst.length=16 dstPos=0 length=-1
	at java.lang.System.arraycopy(System.java:524)
	at com.reandroid.archive2.io.FileChannelInputStream.readBuffer(FileChannelInputStream.java:84)
	at com.reandroid.archive2.io.FileChannelInputStream.read(FileChannelInputStream.java:62)
	at com.reandroid.archive2.block.ZipHeader.readBasic(ZipHeader.java:45)
	at com.reandroid.archive2.block.ZipHeader.readBytes(ZipHeader.java:33)
	at com.reandroid.archive2.model.LocalFileDirectory.visitLocalFile(LocalFileDirectory.java:68)
	at com.reandroid.archive2.model.LocalFileDirectory.visit(LocalFileDirectory.java:41)
	at com.reandroid.archive2.Archive.<init>(Archive.java:47)
	at com.reandroid.archive2.Archive.<init>(Archive.java:62)

Did you checked this commit

Did you checked this commit

Thanks. All known issues seem  to have been fixed.

For future reference, the ZipAlign helper class I've used in my project is located here. The API is identical to the AOSP zipalign.

Is DataDescriptor enabled by default now? ZipAligner#setEnableDataDescriptor(boolean) was removed somewhere in the middle. (I was not actively working on my project, so I skipped a lot of versions into the latest.)

No, but entries with DD keep it ; DataDescriptorFactory NO_ACTION is used by default: https://github.com/REAndroid/ARSCLib/commit/afa4683898e164ea6a3f38d305388eb5c1fbea32

Is DataDescriptor enabled by default now? ZipAligner#setEnableDataDescriptor(boolean) was removed somewhere in the middle. (I was not actively working on my project, so I skipped a lot of versions into the latest.)

No, but entries with DD keep it ; DataDescriptorFactory NO_ACTION is used by default: afa4683

Makes sense. Thanks.


Add tag support for most supported AWS resources

This change adds support for syncing in tags for most of the currently supported AWS resources. It also adds some missing indexes.

It seems this API doesn't support IAM user, role, and group. I'd like to be able to sync tags for them. Would it be OK to fetch the tags via the IAM API? If so, should I stick that in the IAM code, rather than this resource? (I'll do this in a separate PR)

It seems this API doesn't support IAM user, role, and group. I'd like to be able to sync tags for them. Would it be OK to fetch the tags via the IAM API? If so, should I stick that in the IAM code, rather than this resource? (I'll do this in a separate PR)

It seems like IAM is on this list: https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/supported-services.html
Was the resource name specified wrong, or maybe the way this module calls the API doesn't generalize perfectly?

It's on the list, but they have a more detailed list of what's supported
here:
https://docs.aws.amazon.com/ARG/latest/userguide/supported-resources.html#services-iam
Note that IAM user, role, and group aren't supported (very little from IAM
is).
On Mon, Jul 26, 2021 at 3:33 PM Alex Chantavy @.***>
wrote:

It seems this API doesn't support IAM user, role, and group. I'd like to
be able to sync tags for them. Would it be OK to fetch the tags via the IAM
API? If so, should I stick that in the IAM code, rather than this resource?
(I'll do this in a separate PR)
It seems like IAM is on this list:
https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/supported-services.html
Was the resource name specified wrong, or maybe the way this module calls
the API doesn't generalize perfectly?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/lyft/cartography/pull/655#issuecomment-886418256, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AALXSMXVMMQ3DVFRRK6JYBLTZT6VHANCNFSM5AU5YTBQ
.

It's on the list, but they have a more detailed list of what's supported
here:
https://docs.aws.amazon.com/ARG/latest/userguide/supported-resources.html#services-iam
Note that IAM user, role, and group aren't supported (very little from IAM
is).

Ah, bummer. Looks good to me, thanks for the PR!


HDFS-16870 HDFS client ip should also be logged when NameNode is processing reportBadBlocks

There are two scenatio involded for reportBadBlocks
1-HDFS client will report bad block to NameNode once the block size or data is not consistent with meta;
2-DataNode will report bad block to NameNode via heartbeat if Replica stored on Datanode is corrupted or be modified.
As for now, when namenode process reportBadBlock rpc request, only DataNode address is logged.
Client Ip should also be logged to distinguish where the report comes from, which is very useful for trouble shooting.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
42m 42s

trunk passed

+1 :green_heart:
compile
1m 40s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
compile
1m 29s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
checkstyle
1m  8s

trunk passed

+1 :green_heart:
mvnsite
1m 32s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javadoc
1m 34s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
3m 51s

trunk passed

+1 :green_heart:
shadedclient
26m 42s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 27s

the patch passed

+1 :green_heart:
compile
1m 27s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javac
1m 27s

the patch passed

+1 :green_heart:
compile
1m 22s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
javac
1m 22s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 58s

the patch passed

+1 :green_heart:
mvnsite
2m 29s

the patch passed

+1 :green_heart:
javadoc
0m 58s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javadoc
1m 28s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
3m 40s

the patch passed

+1 :green_heart:
shadedclient
27m  1s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
395m 37s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

516m 54s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestLeaseRecovery2

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5237/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5237

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux d20bfd7407bd 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 43d48f53c7e1fa2e0c7efc8bce71ae7c79933dfa

Default Java
Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5237/1/testReport/

Max. process+thread count
2077 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5237/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
2m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
43m 43s

trunk passed

+1 :green_heart:
compile
1m 50s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
compile
1m 28s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
checkstyle
1m  8s

trunk passed

+1 :green_heart:
mvnsite
1m 42s

trunk passed

+1 :green_heart:
javadoc
1m 10s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javadoc
1m 36s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
3m 45s

trunk passed

+1 :green_heart:
shadedclient
27m  9s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 28s

the patch passed

+1 :green_heart:
compile
1m 32s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javac
1m 32s

the patch passed

+1 :green_heart:
compile
1m 29s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
javac
1m 29s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 55s

the patch passed

+1 :green_heart:
mvnsite
1m 32s

the patch passed

+1 :green_heart:
javadoc
1m  0s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javadoc
1m 27s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
3m 43s

the patch passed

+1 :green_heart:
shadedclient
26m 29s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
503m 22s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m 15s

The patch does not generate ASF License warnings.

626m 49s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestLeaseRecovery2

hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5237/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5237

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 30f4e88eb6a3 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 05137dd0ffcd6ca5b4442228db70a75be696df01

Default Java
Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5237/2/testReport/

Max. process+thread count
2087 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5237/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@brahmareddybattula @ayushtkn
Could you pls help to review.
Thanks a lot


Update README [Fix #1]

Updated README to specify that this library will not work with passport extensions which use OAuth. This is an important distinction to make because it is not obvious. Not noting this explicitly could cause developers to waste their time.

It does work with passport strategies that uses oauth but it doesn't do federated logout. This means it will log you out from the application but it will not logout you from the identity provider (github) in your case.

Thank you for you're fast response. I will make the appropriate changes to my pull request.


Unknown command: 'loadtestdata' Type 'manage.py help' for usage.

I typed the command as below:
$ python manage.py loadtestdata entry.Entry:3
but it's now working. I got this:
Unknown command: 'loadtestdata'
Type 'manage.py help' for usage.
Maybe something wrong?

Have you added 'autofixture' to the INSTALLED_APPS setting in your django settings file?

Thanks @shaileshahuja  it worked.

:+1: Thanks @shaileshahuja for helping out here.


Anti error codes added

hehehehe

Whats the exact purpose od this change?

Specifying different error codes so they can be differentiated

And therefore debugged


[Feature]Inlong-Sort-Standalone support JMX metrics listener for pushing.

Description
Inlong-Sort-Standalone support JMX metrics listener for pushing.
Use case
No response
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

fixed by #1930 ,close it


Prevent repeated requests to the same media file by the same requester

While in theory I suppose a sensible cache would help with this, in practise the media repo doesn't cope very well with repeatedly requesting the same media file, and it might be a smart idea to sensibly limit the number of times a given IP can request the same media over and over.
We've seen cases where the same media is requested within a few hundred millseconds, and any reasonable browser / client should be caching those requests and not hitting the server each time.

err, lower the rate limits?

But it's valid to request a lot of different media? Assuming rate limits apply to the endpoint, not endpoint+mediaId?

sure, but it's questionable if this needs an application level block or if it should be at a higher network layer. The application is already serving from cache (or is supposed to be), so should already be performing at maximum capability. A reverse proxy or alternative caching layer might want to de-dupe the requests or limit requests more severely.


Ubuntu18.04 build failed!

Hi there
I built eos on Ubuntu18.04 and it's failed:
$ ./scripts/eosio_build.sh
EOSIO Version: 2.0.7
2020年 07月 18日 星期六 03:35:10 UTC
User: jht
Current branch: master
No installation location was specified. Please provide the location where EOSIO is installed.
Do you wish to use the default location? /home/jht/eosio/2.0? (y/n) y

EOSIO will be installed to: /home/jht/eosio/2.0
=====================================================================================
======================= Starting EOSIO Dependency Install ===========================
Architecture: Linux
OS name: Ubuntu
OS Version: 18.04
CPU cores: 4
Physical Memory: 12G
Disk space total: 171G
Disk space available: 140G
[Ensuring package dependencies]
 - git ok
 - make ok
 - bzip2 ok
 - automake ok
 - libbz2-dev ok
 - libssl-dev ok
 - doxygen ok
 - graphviz ok
 - libgmp3-dev ok
 - autotools-dev ok
 - libicu-dev ok
 - python2.7 ok
 - python2.7-dev ok
 - python3 ok
 - python3-dev ok
 - autoconf ok
 - libtool ok
 - curl ok
 - zlib1g-dev ok
 - sudo ok
 - ruby ok
 - libusb-1.0-0-dev ok
 - libcurl4-gnutls-dev ok
 - pkg-config ok
 - patch ok
 - clang ok
 - llvm-7-dev ok
 - No required package dependencies to install.

[Ensuring CMAKE installation]
 - CMAKE found @ /usr/local/bin/cmake.

[Ensuring Boost 1.71.0 library installation]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 95.4M  100 95.4M    0     0  2017k      0  0:00:48  0:00:48 --:--:-- 2296k
Building Boost.Build engine with toolset gcc...
Failed to build Boost.Build build engine
Consult 'bootstrap.log' for more details

and bootstrap.log show:
###
###
### Using 'gcc' toolset.
###
###
clang++ --version
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
###
###
clang++ -x c++ -std=c++11 -O2 -s -DNDEBUG builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execc
md.cpp filesys.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam.cpp jambase.cpp jamgram.cpp lists.cpp
make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp object.cpp option.cpp output.cpp parse.cpp pathsys.cpp regexp.cpp rules.cpp scan.
cpp search.cpp strings.cpp subst.cpp sysinfo.cpp timestamp.cpp variable.cpp w32_getreg.cpp modules/order.cpp modules/path.cpp modules/prope
rty-set.cpp modules/regex.cpp modules/sequence.cpp modules/set.cpp execunix.cpp fileunix.cpp pathunix.cpp -o b2
sysinfo.cpp:10:10: fatal error: 'thread' file not found
#include <thread>
         ^~~~~~~~
1 error generated.

can somebody help me?

It looks like clang can't find the standard C++ library headers, do you have an alternate library installed?  Can you please try to build on a clean 18.04 box and see if you can reproduce the issue?  Our build system does this multiple times per day and we haven't seen this behavior.


ceph-facts: Fix osd_pool_default_crush_rule fact

The osd_pool_default_crush_rule is set based on crush_rule_variable, which
is the output of a grep command.
However, two consecutive tasks can set that variable, and if the second task is
skipped, it still overwrites the crush_rule_variable, leading the
osd_pool_default_crush_rule to be set to ceph_osd_pool_default_crush_rule
instead of the output of the first task.
This commit ensures that the fact is set right after the crush_rule_variable
is assigned, before it can be overwritten.
Closes #5912
Signed-off-by: Benoît Knecht<EMAIL_ADDRESS>

I just want to add a quick note about the severity of the issue that this PR fixes.
In conjunction with #5910, setting osd_pool_default_crush_rule to the wrong value results in existing pools having their crush rule changed. On pools containing a large amount of data, it's pretty catastrophic, as it remaps a huge amount of PGs and tries to reshuffle massive amounts of data. In our test cluster, this caused several OSDs to become overwhelmed and crash.
And even if we work around #5910, the fact that the osd pool default crush rule setting in /etc/ceph/ceph.conf changes back and forth during a site.yml.sample run causes ceph-handler to restart all Ceph daemons (including ceph-osd), which is a big drawback on large clusters when the playbook was not expected to make significant changes.
So while it may seem like a fairly minor fix on the surface, it has a pretty significant operational impact, and I would love to see it merged and backported ASAP. Let me know if there's anything I can do to help in that regard.


add nf-core/callingcards citation

Adding the nf-core/callingcards paper citation to /publications

@nf-core-bot fix linting


Add Local RAG

A framework for local RAG.
Processing pipeline

Initialize modules below with a given json config
Collect document paths that ends with extensions specified in the config (filter out unsupported ones)
Load document contents with extension-specific DocumentLoader
Parse document contents into structured chunks with extension-specific DocumentParser
Create the embedding for each chunk with the Embedder specified in the config
Add chunks and corresponding embeddings into the VectorIndex specified in the config
On user query, most semantically similar chunks are retrieved from the VectorIndex and re-ranked by the Reranker specified in the config

Supported features

DocumentLoader & DocumentParser: only .txt files
Embedder: only flag_bge_m3 (https://huggingface.co/BAAI/bge-m3)
VectorIndex: only faiss (the IndexFlatL2 variant in https://github.com/facebookresearch/faiss)
Reranker: only identity (no re-ranking, directly return the input)

The default json config is configs/local_rag.json, in which the data_directory is set to "../data/" (relative path) by default.
Test data is in tests/test_data/.
Please change the value of "data_directory" in configs/local_rag.json to "../tests/test_data/" if you want to use the test data for a quick run.

👍 Great job! It is quite a large feature and takes time for me to review. Please wait for my feedback 😉.
Thanks for your contribution!

👍 Great job! It is quite a large feature and takes time for me to review. Please wait for my feedback 😉. Thanks for your contribution!

Sure! Let me know if you have any questions. 😎

Hi, @donggangj,
When I tested this PR locally, I encountered two issues:

The ppilot, configs, and plugins folders are all being recognized as top-level directories, which is not allowed for Python packages.

The first time RAG is launched, our plugin takes a long time to download the checkpoint and fails to respond to queries.

I think we can discuss and try a more lightweight solution to integrate RAG and other planned features.

Thanks for your contribution! Merge it now and revert it temporarily for future refactoring plans.


Authorizing multiple scopes

I'm trying to authorize multiple scopes for a script and having trouble with multiple scopes. If I hit the /devicecode endpoint with two scopes "offline_access 499b84ac-1321-427f-aa17-267ca6975798/user_impersonation https://management.azure.com/user_impersonation" the flow works fine for the user. They go to the site, log in, accept all the scopes and it says they're logged in. Then the /token enpoint returns with "AADSTS28000: Provided value for the input parameter scope is not valid because it contains more than one resource. Scope ....". But now that the user has accepted both scopes, I can do a single /device_code prompt without a scope to get a refresh token, and then get the respective access tokens. It seems that the token endpoint should really accept a scope parameter so that I can get the user to authorize all the scopes, and then get the access tokens for the respective scopes that have been authorized.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 58bc32c3-9e21-a56a-f876-3e750576f228
Version Independent ID: 83231247-55a8-e69b-0712-8c7b1481a02d
Content: Use Microsoft identity platform to sign in users on browser-less devices
Content Source: articles/active-directory/develop/v2-oauth2-device-code.md
Service: active-directory
Sub-service: develop
GitHub Login: @rwike77
Microsoft Alias: ryanwi

@davidcorrigan714 Thank you for your feedback . We will investigate and update this thread.

@davidcorrigan714 Are you suggesting this as a feature request? I am a bit confused about the ask.
AFAIK, you can receive an access token for only one resource at a time and it can contain multiple scopes. So, having multiple scopes from multiple resources at the same time is causing the issue in your scenario.

It's a feature request on the token endpoint (unless there is another way to this that I'm not understanding). The problem is that the device code endpoint is essentially a two in one. It's using the scope parameter to both prompt the user to authorize the desired scopes, and specify the desired access tokens. Prompting the user allows multiple scopes, but getting the access tokens does not. So by allowing the token endpoint to accept a scope parameter in the device code flow it would allow the user to authorize both scopes, and the app to get both tokens. For example say I have scope 1 on Azure DevOps and scope 2 on Graph.
Let's see if I can make this legible with GitHub formatting (assume the client ids & codes are exchanged properly):
Currently an error case:
POST /device_code
scope: scope 1 + scope 2

User opens the browser, it asks for permission for my app to use scope 1 + scope 2. The authorization of the scopes does work.

POST /token
Invalid scope from /device_code to get any access tokens.

What would be nice:
POST /device_code
scope: scope 1 + scope 2

User opens the browser, it asks for permission for my app to use scope 1 + scope 2

POST /token
scope: scope 1

POST /token
scope: scope 2

What I'm currently doing:
POST /device_code
scope: scope 1

User opens the browser, it asks for permission for my app to use scope 1

POST /token
scope: scope 1

POST /device_code
scope: scope 2

User opens the browser, it asks for permission for my app to use scope 2

POST /token
scope: scope 2

Now in the last case and current implementation the user has to do the flow for every separate resource needed by the app but they only have to do it the first time the app needs the resource. After that the app is allowed to use all the resources and the refresh token will work to get all the necessary access tokens. It just seems odd that there doesn't seem to be a way to do it all in one shot up front and adding a scope parameter to the token endpoint for the device code flow would seem to fix that. Another solution would be that an empty scope parameter to the device code endpoint would prompt the user to allow all the app's defined api scopes but it doesn't seem to do that. RFC 6749 section 3.3 says that there should be docs somewhere stating what the predefined scope is when the scope parameter is omitted but I haven't been able to find that doc for Azure. It seems to default to some minimal set of scopes.

I didn't realize that what I was asking for actually works on the v1.0 version of the api. Guess that solves my issue for now.

@davidcorrigan714 I understand. I will pass the feedback to the engineering team regarding this. As you have a workaround for the issue, I will go ahead and close the issue for now.


Add pdb-explorer extension

Description

Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

Hi @pernielsentikaer - thanks for the review!
As requested - changes are addressed in latest push

[x] Created metadata folder and moved screenshots from assets to metadata
[x] Removed unused extension-icon.png from assets folder
[x] Referenced screenshots via links in README instead of showing screenshots directly


Symlink Go binaries to versioned bin folder in GOPATH

Adding symlinks from initial Go installation bin directory to GOPATH/go<major.minor>/bin to maintain parity with existing scripts.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhay-krishna
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhay-krishna]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@abhay-krishna: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

builder-base-tooling-presubmit
f90a2abaff0c4087bc0d81f7b6e460b307845264
link
/test builder-base-tooling-presubmit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhay-krishna, jaxesn
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhay-krishna,jaxesn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Livery request: Avianca (Pride Livery) Airbus A320

I screenshoted the tail section of the aircraft.
there's the link to it: https://ibb.co/x3kTZhx

You should redraw the tail in gimp


Feature/firefox version

This Pull request ported the chrome extension to firefox.
closes #1

cool! do you wanna publish it? also lets take out those 2 console.logs you left in there :)

Ok I will make those changes then publish it.

Just Published https://addons.mozilla.org/en-US/firefox/addon/netlify-browser-extension/

awesome


Update README.md

Motivation and Context

Fixes markdown formatting not displayed correctly in Github
How has this been tested?

Checklist

[X] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[X] Assigned myself to the PR
[X] Added tests to cover my changes

I'm re-running the CI but I suspect these failures are legitimate.

Cherry-picking this on gh-122 for simplicity, thanks @MarkusSagen!


Server start and stop

Consider this 2 functions that start and stop an http server using express.
async function start(options) {
    app = express()
    return new Promise((resolve, reject) => {
        httpServer = app.listen(options.port, err => {
            if (err) {
                return reject(err)
            }
            else {
                return resolve(httpServer)
            }
        })
    })
}

async function stop() {
    return new Promise((resolve, reject) => {
            httpServer.close(err => {
                if (err) {
                    reject(err)
                }
                else {
                    resolve()
                }
        })
    })
}

If i do server.start().then(server.stop())  httpServer won't exist by the time i try to httpServer.close.
I am using a setTimeout around httpServer.close it to make it work.
Could this be fixed?

Hi @bertolo1988 I'm sorry you're having problems. I'm not sure I understand what is going on or even how to run your example. Can you help us by providing the following:

Version of Node.js
Version of Express.js
Full runnable code that we can copy and paste and run to see the issue without needing to make modifications.
Instructions for how to run the code and see the error.

Thanks!

node: v8.9.4
npm: 5.7.1
var express = require('express')
var app = express()

async function start(options) {
    app = express()
    return new Promise((resolve, reject) => {
        httpServer = app.listen(options.port, err => {
            if (err) {
                reject(err)
            }
            else {
                resolve(httpServer)
            }
        })
    })
}

async function stop() {
    return new Promise((resolve, reject) => {
            httpServer.close(err => {
                if (err) {
                    reject(err)
                }
                else {
                    resolve()
                }
        })
    })
}

start().then(stop())

Looks like there are several issues, but none of them are with Express.js. To list a couple:

You're not passing in options object to start which is why you get the port error.
You're calling stop() immediately instead of waiting for the promise to resolve.

start({ port: 3000 }).then(() => stop())

For further support, please drop by our support channel https://gitter.im/expressjs/express 👍

Yeah just noticed my mistake. Have a nice weekend.
Sorry, thanks.


Remove log message which skips TestAspect.silentLogging

Changes being made
Remove log messages on shutdown which skips TestAspect.silentLogging
Additional context
Fixes #128

Codecov Report

Merging #134 (48eba60) into main (fd49704) will decrease coverage by 0.12%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #134      +/-   ##
==========================================
- Coverage   37.60%   37.49%   -0.12%
==========================================
  Files         125      125
  Lines        4797     4801       +4
==========================================
- Hits         1804     1800       -4
- Misses       2993     3001       +8

Files
Coverage Δ

...ain/scala/zio/temporal/worker/ZWorkerFactory.scala
0.00% <0.00%> (ø)

... and 4 files with indirect coverage changes


DecodeCSV outputs an unexpectedly result

I recently feel confused while using DecodeCSV operator. The meaning of record_defaults is different from that described in the documentation.
Here is the description in documentation:
record_defaults: One tensor per column of the input record, with either a
  scalar default value for that column or empty if the column is required.

Problem 1 : The size of record_defaults limit the size of output
>>> value=["a,b,c"]
>>> parts_val = parsing_ops.decode_csv(value, field_delim=",", record_defaults = [[""]])
>>> parts_val
[<tf.Tensor 'DecodeCSV_10:0' shape=(1,) dtype=string>]

Problem 2 : If record_defaults contains empty element, the type of output is always float32. In the same time, it failed to check the existence of value.
>>> value=["a,b,c"]
>>> parts_val = parsing_ops.decode_csv(value, field_delim=",", record_defaults = [[""], [], []])
>>> parts_val
[<tf.Tensor 'DecodeCSV_13:0' shape=(1,) dtype=string>, <tf.Tensor 'DecodeCSV_13:1' shape=(1,) dtype=float32>, <tf.Tensor 'DecodeCSV_13:2' shape=(1,) dtype=float32>]

Maybe I understand it in a wrong way. Any comment is appreciated. Thanks.

@keveman Any idea? It doesn't look like there's bounds checking on the records_default, so that might be a problem.

I think the documentation is correct, but any suggestions to improve it will be appreciated.
RE: Problem 1: The documentation suggests that record_defaults must contain One tensor per column of the input record. So, it needs to match with the number of columns in the input. If it doesn't the op will fail. For example:
value = tf.placeholder(tf.string)
parsed = tf.decode_csv(value, record_defaults=[[""]])
run = tf.Session().make_callable(parsed, [value])

print(run("a")) # Correctly "decodes" this single column CSV into string tensor

print(run("a,b,c")) # Will throw an exception saying  "Expect 1 fields but have 2 in record 0"

RE: Problem 2: An "empty element" means an empty tensor, i.e., a zero-element tensor with the expected type of the value of the column. Without any additional context information, TensorFlow interprets [] as a float32 tensor, just like say numpy does (well numpy typically uses float64):
tf.constant([]).dtype # This will be tf.float32
np.array([]).dtype    # This will be np.float64

To explicitly provide a zero-element tensor of the correct type, e.g., int32, you could use tf.constant([], tf.int32) (and similarly in numpy np.array([], np.int32)).
So, for your example, I'd do something like:
>>> value = ["a,b,c"]
>>> empty = tf.constant([], tf.string)  # Or np.array([], np.string)
>>> parts_val = tf.decode_csv(value, field_delim=",", record_defaults = [[""], empty])
>>> parts_val
[<tf.Tensor 'DecodeCSV:0' shape=(1,) dtype=string>,
 <tf.Tensor 'DecodeCSV:1' shape=(1,) dtype=string>]

Hope that helps. Since this is not a bug, I'm going to close this out.
If you'd like to contribute suggestions to improve the documentation, we'd be more that happy to hear (or even better, feel free to send a PR editing the documentation here)
Hope that helps!

@asimshankar Thank you for your explanation.
Actually, i don't know the number of columns in the input at the first time. What i hope is record_defaults do not has to match with the number of columns in the input.
If the size of record_defaults is greater than the number of columns, fill in output with the defaults.
If the size of record_defaults is less than the number of columns, throw exception to show that column is missing and required.
Perhaps that is more reasonable, isn't it?  :)

@uuleon : tf.decode_csv expects that the number of columns is known before being invoked, so record_defaults has to match exactly. I'm not sure of the genesis of this, but I suspect this is to adhere to RFC 4180 in its strict sense and discourage subtle bugs from invalid input data (e.g., differing number of columns per line or unexpectedly missing columns in the file - see Section 2 of the RFC).


Fix timeline resetting when options are changed

Overview

Timeline range is now locked when the user zooms, pans, or clicks the range-selector buttons.
The "All" button unlocks the zoom and restores the auto-zooming behavior.
Fixed various issues related to zooming and calculating data bounds.
Refactored range-selector, moving most of the zoom/bounds code into FhirChartConfigurationService
Code cleanup

How it was tested

Ran Showcase app with mock FHIR server
Added observation and medication layers in various combinations
Tested hiding/showing each layer with each preset button selected
Tested changing color and interpolation settings with each preset button selected
Tested zoom/pan by clicking in the chart to make sure it locks zoom when layer is added/removed
Tested that preset buttons select/deselect appropriately when zooming/panning the chart. The "All" button is a bit unintuitive because if you zoom out it deselects even though all of the data is still visible. We might want to rename it to "Auto"?

Checklist

[x] The title contains a short meaningful summary
[x] I have added a link to this PR in the Jira issue
[x] I have described how this was tested
[ ] I have included screen shots for changes that affect the user interface
[ ] I have updated unit tests
[ ] I have run unit tests locally
[ ] I have updated documentation (including README)


[#511] Ktlint 0.46.1

This code review checklist is intended to serve as a starting point for the author and reviewer, although it may not be appropriate for all types of changes (e.g. fixing a spelling typo in documentation).  For more in-depth discussion of how we think about code review, please see Code Review Guidelines.
Author

[ ] Self-review: Did you review your own code in GitHub's web interface? Code often looks different when reviewing the diff in a browser, making it easier to spot potential bugs.
[ ] Automated tests: Did you add appropriate automated tests for any code changes?
[ ] Manual tests: Did you update the manual tests as appropriate? While we aim for automated testing of the application, some aspects require manual testing. If you had to manually test something during development of this pull request, write those steps down.
[ ] Code coverage: Did you check the code coverage report for the automated tests?  While we are not looking for perfect coverage, the tool can point out potential cases that have been missed. Code coverage can be generated with: ./gradlew check for Kotlin modules and ./gradlew connectedCheck -PIS_ANDROID_INSTRUMENTATION_TEST_COVERAGE_ENABLED=true for Android modules.
[ ] Documentation: Did you update documentation as appropriate? (e.g README.md, etc.)
[ ] Run the app: Did you run the app and try the changes?
[ ] Screenshots: Did you provide before and after UI screenshots in the description of this pull request?  This is only applicable for changes that modify the UI.
[ ] Rebase and squash: Did you pull in the latest changes from the main branch and squash your commits before assigning a reviewer? Having your code up to date and squashed will make it easier for others to review. Use best judgement when squashing commits, as some changes (such as refactoring) might be easier to review as a separate commit.

Reviewer

[x] Checklist review: Did you go through the code with the Code Review Guidelines checklist?
[ ] Ad hoc review: Did you perform an ad hoc review?  In addition to a first pass using the code review guidelines, do a second pass using your best judgement and experience which may identify additional questions or comments. Research shows that code review is most effective when done in multiple passes, where reviewers look for different things through each pass.
[ ] Automated tests: Did you review the automated tests?
[ ] Manual tests: Did you review the manual tests?
[ ] How is code coverage affected? We encourage you to compare coverage before and after changes and when possible, leaving it in a better place.
[ ] Documentation: Did you review Docs, README.md, and Architecture.md as appropriate?
[ ] Run the app: Did you run the app and try the changes? While the CI server runs the app to look for build failures or crashes, humans running the app are more likely to notice unexpected log messages, UI inconsistencies, or bad output data. Perform this step last, after verifying the code changes are safe to run locally.

Great! I see this update brings us some new checks, which seem all useful to me. Just one question, isn't there a way to suppress the ktlint:filename warning globally?

There is!  I wasn’t sure if we might find the warning useful in the future for certain new files, so I decided to opt out on a case-by-case basis for now.


Updated libuavcan submodule with upstream ci fix.

@LorenzMeier This commit does not affect PX4 it just points to a commit that fixes a CI failure introduced in the changes upstream to  support the nuttx upgrade to 7.16+
I will do a separate PR to update the libuavcan submodule once upstream has merged my branch.

Just capturing this here for now. This is the mem usage of the upstream libuavcan once @pavel-kirienko  does the merge of for_upstream_nuttx_changes_4 to master. I think I recall @pavel-kirienko saying we need to call a shrink function. So that will be in the next PR that updates libuavcan to master.
 free
             total       used       free    largest
Mem:        237408     145440      91968      89920
nsh> WARN  [commander] Not ready to fly: Sensors not set up correctly
uavcan start
WARN  [uavcan] Node ID 1, bitrate 1000000
WARN  [uavcan] sensor bridge 'gnss' init ok
WARN  [uavcan] sensor bridge 'mag' init ok
WARN  [uavcan] sensor bridge 'baro' init ok
nsh> free
             total       used       free    largest
Mem:        237408     155712      81696      76704
nsh> uavcan start fw
nsh> free
             total       used       free    largest
Mem:        237408     168192      69216      61920
nsh> uavcan stop fw
WARN  [uavcan] stopping fw srv thread...
WARN  [uavcan] exiting
nsh> free
             total       used       free    largest
Mem:        237408     156176      81232      76704
nsh> uavcan stop
WARN  [uavcan] exiting.
nsh> free
             total       used       free    largest
Mem:        237408     145728      91680      76736
nsh>

@davids5 Just to clarify why we need shrink(): libuavcan in PX4 is configured to use heap rather than static allocation for its memory needs. The library allocates fixed size (64 bytes) chunks from the heap as needed and keeps them in an internal allocator indefinitely, so the heap footprint stops growing once a peak memory consumption is reached. Calling shrink() hints the library to free() all memory chunks that are unused at the moment.


UX Observation of Immersion Day - Bahaa

Describe the issue
Teams have a need for basic AWS knowledge and a preview of how the LZ is different from regular AWS so that they can plan and estimate their
work for POs and EAs that are deciding whether or not they want to onboard to AWS.
Which Sprint Goal is this issue related to?

Note: 'Milestone' is a ZenHub term that we use synonymously with 'Sprint'.

The 'Milestone' description (created in ZenHub) should clearly list Sprint Goals. This section should indicate which 'goal' this issue is related to.
Additional context

Training is specific to BC Gov SEA provided by AWS in a regular format AWS calls Immersion day.

Definition of done

Review training track with PO prior to attending.
Attend immersion day training provided by AWS and gain understanding of experience for teams
Creation of feedback form to capture any missing pieces/improvements needed from future attendees (Focus of the form: Did you get enough info to make a Go/No-Go decision to BC Gov SEA and what's missing)

Connected with Julian to set a day to join the course. Julia Suggested April 13th, 2021.

Awaiting for user feedback, through my observation of the training:
1- as a non developer that was, it was very condensed with information at the beginning
2- the hands experiential learning at the end could be longer and slower
3- would have been great if the exercise part recorded so if i am-slower i could go back to the recording and practice

Emily sent the feedback and created a deck slide for the demo.


pineapple-pictures v0.8.0 fails to build on i386

I updated pineapple-pictures in Debian today to v0.8.0 but it fails to build for i386. The previous v0.7.4 did not have the build failure.
The build log of v0.8.0 for i386 can be seen at https://buildd.debian.org/status/fetch.php?pkg=pineapple-pictures&arch=i386&ver=0.8.0-1&stamp=1722081327&raw=0
I will be happy to give any extra log or test any patch if needed.

/usr/include/i386-linux-gnu/qt6/QtCore/qfloat16.h: In member function ‘constexpr qfloat16::operator NativeType() const’:
/usr/include/i386-linux-gnu/qt6/QtCore/qfloat16.h:64:52: error: SSE register return with SSE2 disabled
   64 |     constexpr operator NativeType() const noexcept { return nf; }
      |                                                    ^
/usr/include/i386-linux-gnu/qt6/QtCore/qfloat16.h: In function ‘qfloat16 operator+(qfloat16, qfloat16)’:
/usr/include/i386-linux-gnu/qt6/QtCore/qfloat16.h:124:147: error: operation not permitted on type ‘_Float16’ without option ‘-msse2’
  124 |     friend inline qfloat16 operator+(qfloat16 a, qfloat16 b) noexcept { return qfloat16(static_cast<NearestFloat>(a) + static_cast<NearestFloat>(b)); }
      |

I guess it's very likely a Qt6 issue or SSE2 support issue? Since according to the log that you provided, it's unlikely related to Pineapple Pictures itself.

Yeah, seems like you are correct. I ran a pipeline with 0.7.4 on i386 and that now fails. So, something else has changed. I will look more into it. Thanks a lot for the quick reply and support.

Closing this issue as this has been seen on other packages also and  https://codereview.qt-project.org/c/qt/qtbase/+/579205 has been opened.


GitHub Enterprise config gets cleared after adding organisations/users

Rancher versions:
rancher/server: v1.6.10
Steps to Reproduce:

Setup Access Control using GitHub and specify an Enterprise server
Active Access Control by hitting Authenticate using GitHub
After successful authentication, add a user/org and hit Save.

Results:
Config parameters which point to Enterprise install are reset (githubConfig.hostname == null, githubConfig.scheme == 'https'). After the activation of access control, the parameters are there but when clicking Save they are not send.

Logan - on the post the UI is not sending the 'hostname' and the schema is 'https' even thou I had set it to unsecured.  This is causing the access to get messed up.

Available with v1.6.11-rc6

Version - v1.6.11-rc6
Verified fixed


fix(Link): partial query match for Vue

🔗 Linked issue
Follow-up for #2781
❓ Type of change

[ ] 📖 Documentation (updates to the documentation or readme)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
Implement partial exactQuery functionality for Vue's Link component
📝 Checklist

[x] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

Thanks @sandros94! 😊


nuitka fails to find C include files

Tried nuitka-0.5.27 on Gentoo/AMD.
nuitka fails to compile 2-line program import numpy;print numpy.zeros([2,2])
Lots of warnins like
Nuitka:WARNING:/usr/lib64/python2.7/site-packages/pkg_resources/__init__.py:72: Cannot find 'pkg_resources.extern.packaging.markers' in package 'pkg_resources' as absolute import.

and
Nuitka:WARNING:Unresolved '__import__' call at '/usr/lib64/python2.7/site-packages/six.py:82' may require use of '--recurse-directory'.

Then multiple error messages like
n.build/__frozen.cpp(3): catastrophic error: cannot open source file "Python.h"
  #include <Python.h>
                     ^

compilation aborted for n.build/__frozen.cpp (code 4)
n.build/__helpers.cpp(4): catastrophic error: cannot open source file "nuitka/prelude.h"
  #include "nuitka/prelude.h"
                             ^

List of installed nuitka files (truncated for readability):
/usr
/usr/bin
/usr/bin/nuitka
/usr/bin/nuitka-run
/usr/lib
/usr/lib/python-exec
/usr/lib/python-exec/python2.7
/usr/lib/python-exec/python2.7/nuitka
/usr/lib/python-exec/python2.7/nuitka-run
/usr/lib64
/usr/lib64/python2.7
/usr/lib64/python2.7/site-packages
/usr/lib64/python2.7/site-packages/Nuitka-0.5.27-py2.7.egg-info
/usr/lib64/python2.7/site-packages/Nuitka-0.5.27-py2.7.egg-info/PKG-INFO
/usr/lib64/python2.7/site-packages/Nuitka-0.5.27-py2.7.egg-info/SOURCES.txt
/usr/lib64/python2.7/site-packages/Nuitka-0.5.27-py2.7.egg-info/dependency_links.txt
/usr/lib64/python2.7/site-packages/Nuitka-0.5.27-py2.7.egg-info/top_level.txt
/usr/lib64/python2.7/site-packages/nuitka
/usr/lib64/python2.7/site-packages/nuitka/Builtins.py
/usr/lib64/python2.7/site-packages/nuitka/Builtins.pyc
...
/usr/lib64/python2.7/site-packages/nuitka/build/__init__.pyo
/usr/lib64/python2.7/site-packages/nuitka/build/include
/usr/lib64/python2.7/site-packages/nuitka/build/include/coro.h
/usr/lib64/python2.7/site-packages/nuitka/build/include/nuitka
/usr/lib64/python2.7/site-packages/nuitka/build/include/nuitka/allocator.h
...
/usr/lib64/python2.7/site-packages/nuitka/build/include/nuitka/prelude.h
...
/usr/lib64/python2.7/site-packages/nuitka/utils/__init__.pyo
/usr/share
/usr/share/doc
/usr/share/doc/nuitka-0.5.27
/usr/share/doc/nuitka-0.5.27/README.rst.lzma

Whereabouts of Python.h:
/usr/include/python2.7/Python.h
/usr/include/python3.4m/Python.h
/usr/lib64/pypy/include/Python.h
... (android and macos stuff skipped)

If more information from me is required, I will of course provide it.

Can you provide the output for scons, using the --show-scons making sure that the ".build" directory is deleted, so no caching will be going on.
In "SingleExe.scons" there is a check that will be happy with  /usr/include/python2.7/Python.h I suppose, and then that directory is added to the PATH. With the scons output, I will be able to tell, if anything strange is going on.
And of course, this won't be working for even hello world, there is a systematic issue there I suppose.

kayhayen, thank you for help.
I added --show-scons to nuitka command-line. Noticed that nuitka tried to use Intel icc compiler. Consulted nuitka --help output and found no way to tell it to use gcc. Hid icc compiler so nuitka would not find it.
After that, the 2 lines of code compiled successfully. Output of the executable is correct.
I have multiple versions of gcc installed. Is it possible to tell nuitka exactly which of them should be used?
If there is no such feature in nuitka, I suggest that it is implemented.

Nuitka respects the "CC" variable. Using CC=/usr/bin/gcc ought to work.
The "icc" compiler ought to be supported by Scons though. Can you find out, why it doesn't get the include path handed over? Are you using the builtin copy from Nuitka or an installed latest greatest version of scons?
If the "icc" cannot work, I would like to know and disable it for Nuitka.

I got dev-util/scons-2.3.4 installed, don't know how great it is.
I wonder whether a good .ebuild script for nuitka exists (better than my https://gist.github.com/krisk0/d7f66cb5339894c1c92456785470288a).
My icc is very old and nuitka probably wants a capable compiler. And I don't want to compile my python code with icc.  For these reasons, I won't test icc under nuitka. Sorry.
It's good that I can test multiple gcc and clang compilers, thank you for the CC hint.
// nuitka and pyinstaller both work for me. nuitka is a lot slower than pyinstaller when compiling code, but pyinstaller-created executable puts some garbage into my /tmp while nuitka-created executable behaves normally. Still need to check which of them runs faster.


CLI: fix defaults; make consistent with admin cli

Coverage increased (+0.1%) to 68.452% when pulling 7914e25ca51a4914a4ee24e976b9de9effc22c97 on fix-cli-defaults into 51b6724eeb26ec943f9f3e33f3c0fbbfc33fd590 on master.

Coverage increased (+0.1%) to 68.432% when pulling baa6b9d916f3550573cb3f862252d006dd60d6bf on fix-cli-defaults into 51b6724eeb26ec943f9f3e33f3c0fbbfc33fd590 on master.

We probably should just deprecate these commands, which are duplicates from cherami-cli. I don't see the value of having the same commands in two tools, and they're causing code duplicates like this.
Or as @thuningxu has mentioned before, we can fold these two tools into a single binary, and only expose the 'admin' functionalities to admins. Although I personally slightly prefer the current solution.

About admin vs cli . I think Kobe is working on something to that effect. So not dealing with it in this diff.

Coverage decreased (-0.09%) to 68.219% when pulling 9f1ba97c9e7ac66299869649e520475cdfe82f01 on fix-cli-defaults into 51b6724eeb26ec943f9f3e33f3c0fbbfc33fd590 on master.

Yes, I'm working on refactoring the cherami-cli and cherami-admin code. Will send out a PR later.

Coverage decreased (-0.2%) to 68.146% when pulling e34e823b8a96df5fd9f9c00df8bc0c0ba0f51196 on fix-cli-defaults into 51b6724eeb26ec943f9f3e33f3c0fbbfc33fd590 on master.


Font rubicon-icon-font.ttf missing in 2.4.3

react-native, react and native-base version
"native-base": "2.4.3",
"react-native": "0.55.3",
"react": "16.3.1",

I get the following error when running react-native run-ios

+1

are there any updates?
i've tried removing node_modules and doing npm install. i still got the error.
here's my versions:
react-native-cli: 2.0.1
react-native: 0.47.2
"native-base": "^2.3.1",

I still got unexplainable errors after upgrading so I solved this by manually downloading rubicon-icon-font.ttf and pasted it in node_modules\native-base\Fonts\

For the record, I ended up removing node_modules folder, dumping watchman  cache, dumping other caches and using the following versions:
"native-base": "^2.3.9",
"react": "16.3.1",
"react-native": "0.55.3",

Now everything seems to run smoothly.

Latest fixed on master
Release by EOD

Fixed with 2.4.4

Download the file and put in the required location. This fixed in my case.
File Location:
https://github.com/shoutem/school/blob/master/LunchSpotApp/android/app/src/main/assets/fonts/rubicon-icon-font.ttf

Same here on RN 0.60.5 and native-base 2.13.6

@g33ktony i have the same issue on AppCenter, so i decided to fix the version on 2.12.1.
2.12.1 works fine on my xCode.


Out of memory issue

We are using websocket (with stomp) and facing an issue due to a "long[]" object. This object seems to grow really fast, for example after 1mn, we've got an "out of memory" exception.
Here is our configuration:
server.port: 9001

allowed.remotes: my_ip

remote:
    auth:
        prefix: /auth/**
        url: http://localhost:7001
    api:
        prefix: /api/**
        url: http://localhost:9000
    ws-test:
        prefix: /ws-test/**
        url: http://localhost:9000

spring.cloud.gateway.enabled: true

logging:
    level:
        org.springframework.cloud.gateway: TRACE
        org.springframework.http.server.reactive: DEBUG
        org.springframework.web.reactive: DEBUG

Do you have any solution?

Please learn how to properly format code and logs.
Can you provide any more information about how you determined this might come from the gateway? Your configuration does not show any gateway configuration, but old zuul configuration.

Sorry for that code format, I just edited it.
I didn't really determined what is causing this issue (sorry I didn't precise it in the previous post) and my configuration might be wrong or not what it should look like, so it's vague.
After starting the gateway, everything seem to work well. But after ~1-2mn, and due to the websockets because we are requesting every second to update a "clock", my gateway gets an out of memory exception.
I am still investigating on that, so I was wondering if you already faced the same issue.

No


Support python 3.10

When will there be support python 3.10?

This is a good point.
Compatibility has been improved at #2. You can use python 3.10 with the latest version (v0.2.1) of the package.


refactor(mempool): change address-to-tx value type to QueuedTransaction

This change is

Codecov Report
Attention: Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 83.75%. Comparing base (fe164f3) to head (331b790).

Files
Patch %
Lines

crates/mempool/src/transaction_queue.rs
50.00%
1 Missing :warning:

Additional details and impacted files
@@                                 Coverage Diff                                 @@
##           mohammad/transaction-queue/change-to-address-to-tx     #410   +/-   ##
===================================================================================
  Coverage                                               83.75%   83.75%
===================================================================================
  Files                                                      37       37
  Lines                                                    1693     1693
  Branches                                                 1693     1693
===================================================================================
  Hits                                                     1418     1418
  Misses                                                    200      200
  Partials                                                   75       75

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.