using pick on a valibot schema causes superforms to fail to submit when use:enhance is on.

Description
When submitting a superform with valibot when use:enhance is active. Using a pick on valibot to select a partial schema causes submitting the form to fail with a 400 error.  If I make a new valibot schema with only the fields I am interested in, it works as expected.
Also another possible bug, when use:enhance is turned off $submitting also does not work on the form and is not being passed into the form on submit.  I don't know if that is expected behavior or not.
valibot-schemas.ts
import { string, email, object, boolean, minLength, toTrimmed, picklist, literal, date } from 'valibot';

const roleOptions = ['USER', 'PREMIUM', 'ADMIN'] as const;

export const userSchema = object({
	firstName: string([
		toTrimmed(),
		minLength(1, 'Please enter your first name.'),
	  ]),
	  lastName: string([
		toTrimmed(),
		minLength(1, 'Please enter your last name.'),
	  ]),
	email: string([
		toTrimmed(),
		email('The email address is not valid.'),
		minLength(1, 'An email address is required.'),
	  ]),
	  role: picklist(roleOptions, 'You must have a role.'),
	  verified: boolean(),
	  terms: literal(true, 'You must accept the terms and privacy policy.'),
	receiveEmail: boolean(),
	createdAt: date(),
	updatedAt: date()
}
);

export const emailSchema = object({
	email: string([
		toTrimmed(),
		email('The email address is not valid.'),
		minLength(1, 'An email address is required.'),
	  ]),
	});

export type UserSchema = typeof userSchema;
export type emailSchema = typeof emailSchema;

failing when using pick with a 400 error. It doesn't even try and hit the +page.server.ts

Create an account

Enter your email below to create your account

		{#if $message?.status === 'success'}
			<Alert.Root variant="default">
				<Icons.userCreated class="h-4 w-4" />
				<Alert.Title>Account Created</Alert.Title>
				<Alert.Description>{$message.text}</Alert.Description>
			</Alert.Root>
		{:else}
			{#if $message}
				{#if $message?.status === 'error'}
					<Alert.Root variant="destructive">
						<Icons.warning class="h-4 w-4" />
						<Alert.Title>Error</Alert.Title>
						<Alert.Description>{$message.text}</Alert.Description>
					</Alert.Root>
				{/if}
			{/if}
			<form method="POST" use:enhance>
				<Form.Field {form} name="email">
					<Form.Control let:attrs>
						<Input
							{...attrs}
							bind:value={$formData.email}
							id="email"
<EMAIL_ADDRESS>							type="email"
							autocapitalize="none"
							autocomplete="email"
							autocorrect="off"
							disabled={$submitting}
						/>
					</Form.Control>
					<Form.Description />
					<Form.FieldErrors class="w-full" />
				</Form.Field>
				<Form.Button disabled={$submitting} class="mt-2 w-full">
					{#if $submitting}
						<Icons.spinner class="mr-2 h-4 w-4 animate-spin" />
					{/if}
					Sign In with Email
				</Form.Button>
			</form>
			{$submitting ? 'Logging in...' : 'Login'}
			<SuperDebug data={form} />
		{/if}

works fine if I make a separate schema with just the field I am interested in.
import { goto } from '$app/navigation';
	import { Icons } from '$lib/components/icons';
	import { Button } from '$lib/components/ui/button';
	import * as Form from '$lib/components/ui/form';
	import { Input } from '$lib/components/ui/input';
	import { pick } from 'valibot';
	import { emailSchema } from '$lib/config/valibot-schemas';
	import { valibotClient } from 'sveltekit-superforms/adapters';
	import { type SuperValidated, type Infer, superForm } from 'sveltekit-superforms';
	import SuperDebug from 'sveltekit-superforms';
	import * as Alert from '$lib/components/ui/alert';

	type EmailSchema = typeof emailSchema;
	export let data: SuperValidated<Infer<EmailSchema>>;

	const form = superForm(data, {
		validators: valibotClient(emailSchema),
		dataType: 'json'
	});
	const { form: formData, submitting, message, enhance } = form;

Can you reproduce it here? Then I'll take a closer look: https://stackblitz.com/edit/superforms-2-testing-valibot?file=src%2Froutes%2F%2Bpage.server.ts,src%2Froutes%2F%2Bpage.svelte

Couldn't reproduce it, but check if 2.7.0 fixes it anyway.

Closing, as Valibot has been updated and there have also been some improvements in its JSON Schema generator.


Video download error: http://www.good4utah.com/good-things-utah/gtu-featured-guest/utahs-stars-friends-concert-for-a-cause

An error occurred while downloading the following video: http://www.good4utah.com/good-things-utah/gtu-featured-guest/utahs-stars-friends-concert-for-a-cause
This is the output of youtube-dl --verbose
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'-J', u'http://www.good4utah.com/good-things-utah/gtu-featured-guest/utahs-stars-friends-concert-for-a-cause']
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out None, pref ANSI_X3.4-1968
[debug] youtube-dl version 2015.12.21
[debug] Python version 2.7.6 - Linux-3.19.0-39-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6
[debug] Proxy map: {}
WARNING: Falling back on generic information extractor.
ERROR: Unsupported URL: http://www.good4utah.com/good-things-utah/gtu-featured-guest/utahs-stars-friends-concert-for-a-cause
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 1287, in _real_extract
doc = compat_etree_fromstring(webpage.encode('utf-8'))
File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 248, in compat_etree_fromstring
doc = _XML(text, parser=etree.XMLParser(target=etree.TreeBuilder(element_factory=_element_factory)))
File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 237, in _XML
parser.feed(text)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
self._raiseerror(v)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
ParseError: not well-formed (invalid token): line 770, column 69
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 665, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 291, in extract
return self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 1896, in _real_extract
raise UnsupportedError(url)
UnsupportedError: Unsupported URL: http://www.good4utah.com/good-things-utah/gtu-featured-guest/utahs-stars-friends-concert-for-a-cause

Are these reports still automated? If so, please stop right away.

No, they aren't automated.
Now I always check if the URL contains a video player before sending opening an issue.
The current page contains a video that YouTube-dl cannot download, and so does #7981 .

Then feel free to reopen any issues you consider legit.
For example, there is indeed an embedded video in the URL you mention, but good4utah.com is not a supported site at the moment, and the program throws an exception indicating so in the error message. Is this a feature request to support that site? If so, why not to put that information in the issue title? Is the backtrace useful in that case?


Remove all interactive client code & docs

Delete interactive client code
Delete interactive mode documentation
Remove client/core interactive mode interaction
Also, remove link checker

Nice one, this will tidy up things :) I found some other interactive mode references here that we might want to remove:
https://github.com/balena-os/leviathan/blob/vipul/no-longer-blessed/core/lib/common/suite.js#L177
https://github.com/balena-os/leviathan/blob/vipul/no-longer-blessed/core/lib/common/suite.js#L91 (maybe?)
https://github.com/balena-os/leviathan/blob/vipul/no-longer-blessed/client/lib/index.js#L408 (I think this is related to interactive mode, it sends something on stdin to the client to the core?) - related is this one: https://github.com/balena-os/leviathan/blob/vipul/no-longer-blessed/core/lib/main.js#L258 (gives whatever was sent to the stdin of suite - but then suite doesn't have a handler for stdin, so it will never do anything)

Accepted all suggestions, thanks @rcooke-warwick
I have removed the client/core talking mechanism that might link to interactive mode as well. Let's keep an eye if something fails.

@vipulgupta2048 Could you rebase this please?

also lgtm @vipulgupta2048, whenever you get the chance, rebase then merge!


File upload

There's a way to upload a document (in this case an Image) and add a request (Open webui allow this) i'd like to use this library to make an Image recognition.
It is possible?

Yes, use imagesAsBase64 parameter in the Chat.SendAsync or Message.Images property(and pass base64 to it)


 UI sends 'Regression error' to send SDK.

Checklist

[ ] I have added screenshots above for all UI changes.
[ ] I have added e2e tests for all UI changes.
[ ] Documentation was updated if it was needed.

Codecov Report

Merging #1729 (a924fd7) into main (d0f387f) will decrease coverage by 13.42%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #1729       +/-   ##
===========================================
- Coverage   89.00%   75.58%   -13.43%
===========================================
  Files         105       10       -95
  Lines        5577      258     -5319
===========================================
- Hits         4964      195     -4769
+ Misses        613       63      -550

Flag
Coverage Δ

unittests
75.58% <ø> (-13.43%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ranalysis/erroranalysis/_internal/cohort_filter.py

...ponsibleai/responsibleai/_tools/causal/__init__.py

raiwidgets/raiwidgets/responsibleai_dashboard.py

responsibleai/responsibleai/__init__.py

raiutils/raiutils/dataset/__init__.py

...tools/counterfactual/dashboard_schemas/__init__.py

...nsibleai/databalanceanalysis/data_balance_utils.py

erroranalysis/erroranalysis/_internal/utils.py

...leai/responsibleai/modelanalysis/model_analysis.py

...s/erroranalysis/_internal/error_report/__init__.py

... and 85 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Just to make sure, this doesn't break the UI in any way, as it looks like it might be a breaking change? As in the UI doesn't expect to see Error anywhere instead of Regression error in the result right now?

I thought that UI sends "Error" but it was not the case. So this was not working as expected previously. With this change it works now as expected. There is not breaking change on UI side.


Update AWS SDK V2 to v0.18.0

Updates the dependency of github.com/aws/aws-sdk-go-v2 to v0.18.0 which includes the new API action DescribeInstanceTypes.
Previously we vendored the old v1 SDK which has that API action (https://github.com/cilium/cilium/pull/9699#pullrequestreview-327202685 for history on why). Now that SDK has is removed.

This change is

test-me-please

Coverage increased (+0.02%) to 45.959% when pulling 64eed1e26ae288e9939fc5c17c87f4c2310d4e52 on vladu/bump-aws-sdk-v2 into 33e37990078d47467f544f3230b21ec3a34b14c9 on master.


Not compatible with String + Format plugin

The composable accepts only one argument. If you install the CustomParseFormat dayjs can accept 2 to 4 arguments.
Would you recommend to not use the composable in that case? Or you can integrate it in the composable?
Thanks

maybe try adding 3+ more optional parameters to the composable locally and see if that works? if so PR that here

i no longer execute dayjs in the composable, try https://github.com/fumeapp/dayjs/releases/tag/v2.1.1

try the latest @nicolashmln


Enable automagical installation

See https://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Distributing Jupyter Extensions as Python Packages.html#Automatically-enabling-a-server-extension-and-nbextension
Configure the setup.py such that it will automatically install the extension for jupyter.

PRs are welome


Better process metrics

Use psutil to generate process global metrics e.g. cpu/memory/files/io ... .
The problem with rusage is that it's too low level, hard to understand and also
sometimes hard to use since it depend on the platform example:
maxrss which is in kilobytes in Lunix and bytes in OSX.
- http://man7.org/linux/man-pages/man2/getrusage.2.html
- https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/getrusage.2.html
This new process metric aim to give more fine grain and understandable metrics.

+1


Unexpected renderer error occurred after specifying pixel shader path.

Environment
Windows build number: 10.0.19042.0
Windows Terminal version (if applicable): 1.6.10571.0

Any other software?
* Cascadia Code
* Cascadia Code PL
* Cascadia Mono
* Cascadia Mono PL

Steps to reproduce
I had a fresh new Windows Terminal install from Microsoft Store;
downloaded example shaders from here;
added the following line to "Windows PowerShell" profile in settings.json and saved changes to file;
"experimental.pixelShaderPath": "D:\\temp\\Invert.hlsl",

restarted Windows Terminal and a warning popped up:

which translates to:

Warning
Renderer encountered an unexpected error: -2147024773
[OK]

And I think this warning message comes from this line.
I also tried changing the path to D:/temp/Invert.hlsl; changing the path to an empty folder; adding obvious syntax errors to the hlsl file.
But nothing changed, every time the exact same warning message popped up.

Expected behavior
Nothing pops up and the screen color is inverted.

Actual behavior
Described above.

Weird, that error is:

ERROR_INVALID_NAME
The filename, directory name, or volume label syntax is incorrect.
winerror.h

Which I don't think makes any sense. Does the retro terminal effect work for you? Use "experimental.retroTerminalEffect": true instead of experimental.pixelShaderPath

"experimental.retroTerminalEffect": true works fine.

After trying "D:\temp\Invert.hlsl", "D:\temp\\Invert.hlsl" , "D:\\temp\Invert.hlsl" and several other strings, I somehow managed to "glitch" it and made it work properly with shader files under D:\temp\.

However, when I changed the path from D:\temp\ to C:\Users\{Username}\temp\, the same warning message popped up just like before. (these two folders contain the same files)

Then I restarted Windows Terminal; changed path to other shader files in the same folder; tried to "glitch" again. But nothing new happened.

Hey so, I know you've gone through the trouble of hiding your Windows username in your screenshots, but I'm worried now if that might be part of the problem. Does your username have any Chinese characters in it? I'm worried that the shader might fail to load if the path has non-ascii characters in it. If you don't want to share your username, could you try some other path with Chinese characters in it?

Hey so, I know you've gone through the trouble of hiding your Windows username in your screenshots, but I'm worried now if that might be part of the problem. Does your username have any Chinese characters in it? I'm worried that the shader might fail to load if the path has non-ascii characters in it. If you don't want to share your username, could you try some other path with Chinese characters in it?

No, my username is made up of English letters and number digits. It's even a valid variable name in C/C++.

Huh. Well there goes that theory. Here's another totally wild theory - does the drive matter? Can the Terminal consistently load shaders from the D:\ drive, but not the C:\ drive?

Everything is working perfectly today. I've tried both drives, path with Chinese characters, inside and outside $HOME.
But I have no idea on what caused this change. Basically I was browsing the Internet and making PowerPoints these days. There was neither system update, nor Windows Terminal Update.😂

Well. That's annoying that the repro is gone before we could figure it out. But I'm glad that it's fixed! I still have no idea what might have caused this. I suppose I'll keep my eyes open for future reports.
Thanks!


Init/Shutdown sequence cannot be done twice

Describe the bug
Hello,
It seems even after ref counting of Init/Shutdown has been added in https://github.com/aws/aws-sdk-cpp/issues/2411, doing a sequence of Init/Shutdown twice, fails.
In pseudocode:
Init
Do S3Get
Shutdown

Init
Do S3Get  // Fails with curlCode: 35, SSL connect error
Shutdown

Expected Behavior
It seems the SDK requires us to ensure, once Shutdown is called, we can never Init again?
Current Behavior
The outcome of S3 GET API is: curlCode: 35, SSL connect error
Reproduction Steps
Here's a GTest:
TEST_F(S3Test, awsdk){
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    Aws::S3::S3ClientConfiguration config;
    auto client = std::make_unique<Aws::S3::S3Client>(config);

    // GET
    auto get = [&](){
        Aws::S3::Model::GetObjectRequest request;
        request.SetBucket("my-bucket");
        request.SetKey("my-key");

        auto outcome = client->GetObject(request);
        if(!outcome.IsSuccess()){
            std::cout << "GET FAILED " << outcome.GetError().GetMessage() << std::endl;
        }
    };

    get(); // THIS SUCCEEDS
    client.reset();
    Aws::ShutdownAPI(options);

    Aws::InitAPI(options);
    client = std::make_unique<Aws::S3::S3Client>(config);
    get(); // THIS FAILS
    client.reset();
    Aws::ShutdownAPI(options);
}

The relevant output is:
GET FAILED curlCode: 35, SSL connect error
The entire output is:
./fusion_gtest --gtest_filter="*S3Test*sdk*"
Note: Google Test filter = *S3Test*sdk*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from S3Test
[ RUN      ] S3Test.awsdk
GET FAILED curlCode: 35, SSL connect error
[       OK ] S3Test.awsdk (30331 ms)
[----------] 1 test from S3Test (30331 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (30331 ms total)
[  PASSED  ] 1 test.

Possible Solution
No response
Additional Information/Context
No response
AWS CPP SDK version used
1.11.411
Compiler and Version used
Apple clang version 15.0.0 (clang-15<IP_ADDRESS>)
Operating System and version
MacOS 14.4.1

I tried replicating and could not. I do think think this about the warning we have on the repo main page

⚠ If you are using macOS Sonoma, there is a https://github.com/aws/aws-sdk-cpp/issues/2804 where using libcurl version 8.4.0 on macOS can lead to issues. https://github.com/curl/curl/issues/12525. In the meanwhile, please use an updated version of curl from Homebrew. You can include this in your project via the CMAKE_PREFIX_PATH.

cmake -DCMAKE_PREFIX_PATH="/opt/homebrew/opt/curl/" \
 -DAUTORUN_UNIT_TESTS=OFF \
 -DBUILD_ONLY="s3" \
 -DCMAKE_INSTALL_PREFIX="~/sdk-install" \

using a brew install or custom install of curl and not the one that ships with macOS should fix your problem

cool give a shout if you have anymore issues!


"Too many arguments for function: 4" when trying to open any file in neovim

Category

[ ] fzf binary
[ ] fzf-tmux script
[ ] Key bindings
[ ] Completion
[ ] Vim
[x] Neovim
[ ] Etc.

OS

[x] Linux
[ ] Mac OS X
[ ] Windows
[ ] Windows Subsystem for Linux
[ ] Etc.

Shell

[ ] bash
[ ] zsh
[ ] fish

After invoking :FZF in neovim on ubuntu server, pressing enter on any file gives:
E118: Too many arguments for function: 4
This does not happen on my Arch Linux machine and as far as I can tell, I've copied the necessary configuration files over. I just set up this Ubuntu 16.04 VM yesterday so all packages are at the most recent on those repos.

Are you using the latest version of fzf?
It seems that this problem is already solved https://github.com/junegunn/fzf/issues/765#issuecomment-266629073

Still getting this with:
NVIM v0.2.0
0.16.8 (brew)
MacOS El Capitan 10.11.4


Cas/pump session tlr122

Integrating the FDA risk scenarios supported in this simulator. Shouldn't effect iCGM analysis so pushing through to setup Google using the simulator for the glucose prediction project.

Merging without review since I think there is negligible or no impact of this branch on related analysis like iCGM


HP Multifunction Printers - Scanning doesnt work

Hi,
HP Printer Model : LaserJetPro MFP M329dw and 1 more HP MFT Printer(Will share the model number soon).
I have tried using document scanner in ubuntu mate. It always shows as failed to connect to scanner.
ipp-usb-hp- laserjet pro MFP M329dw.zip
Had to remove ipp-usb and installed hplip and hp-plugin for getting scanner to work.
Attached whatever logs were available..
Thanks.

Hi,
with ipp-usb proprietary drivers are not expected to work, but sane-airscan and sane-escl should work. Did you try some of them together with the ipp-usb?

Disconnect from USB.
Install ipp-usb.
Connect to USB
Give outputs from
avahi-browse -rt _ipp._tcp
avahi-browse -rt _uscan._tcp

Hi @debiantriage,
from the ipp-usb log I see no attempt to scan using eSCL. But scanning part of the printer detected and published to Avahi. Looks like sane-airscan/sane-escl not installed or disabled.

Thank you @alexpevzner  I meant to, but completely forgot to look at the file attached by @sasikumar-mhc ! Maybe @sasikumar-mhc would say which OS is being used?

@debiantriage  Im using ubuntu mate 22.04.1

@alexpevzner ubuntu mate comes preinstalled with ipp-usb and im hoping that they also included sane-airscan/sane-escl ??
or should it be installed manually by me?
@debiantriage
ipp_tcp.txt
ipp-usb.zip
timeout.txt
uscan_tcp.txt
https://drive.google.com/file/d/1mWyLhPrQjzNWEmOFzqurmdaTCniCDhtp/view?usp=share_link
Thanks.

Hi @sasikumar-mhc,
looks like for now the problems are are actually at the sane-airscan side. I need its protocol trace logs.
Please, in the /etc/sane.d/airscan.conf file uncomment the following line:
[debug]
trace = ~/airscan/trace

Logs will be written to the ~/airscan/trace directory, which will be created automatically. For now I only need *.log files, *.tar files are not needed.

@alexpevzner
simple-scan-zeroconf.log
simple-scan-HP-LaserJet-Pro-M329-[8F773A]-(USB).log
Thanks.

@sasikumar-mhc,
From the printer's response to the ScanJobs command, looks like scanning is administratively prohibited:
<Details>
    <ElementXPath>/ScanSettings/ContextID</ElementXPath>
    <Description>Scan State is disabled, context_id missing!</Description>
 </Details>

Manual (http://h10032.www1.hp.com/ctg/Manual/c06184015.pdf) on a page 78 says the following:

IMPORTANT: For security purposes, scanning remotely is disabled by default. To enable this feature, in the Embedded Web Server, open the Settings tab, click Security, click Administrator Settings, select the Enable checkbox beside Scan from a Computer or Mobile Device, and then click Apply

This is very likely, that from the administrative point of view, the device doesn't distinguish between eSCL scanning from network and from USB
So I suggest to try to tweak this setting.

@alexpevzner
It worked. Thank you.

Fixed by proper printer configuration


Per-field @auth terminates subscriptions instead of setting field to null

Describe the bug
Given a multi @auth schema (abbreviated):
type Location
@model
@auth(
rules: [
{ allow: owner,  operations: [create, update, delete, read] },
{ allow: private, provider: iam, operations: [create, update, delete, read] },
{ allow: groups, groups: ["Admins"] },
{ allow: groups, groups: ["Users"], operations: [read]  },
])
@searchable {
id: ID!
owner: String
name: String!
clientUUID: String @auth(rules: [{ allow: owner}, { allow: private, provider: iam}, { allow: groups, groups: ["Admins"] }])
clientId: String @auth(rules: [{ allow: owner}, { allow: private, provider: iam }, { allow: groups, groups: ["Admins"] }])
clientSecret: String @auth(rules: [{ allow: owner}, { allow: private, provider: iam}, { allow: groups, groups: ["Admins"] }])
ratings: Int
rating: Float @auth(rules: [{ allow: private, provider: iam}, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])
}
Subscriptions terminate due to unauthorized access for the specific fields. Why not set these fields to null and continue? This is what is implied in documentation: https://docs.amplify.aws/cli/graphql-transformer/directives#field-level-authorization in "Per-Field with subscriptions" section.
2020-07-25 20:24:11.001541-0400 ReGroup[25243:6464607] [InitialSyncOperation] Beginning sync for Account
2020-07-25 20:24:11.178240-0400 ReGroup[25243:6464625] [InitialSyncOperation] Beginning sync for Location
2020-07-25 20:24:11.415044-0400 ReGroup[25243:6464651] [InitialSyncOperation] Beginning sync for User
2020-07-25 20:24:11.579121-0400 ReGroup[25243:6464605] [InitialSyncOperation] Beginning sync for UserLocation
2020-07-25 20:24:11.719556-0400 ReGroup[25243:6464651] [RemoteSyncEngine] One or more errors occurred syncing models. See below for detailed error description.
2020-07-25 20:24:11.719758-0400 ReGroup[25243:6464651] [RemoteSyncEngine] DataStoreError: An error occurred syncing Location
Caused by:
DataStoreError: GraphQL service returned a partially-successful response containing errors: [Amplify.GraphQLError(message: "Not Authorized to access clientId on type String", locations: Optional([Amplify.GraphQLError.Location(line: 8, column: 7)]), path: Optional([Amplify.JSONValue.string("syncLocations"), Amplify.JSONValue.string("items"), Amplify.JSONValue.number(0.0), Amplify.JSONValue.string("clientId")]), extensions: Optional(["data": Amplify.JSONValue.null, "errorType": Amplify.JSONValue.string("Unauthorized"), "errorInfo": Amplify.JSONValue.null])), Amplify.GraphQLError(message: "Not Authorized to access clientSecret on type String", locations: Optional([Amplify.GraphQLError.Location(line: 9, column: 7)]), path: Optional([Amplify.JSONValue.string("syncLocations"), Amplify.JSONValue.string("items"), Amplify.JSONValue.number(0.0), Amplify.JSONValue.string("clientSecret")]), extensions: Optional(["errorType": Amplify.JSONValue.string("Unauthorized"), "errorInfo": Amplify.JSONValue.null, "data": Amplify.JSONValue.null])), Amplify.GraphQLError(message: "Not Authorized to access clientUUID on type String", locations: Optional([Amplify.GraphQLError.Location(line: 10, column: 7)]), path: Optional([Amplify.JSONValue.string("syncLocations"), Amplify.JSONValue.string("items"), Amplify.JSONValue.number(0.0),
...
To further illustrate issue, there is no error generated about the "rating" field as Users have read access.
To Reproduce
Create graphql api with similar schema, authenticate as user in "Users" group. Launch application to start subscriptions.
Expected behavior
Expectation for a subscription if in "Users" group would be to receive subscriptions for Location objects with clientUUID field set to null as indicated in documentation.
Screenshots
Not helpful in this case... error log shows subscriptions failing due to the issue.
Environment(please complete the following information):
Swift version 5
Relevant items from Podfile.lock:

Amplify (1.0.4):

Amplify/Default (= 1.0.4)

Amplify/Default (1.0.4)
Amplify/Tools (1.0.4)
AmplifyPlugins/AWSAPIPlugin (1.0.4):

AppSyncRealTimeClient (~> 1.1.0)
AWSAuthCore (~> 2.14.0)
AWSCore (~> 2.14.0)
AWSPluginsCore (= 1.0.4)
ReachabilitySwift (~> 5.0.0)

AmplifyPlugins/AWSCognitoAuthPlugin (1.0.4):

AWSAuthCore (~> 2.14.0)
AWSCore (~> 2.14.0)
AWSMobileClient (~> 2.14.0)
AWSPluginsCore (= 1.0.4)

AmplifyPlugins/AWSDataStorePlugin (1.0.4):

Device Information (please complete the following information):
iPhone 11 simulator and iPhone 11 device.
Additional context
The intent is for users to be able to read Locations while having sensitive data redacted.

Interesting note... The error does not occur any longer on Simulator iPhone 11 (13.5) after upgrading to Amplify 1.0.5, but continues to occur on an iPhone 11 Pro Max device (13.5.1). I believe this was happening on both simulator and device earlier before upgrading to 1.0.5.

hi @jimkqwantegy, currently we have not tested support for { allow: private, provider: iam, operations: [create, update, delete, read] }, or the @searchable so it will be difficult to debug this issue. We do support auth rules with allow owner and static groups. I'll mark this as a follow up to provide more details regarding your use case.

The use case is that asynchronous business logic needs access to these fields from other AWS services using IAM, while other Cognito groups should not have visibility whatsoever. This seems to work well at the entity level but breaks at the field level. Could probably abstract these fields to another entity but documentation implied this was possible.

@jimkqwantegy we're currently reviewing the implementation of the @auth across all platforms and I'll keep you posted. Unfortunately I don't have an update right now.
Quick question, is your field protected with @auth required?

Hi Daniel,
Yes… That model looks like the following, but the rules have been loosened a bit so things will work. In actuality, the Users group wouldn’t be able to read, etc…
Thanks
Jim
type Location
@model

@auth(

rules: [

    { allow: owner, ownerField: "owner", operations: [create, update, delete, read] },

    { allow: groups, groupsField: "locationAccountId", operations: [create, update, delete, read] },

    { allow: private, provider: iam, operations: [create, update, delete, read] },

    { allow: groups, groups: ["Admins"] },

    { allow: groups, groups: ["Owners"] },

    { allow: groups, groups: ["Users"], operations: [read]  },

])

@searchable {

id: ID!
owner: String
locationAccountId: ID!
name: String!
address: String!
region: String
country: String
coordinates: Coordinates
status: Status
api: String
clientUUID: String @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: private, provider: iam}, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])
clientId: String @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: private, provider: iam }, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])
clientSecret: String @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: private, provider: iam}, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])
ratings: Int
rating: Float @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: private, provider: iam}, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])
capacity: Int
targetCapacity: Int
currentCapacity: Int
requiresAssessment: Boolean!
requiresAuthentication: Boolean!
requiresTemperature: Boolean
requiresMask: Boolean
requiredAccessControl: Boolean!
entries: [EntryArray]
type: LocationType
dwellTime: Int
account: Account @connection(fields: ["locationAccountId"])
}
From: Daniel Rochetti<EMAIL_ADDRESS>Reply-To: aws-amplify/amplify-ios<EMAIL_ADDRESS>Date: Tuesday, October 13, 2020 at 2:44 PM
To: aws-amplify/amplify-ios<EMAIL_ADDRESS>Cc: jimkqwantegy<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [aws-amplify/amplify-ios] Per-field @auth terminates subscriptions instead of setting field to null (#673)
@jimkqwantegyhttps://github.com/jimkqwantegy we're currently reviewing the implementation of the @auth across all platforms and I'll keep you posted. Unfortunately I don't have an update right now.
Quick question, is your field protected with @auth required?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/aws-amplify/amplify-ios/issues/673#issuecomment-707936485, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQM35CP2IDGX3A2RLR6WEC3SKSNXFANCNFSM4PHW7ZWQ.

Pasting schema to apply markdown
type Location
    @model
    @auth(
    rules: [

        { allow: owner, ownerField: "owner", operations: [create, update, delete, read] },
        { allow: groups, groupsField: "locationAccountId", operations: [create, update, delete, read] },
        { allow: private, provider: iam, operations: [create, update, delete, read] },
        { allow: groups, groups: ["Admins"] },
        { allow: groups, groups: ["Owners"] },
        { allow: groups, groups: ["Users"], operations: [read]  },
    ])
    @searchable {

  id: ID!
  owner: String
  locationAccountId: ID!
  name: String!
  address: String!
  region: String
  country: String
  coordinates: Coordinates
  status: Status
  api: String

  clientUUID: String @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: private, provider: iam}, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])

  clientId: String @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: private, provider: iam }, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])

  clientSecret: String @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: private, provider: iam}, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])

  ratings: Int

  rating: Float @auth(rules: [{ allow: owner, ownerField: "owner"}, { allow: groups, groupsField: "locationAccountId", operations: [read] }, { allow: private, provider: iam}, { allow: groups, groups: ["Admins"] }, { allow: groups, groups: ["Users"], operations: [read] }])

  capacity: Int
  targetCapacity: Int
  currentCapacity: Int
  requiresAssessment: Boolean!
  requiresAuthentication: Boolean!
  requiresTemperature: Boolean
  requiresMask: Boolean
  requiredAccessControl: Boolean!
  entries: [EntryArray]
  type: LocationType
  dwellTime: Int
  account: Account @connection(fields: ["locationAccountId"])
}

The above schema contains a couple complex use cases which are in question of support:

containing multiple @auth rules, allow owner, private, groups
containing per field level @auth rule.

This is the set of use cases that have been tested for DataStore with @auth directive: https://docs.amplify.aws/lib/datastore/setup-auth-rules/q/platform/ios
If you'd like to use that schema right now, I'd say the the workaround is to disable conflict resolution, and make calls directly to AppSync using Amplify.API

We recently launched support for 1. containing multiple @auth rules on a single model. I'm not aware of the effort to support 2. per field level @rule rules on iOS at the moment.

@jimkqwantegy you can read more about Multiple Authorization types in DataStore here

Field level auth feature request is being tracked here:
https://github.com/aws-amplify/amplify-swift/issues/1749


[WIP] Add JNIFUSE logging for debugging

What changes are proposed in this pull request?
Please outline the changes and how this PR fixes the issue.
Why are the changes needed?
Please clarify why the changes are needed. For instance,

If you propose a new API, clarify the use case for a new API.
If you fix a bug, describe the bug.

Does this PR introduce any user facing changes?
Please list the user-facing changes introduced by your change, including

change in user-facing APIs
addition or removal of property keys
webui

@LuQQiu Now you add logd all the time？

THIS IS FOR DEBUGGING PURPOSE, NO PLAN TO MERGE


feat: add analysis for emu run

This PR will try to:

Create consensus sequence (similar to ASV?) from emu Run (https://github.com/treangenlab/emu/issues/9)
Perform PCoA analysis
Visualize distribution per taxa level

HI, I just saw this. Do you have any update on this consensus sequence?


[BUG] <关于使用finetune_lora_ds.sh显存爆炸的问题>

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

[X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

[X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior
在使用lora对微调后并合并的模型文件进行二次微调的时候遇到显存爆炸的情况，我是一机多卡，两张32G的V100，训练阶段显存爆炸后遇到如下错误：
bash finetune/finetune_lora_ds.sh
[2024-01-23 11:24:19,300] torch.distributed.run: [WARNING]
[2024-01-23 11:24:19,300] torch.distributed.run: [WARNING] *****************************************
[2024-01-23 11:24:19,300] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-01-23 11:24:19,300] torch.distributed.run: [WARNING] *****************************************
[2024-01-23 11:24:21,344] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-23 11:24:21,523] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-23 11:24:22,466] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-23 11:24:22,466] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary
Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm
Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention
[2024-01-23 11:24:22,697] [INFO] [comm.py:637:init_distributed] cdb=None
Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary
Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm
Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00,  8.63it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00,  8.26it/s]
trainable params: 1,387,790,336 || all params: 9,109,114,880 || trainable%: 15.235183157553942
Loading data...

Detected CUDA files, patching ldflags
Emitting ninja build file /home/ubuntu/.cache/torch_extensions/py310_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.0702824592590332 seconds
Using /home/ubuntu/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/ubuntu/.cache/torch_extensions/py310_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.06588077545166016 seconds
/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
self._dummy_overflow_buf = get_accelerator().IntTensor([0])
/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
self._dummy_overflow_buf = get_accelerator().IntTensor([0])
0%|                                                                                | 0/100 [00:00<?, ?it/s]/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
warnings.warn(
/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
warnings.warn(
1%|█▌                                                                                | 1/100 [00:01<02:54,  1.76s/it]tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 0.4191, 'learning_rate': 0, 'epoch': 1.0}
1%|█▌                                                                                | 1/100 [00:01<02:54,  1.76s/it]Traceback (most recent call last):
File "/data/app/Qwen-main/finetune.py", line 374, in
train()
File "/data/app/Qwen-main/finetune.py", line 367, in train
trainer.train()
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/transformers/trainer.py", line 1837, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/transformers/trainer.py", line 2693, in training_step
self.accelerator.backward(loss)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/accelerate/accelerator.py", line 1958, in backward
self.deepspeed_engine_wrapped.backward(loss, **kwargs)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/accelerate/utils/deepspeed.py", line 167, in backward
self.engine.backward(loss, **kwargs)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1964, in backward
self.optimizer.backward(loss, retain_graph=retain_graph)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 2040, in backward
self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
scaled_loss.backward(retain_graph=retain_graph)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 886, in reduce_partition_and_remove_grads
self.reduce_ready_partitions_and_remove_grads(param, i)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1399, in reduce_ready_partitions_and_remove_grads
self.reduce_independent_p_g_buckets_and_remove_grads(param, i)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 915, in reduce_independent_p_g_buckets_and_remove_grads
self.reduce_ipg_grads()
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1350, in reduce_ipg_grads
self.average_tensor(self.ipg_buffer[self.ipg_index])
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1110, in average_tensor
self.allreduce_and_scatter(buckets[bucket_key],
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1014, in allreduce_and_scatter
self.allreduce_and_copy_with_multiple_ranks(small_bucket,
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 987, in allreduce_and_copy_with_multiple_ranks
allreduced = self.allreduce_bucket(small_bucket, log=log, divide=divide, process_group=process_group)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1464, in allreduce_bucket
tensor = self.flatten(bucket)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/_utils.py", line 498, in _flatten_dense_tensors
return torch._C._nn.flatten_dense_tensors(tensors)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 274.00 MiB. GPU 1 has a total capacty of 31.74 GiB of which 60.38 MiB is free. Including non-PyTorch memory, this process has 31.67 GiB memory in use. Of the allocated memory 29.24 GiB is allocated by PyTorch, and 1.97 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-01-23 11:24:49,334] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 523571 closing signal SIGTERM
[2024-01-23 11:24:49,548] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 523572) of binary: /data/anaconda/envs/qwen-lora/bin/python3.10
Traceback (most recent call last):
File "/data/anaconda/envs/qwen-lora/bin/torchrun", line 8, in
sys.exit(main())
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/data/anaconda/envs/qwen-lora/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
finetune.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2024-01-23_11:24:49
host      : 10-60-69-204
rank      : 1 (local_rank: 1)
exitcode  : 1 (pid: 523572)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
期望行为 | Expected Behavior
No response
复现方法 | Steps To Reproduce
首先针对原Qwen-7B模型进行lora微调，微调后的文件使用lora合并，合并之后运行微调的模型没有问题
然后针对微调并合并后的这个模型文件进行二次微调的时候出现了显存爆炸问题，问题log如上。
运行环境 | Environment
- OS:Ubuntu 20.04
- Python:3.10.13
- Transformers:4.32.0
- PyTorch:2.1.2+cu121
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):12.1
- nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0
- nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100S-PCIE-32GB          Off | 00000000:00:03.0 Off |                  Off |
| N/A   30C    P0              25W / 250W |      5MiB / 32768MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla V100S-PCIE-32GB          Off | 00000000:00:08.0 Off |                    0 |
| N/A   29C    P0              23W / 250W |      5MiB / 32768MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1441      G   /usr/lib/xorg/Xorg                            4MiB |
|    1   N/A  N/A      1441      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+

备注 | Anything else?
No response

我也有这个问题

我也有这个问题

解决了吗

我也有这个问题

解决了吗

还没有，我觉得 #619 应该也是同样的问题，不过没有给出答案。如果有解决方案了能麻烦分享一下吗，我如果找到原因了也会在这发一下


make_travis_yml_2.hs doesn't detect compiler error

Check out https://travis-ci.org/peti/pico-prolog/jobs/135561591:
Data/Vector/Fusion/Stream/Size.hs:25:10:
    Warning: No explicit method nor default method for `signum'
    In the instance declaration for `Num Size'
[ 4 of 19] Compiling Data.Vector.Internal.Check ( Data/Vector/Internal/Check.hs, dist/build/Data/Vector/Internal/Check.o )
[ 5 of 19] Compiling Data.Vector.Fusion.Stream.Monadic ( Data/Vector/Fusion/Stream/Monadic.hs, dist/build/Data/Vector/Fusion/Stream/Monadic.o )
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package transformers-<IP_ADDRESS> ... <command line>: can't load .so/.DLL for: HStransformers-<IP_ADDRESS>-330efd28cf24d8cb9a5fe8fa655d885d5db855228308e159f3d7459f5576a972 (libHStransformers-<IP_ADDRESS>-330efd28cf24d8cb9a5fe8fa655d885d5db855228308e159f3d7459f5576a972.so: cannot open shared object file: No such file or directory)

travis_time:end:158a883c:start=1465218060448809022,finish=1465218089177765635,duration=28728956613

The command "cabal new-build ${TEST} ${BENCH}" exited with 0.

The build fails with a nasty compiler/linker error, but Travis-CI won't detect that; the build is marked as successful in the end.

Another, similar error happened in https://travis-ci.org/peti/hledger-interest/jobs/135560940:
Preprocessing library hledger-lib-0.27.1...

Hledger/Data/Types.hs:23:8:
    Could not find module `GHC.Generics'
    It is a member of the hidden package `ghc-prim'.
    Perhaps you need to add `ghc-prim' to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.

travis_time:end:13841d3c:start=1465213162062566762,finish=1465213163946251238,duration=1883684476

The command "cabal new-build --disable-tests --disable-benchmarks" exited with 0.

The build is also marked as successful despite the obvious error.

@peti unfortunately this is a Bug in cabal, so we'll have to wait until there's an upstream fix

Fyi, the 1.24 branch of cabal has a fix for that silent exit-0 failures, and I just rebuilt the cabal-1.24 package in my ppa with the latest 1.24 branch tip for Travis' benefit
so this should work now... if not, please reopen

I've switched several of my more involved projects to the new build script now and it works like a charm. Thank you very much for the effort you've put into this tool; it's very useful to me and many others and I believe it has a great impact towards making Haskell packages more reliable and thus easier to build, distribute, and re-use.


Imports for wasm-in-wasm

Summary
The current wasm-in-wasm example creates and instantiates a WebAssembly module using this call: WebAssembly::instantiate_buffer(WASM, &Object::new()). From the docs, the second argument is the imports object for the WebAssembly module. The example just uses and empty object, and it isn't immediately clear how to create actual imports for the WebAssembly module.
Additional Details
I've taken a stab at implementing this here, but I'm running into a bug I can't track down. I'm not sure if my approach is wrong, or if there is just something I'm misunderstanding:
https://github.com/Perseus101/wasm-bindgen/tree/wasm-imports-example/examples/wasm-in-wasm-imports

Unfortunately when you provide a JS function as an input to the import object whenever it's invoked the this parameter is always null. You'd need to use .bind somehow if you'd like to set the this parameter for later usage.
There's probably not a super ergonomic way to do this in Rust right now, you'll likely need some gnarly-ish JS glue.

Thanks, that was the insight I was missing! It wasn't too hard to get it working with some judicious use of Reflect. The link above should be a working example now.
On another note, do you think it is worth merging this example upstream? I know this would have been useful to me, but it is a pretty niche example. Let me know, I'll open a PR if you think it's worth it.

Nice! Seems reasonable to me to add :)


Github badge failing for c++ repo

Hi! Thanks for this useful library! Since a few weeks this C++ repo is not being counted any more, any suggestions? Seems like it's related to the repo, as others still work.

![](https://tokei.rs/b1/github/ethz-adrl/towr?category=code)

Thank you for issue! I've pushed a fix for this to master and will try to push out a deploy soon.


feat: improve starter and interactions

Summary

This PR

Enhanced interactions when users run pnpm create rspress@latest
More comprehensive starter template

Details

Currently, if we run pnpm create rspress@latest, the starter website looks like this

In my personal opinion, it's a little bit confusing. Inspired by VitePress, I made some changes to the starter.

Modify template files
Add interactions for users to input site title and description.

Now users can input site title and description when they init project.

The page with the above data looks like this

Guide page is from the Use MDX in rspress.dev.
Hello page is just a simple md with few words.
API page is an external link to Rspress API
The whole process looks like this
https://github.com/web-infra-dev/rspress/assets/55378595/699bff18-15bf-4234-8e27-c7385bf9642d
Related Issue

Checklist

[ ] I have added changeset via pnpm run change.
[ ] I have updated the documentation.
[ ] I have added tests to cover my changes.

I change the doc folder to docs folder because I suppose it should be consistent with the Rspress Doc structure.
rspress.dev also presents an example for docs

Thanks.

Thanks.

Nothing 🙌


License information

There is no license information in the repository. Can you please provide one?
TIA!

Done, we added the MIT license

Thanks !


(Feature request) Terminal moon sorting

Hello. It would be nice if you could add an option in mod settings which would change moons (including mods moons) list sorting in the terminal; for example, sort by price or sort by danger. It would be a subtle but nice addition and much appreciated. And maybe in addition, add an option to keep one line empty between moons with different danger levels. It would be nice if you could like separate moons with different danger levels and also sort them by price.

i keep the request in mind

I would also like to request this feature, having the moons that require money bundled with the ones that don't lead to a lot of confusion in my group


Support running e2e tests for dev

This change is

Reviewed 3 of 3 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

docs/devel/build-tool.md, line 113 at r1 (raw file):
Previously, istalker2 (Stan Lagun) wrote…
-ginkgo.focus="Should have default route"
why is this? The same for documentation

It's meant to demonstrate that you can run just some e2e tests. Perhaps should add a note that it's optional.

Comments from Reviewable

Review status: 1 of 3 files reviewed at latest revision, 1 unresolved discussion.

docs/devel/build-tool.md, line 113 at r1 (raw file):
Previously, ivan4th (Ivan Shvedunov) wrote…
It's meant to demonstrate that you can run just some e2e tests. Perhaps should add a note that it's optional.

Done.

Comments from Reviewable

Reviewed 1 of 3 files at r1, 2 of 2 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable


Added a functionality of logo clicking will redirect to home page

Related Issue
This is just a feature enhancement
Fixes:  #506
Description
Added an anchor without text decoration such that it links to home page
#506

Type of PR

[ ] Bug fix
[x] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
[Attach any relevant screenshots or videos demonstrating the changes. Make sure to attach before & after screenshots in your PR.]
Checklist:

[x] I have made this change from my own.
[x] I have taken help from some online resources.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

@jvkousthub Please review the conflicts and resolve them in the mentioned files. Make sure to pull the latest changes before pushing your feature or bug fixes to the main branch. After resolving the conflicts, kindly open a new PR for review.


Generate key colors for rank 2 MOS

Same as in the MOS dialog.

Due to the free size of the scale I couldn't do exactly the same as in the MOS dialog, but close enough.


display "Recent Chats" only if there are recent chats

The types don't check probably an easy fix. VSCode should give you some feedback?


Compatibility with segment replication

Summary
With 2.9.0 release, there are lot of enhancements going in for segment replication[1][2] feature (went GA in 2.7.0), we need to ensure different plugins are compatible with current state of this feature. Previously, we ran tests on plugin repos to verify this compatibility but want plugin owners to be aware of these changes so that required updates (if any) can be made. With 2.10.0 release, remote store feature is going GA which internally uses SEGMENT replication strategy only i.e. it enforces all indices to use SEGMENT replication strategy. So, it is important to validate plugins are compatible with segment replication feature.
What changed
1. Refresh policy behavior

RefreshPolicy.IMMEDIATE will only refresh primary shards but not replica shards immediately. Instead post refresh, primary will start  a round of segment replication to update the replica shard copies leading to eventual consistency.
RefreshPolicy.WAIT_UNTIL ensures the indexing operation is searchable in your cluster i.e. RAW (Read after write guarantee). With segment replication, this guarantee is not promised due to delay in replica shared updates from asynchronous background refreshes.

2. Refresh lag on replicas
With segment replication, there is inherent delay in documents to be searchable on replica shard copies. This is due to the fact that replica shard copies over data (segment) files from primary. Thus, compared to document replication, there will be on average increase in amount of time the replica shards are consistent with primaries.
3. System/hidden indices support
With https://github.com/opensearch-project/OpenSearch/pull/8200, system and hidden indices are now supported with SEGMENT replication strategy. We need to ensure there are no bottlenecks which prevents system/hidden indices with segment replication.
Next steps
With segment replication strong reads are not guaranteed. Thus, if the plugin needs strong reads guarantees specially as alternative to change in behavior of refresh policy and lag on replicas (point 1 and 2 above), we need to update search requests to target primary shard only. With https://github.com/opensearch-project/OpenSearch/pull/7375, core now supports primary shards only based search. Please follow documentation for examples and details
Open questions
In case of any questions or issues, please post it in core issue
Reference
[1] Design
[2] Documentation

Request owners to add v2.9.0 label on this issue.

Hi Plugin Owners,
Gentle reminder to look into this issue as code freeze date for 2.9.0 release is near i.e. July 11th.

refer https://github.com/opensearch-project/alerting/issues/974#issuecomment-1631567927
The same points for system/hidden indices and scheduling are applicable for ISM as well.
cc : @dreamer-89

Ping @getsaurabh02 @bowenlan-amzn as we are approaching 2.10 code freeze date. Please have a look.

@dreamer-89 what's the behavior of an index created in an OS version without segRep, and later upgrade to a version that enforces segRep? It will still use docRep right?

@dreamer-89 what's the behavior of an index created in an OS version without segRep, and later upgrade to a version that enforces segRep? It will still use docRep right?

This is a good question! Yes, you are right. AFAIK, once an index is created, it's replication type change wouldn't be allowed i.e. index created with DOCUMENT replication wouldn't be changed to SEGMENT and vice-versa.
When cluster level settings enforces a specific replication strategy, it will govern replication type for new index creations. There wouldn't be change in replication type for existing indices. Tagging product @nandi-github @elfisher if this is not correct!
CC @mch2 @anasalkouz

@bowenlan-amzn : Is there anything pending on this issue ? If not, do you mind closing this issue.

Yes, we can close this now


Fix get_where_subquery macro link

Description & motivation
Update a link for get_sub_query in the https://docs.getdbt.com/reference/resource-configs/where docs.  It currently points to a page that returns a 404.

@p11o : thanks for much for fixing the 404! ❤
looking at this now. it's fixed! it came from this PR https://github.com/dbt-labs/docs.getdbt.com/pull/1457 what was merged back on May 19. i'll go ahead and close this PR. feel free to reopen if this hasn't been fully resolved.

@p11o : thanks for much for fixing the 404! ❤
looking at this now. it's fixed! it came from this PR https://github.com/dbt-labs/docs.getdbt.com/pull/1457 what was merged back on May 19. i'll go ahead and close this PR. feel free to reopen if this hasn't been fully resolved.


robyn_chain function not found error

Project Robyn
Describe issue
Describe your issue and provide warnings and error messages.
The function robyn_refresh utilizes robyn_chain internally , which seems to have been deprecated/removed in the latest version.
Below is the error message :
Error in robyn_chain(json_file) : could not find function "robyn_chain"
Provide reproducible example
Issues are often related to custom input data that is difficult to debug without. If necessary, please modify your data to mask real values and share a dataset that is able to reproduce the issue. Please also share your model configuration and exported JSON files if available.
Environment & Robyn version
Make sure you're using the latest Robyn version before you post an issue.

Check and share Robyn version: packageVersion("Robyn")
R version (Please, check and share: sessionInfo() or R.version$version.string)

All the packages are being run as the latest version in Windows.
Can someone help me with this or point me towards the resolution

The function is not deprecated. You can see it here. Please check if you're using the latest GitHub version and retry.

My apologies, it was because of the latest version not being used. However, now it says the json file is not found :

I think it is because the function robyn_chain is not being able to locate the file correctly, because

returns NULL.

hm this is strange. I just tried refresh on the demo data and can't reproduce your error. In your screenshot, the model recreation is successful using the json. So it's strange that it misses the second time. Let's see if Bernardo's fix on your other issue can solve this first?


set-hook -> set-option?

I don't fully grasp the issue, but there seems to be some corner-case with hooks here: https://github.com/carlocab/tmux-nvr/blob/main/nvr.tmux#L18
tmux v2 change notes suggests set-hook can be replaced w/ set-option that already allows multiple values:
* Hooks are now stored in the options tree as array options, allowing them to
have multiple separate commands. set-hook and show-hooks remain but
set-option and show-options can now also be used (show-options will only show
hooks if given the -H flag). Hooks with multiple commands are run in index
order.

I'm not sure what corner case you're referring to. Could you elaborate on that?
set-hook and set-option are equivalent for hooks, as explained here. I prefer set-hook because the intent of the command is more transparent. They both need an index to be specified to define multiple hooks.

I'm referring to this:
It should just work, unless you have existing session-created tmux hooks.
If this applies, you need to specify an unused index in the session-created array using the @tmux-nvr-hook-index option in tmux.conf. For example,

Yes. Replacing set-hook with set-option will not change that part of the README. You will still need to specify indices of the session-created hook array even when setting it via set-option, as stated in the section in the manual on hooks.

Oh well.

Yup, I don't like having to choose indices for hooks either. I guess it's part of having written tmux in C.
Thanks for trying to think of a fix though!

By the way, I've fixed this now. Thanks for pushing me to look into it.

Glad to hear!
Is it this? https://github.com/carlocab/tmux-nvr/commit/1f252ce4ea67ee4c9916eb2734d1a7d9ab04ea8a

Yes, that one.


[SPARK-22188] [CORE] Adding security headers for preventing XSS, MitM and MIME sniffing

What changes were proposed in this pull request?
The HTTP Strict-Transport-Security response header (often abbreviated as HSTS) is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS, instead of using HTTP.
Note: The Strict-Transport-Security header is ignored by the browser when your site is accessed using HTTP; this is because an attacker may intercept HTTP connections and inject the header or remove it. When your site is accessed over HTTPS with no certificate errors, the browser knows your site is HTTPS capable and will honor the Strict-Transport-Security header.
The HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks.
The HTTP X-Content-Type-Options response header is used to protect against MIME sniffing vulnerabilities.
How was this patch tested?
Checked on my system locally.

Can one of the admins verify this patch?

I can see them from the browser.
LGTM except two minor comments, @krishna-pandey .

@dongjoon-hyun Thanks for the review. Made the changes as suggested.

@dongjoon-hyun Made the changes as suggested.

Test build #3944 has started for PR 19419 at commit d00a1dc.

Test build #3944 has finished for PR 19419 at commit d00a1dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@vanzin @tgravescs @ajbozarth  what is your opinion on this PR? Is it a necessary fix for Spark?

I think the change is OK, to give a mechanism to set these headers if desired. There are still several comments to address though.

overall I think the headers are fine, more security the better.  I'm not exactly sure the attack vector with the spark UI though.  Normally I would expect your UI to be on a corporate network and you vpn in, but I guess maybe if you are running in AWS or similar public cloud and you go somewhere to access, but I'm not sure what data they can get there or why you would be using http in the first place but there are lots of setups.
@krishna-pandey  do you have specific use case/attack vector in mind here?
I was wondering if there was a more generic way to allow user to specify desired headers without having a config for each one. Downside to that is its not as obvious though too so I need to think about that a bit more.

@tgravescs These generic headers are about providing available client-side protection for the application. I also think even if there is no sensitive data to formulate an attack by itself here, the information can be used in conjunction to target other ecosystem components. Also, in future we may add an interface for data access. Now is the time to think of Security First. Cross-site Scripting is one of the most prevalent attack vector and has been an OWASP Top 10 risk for web applications for decades. As the effort to have these in place here is minimal, IMHO we should set these.
As you rightly mentioned, deployment on cloud can expand the attack surface pretty wide in absence of right firewall policy. Also let's not forget insider threat inside corporate networks.
Going forward may be we will have enough insight to choose which headers are needed to be enabled by default and enforce them from application side and not leave it to Users.

ok to test.

Test build #82736 has started for PR 19419 at commit 10f6f30.

@srowen @tgravescs @dongjoon-hyun @jerryshao
Please review the PR. I have incorporated all changes as suggested.

Test build #82736 has finished for PR 19419 at commit 10f6f30.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82736/
Test FAILed.

/home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/internal/config/package.scala:440:0: Whitespace at end of line

Please fix the style issue.

Test build #82737 has started for PR 19419 at commit 10f6f30.

Test build #82737 has finished for PR 19419 at commit 10f6f30.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82737/
Test FAILed.

Test build #82741 has started for PR 19419 at commit 1e61484.

Test build #82741 has finished for PR 19419 at commit 1e61484.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82741/
Test FAILed.

@jerryshao removed Whitespace at end of line 440 in package.scala. ok to test.

Test build #82742 has started for PR 19419 at commit 5c76b91.

Test build #82742 has finished for PR 19419 at commit 5c76b91.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82742/
Test FAILed.

Test build #3947 has started for PR 19419 at commit 5c76b91.

Test build #3947 has finished for PR 19419 at commit 5c76b91.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #82791 has started for PR 19419 at commit 85cb02b.

Test build #82795 has started for PR 19419 at commit acfd227.

LGTM.

Test build #82791 has finished for PR 19419 at commit 85cb02b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82791/
Test PASSed.

Merged build finished. Test PASSed.

Test build #82795 has finished for PR 19419 at commit acfd227.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82795/
Test PASSed.

Test build #82807 has started for PR 19419 at commit 1cc34e9.

Is there a reason why this cannot be always enabled?

The downside of additional header traffic is trivial. I suppose it is not needed in most contexts so it was conservative to leave it off by default. That said I am not aware of any particular negative to enabling it. I would be OK with leaving it on by default if there's no particular downside.

Test build #82807 has finished for PR 19419 at commit 1cc34e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82807/
Test PASSed.

Yea in general for security features it seems like it's good to turn on them by default.

@rxin , @srowen I think we can enable X-XSS-Protection and X-Content-Type-Options response header by default. STS Header can be left configurable or enabled by default when Spark UI is running on HTTPS.
Word of caution: When X-Content-Type-Options response HTTP header is set to "nosniff", it will block a request if the requested type is  "style" and the MIME type is not "text/css", or when requested type is "script" and the MIME type is not a JavaScript MIME type.

For the content type header, does the UI seem to work normally with it on? I don't think we should have any problem with that restriction or else need to fix it .  You could move the defaults to on then.

Test build #82876 has started for PR 19419 at commit de54313.

Test build #82878 has started for PR 19419 at commit b6d4885.

@srowen @rxin Made changes to enable the X-Content-Type-Options and X-XSS-Protection values by default. Please review.

Test build #82876 has finished for PR 19419 at commit de54313.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82876/
Test PASSed.

Test build #82878 has finished for PR 19419 at commit b6d4885.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82878/
Test PASSed.

Merged to master


ZEPPELIN-371 : Import a notebook

Import a notebook by either an external URL or uploading a valid JSON.

could we add test for this?

Sure, I'll write a unit test case for this.

Added test case for the same as well.
CI fails in this PR. with following reason
[32;1mThe command "mvn package -Pbuild-distr -Phadoop-2.3 -Ppyspark -B" exited with 0.[0m
travis_time:start:088fd37b
$ ./testing/startSparkCluster.sh 1.4.0 2.3
SPARK_HOME is /home/travis/build/apache/incubator-zeppelin/spark-1.4.0-bin-hadoop2.3

No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

The build has been terminated

Thanks @prabhjyotsingh for the nice feature. I manually tested and working well for me.
CI failing is fixed by #380. Could you try rebase this branch?

The build is failing with following error
    06:21:48,776  INFO org.apache.zeppelin.rest.AbstractTestRestApi:226 - Test Zeppelin terminated.
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.115 sec <<< FAILURE! - in org.apache.zeppelin.rest.ZeppelinRestApiTest
org.apache.zeppelin.rest.ZeppelinRestApiTest  Time elapsed: 4.115 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.zeppelin.rest.AbstractTestRestApi.startUp(AbstractTestRestApi.java:123)
at org.apache.zeppelin.rest.ZeppelinRestApiTest.init(ZeppelinRestApiTest.java:57)

Running org.apache.zeppelin.security.SecurityUtilsTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec - in org.apache.zeppelin.security.SecurityUtilsTest
Running org.apache.zeppelin.server.CorsFilterTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.29 sec - in org.apache.zeppelin.server.CorsFilterTest
06:21:49,112  INFO org.apache.zeppelin.server.ZeppelinServer:112 - Shutting down Zeppelin Server ...
06:21:49,112  INFO org.apache.zeppelin.server.ZeppelinServer:112 - Shutting down Zeppelin Server ...
06:21:49,113  INFO org.apache.zeppelin.server.ZeppelinServer:118 - Bye
06:21:49,113  INFO org.apache.zeppelin.server.ZeppelinServer:118 - Bye

Results :

Tests in error:
ZeppelinSparkClusterTest.init:46->AbstractTestRestApi.startUp:123 Â» NullPointer
ZeppelinRestApiTest.init:57->AbstractTestRestApi.startUp:123 Â» NullPointer

Tests run: 16, Failures: 0, Errors: 2, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Zeppelin ........................................... SUCCESS [ 10.558 s]
[INFO] Zeppelin: Interpreter .............................. SUCCESS [01:19 min]
[INFO] Zeppelin: Zengine .................................. SUCCESS [  4.987 s]
[INFO] Zeppelin: Spark dependencies ....................... SUCCESS [ 32.026 s]
[INFO] Zeppelin: Spark .................................... SUCCESS [01:20 min]
[INFO] Zeppelin: Markdown interpreter ..................... SUCCESS [  0.700 s]
[INFO] Zeppelin: Angular interpreter ...................... SUCCESS [  0.279 s]
[INFO] Zeppelin: Shell interpreter ........................ SUCCESS [  0.356 s]
[INFO] Zeppelin: Hive interpreter ......................... SUCCESS [  2.433 s]
[INFO] Zeppelin: Apache Phoenix Interpreter ............... SUCCESS [ 10.156 s]
[INFO] Zeppelin: PostgreSQL interpreter ................... SUCCESS [  1.127 s]
[INFO] Zeppelin: Tajo interpreter ......................... SUCCESS [  1.565 s]
[INFO] Zeppelin: Flink .................................... SUCCESS [ 18.759 s]
[INFO] Zeppelin: Apache Ignite interpreter ................ SUCCESS [ 15.523 s]
[INFO] Zeppelin: Kylin interpreter ........................ SUCCESS [  0.678 s]
[INFO] Zeppelin: Lens interpreter ......................... SUCCESS [  3.232 s]
[INFO] Zeppelin: Cassandra ................................ SUCCESS [ 39.078 s]
[INFO] Zeppelin: web Application .......................... SUCCESS [ 37.655 s]
[INFO] Zeppelin: Server ................................... FAILURE [ 15.121 s]
[INFO] Zeppelin: Packaging distribution ................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:55 min
[INFO] Finished at: 2015-11-06T06:21:49+00:00
[INFO] Final Memory: 66M/362M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project zeppelin-server: There are test failures.
[ERROR]
[ERROR] Please refer to /home/travis/build/apache/incubator-zeppelin/zeppelin-server/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :zeppelin-server
travis_time:end:1b88a8e0:start=1446790550157092954,finish=1446790909997950807,duration=359840857853

The command "mvn verify -Pusing-packaged-distr -Phadoop-2.3 -Ppyspark -B" exited with 1.

My guess is, in AbstractTestRestApi:116
ZeppelinServer.notebook.getInterpreterFactory().get() is empty
any pointers what would have gone wrong ?

I'm in the same CI test error in #341. Let me try to figure out.

Please take a look NotebookServerTest. Extending AbstractTestRestApi helped me to make ci build passes in #341.

Thanks @Leemoonsoo, have made relevant changes.
now it fails for  "ZeppelinIT.testAngularDisplay:173->createNewNote:306 » Timeout Timed out after"
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.zeppelin.ZeppelinIT
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.224 sec <<< FAILURE! - in org.apache.zeppelin.ZeppelinIT
testAngularDisplay(org.apache.zeppelin.ZeppelinIT)  Time elapsed: 22.152 sec  <<< ERROR!
org.openqa.selenium.TimeoutException: Timed out after 10 seconds waiting for visibility of element located by By.id: noteNameModal
Build info: version: '2.44.0', revision: '76d78cf323ce037c5f92db6c1bba601c2ac43ad8', time: '2014-10-23 13:11:40'
System info: host: 'testing-worker-linux-docker-095ed8f0-3208-linux-3', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-40-generic', java.version: '1.7.0_76'
Session ID: e6ed0c8a-b983-40da-b950-09ead342ffe0
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=LINUX, acceptSslCerts=true, javascriptEnabled=true, cssSelectorsEnabled=true, databaseEnabled=true, browserName=firefox, handlesAlerts=true, nativeEvents=false, webStorageEnabled=true, rotatable=false, locationContextEnabled=true, applicationCacheEnabled=true, takesScreenshot=true, version=31.0}]
    at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:79)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:228)
    at org.apache.zeppelin.ZeppelinIT.createNewNote(ZeppelinIT.java:306)
    at org.apache.zeppelin.ZeppelinIT.testAngularDisplay(ZeppelinIT.java:173)

Results :

Tests in error:
  ZeppelinIT.testAngularDisplay:173->createNewNote:306 » Timeout Timed out after...

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

Trying to debug, will update once finished.

Thanks @Leemoonsoo while importing any external library I'll keep this in mind for next time.
Resolved build failure, ready for review.

Working nicely, Looks good to me.

Merge if there're no more discussions

@prabhjyotsingh thank you, great feature.
Huge :+1: for having a tests!
This comes very handy with services like github or https://www.zeppelinhub.com/viewer


fix AUTH failed: ERR Client sent AUTH, but no password is set

for example:
in laravel or other projects  .env file
REDIS_HOST=redis
REDIS_PASSWORD=
REDIS_PORT=6379

@hncg Thank you! This has been driving me insane with Laravel until I saw your PR. I hope this gets merged ASAP!

REDIS_PASSWORD should be set to 'null' instead of nothing.
Ex:
REDIS_HOST=redis
REDIS_PASSWORD=null
REDIS_PORT=6379

Yes, but we shouldn’t  expect that everyone knows exactly how to set.
@the-veloper

Fixed for v1.1 in 31baa9b (will be included in v1.1.4).


supress Actions jobs when only text files are changed

A follow-up of #674
It wouldn't be needed to run check jobs when only text files (CHANGES, etc.) are changed.
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-excluding-paths

The syntax is fine - I assume there is no harm in merging this.

I wouldn't do it. You never know what those text files - they can be resources that are compiled or used at runtime, etc. Why bother? This is what the CI is for - to make sure we (committers) didn't screw up.


Update reqs for Mac M1 ARM. Remove dependencies: TF, Keras, and Albumentation. Remove FSLeyes (temporarily)

Checklist

[x] I've given this PR a concise, self-descriptive, and meaningful title
[x] I've linked relevant issues in the PR body
[x] I've applied the relevant labels to this PR
[ ] I've added relevant tests for my contribution
[ ] I've updated the documentation and/or added correct docstrings
[x] I've assigned a reviewer
[ ] I've consulted ADS's internal developer documentation to ensure my contribution is in line with any relevant design decisions

Description
As we're moving to IVADOMED, I think we can scrap the following requirements: Tensorflow, Keras, and Albumentations. This should also conveniently make AxonDeepSeg compatible with Macs using the ARM M1 processor, as long as there's no issues on the IVADOMED side (?).
I've also updated the requirements for Mac M1 ARM compatibility (MiniForge isn't compatible with Python 3.7). I've increased it to Python 3.9, but am not sure if this will break FSLeyes compatibility. If it does, maybe we should continue anyways (or downgrade to 3.8) and see if we can resolve it on the FSLeyes side.
Linked issues
#523

@mariehbourget this PR is ready for review. I had to remove FSLeyes, hopefully just temporarily, as it doesn't seem to be compatible with either Python 3.9 or the upgraded sci-kit versions. However, now I'm finally able to run tests locally on my Mac M1 machine, which will help with developing ADS v4; we can (hopefully) reinstate FSLeyes after IVADOMED has been integrated.

Oops just saw you already left messages!


added uuid library to generate random uuid

imported uuid library to generate a random uuid4 (36 character unique identifier) at each run rather than sticking to the same static uuid.

Great idea, this makes the testing of the functionality much more reliable and heterogeneous.


Fix some issues causing errors in TiKV on my machine

I'm running an AMD machine with Linux kernel version 5.3. This causes two issues:

The stat entries in /sys/block/ have 15 columns instead of 11.
The layout of data returned by the cpuid instruction is completely different. The cache-size crate is based on the rust-cpuid crate, which does not yet support AMD. In order to distinguish this state I need to know the vendor_id of the CPU.

These are addressed in the first and second commits, respectively.
PTAL @lonng

BTW, I released a new version of sysinfo with new functionalites (such as "get_vendor_id"). You might want to take a look.

@GuillaumeGomez  Thanks, @nrc, it seems this PR does not necessary anymore.


Geolocate /ingest endpoint

What does this pull request do? Explain your changes. (required)
For /ingest endpoint find closes server. Uses same code as /broadcast endpoint.
Specific updates (required)

How did you test each of these updates (required)

Does this pull request close any open issues?

Screenshots (optional):

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.

@iameli so there will own API service in the mdw.livepeer.com and own in the fra.livepeer.com?


Release 1 6 0

Config updates. Not sure why https://release-1-6-0.kyverno.io/ isn't working at this point. I thought it should after I created the subdomain. But these config file changes are needed anyhow.

@JimBugwadia can you take a look when you get a chance, please? Changes made on the Netlify side and repo side. Compared config file changes in release-1-6-0 to release-1-5-0 and they appear to be correct yet 1.6.0 isn't showing in the drop-down and kyverno.io is reflecting the state of release-1-5-0.

Fixed. Netlify caching issue, I think


Check if "x is greater than y" is currently defined in ECMA-262 when x or y can be NaN

Intl.DateTimeFormat.prototype.formatRange[ToParts] currently have a step:

If x is greater than y, throw a RangeError exception.

I don't know if is greater than is currently defined in ECMA-262 when the operands can be NaN.
(An easy way to sidestep this issue, is to move the step into PartitionDateTimeRangePattern, after step 4.)
See also: https://github.com/tc39/ecma262/pull/2007

Moved this check into PartitionDateTimeRangePattern after it checks that neither x nor y are NaN or infinite.
We will need to update some test descriptions in Test262. @anba, @FrankYFTang, would SpiderMonkey or V8 need to be updated in any way?

We will need to update some test descriptions in Test262. @anba, @FrankYFTang, would SpiderMonkey or V8 need to be updated in any way?

so. is this a pure editorial change or a normative change?

This would be an editorial change.


ROX-15780: set file modification date in diagnostic bundle

Description
Currently we do not set any time in diagnostic bundle files. This result with using 1980 as a default which is confusing for users as they expect current date. This PR fixes it by setting current timestamp.
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
TODO(replace-me)
Use this space to explain how you tested your PR, or, if you didn't test it, why
you did not do so. Valid reasons include, for example, "CI is sufficient",
"No testable changes". Feel free to attach JSON snippets, curl commands,
screenshots.
In addition to reviewing your code, reviewers must also review your testing
instructions and make sure they are sufficient.

Current dependencies on/for this PR:

master

PR #5426

PR #5231   👈

This comment was auto-generated by Graphite.


bug: Content Overflow on Header and Theme Icon Redundancy

Description:
There are two primary issues observed in the current UI:
Content Overflow on Header:
The content pane (on the right side, "Course Content") is overlapping onto the header section when expanded. This creates a visual disruption as the content is not contained within its own section and instead bleeds over onto the header.
Redundant Change Theme Icon:
The "Change Theme" icon is currently placed both in the profile section and the header. Since the header already contains a theme toggle option, the one in the profile section is redundant and should be removed to avoid confusion and streamline the UI.

I have already solved this both as they are not major issues if u want i will raise pr

Hey!
I tried to fix this issue

#1274

yeah but i had already fixed this and you forgot to add overflow-y-auto in lower div . what we had to do was restrict overflow-y to only lower div as it was added on main div the content was overflowing from lower div

Oops my bad!
I don't know...this is my first pr and it seemed a easy task
Should I close my pr or like delete it or something?

I did close my PR


changed input exception to temporary red innerHTML

Should now display an error message above the searchboxes in red instead of a popup, goes away when the filter input is changed.

Updated tests to match new reset alert function


Bundle size increase concern

Hi @mrmartineau, this is more of a question/concern than a definitive issue. I've been testing including this in the production bundle and I noticed that just importing the DesignSystem increases my bundle size by 500K which is unusual.
Looking at the package.json, it seems like maybe more than just the compiled index.js is being included, and instead having a compiled index.js as the only thing that the package.json is linked to would help?
So, I was wondering if you have seen this issue as well? Or have any ideas on what might be going on?
FWIW, I was testing with this starter kit (which is what my setup is based on) incase you wanted to try it out as well: https://github.com/ctrlplusb/react-universally
Thank you!

@oyeanuj thanks for your concern, the footprint of this bundle should be very small. There maybe a bundling issue that meant it is larger than it should be, but I'm pretty sure there isn't because when I look at the cjs (dist) directory, it looks like this unminified: https://www.dropbox.com/s/trc6eahgxnrp3oz/Screenshot 2017-10-30 07.21.13.png?dl=0
There are two dependencies:

modularscale-js: 966 bytes minified (505 bytes gzipped)
object-get: 740 bytes minified (421 bytes gzipped)

Now I don't know how large your design system object is, but that might be the issue..
If there's something that I'm missing, it would be great to understand what it is so we can resolve this.

@oyeanuj are you certain that this package increased your bundle by 500k? Have you had a chance to test it?
Also, watch this space for some updates. There's an open PR at the moment, that changes the API slightly, and I will be working on some of your other suggestions soon.

@mrmartineau I do agree that is weird, hence the framing as a question. So, I've tried to reproduce it in a couple of ways:

Create React App: This failed to compile the production build as it couldn't find a pre-compiled file in the repo. Here is the reproduction (https://github.com/oyeanuj/my-errors/tree/design-system-utils) and here was the errors:

→ yarn run build
yarn run v1.3.2
$ react-scripts build
Creating an optimized production build...
Failed to compile.

Failed to minify the code from this file:

 	./node_modules/design-system-utils/src/index.js:6

Read more here: http://bit.ly/2tRViJ9

error Command failed with exit code 1.

So that might be something to fix in general?

With React-Universally: I've been able to reproduce it with a clean branch from the master of the kit. If you run yarn run build before and after this one commit, you see the weird increase in bundle size.

I'll try to dig more to isolate the cause to see if it was my setup or can be reproduced generally (hence the attempt to use CRA originally).

This is a strange one... Bundlephobia says the package is 887b
Regarding the minify error. I am using this info in the package.json, which means webpack will try to use that if it can.. so perhaps your webpack config needs amending.. I realise this isn't a solution for everyone so it would be good to fix.
"main": "./cjs/index.js",
"module": "./src/index.js",

On more investigation, it does seem to be a minification error of some sort while bundling. I imported the files in the package to use them as a local library and the error went away.
I imagine it has to do something with UglifyJS not working well with this library for some reason. If it is unable to minify and uglify, that could explain the bundle size increase. Both my config (via React-Universally) and CRA use Uglify and both are having those issues.
I think we should resolve the CRA minify issue above as per http://bit.ly/2tRViJ9 since that would help a lot more people.

Another update: So, it turns out that Webpack v3 so far uses the 0.4.6 of the UglifyJS plugin and with v4, they will start using 1.x of the UglifyJS plugin (which internally uses Uglify-ES).
So, I upgraded to v1 and the errors seem to go away, and the bundle size seems back to normal.
Closing issue to reduce noise now that we know the cause.
PS. @mrmartineau thanks for engaging in this conversation!

@oyeanuj thanks for looking into this again. I'm not sure of the best way to proceed. I think it could be due to the main/module fields, perhaps I also need a browser field as well (See https://webpack.js.org/configuration/resolve/#resolve-mainfields for more info), but even that probably wouldn't solve the issue because that bundle ought to be a UMD version.. Any ideas?