Research - What generative AI tools can be used?

The project requirements:
"Use [AI tools] for everything:  Project planning, scaffolding, coding, debugging, TDD, documentation, build scripts, docker implementations, database creation, server implementation, and anything else you can think of! "
"Trying the same thing in 3 different ways, and figuring out simple metrics to judge the efficiency of the process and quality of the outcome."
Link to AI Coder in Project portal:
AI Coder Project.pdf
We've created a spreadsheet for documenting the AI tools we've researched and will also track which ones we use: https://docs.google.com/spreadsheets/d/1xEG2II8gC8OrlHAYuLT6_YXUkAReKlAekPmJr2KiQ_Q/edit?usp=sharing

Feel free to keep adding tools to the spreadsheet as needed


Not working with react native 0.76.1

Unable to build after upgrading my project from 0.74.4 to 0.76.1

can you please try with patch:
https://github.com/SimformSolutionsPvtLtd/react-native-audio-waveform/issues/128#issue-2643374803
let us know if this is working for you or not.
We have tested with this patch in 0.76.1 with newArchEnabled=true and we are able to run app in android and iOS properly with all the features.


sliced null buffers lead to incorrect result in take kernel (and probably on other places)

Describe the bug
Sliced arrays lead to incorrect null buffers. I have found this in code where I copied the null buffer of one array to that of a new one (without offset).
I could pinpoint this bug in the take kernel, but I think it may occur in other parts as well.
In the code snippet below, you see the output of the array having two zero values where that should be null. As these are the values that should not have their validity bits set, it could probably be any other value at runtime.
I could also reproduce this with other StringArray
To Reproduce
use arrow::array::{UInt32Array, Array, LargeStringArray, PrimitiveArray, Int32Array};
use arrow::compute::take;
use arrow::datatypes::UInt32Type;

fn main() {
    let idx0 = UInt32Array::from(vec![Some(0), Some(1), Some(2), Some(3), Some(4), Some(5), Some(6), None, None]);
    let sliced_idx_arr = idx0.slice(5, 4);
    let sliced_idx = sliced_idx_arr.as_any().downcast_ref::<UInt32Array>().unwrap();

    let idx1 = UInt32Array::from(vec![Some(5), Some(6), None, None]);

    println!("indices:\n{:?}\n{:?}\n\n\n", &idx1, &sliced_idx);
    // this is true
    assert_eq!(&sliced_idx.into_iter().collect::<Vec<_>>(), &idx1.into_iter().collect::<Vec<_>>());

    let array = Int32Array::from(vec![0, 1, 2, 3, 4, 5, 6, 7, 8]);
    let taken1_arr= take(&array, &idx1, None).unwrap();
    let sliced_taken_arr = take(&array, &sliced_idx, None).unwrap();

    let taken1 = taken1_arr.as_any().downcast_ref::<Int32Array>().unwrap();
    let sliced_taken = sliced_taken_arr.as_any().downcast_ref::<Int32Array>().unwrap();

    let vec1: Vec<_> = taken1.into_iter().collect();
    let sliced_vec: Vec<_> = sliced_taken.into_iter().collect();

    println!("taken arrays:\n{:?}\n{:?}", &taken1, &sliced_taken);

    assert_eq!(vec1, sliced_vec)
}

Output
indices:
PrimitiveArray<UInt32>
[
  5,
  6,
  null,
  null,
]
PrimitiveArray<UInt32>
[
  5,
  6,
  null,
  null,
]

taken arrays:
PrimitiveArray<Int32>
[
  5,
  6,
  null,
  null,
]
PrimitiveArray<Int32>
[
  5,
  6,
  0,
  0,
]
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[Some(5), Some(6), None, None]`,
 right: `[Some(5), Some(6), Some(0), Some(0)]`', src/main.rs:29:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Expected behavior
I'd expect the sliced array to have the same result as the array that is not sliced.
Additional context
https://github.com/pola-rs/polars/issues/878

I think the problem is this [clone of the null buffer in take_indices_nulls]]1  where it should instead take a slice:
The other combinations of nullable/nonnull indices/values look ok which probably explains why this wasn't found earlier.

I think the problem is this [clone of the null buffer in take_indices_nulls]]1 where it should instead take a slice:

Good catch, I think you're right.

@jhorstmann Should there be done some arithmetic to compute the offset of the slice. Eg. bytes to bits. And what if the offset of the indices is not a multiple of 8? Should we then copy the right bits to a new buffer?

@ritchie46 there is a bit_slice method in Buffer for this usecase, if the offset is a multiple of 8 it returns a shallow slice, otherwise a new buffer with the shifted bits is constructed. And I agree this api is really easy to miss or error prone to use and we need to somehow come up with an api that doesn't require us to constantly think about bit offsets.


SPECT Y90 data

The SPECT notebook(s) contain
!pwd
!cp -r /mnt/materials/SIRF/PSMRTBP2022/Y90-NEMA-NPL/SPECT/* .
!ls .

and commented-out lines to download the data.
ideally this data-set is on Zenodo, and we add it to the download_data.sh script. In addition, the above lines rely on Linux and should  be replaced by Python lines (like in the PET notebooks).

@danieldeidda we probably have no time to fix that this week I guess. Let me know.

That data is on OSF https://osf.io/pcfb4/ which is the one used at NPL. There is a way to download data from there but at the time of the PSMR school it was not working

I've modified the notebook accordingly but keep the issue as it should be added to download_data.sh

sure, I guess it would would need to go on the SPECT data download since there isn't one. However, when running the line above will also download PET data

@danieldeidda is there a chance this will ever be split in PET and SPECT data? Or even on Zenodo? If not, we should close this issue.
As the notebook works fine, I certainly don't want to modify anything now before 3.5.

Hi, OSF was the NPL select way of sharing this. I'd rather not creating zenodo copies of the same data as this data has already been cited. If we want a separate PET and SPECT repository we possibly can create another one.

ok, if it already has a DOI, then I wouldn't duplicate it. That seems the wrong thing to do, even if it is for splitting.
So yes, if we ever got some other experimental data, then we can use that. Can still be NPL's preferred way of sharing of course.
So I'll close this issue.


Update bevy_egui dependency

bevy_egui has bumped breaking semver, so for the feature to still work with the latest version we need to bump the dependency

Fixed in #27


Add DI extensions for Azure and SCM-based clients

Now that Microsoft.Extensions namespaces are considered part of the BCL, we can simplify our story around how Azure and SCM-based clients can be added to DI service collections.
I have a sketch showing how this may work in this prototype PR: https://github.com/Azure/azure-sdk-for-net/pull/46593
Remaining work here includes:

[ ] Close on IConfiguration schema/story
[ ] Validate this approach will work for Azure clients - need to list functional scenarios supported by Microsoft.Extensions.Azure and ensure they can be implemented via new extensions
[ ] Finalize API shapes and patterns

Once we make a go/no-go decision here, we'll want to solve back-compat with Microsoft.Extensions.Azure packages and make a plan for updating docs.

Now that Microsoft.Extensions namespaces are considered part of the BCL

Somewhat tangential, but @annelo-msft could you elaborate on this please?


Bootstrap prevents 'X' element of search input boxes to clear value

Environment: Linux, Chromium latest
When a input element with type=search is used as the trigger of a dropdown menu the shadow DOM element 'X' will not work as expected, the input will not be cleared. It looks like Bootstrap is preventing click event propagation.

We nuke the × icon from search inputs in Reboot for standardizing the appearance of input types. I think we could potentially bring it back in v6, but for now, it'll stay as-is. This isn't a problem with dropdowns as far as I know, but the search input itself.


[News] iai_kinect2/libfreenect2 at ICRA 2015

Three teams at Amazon Picking Challenge on the first day used iai_kinect2/libfreenect2 for their manipulation tasks.
JSK, University of Tokyo reported overall success with less than ideal performance of pointcloud framerate being 10Hz with a GTX 780.

MIT reported flawless success with a GTX Titan X and dual Kinect v2.

Alten the Netherlands had linking issues with the binary ROS package and could not enable GPU processing. But they were able to work it through using CPU only processing.

PR2 maintainer Clearpath Robotics also sets to integrate Kinect v2 in its indigo upgrade kit for PR2. They will have two people working on contributing to the driver.

Thanks for sharing. It's really nice to see that people from all over the world are using this software. Also interesting to see how much the individual setups differ.

Uber cool! Congrats Thiemo!


Make the offer process learned (rather than rules-based)

Currently offers are determined by fixed rules based on the agent's inventory.
For truly emergent trading, we need decisions on offers to be part of the action space.

Suggested process:

Trader uses each agent's posted_offer to resolve possible trades (this process must ensure offers are still valid at this point)
As before, Traders are ensured to only trade once using the traded vec
automatic resource consumption takes place
Forager takes an action based on agent_state (which now should include posted_offer)
check posted_offer is valid, given the inventory levels of the agent (if not, set to trivial - agent must learn which offers are possible given resource levels)

Useful analysis:

episode vs. automatic posted_offer cancellation (due to incorrectly judged offer action)
episode vs. average time with offer posted
episode vs. rate of successful trades


Make a Docker container

containers are cool
#4

[x] Docker basics #5
[ ] docker compose
[x] dockerhub ci

[x] Docker basics #5
[ ] docker compose
[ ] dockerhub ci

Using docker compose gives me a weird error on WSL:
[+] Running 0/0
 ⠋ continentbot Pulling                                                                                0.1s
error getting credentials - err: docker-credential-desktop.exe resolves to executable in current directory (./docker-credential-desktop.exe), out: ``

I'll try it on my Linux machine tomorrow


Locale support for type conversions

Current behavior
As far as I can determine, all type conversions performed by Jodd BeanUtil or rather its associated converters are all with respect to the default locale.
Expected behavior
I would love to replace Apache BeanUtils with Jodd, but we support arbitrary locales for conversion of all data types. If Jodd could add this feature, we might be able to migrate.

Absolutely make sense. Will prioritize.

Hey, sorry for the long response, had a medical condition... Do you need locale support for e.g. numbers (1.000.00) and dates?

We do dates through the parsing and formatting classes Java provides (SimpleDateFormat, DateTimeFormatter), so no, we don't need that. It's about numbers, but that does include BigInteger and BigDecimal.
And thank you for even considering our request!


compat: Mycelium on Android supports sending to P2WPKH and P2SH

Reported by @andrewtoth:

Also it looks like you tested Mycelium for iOS. Their Android wallet is far more popular and better maintained, and I believe it also supports sending to P2WPKH and P2WSH addresses.

Mycelium on Android can also generate P2PKH (Legacy), P2SH (Segwit compatible), and Bech32 (Segwit native) addresses.
The matrix is out of date.

@ksedgwic Thanks for adding to the discussion. The matrix, I believe, is correct. The testing on Mycelium was on iOS. I believe the matrix information is correct for iOS.
However, based on feedback it seems Mycelium Android is significantly different and supports more features. It makes sense to have an additional entry for Mycelium on Android. This is in my queue. I will update here when it is tested.

Oh oops, I didn't realize that Mycelium's Android and iOS implementations were so different, when I updated the segwit compatibility matrix here: https://github.com/bitcoinops/bitcoinops.github.io/pull/510/commits/6657e350a2215bf8fa3c967b64a126d13c9f4d74#diff-0a1e23d669a81c0371ed7b32fdf413c8bb6ca271f3481ca66087a725814562bf
@bitschmidty: Perhaps I should have rather created a separate entry for Android there? Should we still do this: revert the original entry (but label it "iOS"), and add a separate Android entry with the information I collected?


Add example DSP yamls for V2, and update existing yamls with newer api fields

Acceptance criteria:

Add new example DSP (for simple, external data connection/obj store, etc.)
Ensure existing v1 samples still work in v2 dspo

Migrated to Jira: https://issues.redhat.com/browse/RHOAIENG-1710


tray-icon = "0.14.0" Failed to reference on win11

error: failed to select a version for gtk-sys.
... required by package rfd v0.9.0
... which satisfies dependency rfd = "^0.9" of package tauri v1.0.0
... which satisfies dependency tauri = "^1" of package linyu-client v0.0.1 (C:\Users\duwei\Desktop\linyu\linyu-client\src-tauri)
versions that meet the requirements ^0.15.1 are: 0.15.3, 0.15.1
the package gtk-sys links to the native library gtk-3, but it conflicts with a previous package which links to gtk-3 as well:
package gtk-sys v0.18.0
... which satisfies dependency ffi = "^0.18" of package gtk v0.18.1
... which satisfies dependency gtk = "^0.18" of package libappindicator v0.9.0
... which satisfies dependency libappindicator = "^0.9" of package tray-icon v0.14.0
... which satisfies dependency tray-icon = "^0.14.0" of package linyu-client v0.0.1 (C:\Users\heath\Desktop\linyu\linyu-client\src-tauri)
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the links ='gtk-sys' value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

You can't use tray-icon with Tauri v1, only in v2 (it's part of tauri v2 so no need to use this crate). This is not a bug, more like a limitation.

So how do I use it in v1

Or do I have to use v2 to use this tray event

Like I said, you can't use this library in v1. You need to migrate to v2 if Tauri's v1 tray apis don't work for you.

Ok, so is there a good way to migrate v1 to v2?

We're working on migration docs here https://beta.tauri.app/start/migrate/from-tauri-1/ (not perfect yet). If your app is new/small i'd recommend to just create a new one via create-tauri-app --beta and move your old files over.

In the v2 version, the tray icon version uses 0.13.5. How do I use 0.14.0.
@FabianLars

You can wait for us to update it, open a PR to update it, or use the tray-icon crate directly (a bit harder because of the threading requirements)

Nevermind, Tauris was already updated. In the next tauri release you'll have access to 0.14


Autodiscovery should skip printer ink

ink entity:
state_class: measurement
marker_high_level: 100
marker_low_level: 15
marker_type: ink
unit_of_measurement: %
friendly_name: EPSON ET-3760 Series Black ink

Power profile

Seems fine for me for know. Thanks.


Add a functional F# API

A la Keras functional API

I’d be very much interested by this!

I'm closing this as I'm not currently in favour of having a separate API for F# in this repository - espcially as from the F# perspective a lot of the rationale for doing work here is to provide a suitable backend for DiffSharp.
We should just make the API equally usable from C# and F# for now.


ESP8266 Exception when trying to connect (over TLS to AWS)

I have a large sketch that worked a couple days ago, and now it doesnt after adding FS.h to it. All I did was add the ability to read/write SPIFFS, and as far as i know, it had nothing to do with my mqtt functions.
my ask is that based on the trace below, can someone help me figure out where to look for the issue?
here is the function that throws the error:
void reconnect() {   // Loop until we're reconnected   while (!mqttClient.connected()) {     Serial1.printf("[MAIN] Free heap: %d bytes\n", ESP.getFreeHeap());     if (DEBUG) {       Serial1.print("Attempting MQTT connection...");     }     Serial1.printf("[MAIN] Free heap: %d bytes\n", ESP.getFreeHeap());     // Attempt to connect     ESP.wdtDisable();     if (mqttClient.connect(mqttClientId)) {       ESP.wdtEnable(5000);       if (DEBUG) {                 Serial1.println("connected");       }  // Once connected, publish an announcement...       mqttClient.publish("dev/controller", "connected");       // ... and resubscribe       mqttClient.subscribe("dev/controller/#");     } else {       if (DEBUG) {         Serial1.print("failed, rc=");         Serial1.print(mqttClient.state());         Serial1.println(" try again in 5 seconds");       }       delay(500);     }   } }
specifically the line if(mqttCLient.connect(mqttCLientId)){
exception29.txt

You're probably running out of memory there.


Masking for affine registration

Hello,
I've tried to perform an affine registration with initially setting a fixed binary mask. The registration works, however it does not differ from the result without setting a fixed mask prior to the same registration.
I did masking before and for a bspline registration it worked perfectly fine. For the affine registration, the whole image is transformed despite setting a mask before. Can anyone relate to this problem? Thank you in advance!
All the best

An affine transform is global to the image. It maps every voxel in the
fixed image to a point in the moving image space - regardless if you use a
mask or not.  The masking is used during registration, not during the final
resampling.
So using a mask/not using masking is not bound to give you a different
result for a global transformation. It depends on to what extent the mask
exterior would have led to a different result. For example, if the exterior
is different between moving and fixed, then the result would likely be
better with masking. If the mask-exterior is similar, eg zeros, then it is
likely masking would have limited to no effect (for a global transform).
Soren
On Wed, Dec 16, 2020 at 10:07 AM gazile17<EMAIL_ADDRESS>wrote:

Hello,
I've tried to perform an affine registration with initially setting a
fixed binary mask. The registration works, however it does not differ from
the result without setting a fixed mask prior to the same registration.
I did masking before and for a bspline registration it worked perfectly
fine. For the affine registration, the whole image is transformed despite
setting a mask before. Can anyone relate to this problem? Thank you in
advance!
All the best
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/SuperElastix/SimpleElastix/issues/410, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHBDDNFFLXHW254KTC6HCTSVB2FLANCNFSM4U5UQOJA
.

Alright, thank you very much for the quick and clear explanation!


Fix SQL queries for inline datasource with null values

Fixes a bug because of which some SQL queries cannot be parsed using druid convention. Specifically, these queries translate to an inline datasource and have some null values. Calcite internally uses NULL as SQL type for these literals and that is not supported by the druid. In this PR, we are marking such null literals as the STRING column type.
This PR has:

[ ] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[ ] been tested in a test Druid cluster.

@LakshSingla thank you for your review. Please let me know if you feel there is a code change required.

Minor doubt, thanks for addressing the comments. LGTM.


emit more useful logs from failed imagetest helm installs

standard helm emits some pretty useless logs when it errors out:
https://github.com/chainguard-images/images-private/actions/runs/7845026704/job/21411213191#step:15:89
INSTALLATION FAILED: client rate limiter Wait returned an error: context deadline exceeded

This wraps the install in a block and captures the error, and prints a whole bunch more diagnostics similar to the k8s-diag error we use.
these will get captured by the provider and emitted as diagnostics.details, which in turn will show up in the CI logs as the <detail></detail>

I'm going to force merge this given the analysis above ^


Make AssetAction::Ignore not copy assets to imported_assets

Objective
Lets say I have the following .meta file:
(
    meta_format_version: "1.0",
    asset: Ignore,
)

When a file is inside the assets directory and processing is enabled, the processor will copy the file into imported_assets although it should be ignored and therefore not copied.
Solution

I added a simple check that does not copy the assets if the AssetAction is Ignore.

Migration Guide

The public ProcessResult enum now has a ProcessResult::Ignore variant that must be handled.

@alice-i-cecile is this migration guide ok?:D I don't know what else to write

Updated your PR description to amend the migration guide :) It's unlikely to be particularly useful to folks, but this adding enum variants is a semver breaking change in Rust.


Image presentation crashes the application after window resize

If my window gets resized, I get Assertion failed: vkRes != VK_ERROR_OUT_OF_DATE_KHR due to OUT_OF_DATE swapchain image acquire result not being handled. This logic doesn't allow to properly launch an application on tiling window managers at all (without forcing floating mode at launch).
I think it's better to get rid of VK_KHR_Swapchain usage completely because Mantle doesn't use swapchain logic at all and only uses presentation images, while leaving handling window events (like resizing), which leads to swapchain invalidation to the client. I have already fixed this in ddef5b656d8cd83732f6fd94636c2e719bb8cb2d.

Was mostly fixed with 5f603d599. Closing.


Can't override dashboard form

Issue Summary
I want to customize product model. I got trouble at overriding ProductForm for dashboard. My app had hooked up from INSTALLED_APPS(checked by putting assert False in models.py), but for something reason, get_class (oscar/apps/dashboard/catalogue/views.py:24) doesn't want load local classes from .forms. Thanks.
Steps to Reproduce
./manage.py oscar_fork_app catalogue apps/
./manage.py oscar_fork_app catalogue_dashboard apps/
Replace oscar apps in INSTALLED_APPS
Put assert False in apps/dashboard/catalogue/forms.py
Technical details

Python version: 3.7.5
Django version: 2.2.9
Oscar version: 2.0.4

Looks like you haven't forked the core dashboard app - see https://github.com/django-oscar/django-oscar/issues/3232 . I'm closing as this isn't a bug/feature request. If you need further assistance please ask for help on Stack Overflow using the django-oscar tag, or joining our slack workspace and asking a question there. Thanks.


Requester can sign their own cert (when also listed as a signer)

It looks like an AuthorizedRequester can sign their own cert if they are also an AuthorizedSigner for the same environment. It seems intuitive for me that some requesters would also be signers for the same environment, and that requesters should not be able to sign a request for their own pubkey by default.

You've described the current behavior correctly. Is your expectation just that as a default a key cannot be used to sign a request if the signing key is == requesting key? And make an exception for if the config only has 1 key in it?
The last time I ran this in production I actually worked around this via lots of keys. I get this isn't appropriate for everyone. But I, as an individual, had signing keys for prod, staging and dev and user/requester keys for prod, staging and dev. This seems unintuitive and kind of lame actually, however, once you realize that ssh-agent can only store one certificate per public key you may end up wanting lots of keys.
Anyway, I'm interested in hearing what behavior you'd like to see.

Thanks for the quick reply!
I’d like to see disallowing the same key fingerprint being allowed to request and sign. For example, if you’re using some form of hardware token for your SSHing, you don’t have the luxury of multiple keys per environment.

On Aug 28, 2019, at 6:51 PM, Bob Van Zant<EMAIL_ADDRESS>wrote:
You've described the current behavior correctly. Is your expectation just that as a default a key cannot be used to sign a request if the signing key is == requesting key? And make an exception for if the config only has 1 key in it?
The last time I ran this in production I actually worked around this via lots of keys. I get this isn't appropriate for everyone. But I, as an individual, had signing keys for prod, staging and dev and user/requester keys for prod, staging and dev. This seems unintuitive and kind of lame actually, however, once you realize that ssh-agent can only store one certificate per public key you may end up wanting lots of keys.
Anyway, I'm interested in hearing what behavior you'd like to see.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Do you anticipate making a fix for this?
Wondering if I should try and block off some time.

Here you go! https://github.com/cloudtools/ssh-cert-authority/pull/42

Fixed with pr #42 . I updated the README and pushed the version to 2.0.0 since this may break backwards compatibility for folks. Thank you!


Add LICENSE file

We need to add a license file to legally allow companies to use the package.

resolved in 1d1fdd2655edcea4b59bb0ce8b9a73302d8e63cc


Pretty Print for Requests

Hello,
I have enabled elastic.SetTraceLog(log.New(os.Stdout, "", log.LstdFlags)) and have set the Pretty as true
	searchResult, err := client.Search().
		Index("twitter"). // search in index "twitter"
		Query(termQuery). // specify the query
		From(0).Size(10). // take documents 0-9
		Pretty(true).     // pretty print request and response JSON
		Do()

This prints the response in the pretty format. However the request is not pretty. I’m using "gopkg.in/olivere/elastic.v3"

@sundarv85 You're right. Using Pretty(true) simply makes Elastic pass pretty=true as a URL query parameter to Elasticsearch. So in the end, Elasticsearch will pretty print the response and I simply output the pretty-printed response. Elastic doesn't do anything to pretty print anything, neither request nor response. So it first of all is a documentation issue.
I will have to dig into net/http/httputil to see if there's anything I can do to get the HTTP request pretty printed.

I think I'll not be doing pretty-printing for the request side.


⚠ Bofrost.de has degraded performance

In aab40b8, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 924 ms

Resolved: Bofrost.de performance has improved in 84082ae.


fix(PolyhedronFacetVertices): Set InlinedVector size to 3.

:tada: This PR is included in version 2.5.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


[SPARK-10570][Core] Add version info to json api

Add a new api endpoint /api/v1/version to retrieve various version info. This PR only adds support for finding the current spark version, however other version info such as jvm or scala versions can easily be added.

cc @JoshRosen (you mentioned this in the JIRA comment)

Can one of the admins verify this patch?

ok to test. This change looks OK to me. @JoshRosen

Test build #50507 has started for PR 10760 at commit 65d74b0.

Test build #50507 has finished for PR 10760 at commit 65d74b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50507/
Test PASSed.

cc @JoshRosen

retest this please

Test build #54390 has started for PR 10760 at commit 65d74b0.

Test build #54390 has finished for PR 10760 at commit 65d74b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54390/
Test PASSed.


New commits is updated but the lgtm label is not canceled

Maybe a bug @wzshiming
New commits is updated but the lgtm label is not canceled
Originally posted by @wuyingjun-lucky in https://github.com/clusterpedia-io/clusterpedia/issues/387#issuecomment-1255729726

/assign


Can't add relative timelocks (nSequence field) to transaction using the playground

You can't add relative timelocks (nSequence field) to transaction using the playground. Because of this it's not possible to make transactions with the Licho's Mecenas# example smart contract in the playground.

We'll move this issue to the Next.js version of the playground.


After the ~2 minute timeout for background services, the notification is removed

General way to replicate with demo-session:

Play some music.
Navigate to home screen.
Pause music through the notification.
After ~2 minutes, onDestroy() of the PlaybackService is called and the notification is removed.

After that, the user has no way to resume playback besides going back into the app, the notification is removed and app no longer responds to play command from bluetooth headset.
Seeing how other apps like YouTube Music persist the notification and continue to respond to media buttons, is there a general recommended way to mimic this behavior with Media3?

It's a design decision to take the service off the foreground when the player is paused. When in this state and the user does not resume playback within a given duration of time, the system will kill the idle service. This is working as intended and helps the system be in a healthy state instead having several services running silently in the background.

After that, the user has no way to resume playback besides going back into the app,

Your app can implement playback resumption. Then the system places a notification that allows the user to press the play button on that notification to restart your service and get back to the app. WIth playback resumption a user can also use the BT headset to resume.
This is how Androids designs this for not having various idle services running and giving the user a way to resume playback after an app has terminated.

general recommended way to mimic this behavior with Media3?

You can override MediaSessionService.updateNotification() and handle the entire foreground management on your own.

It's a design decision to take the service off the foreground when the player is paused. When in this state and the user does not resume playback within a given duration of time, the system will stop the idle service. This is working as intended and helps the system be in a healthy state instead having several services running silently in the background.

Couldn't it post the notification without keeping the service alive at all? As you said we could override MediaSessionService.updateNotification(), but I feel like keeping the notification would be what most media playing apps would want by default. At least, that's what most of the apps I use do, and mine did before changing to Media3

Couldn't it post the notification without keeping the service alive at all?

Only when the library knows that the app supports restarting without crashing, which is the case when an app ops-in to playback resumption.
Generally when not running, it's unclear what to do with controls like next/previous, seek and the like when the service is not running. Because, when the service is started when not running, it needs to be started in the foreground which requires the service to start playback. That's only the case for the play button.
So from the library point of view, an app can implement playback resumption to get exactly that, a notification with just a play button. That's the system way and it has been designed like this for a reason.
With Media3 it's two simple steps:

Opt-in to playback resumption by adding androidx.media3.session.MediaButtonReceiver to the manifest.
Implement Callback.onPlaybackResumption().

That's for the reason that the library knows that playback resumption is supported by the app. If that's not the case and some actor tries to do playback resumption, you land on #167. No one wants to be there.
Media3 doesn't post such a notification at the end because there is an API for this that removes uncertainties. Media3 offers the System UI playback resumption with a play button which is easy to implement and designed to work well together with system restrictions regarding starting a foreground service.

that's what most of the apps I use do, and mine did before changing to Media3

And, an app like yours is ultimately free to ignore all I've said here and what the library intents to do.
An app can post it's own notification at any time. You can for instance do this in onDestroy() of your service (you could even reuse DefaultMediaNotificationProvider. Just to store the last notification provided by it and re-post that when the app terminates. As mentioned, it's unclear how you'd respond to the buttons different than play without crashing, but technically this is possible). I think I don't need to explain in detail though that we don't fancy giving too much support in case an app posts such a notification and then runs into problems with service restrictions imposed by the system. :)

So from the library point of view, an app can implement playback resumption to get exactly that, a notification with just a play button.

Not sure if this is intended, but this notification doesn't show with the two steps completed with my MediaSessionService. It does show when I use MediaLibraryService instead. Thought I would add this as a note. I read a lot of issue comments and did not find this mentioned anywhere.

@DimitarStoyanoff Thanks for your comment!
That's correct. System UI uses onGetRoot() and onGetChildren() of the MediaLibraryService (or equivalent methods in the legacy MediaBrowserServiceCompat) to ask whether to post the resumption notification. The MediaSessionService does not have these domain methods and hence does not support this notification.
I will make this clear in the documentation.

Thank you, @marcbaechinger, for your quick responses and for all the work you put into this project.

Since I still have this problem even after implementing playback resumption with a MediaSessionService, I'm wondering why the library requires a MediaLibraryService and not just a MediaSessionService to display a notification with a play button, since playback resumption callback already requires a playlist to play for the play button.
Is it possible to make this happen the notification also appear with MediaSessionService, and if not what is the minimum amount needed to implement in MediaLibraryService to make this happen.
Also why does MediaSessionService with playback resumption implemented show the notification with play button on API 30, but not API 34? (haven't check the exact API range where it works / doesn't work). For example here's a screenshot of the working notification on API 30:

Thanks for your report!

Since I still have this problem even after implementing playback resumption

As mentioned in a comment above, the notification is removed when the system destroys the service after about 2 minutes of not being in the foreground which is when the player is paused. This is working as intended.

Is it possible to make this happen the notification also appear with MediaSessionService
and if not what is the minimum amount needed to implement in MediaLibraryService to make this happen.

Under the hood, MediaLibraryService offers the legacy API of the MediaBrowserService that System UI uses to detect whether an app wants to show a resumption notification.
From your report I assume that you are advertising your MediaSessionService in the manifest as a MediaSessionService and MediaBrowserService:
    <service
        android:name=".PlaybackService"
        android:foregroundServiceType="mediaPlayback"
        android:exported="true">
      <intent-filter>
        <action android:name="androidx.media3.session.MediaSessionService"/>
        <action android:name="android.media.browse.MediaBrowserService"/>
      </intent-filter>
    </service>

I just noticed that the demo app is doing this as well. While this apparently works, I think it confusing to do so in the demo app. I'll send a CL to change this from MediaSessionService to MediaLibraryService.
Regarding your question, I guess that you are having a service that extends from MediaSessionService. Then as above the manifest declares the MediaSessionService and MediaBrowserService. Please correct me if I'm wrong.
I think you kind of discovered a loop-hole in the implementation or, in a more positive way, you discovered the feature that we actually can offer a resumption notification for a MediaSessionService as well. Thank you! :)
Lemme explain: System UI picks the MediaBrowserSerrvice declaration in your manifest and Media3 returns your MediaSessionService. System UI then tries the legacy service contract to figure out whether we want a resumption notification. Because playback resumption is enabled, Media3 responds to System UI under the hood, even if the service doesn't have the MediaLibrarySession.Callback implemented properly (which is correct for a MediaSessionService). Media3 does this here and here.  Like you mention above, there is actually no need to implement  MediaLibrarySession.Callback.onGetLibraryRoot() and MediaLibrarySession.Callback.onGetChildren(). Media3 responds to the calls from System UI that gets all the information about what to display in the resumption notification from the metadata of MediaSessionCompat which is provided for the library and the session service in the same way.
While this is unexpected and confusing, I think we should keep this behaviour because it can easily be removed from a MediaSessionService by not declaring MediaBrowserService in the manifest. Unlike MediaLibraryService where removing the declaration would mean the for instance Android Auto strops working, for a MediaSessionService this works to prevent System UI doing a resumption notification by still allowing resumption over for instance Bluetooth.
So for this purpose, I think you can either remove MediaBrowserService from the manifest if you wish the MediaSessionService doesn't have the System UI resumption notification. I understand though you would actually want this to happen, so as you discovered this works.

MediaSessionService with playback resumption implemented show the notification with play button on API 30, but not API 34?

Guess the first part is explained above. I'm not sure why it doesn`t work on 34 without further information.
Can you repro this with the demo app?
I think since API 33 there is need for some additional permission regarding notifications that may have an impact. I assume you got this already as else you would possibly see some warnings from the system. The demo app has the here in the manifest and here in the MainActivivty as well as here.
If you can repro that the notification is not shows with the notification permission in place, can you please make a bug report from your app that is made just after you started playback (aka when the first playback notification is displayed by System UI). You can also filter logcat for MediaResumeListener (when looking at all logs, not only mine). The interesting bit is whether you see
2023-12-29 12:54:23.247   877-877   MediaResumeListener     com.android.systemui                 D  Checking for service component for androidx.media3.demo.session
2023-12-29 12:54:23.252   877-958   MediaResumeListener     com.android.systemui                 D  Testing if we can connect to ComponentInfo{androidx.media3.demo.session/androidx.media3.demo.session.PlaybackService}
2023-12-29 12:54:23.762   877-958   MediaResumeListener     com.android.systemui                 E  Cannot resume with ComponentInfo{androidx.media3.demo.session/androidx.media3.demo.session.PlaybackService}

or
2023-12-29 12:52:14.017   877-877   MediaResumeListener     com.android.systemui                 D  Checking for service component for androidx.media3.demo.session
2023-12-29 12:52:14.018   877-958   MediaResumeListener     com.android.systemui                 D  Testing if we can connect to ComponentInfo{androidx.media3.demo.session/androidx.media3.demo.session.PlaybackService}
2023-12-29 12:52:14.518   877-958   MediaResumeListener     com.android.systemui                 D  Connected to ComponentInfo{androidx.media3.demo.session/androidx.media3.demo.session.PlaybackService}
2023-12-29 12:52:14.578   877-958   MediaResumeListener     com.android.systemui                 D  Can get resumable media for 0 from ComponentInfo{androidx.media3.demo.session/androidx.media3.demo.session.PlaybackService}


Handle multiple relations in getEndpoint

versions:
"js-data": "2.8.2",
"js-data-angular": "3.1.0",

Hi there! I have a model that looks like this:
{
  name: 'TelephoneNumber',
  endpoint: 'telephonenumber',
  relations: {
    belongsTo: {
      Account: {
        parent: true,
        localKey: 'accountId',
      },
      Partition: {
        parent: true,
        localKey: 'partitionId',
      }
    }
  },
}

As you can see a TelephoneNumber belongs to both an Account and Partition (incidentally, an Account belongs to a Partition). Odd, I know. When I call getEndpoint like so:
const item = {accountId: 'foo', partitionId: 'bar', ...otherStuff}
const url = httpAdapter.getEndpoint(TelephoneNumberDefinition, item)

Here this returns: partition/bar/telephonenumber
The issue is that right now the defineResource method doesn't support having a definition with multiple parent, parentKey, and parentId properties (here. So when that runs the account is set as the parent, and then the partition overrides it (though this is not 100% deterministic because iteration on objects is not 100% consistent across browsers).
What I would expect is if I provide an accountId it should factor the account in generating the URL, so it would become: partition/bar/account/foo/telephonenumber
Problem I can see from this is with me providing both an accountId and a partitionId, you don't know which one to use... So, I think maybe the solution could be that I don't provide a partitionId and you look that up on the account which would happen naturally...
To sum up, what I think needs to happen is this code needs to account for multiple parents. Then all code using that needs to account for a potential of an array of parents (in particular, the getEndpoint method). That would resolve my issue...
I expect that this would be a breaking change for people depending on and using the parent with multiple belongsTo.

We found a workaround that works pretty well. Technically I still think this is a bug because the implication is that you can have multiple parents but the reality is that it's not well supported. However, because we found a workaround, we no longer have an urgent use case and you can close this issue...
If anyone comes across this and is wondering what we ended up doing, we pretty much just hardcoded the URLs needed for the Partition relationship. There weren't many places we needed that anyway.

cc @TheMcMurder. This getting fixed might help you out at some point.


[Enhancement]: Make MsSqlBuilder.WithDatabase public

Problem
MsSqlBuilder.WithDatabase is private for any reason, so you are really limited in what you can overwrite. The only currently available public method is WithPassword. There is private WithUsername as well, it's not that important, but can also be made public.
Solution
Make this method(s) public
Benefit
You can use a user-defined database name. It allows us to use some features that are not available for "system databases", such as change tracking on tables.
Alternatives
Implementing custom MsSqlBuilder that is the same as original but with public modifier on WithDatabase method.
Would you like to help contributing this enhancement?
Yes

Its not configurable in the official docker image  used. That is probably why they are private and not configurable.

@DanielHabenicht yeah, connection string modification sounds great

Could someone help me comprehend and elucidate how the use case looks like when someone chooses to override the database name (Initial Catalog) and attempts to connect to the non-existent database using a basic SQL connection? This results in the following error:

Microsoft.Data.SqlClient.SqlException
Cannot open database "foo" requested by the login. The login failed.
Login failed for user 'sa'.

Which part is responsible for creating the database? How do developers know and be aware that they are responsible for database creation?

@HofmeisterAn  I have run into problems with calling .Migrate() to setup the database, which for some reason requires to set the database into SINGLE_USER mode which does not seem to be supported for master databases.
So I need a database which is not master, thus using Inital Catalog=

Simply changing the database value in the connection string without actually creating it will work in your specific example. This is due to the Entity Framework's capability to automatically generate the database if it doesn't exist. However, this approach will not work for other configurations, potentially leading developers to encounter issues. You can work around this utilizing e.g. the SQL connection string builder (while other cannot):
var connectionString = new SqlConnectionStringBuilder(_sqlContainer._msSqlContainer.GetConnectionString());
connectionString.InitialCatalog = Guid.NewGuid().ToString("D");

In relation to this matter, I proposed several alternatives. One of these involves extending the MSSQL module's functionality to enable the execution of custom SQL scripts during startup, similar to PostgreSQL. This mechanism can then be harnessed to create a custom database or user, providing a versatile solution that accommodates various configurations.
It's important to note that my intention is to avoid providing an incorrect or invalid connection string for a module (or container) and relying on the assumption that the developer is using Entity Framework, which automatically generates the database upon initialization.
Furthermore, if the capabilities of the master database are limited (initializing the MSSQL container), I believe it would be better to provide this feedback to the upstream repository (Microsoft) rather than introducing vague behavior to Testcontainers.

Similar for me, I use migrations and the change tracking feature on a database is required, but it cannot be set up on so-called "system databases", including master. EF Core will take care of database creation, so I really need only to set the Initial Catalog parameter.

Of course, I can use self-written logic to modify the connection string received from Testcontainers.MsSql
And I agree with providing a connection string for a non-existent database is confusing and misleading

Introducing the support of this feature on Microsoft side seems to be the best option so I close the issue here. Will think about reaching image maintainers. Thanks

I would like to point out that if someone is interested in contributing and enhancing the capabilities of the Testcontainers MSSQL module to execute custom SQL scripts during startup (by overriding the entrypoint, etc.) and utilizing this feature to pre-create a database and user, I am completely open to that. However, I personally do not consider it a high priority.

@HofmeisterAn I will take a look if I can help with that


This functionality seems Broken.

I have followed this step and sounds like its not functioning.
It gives 401- Unauthorized even after successfully login using connect-pnp Online.
I have opened similar issue for PNP community as well.
https://github.com/pnp/PnP-PowerShell/issues/3028
Thanks!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 34a78fe1-1c93-1014-d773-29085ec97b9e
Version Independent ID: c1f7a5e1-ab36-29ad-aa29-dbe72efa6b16
Content: Get-PnPContext (SharePointPnPPowerShell)
Content Source: sharepoint/sharepoint-ps/sharepoint-pnp/Get-PnPContext.md
Service: sharepoint-powershell
GitHub Login: @pamgreen-msft
Microsoft Alias: pamgreen

How are you?
Even though the PnP documentation articles are on this repository, I need to kindly redirect you to open PnP cmdlets issues here, in that repository, where a member of the PnP community will be able to correct the article with your feedback.
The documentation for PnP cmdlets is then sync from that repo into this one.
Thank you!

@get-itips Thank you very much for the contribution and sharing this explanation. @AnkitSDesai Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Theme attributes don't work with MaterialComponents themes

Library Version:
3.0.0-rc4
Describe the Bug:
Theme attribute md_corner_radius doesn't work when app theme is inherited from Theme.MaterialComponents.*
To Reproduce:

Create an app theme <style name="Theme.App" parent="Theme.MaterialComponents.Light"></style>
Add <item name="md_corner_radius">16dp</item> to the theme.

Expected Behavior:
Expected behavior is that corner radius is applied to dialogs. Instead, it doesn't work at all. If corner radius is set in code via cornerRadius() method applied to MaterialDialog instance, it does work.
I guess other themes attributes also don't work.

Hmmm. The parent theme should not make a difference.

The sample project uses MaterialComponents so I suspect something else is wrong on your end.

I've tried to use this theme  Theme.MaterialComponents.Light.Bridge many times and it's fine


[Feature Request] Improve the UI on tablets

Please improve the UI on large screens (tablets), the way the original FreeOTP works on those is much better.
On the original FreeOTP i can access 24 tokens per screen without scrolling:

On FreeOTP+ the same screensize lets me access six tokens:

Could this be fixed?

I think it is because of moving to RecyclerView. We can consider using a Grid LayoutManger for tablet.

The issue is fixed in https://github.com/helloworld1/FreeOTPPlus/commit/f5a50c92159bb7f97286015aea02c39ab7558133

I don't have a tablet wide enough to test. Feel free to let me know if you can test a debug version (It won't override your current tokens).

sure, i can test... :)

You can download the apk here
https://drive.google.com/file/d/1dCIJ1Xr4FYxsabJhCznUPJenNcfp3xxf/view?usp=sharing
The debug build is a separate app with orange icon. It won't mess up your existing one. Feel free to test functionalities. Note this version has major internal updates. Please create separate issues for other bugs.

tested, the new layout is SO much better.

Thanks for getting back. I will work on the importing sorting problem.
The ordering is currently only on rows (up and down motion). I need to find ways to support left and right motion.

@lemmy04 Could you try a newer APK that should fix the import order and horizontal drag of the tokens.
https://drive.google.com/file/d/1MjCoTqPu-3SVx1h0Q8v2ezV_hu_mbw4T/view?usp=sharing

finally got around to testing that build, works fine.

Thanks for reporting back!

so when will this improved UI be available in a released version...?

I plan to update the release version to F-Droid in the weekend. Thanks for reminding!

The version 2.4 is now released (on Github). Will need to wait a few days for F-droid to pick up.
Google play release will be also a bit later.

so far neither google nor f-droid have picked up the new version. Is that actually normal for how it works with android apps?

@lemmy04 F-Droid had some trouble automatically picking up the new version, but this is fixed now. It already tried to build the new version but failed


DIY tool doesnt find my Sonoff mini

Jumper mode set, ssid sonoffDiy created with password set. The tool does not see the device, no matter what I try, including public/private mode
Any suggestions?

Suggest you check the computer network card is a dual network card

Done that, all disabled except wifi
Checked the sonoffDiy SSID and i dont see the mini connected..

Is the mini version 3.3.0?After operating the network needs to reopen the software

Yes, its 3.3
Sorry, that didnt make any sense. After operating the network needs to reopen the software>
The issue so far isnt the software, i cant see the sonoff mini connect to the sonoffDiy SSID.

What is the wifi status light on the device?

Single flash or double flash？

Its a single flash

This must be because wifi is not connected. It is suggested to check whether wifi is too far away from the device, and then check whether the SSID and password of wifi are wrong

its about 2 meters away. Yep, its not connected as indicated by the unifi wireless output
This is my wifi setup

Now its flashing 3 times

I suggest you go to our after-sale service. It should be the damaged equipment

I doubt it. Ive got 4, they all do the same thing.

You can test it to see if it doesn't work with sonoffDiy

It looks from the screenshot that you have a Ubiquiti Access Point. I found that I could not get pairing to work via my Ubiquiti AP if the sonoffDiy SSID was radiating on both the 2G and 5G WiFi bands. I got pairing to work by disabling the 5G band on the sonoffDiy SSID. Setup the sonoffDiy SSID as normal/defaults and then go to the WLANS config section of the AP and disable the 5G band for the sonoffDiy SSID. Oddly the Mini works fine on dual band SSIDs once you have it setup. It was just eWeLink pairing that had an issue with a dual band SSID, in my case.

Same fail situation for my 2 Sonoff minis. I have tried the iphone hotspot (cannot disable 5 ghz) and and dedicate Unifi 2.4 Ghz sonoffDiy SSID, both of them are failed to find my Sonoff minis when launched the DIY tool. Has upgraded ewelink firmware to 3.3.0 and confirmed both win10 and minis connected to sonoffDiy (blue indicator blinks 2 times/sec & sonoffDiy client list in the Unifi). Any recommendation?

Hi ferry133
Here is a screenshot of the 5G band disabled as an override in the WLAN setting section of the UniFi Access Point(AP). Is your AP set the same?

Hi, davecoutts, thanks for help.
Not exactly the same with your setting. I created another WLAN1 with sonoffDiy SSID and enable it only on 2.4GHz.

I agree. That should give the same result as my UniFi setup.
Maybe as an elimination step try setting up a duel 2G+5G network in the default WLAN and then disable 5G in the WLAN section, as worked in my case.
I'm clutching at straws but there may be a an expected configuration quirk with the Ubiquiti or Sonoff Mini software.
My Mini would not tether to the sonoffDiy SSID(2G WiFi only) on my Android phone when other devices would. So the Mini in pre-DIY upgrade state appears to be very particular about the WiFi network it will attach to.
Below are my Ubiquiti component versions and a screenshot of the sonoffDiy network settings in case its of any help.
AP Model:  UniFi AP-AC-Pro
AP Version:  <IP_ADDRESS>25
AP Board Revision:  26
UniFi Controller:  <IP_ADDRESS>

I will also mention that I could not get the Sonoff DIY tool to run. It failed to run with dependency errors. I did have success with the Tasmota manual-flash method on Linux.

Thanks a lot. I’d better try the manual way.

It's work for the Tasmota manual-flash method on Mac, too.
Many thanks for davecoutts' sharing.

I figured that the firmware on the sonoff needs to be the latest version (3.3 +) before the diy mode password and ssid work. did the itead ewelink pairing, it prompted to upgrade firmware, upgraded and then tried the steps for DIY mode and they worked.

I had upgraded ewelink 3.3 before trying DIY tool but not work, in my case, and I'm not alone. You may check others' case including Sonoff R3. I finally solved it by manual-flash way.

this way not working for new sonoff mini v 1.2 based on 2019.06.25 it comes loaded with 3.3.0 and tool does not recognize sonoff
i can see sonoff connected to network and i can ping it


fix: change password UI/UX issues

Fixes: #500

Once the user types the password again and now is correct, it shouldn't have an error, but currently, the user needs to leave the input to see that it is correct. Cleaning the error once start to type again it solves this issue.

Once the user types the password again and now is correct, it shouldn't have an error, but currently, the user needs to leave the input to see that it is correct. Cleaning the error once start to type again it solves this issue.

on this way the user need to blur again to know if error is really fixed why don't just validate on change instead of blur then?

Avoid the error to be shown as the user type.

Avoid the error to be shown as the user type.

@luizstacio then what we need is don't show the error straight when typing in first interaction.
1- fiels are empty, user start typing. At this point we will show any error only after user blur the "Confirm Password" field
2- user finished fill "Confirm Password" field, and blurred.
3- passwords don't match, now errors are showing
4- after this point, we check on "change"
does that make sense?
also other option is splitting this in 2 steps (trust wallet app does in this way if you wanna check and test)
complexity would be less and more isolated space to show error

User type the "Password" field

popup password strength gets shown and validate the pass

User clicks in "Next"
User type the "Confirm Password" field

at this point could use logic mentioned above also. first check only on blur, then on change if has errors.
show "Back" button to move back and change password in step 1 again
show "Confirm" button that is disabled until passwords are matching


Fix user based api calls and forms.

Fix login and sign-up pages.
Add login nevbar option when no user is logged in, change to logout when the user has logged in.
Fix user based api calls with mongoDB.
Refactor component and page function calls for more consistency across the app.
Create useStates for currentUserEmail and currentUser to keep track of current user after logging in and to be able to access the user information from anywhere within the app.

Some funky things happening when I pull down and test and try sign up or login. Console shows endless errors of user unauthorized and doesn't show token in applications. Maybe I'm missing something?

Thanks, seeing token in the cookies. Would like to have the sign up be able to login automatically instead of sing up  then login. Will make a feature development ticket for it.

Thanks, seeing token in the cookies. Would like to have the sign up be able to login automatically instead of sing up then login. Will make a feature development ticket for it.

Ya thats fine, this is the simpler approach so to save time I did it this way but we can change it later no problem.


[DO NOT PUSH] test: CiderCollective.Cider.Nightly version 1.1.471

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.0 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Testing before committing to CircleCI


ConvertTo-Json unexpectedly serializes AutomationValue.Null as an empty object, not null

PowerShell commands that output nothing output the special [System.Management.Automation.Internal.AutomationNull]::Value singleton, which typically behaves like $null.
Unexpectedly, ConvertTo-Json doesn't treat it as $null, but treats it as an empty object, which is undesired.
Steps to reproduce
# Note: `. {}` yields the [System.Management.Automation.Internal.AutomationNull]::Value
# singleton, not $null.
[pscustomobject] @{ foo = . {}; bar = $null } | ConvertTo-Json

Expected behavior
{
  "foo": null,
  "bar": null
}

Actual behavior
{
  "foo": {},
  "bar": null
}

Environment data
PowerShell Core 6.2.0-rc.1
Windows PowerShell v5.1

cc @markekraus

Also . {} is equal (-eq) to $null but doesn't serialize to null.

heh.. that's a fun one... we may need to create a custom JSON handler for AutomationNull or something.


MacOS Ventura

It's definitely broken on macOS, even the latest Ventura. It can install fine after reboots, however it is impossible to have it select halmak - it just hangs and reverts back to the system language.
Will go with a hardware based solution and pick it up again then.

I've the same issue.

Halmak.bundle.zip
My customised version that works on Ventura

Halmak.bundle.zip
My customised version that works on Ventura

Thanks a lot! It's works.

Also here is working copy without reversed digits https://github.com/kaievns/halmak/pull/26


Cant Get It to work

All I get our these errors
Value of type 'AuthDataResult' has no member 'uid'
Value of type 'StorageMetadata' has no member 'downloadURL'

@Redthered fixed this on latest release


[Feature]: Add Support for Azure Container Instance Deployment

Is there an existing issue for this?

[X] I have searched the existing issues

Summary
Add Deploment support for Azure Container Instances.
Why should this be worked on?
Self-hosted does not support this configuration. An Azure VM is overkill when this can be implemented as a PAAS service

I am currently hung up getting things running on Azure Container Instance. The main issue appears to be with the Azure Storage Driver for Docker. File permissions don't play nicely with mongo key. See the following thread from Discord for logs and error messages (https://discordapp.com/channels/725602949748752515/1013810425546149889).

Hey @jrotello, thank you for your interest in Appsmith, and the offer to help. I'm not very familiar with Azure myself so I'll call upon your offer to help on this. Would it be possible to put up some steps to do to deploy Appsmith on Azure Container Service (AZS)? I'd like to try with those exact steps and, if I reproduce the problem, see what we can do to resolve this.
I think the problem with working on this issue right now is less of a lack of interest, and more of a lack of the right know-how. Steps to reproduce this would be gold. Thank you!

Hi @sharat87,
Thanks for the quick response! Here are some basic instructions that should get you up to the point where it is failing. Let me know if you have any questions.

Ensure you have an active Azure subscription
docker login azure and follow the prompts
docker context create aci <context name> (ex: docker context create aci appsmithctx)
docker context use appsmithctx
Download the attached compose file and update <storage_account_name> with the name of the storage account to be created in your Azure subscription.

NOTE: You can probably safely update the resource reservations to 1 CPU and 1G of memory for this testing.
docker-compose.zip

docker compose --file docker-compose.aci.appsmith.yml up

All the necessary Azure resources should be created for you automatically. They will be created in the resource group you assigned when creating your Docker context in step 3.
To tear down the resources use docker compose --file docker-compose.aci.appsmith.yml down This we remove the Azure Container Instance resources, bur leave the storage account around. If you want to remove the storage account as well, then you will also need to run docker volume rm <storage_account_name>/stacks

I think the underlying problem as it stands right now is that the azure_file Docker Storage Driver uses SMB under the hood and setting linux file permissions on it doesn't seem to do anything. I haven't tested it, but I'm wondering if there might be a similar issue when running on Windows Subsystem for Linux. See /appsmith-stacks/logs/mongodb/db.log for the permission error.
Please let me know if I can be of further help!

this would be just great

@hockeyrjc  @gguille077 @jrotello  The documentation for installing Appsmith on Azure Container Instance had been published here.


where Exists (select)

add please function exists
Dialect

[x] postgres
[x] mysql
[x] sqlite3

Additional context
Add any other context or screenshots about the feature request here.

"exists" function can be workarounded as:
goqu.L("EXISTS ?", subquery)

I've tried to implement it using goqu.Func("EXISTS", subquery). But it fails on SQLite because of redundant parenthesis.
Such SQL:
SELECT EXISTS((SELECT `id` FROM `admins` WHERE (`id` > 20)));

is not valid for SQLite, although works for MySQL.

I've tried to solve this in the PR above.  Looking forward for your feedback :)


Clarify which resource group needs to be entered when approving the connection for the Shared Private Link

In this section - https://docs.microsoft.com/en-us/azure/search/search-manage-powershell#create-a-shared-private-link-resource
It should be clarified as to which resource group is being specified in each command. There are scenarios where network resources are deployed in separate resource groups and application resources are deployed in separate resource groups.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 170b92a2-bba8-0245-6130-47b5fc685e4a
Version Independent ID: 70c81982-b37d-1e6b-6dc0-9eb588424e7a
Content: PowerShell scripts using Az.Search module - Azure Cognitive Search
Content Source: articles/search/search-manage-powershell.md
Service: cognitive-search
GitHub Login: @HeidiSteen
Microsoft Alias: heidist

Hello @blueelvis
I have assigned the issue to the content author for further review and update the doc as appropriate.

Sorry for the delayed response.  The issue was fixed last week, but we forgot to close the issue. All of the resource group names are now qualified to say which resource they apply to.
#please-close


[Feature] Fix VOL nesting [OSF-5507]

Purpose
VOL view on the contributors page nests very poorly. This flattens the nesting.
Changes
Nodes no longer nest in the list of associated nodes. Set to a static 100px width, trims overflow and adds ellipsis.
Before:

After:

Side effects
None known.
Ticket
https://openscience.atlassian.net/browse/OSF-5507
[#OSF-5507]
QA Considerations

Should show a flat list
Long titles should be truncated.
The modal that used to create a VOL should still indent properly (related code)

Added before/after.


remove ambiguous mention of 'maybe dag-cbor'

It's not clear right now whether dag-cbor is definitely allowed by the spec. IMO this patch makes the REAMDE more clear for now. Personally I would discourage using dag-cbor, but it's a weakly held opinion atm. Nothing precludes adding it back in when/if it can be done without the 'maybe' caveat.

Actually it supports dag-cbor, and uses it :)
Why would you discourage?

maybe just remove the (maybe) if it's out-of-date and it's no longer a maybe? unless it means something else?

Yeah I would remove the maybe but I want to hear why @gobengo thinks it shouldn't be there at all!

Why would you discourage?

I like CBOR but dag-cbor bans parts of it that imo are worth using


Opsec audit script

Description
A script for testing aspects of a deployment's security.
Tested
On centos, debian.
Backwards compatibility
Yes

Is this still relevant or should this be closed?


Send endpoints via API findings

Hello,
I used script python to open Findings using file.csv
The problem is the Endpoint is not attached in Finding.
When I open the Finding who was created by script and click "edit" I see the Endpoint in " Systems / Endpoints".
So, When I Click "Endpoints" -> "Show Endpoints", the "Active Verified Findings" is 0.
The data I send is:
                      data = {
                              "test": teste,
                              "thread_id": 0,
                              "found_by": None,
                              "push_to_jira": False,
                              "title": "TEST TITLE",
                              "date": date,
                              "sla_start_date": date,
                              "cwe": 0,
                              "cve": cve,
                              "cvssv3": None,
                              "cvssv3_score": 0,
                              "severity": "MEDIUM",
                              "description": "TEST DESCRIPTION",
                              "mitigation": "TEST MITIGATION",
                              "impact": "TEST IMPACT",
                              "steps_to_reproduce": None,
                              "severity_justification": None,
                              "references": None,
                              "endpoints": 20,
                              "active": True,
                              "verified": True,
                              "false_p": False,
                              "duplicate": False,
                              "out_of_scope": False,
                              "risk_accepted": False,
                              "under_review": False,
                              "under_defect_review": False,
                              "is_mitigated": False,
                              "numerical_severity": 0,
                              "line": 0,
                              "static_finding": False,
                              "dynamic_finding": False,
                              "nb_occurences": 0,
                              "publish_date": date,
                              "review_requested_by": None,
                              "defect_review_requested_by": None,
                              "sonarqube_issue": None,
                              "reviewers": None
                        }

Any suggestions ?

Double check the format as declared inhttp://defectdojo-dev:8000/api/v2/oa3/swagger-ui/?docExpansion=none#/findings/findings_create
endpoints should be an array I think, but I would actually expect an error if the format is wrong

I see here.
I need to access the Finding e click update.... after ...works.

I am currently having this issues with similar data, but I did put my endpoints in a list. Like @pgabriel10 said, you can force the endpoint to appear after entering editing the issue and clicking finish without any changes. The endpoints are highlighted in the edit menu as if they are selected.

Pushing the issue to Jira also seems to make the endpoints appear as well.


How to use it in Xamarin.Android

I have just watched a thread on stackoverflow, in the Answers section a person recommended this so I was curious to know how am I going to reference in my Xamarin Applications?

Quick research says that you need to use Binding of Java Library.

Download .aar file here.
Check this manual and try follow its steps.

As alternative you can try to use this plugin.

I managed to add the references and put the suggested xml element in the activity. But then suddenly this runtime exception occured that it is not going to get inflated.

Android.Views.InflateException: Binary XML file line #20: Binary XML file line #20: Error inflating class com.cleveroad.loopbar.widget.loopbarview

We don't have expertise in Xamarin development, so you need to find the way how to run this (or any other) lib on your own. It would be great if you post your solution here for anyone who also interested in using this lib in Xamarin.

Sure, I will post the solution if I am able to find one.

Hi @jamsshhayd, are you able to bind the LoopBar into your Xamarin applications? Thanks

@rolloqueritchie I had to move towards another thing because of a short deadline and I had used bottombar to achieve this feature.


can you get the coordinates directly?

Hi guys. According to your paper, maybe you guys had some difficulty getting the coordinates directly. I'm trying to modify your code for the docking pose prediction model to make it a generative model and also directly output coordinates instead of the distance matrix.
I'm thinking of mainly utilizing the coordinate prediction part from the conformation model. and also add the element prediction model from the conformation model as well.  For the coordinate prediction part, i would also use the attention bias with the pocket atoms to update the masked ligand atoms. Have you guys tried this method?
there are some concerns here
1.the coordinate head seems equivariant, does that ensure the model is too? If it isn't equivariant, should we augment the data(reflection, translation etc.)
2.concerning the noise for the masked parts, i'm assuming that too much noise would make the model hard to predict. I wonder what you think of this part as well.
3. Maybe we can add an additional equivariant layer after getting the distances, that also takes the atom features
Thank you as always

for your questions.

in the model backbone, it uses the invariant representation, i.e. pair-wise distance, so you don't need data augmentation.
Actually, the noise is quite easy to predict, as long as it is not too noisy.
It is worthy trying.

Actually, we tried to predict the coordinates directly, but the training is very unstable. We think the random initialization of protein and ligand caused this issue. It is hard to decide on an appropriate position and orientation for the initialization.


FATAL Failed to read file: , 314, , errno: , 2, , msg: , No such file or directory

Hi, I am running the readme_example.py using one A6000 GPUs. Everything follows the default setting.
However, when generating token, I get:
Model create:  99%|██████████████████████████████████████████▍| 4514/4578 [00:16<00:00, 2387.87it/s]2024-11-07 00:27:39.907 FATAL Failed to read file: , 314, , errno: , 2, , msg: , No such file or directory
terminate called after throwing an instance of 'std::runtime_error'
what():  Logged a FATAL error
Aborted (core dumped)
Looks like the files cannot be read.
I can see the offload files in the offload path.

I am not able to reproduce the issue, have you tried to clean up the offloading dir and try again?


Allow WebP images if supported by WP

Feature Description
Now that initial WebP support has landed in WP trunk (https://core.trac.wordpress.org/changeset/50810), it would be good to add support in the editor as well.
In \Google\Web_Stories\Traits\Types::get_allowed_mime_types() and \Google\Web_Stories\Traits\Types::get_allowed_image_mime_types() we should add support the WebP mime type if it's supported by the site (either via core or a plugin).
So we should not be doing $wp_version comparisons or anything, but feature detection.
Alternatives Considered

Additional Context

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

WebP images are visible in the editor media library
WebP images can be uploaded in the editor
WebP images can be uploaded as publisher logos

Implementation Brief

This should be extremely simple.
If we add 'image/webp' to Types::get_allowed_mime_types, then if WordPress doesn't support it, this this line will filter it out.
https://github.com/google/web-stories-wp/blob/515eb8694efa49a3947bb0e431fcce3eb8bd807a/includes/Traits/Types.php#L119
Should webp supported by Types::get_allowed_image_mime_types()

Should webp supported by Types::get_allowed_image_mime_types()

WebP should be allowed for things like publisher logos, so yes :-)


Feat:Add a reservation section above feedback section

Describe the feature
Add a reservation section above feedback section so that user can book a table in that hotel,canteen,etc
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

pls assign me these issue under GSSOC'24 @VanshKing30


Feat:Want to Write Backend Of FeedBack Page Of Canteens

Describe the feature
Want to Write Below Component Backend
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Ok do it


FIFOQueue: dequeue many operation very slow?

When training a relatively simple model (1-layer LSTM, 256 units) my Titan X GPU keeps spiking from 0% to 30% GPU utilization. Conclusion: somewhere in the pipeline there is a bottleneck which limits the GPU to be processing the training batches continuously. I use a FIFOQueue to which examples are being fed in one or more separate threads:
queue = tf.FIFOQueue(
     capacity=self.config.queue_capacity,
     dtypes=[tf.float32, tf.float32],
     shapes=[[30, 49, 512], [30]],
     name="FIFOQueue"
)

For the training operation I use queue.dequeue_many to get examples from the queue. As you can see the batch size is 64 examples. So in the end the input tensor is 64x30x49x512 of type tf.float32:
# Model inputs, either use the queue (training) or feed_dict (evaluation)
inputs, targets = queue.dequeue_many(64)

To find out why my code is running "slow" (i.e. spiking GPU allocation and no temperature increase) I use the Timeline object (see here) to measure execution times of individual operations. The results displayed below show the measurements for one training iteration at which point the queue was filled with more than 1000 examples. I have included screenshots for both GPU and CPU-only runs (forced with export CUDA_VISIBLE_DEVICES=-1.
What strikes me from these results is that it takes a really long time to dequeue examples from the FIFOQueue. What is happening here...something wrong or is the dequeuing operation just very slow? Overall the dequeuing operation and sending the data to the GPU takes up half of the time of a training iteration. No wonder that the GPU utilization is spiking. Any help is welcome optimizing my training pipeline! As I understand correctly the examples are all queued in RAM, is there also a way to queue them ahead on GPU memory so when they are needed they do not have to be moved CPU => GPU?
This is tested on TensorFlow v9.0 build from sources about 1.5 week ago.
GPU running on Titan X

CPU running on Xeon CPU E5-2640

Thanks for including the timelines!  Nice to see people using that ;-)
It's hard to say very much about what's happening here without seeing the whole program.  I don't believe that dequeueing 192MB is inherently that expensive.  (@mrry - does this sound right?)
There are a number of things which might be happening.  Obviously DequeueMany is a blocking operation and the time it appears to take in the timeline will include all the time from when it was issued until when the final element was enqueued into the FIFO.  This timeline will not show the other, potentially concurrent steps which are happening.
Are you enqueueing all 64 required elements before calling the step which dequeues them?
Can you contrive a test case where you put 64 elements into the queue in advance, and just run a step which does the dequeue?

@prb12: At the very least, the DequeueMany op is going to do 64 separate Eigen slice assignments to build up the output batch. If the capacity is less than 64, it will also have the copies out of any corresponding EnqueueMany op arguments accounted to it. The most likely cause is that it's blocking, however, so it would be great if @tomrunia could do the contrived test that you're suggesting. Otherwise using a sampling profiler might reveal a suprisingly costly copy.

Oke, I wrote a simple script (see below) in which the threads simple enqueue batches of np.ones([64, 30, 49, 512], dtype=np.float32) to the FIFOQueue. The main loop simple dequeues 64 examples and performs a simple tf.square of the input tensor. Before the main loop starts I make sure that the queue is filled with a sufficient number of examples. Again, the dequeue operation takes a "long" time to finish as you can see in the timelines below.
GPU running on Titan X

CPU running on Xeon E5-2640

import time
import numpy as np
import threading

import tensorflow as tf
from tensorflow.python.client import timeline

def test_queue():

    feature_input = tf.placeholder(tf.float32, shape=[None, 30, 49, 512], name="queue_inputs")
    target_intput = tf.placeholder(tf.float32, shape=[None, 30], name="queue_targets")

    queue = tf.FIFOQueue(
        capacity=1000,
        dtypes=[tf.float32, tf.float32],
        shapes=[[30, 49, 512], [30]],
        name="FIFOQueue"
    )

    # Enqueue and dequeue operations
    enqueue_op = queue.enqueue_many([feature_input, target_intput])
    queue_inputs, queue_targets = queue.dequeue_many(64)

    queue_size = queue.size()

    # Very simple training operator, dequeue examples from the FIFOQueue and
    # square the matrix elementwise. Just for testing of course.
    square_op = tf.square(queue_inputs)

    # Coordinator for threads
    coord = tf.train.Coordinator()

    # Initialize the TensorFlow session
    gpu_options = tf.GPUOptions(
        per_process_gpu_memory_fraction=0.75,
    )

    sess = tf.Session(config=tf.ConfigProto(
        gpu_options=gpu_options,
        log_device_placement=False,
        allow_soft_placement=False
    ))

    def load_and_enqueue():
        while True:
            # Just feed random stuff to the queue
            #features = np.random.rand(64, 30, 49, 512)
            features = np.ones([64, 30, 49, 512], dtype=np.float32)
            targets  = np.ones([64, 30], dtype=np.float32)

            # Feed example to Tensorflow placeholder
            feed_dict = {
                feature_input: features,
                target_intput: targets
            }

            # Push all the training examples to the queue
            sess.run(enqueue_op, feed_dict=feed_dict)

            if coord.should_stop():
                break

    # Start the threads
    num_threads = 4
    for i in range(num_threads):
        t = threading.Thread(target=load_and_enqueue)
        t.setDaemon(True)
        t.start()

    # Make sure the queueu is filled with some examples (n = 500)
    num_samples_in_queue = 0
    while num_samples_in_queue < 500:
        num_samples_in_queue = sess.run(queue_size)
        print("Initializing queue, current size = %i" % num_samples_in_queue)
        time.sleep(1)

    # Initialize the session
    init = tf.initialize_all_variables()
    sess.run(init)

    run_options  = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()

    for step in range(1000):

        print("Step = %i, QueueSize = %i" % (step, sess.run(queue_size)))

        if step == 10:
            # Perform a step with saving the results as timeline object
            result = sess.run(square_op, options=run_options, run_metadata=run_metadata)

            # Create the Timeline object, and write it to a json
            tl = timeline.Timeline(run_metadata.step_stats)
            ctf = tl.generate_chrome_trace_format()
            with open('timeline.json', 'w') as f:
                print("writing to timeline.json")
                f.write(ctf)

        else:
            # Perform a step without saving the results
            result = sess.run(square_op)

    # Ask for the threads to stop
    coord.request_stop()
    sess.close()

if __name__ == "__main__":
    test_queue()

@tomrunia Thanks for the clear and self-contained repro code. It really helps us to have good quality reports for performance issues like this (and I especially like having the timelines!)
We'll take a look into this.

I spent some time looking into what's going on here...
As I mentioned earlier, your input batch of  [64, 30, 49, 512] * tf.float32 equates to 192MB...
First, I established a baseline cost for memcpying a tensor of the size you are feeding.  A native C++ program calling memcpy in a tight loop between buffers of this size takes around 22ms per iteration (single threaded).  The TensorFlow enqueue/dequeue operations actually use the Eigen library to copy out slices of the tensors and this is likely to be less efficient than a flat memcpy since it needs to handle general shapes of n-D arrays.   Theoretically Eigen might be able to do this in parallel, but doesn't appear to be in this case (from looking at the CPU usage).
Because you are feeding in numpy arrays, and retrieving a large result, you incur a copy in each direction within the session.Run() call,  here and here for TensorFlow 0.9.
Note - this code path is slightly different in various versions of TensorFlow.  Also, if you ever move to the distributed runtime these 192MB tensors will get serialized as ProtoBufs and this is very expensive.
Also, as @mmry says, some of the CPU time related to enqueue operations is most likely being accounted to the DequeueMany op due to this code - once the queue is full, enqueue ops are blocked and wait in a list.  They can then get executed the next time a dequeue succeeds.
Note that a queue with max size of 1000 elements is about 3GB, so depending on your machine config you may be causing a lot of virtual memory (and allocator) pressure.  This isn't happening on my machine, but you may want to check by running htop or vmstat while executing your program.  A smaller queue may be more sensible.
In general, you may be better off using one of the TensorFlow input ops which reads image data directly from the file sytem, and then do any preprocessing as part of the graph.  (as opposed to preprocessing in Python and feeding in the raw tensor data).
For reference, I've attached a pprof profile of your code (with a few relatively insignificant changes).  You can see that the bulk of the cycles are spent in TF_Run_wrapper_helper (doing memcpys of the numpy arrays), and under QueueBase::FlushUnlocked doing both the enqueue and dequeue copy ops via Eigen (one of which ends up turning into a memcpy)
Hope this helps ....
Paul

Hi Paul, thanks a lot for your incredibly detailed analysis of what's happening here. This give me much more insight in what's is happening under the hood. I will rethink about the data processing (input) pipeline and see if I can optimize it here and there and decrease the queue size. Great work with the analysis!

I was facing similar issues, copying large amounts of data from numpy to TF. I moved all the data loading and preprocessing into the graph (using https://www.tensorflow.org/versions/r0.9/how_tos/reading_data/index.html) and jpeg_decode, resize_images etc functionalities, and it sped up the training significantly.

I've noticed memcpy's being mentioned in this and other issues related to performance. I'm guessing memcpy is used to avoid thread-safety issues, in particular to support operations that take references to tensors such as assign.
Has any thought been given to introducing Copy-On-Write functionality to tensors within the graph? Numpy handles this but obviously without the tricky thread safety issue.
I ask because it seems wasteful in terms of I/O and memory to spend a memcpy feeding a numpy array onto a FIFOQueue and another memcpy (within a graph) copying the resultant tensor off the FIFOQueue. Ideally it would be nice to use FIFOQueue as a a pure synchronisation construct ala. go channels, python Queues rather a sychronised buffer. Clearly this needs to happen in cases where tensors cross device boundaries (GPU's/Network interfaces) but might there be scope for performance gains here?
I also notice that memcpy's happen within slicing operations. Could this be replaced with a Eigen::TensorMap (homologous to a numpy view) for e.g.?

I think @prb12 's really detailed analysis answers the question. There is always a tradeoff in using convenient and general operations for data manipulation as well as using numpy source data. We are continually working to improve performance. Automatic fusion of ops as alluded in the original paper's future work and issue #164 will provide generality and good performance.
Closing for now.

@prb12 Thanks for your detailed analysis! I am facing similar issues (timeline attached). I'm using tensorflow read operations throughout, so I believe my problem is not the numpy<->TF data copy. However your comment on virtual memory/allocator pressure seems related to my problem. Here's a screenshot from htop, and also the vmstat output
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0 9744848 1865340 2888112 45542040    1    8   118    50    0    0 21 18 61  0  0

It seems my virtual memory use is 160G, do you think that might be the problem? I am not really familiar with these system level details, so I would greatly appreciate if you could elaborate on this problem. Thank you!

I have an additional observation that might help us debug this issue. The slowdown for me happens when my cache memory gets full. I tried clearing the cache using:
$ sync && sudo echo 3 > /proc/sys/vm/drop_caches

The training then runs perfectly until the cache gets ~3/4 full again (as seen by the yellow bars on htop), and then it gets slow. So far my solution is to clear cache and restart from the last snapshot. Hoping there's a better way to fix this!

@rohitgirdhar clearing cache works for me, when i was training inception v3.

@rohitgirdhar I'm not 100% sure I want to add to an closed issue, commenting on something which may not even be related to the original problem, but...
From the htop output and your drop_caches observation it sounds to me like there may be some memory pressure caused by virtual address space fragmentation and high system buffer cache churn (reading large training datasets from the file system).  Can you see if your program is causing lots of page faults?  e.g. like this
Internally at google we almost always use the TCMalloc memory allocator which appears to be much better for TensorFlow workloads (and most others at google!).  When running the open source TensorFlow build via a standard Python binary the only way to use TCMalloc would be to inject a different heap implementation into Python using LD_PRELOAD, e.g.
LD_PRELOAD="/usr/lib/libtcmalloc.so" python myprogram.py
It would be very useful if you could try running with TCMalloc and see if it makes any difference.

The relevant issue is https://github.com/tensorflow/models/issues/170

I think TCMalloc was the solution for me, as I no longer need to clear the cache to maintain the same training speed. Thanks @prb12 !

Is there any progress on improving Numpy -> GPU throughput with Tensorflow?
I have a similar benchmark to @tomrunia of passing a O(100MB) tensor from Numpy to GPU and returning a small slice of the tensor. This would ideally be bottlenecked by PCI-e bandwidth, but runs at ~1/10th of that rate. Additionally, I see very poor scaling when attempting to run this tasks over multiple GPUs (on a single node), while theoretically I should benefit from the additional PCI-e bandwidth. (all stats from Tensorflow 0.9 built from source).
@prb12's description is very useful, but I'd like to understand more of what's going on when passing through feed_dict, enqueuing, dequeuing.
My understanding:
When a Numpy tensor is passed through a feed_dict it is copied into a host side Eigen tensor.
When a tensor is enqueued, it is added to a queue that lives on the host.
When a tensor is dequeued on the GPU, a cudaMemcpy copies the tensor from host to device. This should run near PCI-e bandwidth.
Questions about my understanding:
Is enqueueing a tensor done by reference or with a full copy (and memory allocation)?
Are the memcpy's on feed_dict passing (and enqueue if there's a memcpy there) performed by a single thread for the full Tensorflow session? Even if the Session.run calls come from multiple Python threads?
Are there plans to fix this, such as a queue that is resident on the GPU?

BTW, I tried TCMalloc on AlexNet with 1-4 Pascal-grade GPUs (Titan X, P100). The queues are doing the threading for the CPU-side JPEG decoding, and since GPUs have very high throughput, threading of the CPU-side code is critical here. TCMalloc does speed things up by 20% for 1-4 threads (inter/intra threads, threads associated with custom queues, etc.), but there's a break-even point at 8 threads, and beyond that TCMalloc does more harm than benefit. At 20 threads (e,g, on a 20-core Intel Xeon E7-8870 v4), perf. is actually 40% down by using TCMalloc in such a heavily threaded environment. So, I doubt that TCMalloc is a universally beneficial solution,

TCMalloc also seems to suffer from page faults, but it deals with them better for some reason. You can prominently see page fault activity in the kernel using perf top -p <PID> -g. Jemalloc gets completely horked on this when training on large datasets like ImageNet, though. 50+% drop in performance, and never recovers on its own. This is really bad user experience. I would wager most TF users don't have the skills to diagnose or fix this. I'd much prefer something as straightforward as a data loader would work out of the box.

As mentioned by @prb12 in the comment dated June 27, 2016,  based on my Cuda profiles using nvprof, I also see that that host to device transfers are not pinned in Tensor Flow v 1.2.1. Any ideas on when this will be fixed?

it's probably better off opening a separate issue to address the "not pinned" problem and link to relevant info, so that it could get triaged properly

@rohitgirdhar ,could you paste your code about function _read_image_from_disk()? I have verify that parse_example is much faster than parse_single_example.


fix: add missing property for some secrets in mapping

Signed-off-by: ankitm123<EMAIL_ADDRESS>

/cc @rawlingsj


Proof harness for aws_byte_buf_cat with a fixed number of arguments

Description of changes:

Added proof harness for aws_byte_buf_cat function using 3 buffers
Added pre and postconditions to aws_byte_buf_cat

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@feliperodri @tautschnig @karkhaz @angelhof @danielsn @markrtuttle could you review the proof harnesses?


Firmware upgrade fails from 3530 to 5370

Hello,
I´m trying to update the firmware of the devkit to 0.5370_perf. I can see my device connected via adb devices but when I click in FastbootUpgrade_v6_5.bat like the instructions said the device become disconnected (red lights) and it fails. I also follow the instructions from here to solve but it fails too at adb reboot bootloader step with same exception.
This is the error I get when I execute FastbootUpgrade_v6_5.bat
"Check adb device..."
"Device is not attached(X)"
"Fastboot upgrade fail(X)"
Press any key to continue . . .
Docker PS also not connecting after first attempt of update, details below of docker ps command.
DevKitFirmware>adb shell
/ # docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
/ #
Few things tried:

Charged battery to 90+ and tried with no luck
Hard reboot by pressing for 12sec(from this recommendation). Device is identified by adb devices but fails with same exception when try to update.
Tried recovery but it also fails with same exception.

As suggested here , I tried all solutions recommended except trying from MAC OS(I do not have MAC OS but tried from 3 different windows 10 devices with no luck).
Can you please help?  Thanks!

Update: I tried with my PI 4 linux and it worked after lot of issues. Major issues found are listed below and request to correct the linux version .sh scripts.

Upgrade bash is not installing when the device battery is 100%. Screenshot for reference
Script is not checking if the device/OS has adb and fastboot installed. It would be good if we can generate some warning message with steps to install both adb and fastboot which are required for this installation.
We may need to increase the initialize time as 100 * 4 is not helpful. Few times it took more time and verification bash failed.

Also, would be interested on what was the issue with windows version of script. Thanks!

@raaznaren Please allow us some time, we are looking into this issue.

@raaznaren Hi, I would suggest instead of upgrading it in this way i.e. try to install the firmware from here-https://store.altek.com.tw/qualcomm/downloads/Azure-IoT-Starter-Kit and run it.
For me it is working fine. Try to do a factory reset if possible.

@v-prsasa - As I mentioned, I already resolved using linux variant OS. My overall experience was too bad with AI Dev kit as I'd spent almost 4-5 hours to set it up and running and now having issues with pushing models.
I may not waste anymore time on this. Please go ahead and close this issue. Thanks!