Memory exhaustion issues when deploying a large site

I'm trying to deploy a pretty large (35gb) Gatsby site for the first time to S3 via gatsby-plugin-s3 deploy with default settings and getting numerous memory exhaustion errors such as:
# Fatal error in , line 0
# API fatal error handler returned after process out of memory on the background thread

And:
class FastBuffer extends Uint8Array {}
^

RangeError: Array buffer allocation failed
    at new ArrayBuffer (<anonymous>)
    at new Uint8Array (<anonymous>)
    at new FastBuffer (internal/buffer.js:788:1)
    at createUnsafeBuffer (buffer.js:111:12)
    at allocate (buffer.js:322:10)
    at Function.allocUnsafe (buffer.js:285:10)
    at allocNewPool (internal/fs/streams.js:36:19)
    at ReadStream._read (internal/fs/streams.js:141:5)
    at ReadStream.Readable.read (_stream_readable.js:457:10)
    at ManagedUpload.fillStream (G:\dev\dunedinsound-gatsby\node_modules\aws-sdk\lib\s3\managed_upload.js:420:25)

And:
 Error: write ECONNRESET

  - stream_base_commons.js:67 WriteWrap.onWriteComplete [as oncomplete]
    internal/stream_base_commons.js:67:16

And (possibly unrelated):
 RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

  - s3.js:585 Request.extractError
    [dunedinsound-gatsby]/[aws-sdk]/lib/services/s3.js:585:35

  - sequential_executor.js:106 Request.callListeners
    [dunedinsound-gatsby]/[aws-sdk]/lib/sequential_executor.js:106:20

  - sequential_executor.js:78 Request.emit
    [dunedinsound-gatsby]/[aws-sdk]/lib/sequential_executor.js:78:10

  - request.js:683 Request.emit
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:683:14

  - request.js:22 Request.transition
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:22:10

  - state_machine.js:14 AcceptorStateMachine.runTo
    [dunedinsound-gatsby]/[aws-sdk]/lib/state_machine.js:14:12

  - state_machine.js:26
    [dunedinsound-gatsby]/[aws-sdk]/lib/state_machine.js:26:10

  - request.js:38 Request.<anonymous>
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:38:9

  - request.js:685 Request.<anonymous>
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:685:12

  - sequential_executor.js:116 Request.callListeners
    [dunedinsound-gatsby]/[aws-sdk]/lib/sequential_executor.js:116:18

This is on Windows 10 with Node 11 64-bit and 16gb of memory.

Not entirely sure what's going on here but we should probably just throttle the amount of ManagedUpload's it creates simultaneously with a queueing system.
The current implementation is to create one for each file the fs can find, which probably isn't that good of a practice.

@FraserThompson I noticed you have your project set up over at https://github.com/FraserThompson/dunedinsound.com/tree/gatsby
Can it be used as a reproduction for the issue?

@jariz possibly not super easily  because I don't check in all of the media and those seem to be what causes the problem.... If you put a subfolder in one of the content/gigs folders and filled it with lots of big images or MP3s it might reproduce the problem but it could cause other errors, I think the site is too complex to be used as a simple test case.
I started with https://github.com/gatsbyjs/gatsby-starter-blog so possibly just loading up one of those content/blog folders with large files like MP3s or photos might reproduce it?

The parallelLimit function from the async library looks like a fairly nice way to limit concurrency.

I'm taking a stab at fixing this myself. I've got it limiting concurrency with an async queue, however this still doesn't seem to fix the issue, it just makes it happen slower. I still eventually run out of memory and then it crashes with this crazy stacktrace:

  ptr1=000001FA8EA7D3D1
    ptr2=0000000000000000
    ptr3=0000000000000000
    ptr4=0000000000000000
    failure_message_object=000000C82F6F6980

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF6E5E83916]
Security context: 0x02389b91a299 <JSObject>
    1: _read [000001BC120CE669] [G:\dev\gatsby-plugin-s3\node_modules\klaw\src\index.js:~27] [pc=000000FE5807D65E](this=0x038f8
4628cc1 <Walker map = 00000149DD3B53D9>)
    2: arguments adaptor frame: 1->0
    3: read [000001FFE70308C9] [_stream_readable.js:~377] [pc=000000FE58004C43](this=0x038f84628cc1 <Walker map = 00000149DD3B5
3D9>,0)
    4: maybeReadMore_(aka maybeReadMore_) [000001BC12082B61] [_stream_readable.js:~572] [pc=000000FE5827BAE7](this=0x030807f804
d1 <undefined>,0x038f84628cc1 <Walker map = 00000149DD3B53D9>,0x038f84628d41 <ReadableState map = 0000015BEC7B00B9>)
    5: processTicksAndRejections [0000036F705043E1] [internal/process/task_queues.js:~61] [pc=000000FE582226CA](this=0x036f7050
2581 <process map = 00000381C69B5C99>)
    6: InternalFrame [pc: 00007FF6E5DEA7F1]
    7: EntryFrame [pc: 00007FF6E5DEA3DC]

==== Details ================================================

[0]: ExitFrame [pc: 00007FF6E5E83916]
[1]: _read [000001BC120CE669] [G:\dev\gatsby-plugin-s3\node_modules\klaw\src\index.js:~27] [pc=000000FE5807D65E](this=0x038f846
28cc1 <Walker map = 00000149DD3B53D9>) {
// optimized frame
--------- s o u r c e   c o d e ---------
function () {\x0a  if (this.paths.length === 0) return this.push(null)\x0a  var self = this\x0a  var pathItem = this.paths[this
.options.queueMethod]()\x0a\x0a  var statFunction = this.options.preserveSymlinks ? self.fs.lstat : self.fs.stat\x0a\x0a  statF
unction(pathItem, function (err, stats) {\x0a    var item = { path: pathItem, s...

-----------------------------------------
}
[2]: arguments adaptor frame: 1->0 {
  // actual arguments
  [00] : 16  // not passed to callee
}

[3]: read [000001FFE70308C9] [_stream_readable.js:~377] [pc=000000FE58004C43](this=0x038f84628cc1 <Walker map = 00000149DD3B53D
9>,0) {
// optimized frame
--------- s o u r c e   c o d e ---------
function (n) {\x0a  debug('read', n);\x0a  n = parseInt(n, 10);\x0a  const state = this._readableState;\x0a  const nOrig = n;\x
0a\x0a  if (n !== 0)\x0a    state.emittedReadable = false;\x0a\x0a  // If we're doing read(0) to trigger a readable event, but
we\x0a  // already have a bunch of data in the buffer, then just trigger\x0a  // the 're...

-----------------------------------------
}
[4]: maybeReadMore_(aka maybeReadMore_) [000001BC12082B61] [_stream_readable.js:~572] [pc=000000FE5827BAE7](this=0x030807f804d1
 <undefined>,0x038f84628cc1 <Walker map = 00000149DD3B53D9>,0x038f84628d41 <ReadableState map = 0000015BEC7B00B9>) {
// optimized frame
--------- s o u r c e   c o d e ---------
function maybeReadMore_(stream, state) {\x0a  // Attempt to read more data if we should.\x0a  //\x0a  // The conditions for rea
ding more data are (one of):\x0a  // - Not enough data buffered (state.length < state.highWaterMark). The loop\x0a  //   is res
ponsible for filling the buffer with enough data if such data\x0a  //   is available. I...

-----------------------------------------
}
[5]: processTicksAndRejections [0000036F705043E1] [internal/process/task_queues.js:~61] [pc=000000FE582226CA](this=0x036f705025
81 <process map = 00000381C69B5C99>) {
// optimized frame
--------- s o u r c e   c o d e ---------
function processTicksAndRejections() {\x0a  let tock;\x0a  do {\x0a    while (tock = queue.shift()) {\x0a      const asyncId =
tock[async_id_symbol];\x0a      emitBefore(asyncId, tock[trigger_async_id_symbol]);\x0a      // emitDestroy() places the async_
id_symbol into an asynchronous queue\x0a      // that calls the destroy callback in the future. It's ...

-----------------------------------------
}
[6]: InternalFrame [pc: 00007FF6E5DEA7F1]
[7]: EntryFrame [pc: 00007FF6E5DEA3DC]
=====================

So perhaps there's something else going on? I also noticed that it looks like the file is buffered twice, once to make the hash and then again for the upload, which doesn't seem optimal...

@FraserThompson Good point. We should probably just pass the buffer var to ManagedUpload instead of creating a new read stream.
Can you give that a try?
Regardless, still not sure what's causing the memory leak here, at no point are we loading the entire file into memory or something.

It's possible to pipe a read stream directly into a crypto Hash object, but I think it means you also have to read the finished hash via a stream.
Instead of:
const buffer = await readFile(path);
const tag = `"${createHash('md5').update(buffer).digest('hex')}"`;

We can do something like: (untested)
const stream = fs.createReadStream(path);
const hashStream = stream.pipe(createHash('md5'));
const finishedHash = await streamToPromise(hashStream); // Uses function from here: https://www.npmjs.com/package/stream-to-promise
const tag = `"${finishedHash.toString('hex')}"`;

Thanks, that seems to work although I can't figure out how to use the same read stream in the S3 upload... Passing 'stream' seems to silently fail.
I've implemented an uploading queue here https://github.com/FraserThompson/gatsby-plugin-s3/tree/queues which limits it to 20 concurrent uploads. It seems to have fixed my issue, now NodeJS is only using around 700mb of memory whereas before it would exhaust the full 16gb.
I can make a pull request if you'd like, I'm not sure about the quality of my code though... I ended up using a lot of callbacks instead of async/await which is a little ugly.

@FraserThompson Looks good! Glad to hear it's resolved.
If you'd open a PR I wouldn't mind ironing out the callback stuff.

Thanks, that seems to work although I can't figure out how to use the same read stream in the S3 upload... Passing 'stream' seems to silently fail.

Streams can only be consumed once, so this is expected behaviour. You need one stream for creating the hash and a second for uploading the file.


Feature proposal: Autocomplete with Open File

Will be good to have autocomplete when opening files inside the editor.

Good idea! Would be nice to have until I get the proper file explorer implemented.

Yes please. That was the first thing I noticed missing.

I've taken this on. My initial progress is in the https://github.com/richrd/suplemon/tree/file-autocomplete branch. Maybe you guys could test? I'll be testing it with my setups too.

Doesn't seem to work for directories, only files.


Remove dependency on django-crispy-forms

As much as I like the library, I'm unhappy with it's presence. This is because it removes from us the easy ability to more readily create custom features like the readonly_fields attribute without even more hacking on form objects. In fact, all we use it for is simple rendering of the form in templates, which isn't enough.
Tasks:

Remove from dependency list
Remove from INSTALLED_APPS.
Create an includes/form.html template that renders forms.

fixed


Failed to install fm with proper permissions using npm

When trying to run it after copy pasting the npm install command in powershell, I get:
PS C:\Users\olofs> fm
fm : File C:\Users\olofs\AppData\Roaming\npm\fm.ps1 cannot be loaded because running scripts is disabled on this
system. For more information, see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ fm
+ ~~
    + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess

you must install king-lang package and use the command kind, not fm. Are you using Windows, right? You may face some issues running Kind on your machine if so. We recommend that you use WSL to assist you

Ah, I see the language has changed name. I was looking at the old formality repo. Sorry for the trivial question

no problem @saolof

@saolof the name of the language happened at the beginning of theis year, not everyone may be familiar with that. Feel free to ask any question you have 😊


Request to make the project available on conan

Although one could simply add incbin to the version control of an application as it is quite small, this has the caveat that you cannot tell automatically if a new version is available. Being able to use conan to download and install the project would facilitate integrating the tool in projects.

I'm not familiar with Conan but I don't mind if someone else wants to do that and we can support it officially upstream.

We created a conanfile.py and CMakelists.txt build wrapper, in case someone wants to base their work off of it. I think it would be important to start versioning in this repository for a Conan package to make any sense though. We used a "virtual version" based off the number of commits.
See: https://github.com/eclipse/cloe/tree/master/vendor/incbin


CI: build pandas on windows (appveyor)

Here is a collection of build scripts/tools for windows:
a nice conda recipe:
https://github.com/rmcgibbo/python-appveyor-conda-example/blob/master/appveyor.yml
https://github.com/pydata/pandas/tree/master/scripts/windows_builder
I have found many confusing explanations/suggestion on the web on how to build python extensions on windows 32/64 bit. The following is very simple to build from master.
Follow the instructions on the following link to build on windows with a binary compat to the existing python. Bascially.
http://pybytes.com/pywavelets/dev/preparing_windows_build_environment.html#installing-windows-sdk-c-c-compiler
Essentially:

download the Microsoft SDK (you need the 64-bit one of you are on a 64-bit machine, even if you are going to want to build 32-bit builds); this is just the install kit. I used http://www.slysoft.com/en/virtual-clonedrive.html to mount the .iso and install it
open the CMD Shell (installed onto the start menu)
set the env variables (I just copy pasted, but you can make a bat file if you want). for either 32 or 64 bit.

you can then just
c:\python27\python.exe setup.py build_ext --inplace
c:\python27\scripts\nosetests pandas

this finally works and passed on py 2.7/3.4/3.5
& we have a green badge: https://github.com/pydata/pandas


Enum is weird in php 8.1

Laravel Version: 8.82.0
PHP Version: 8.1.0
Database Driver & Version: 10.6.4-MariaDB

Description:
Retriving and displaying Enum values raise error after changing Enum values in Enum class.
Steps To Reproduce:
in Post model
    namespace App\Models;

    use App\Enums\PostStatus;

    class Post extends Model
    {
        protected $fillable =[
            'status',
        ];

        protected $casts = [
            'status'=> PostStatus::class,
        ];
    }

in PostStatus Enum
    namespace App\Enums;

    enum PostStatus: string
    {
        case ACTIVE ='ACTIVE';
    }

in controller
    function save() {
        $post_cat = new Post;
        $post_cat->status = PostStatus::ACTIVE ;
        $post_cat->save();
    }

in router
    Route::get('/test', function ( Request $request ) {
        return view('test', [
            'post'=> Post::find(1)
        ]);
    });

in test.blade.php
{{  $post->status->value  }}

And then I visted '/test' . It is ok.
ACTIVE

But I changed PostStatus Enum value.
    namespace App\Enums;

    enum PostStatus: string
    {
        case ACTIVE = 'I changed Value';
    }

and the '/test' shows error .

"ACTIVE" is not a valid backing value for enum "App\Enums\PostStatus"

I wonder why I must not change Enum values.
Is it just intended feature?

Hi there,
Thanks for reporting but it looks like this is a question which can be asked on a support channel. Please only use this issue tracker for reporting bugs with the library itself. If you have a question on how to use functionality provided by this repo you can try one of the following channels:

Laracasts Forums
Laravel.io Forums
StackOverflow
Discord
Larachat
IRC

However, this issue will not be locked and everyone is still free to discuss solutions to your problem!
Thanks.


matrix free solution using PetscNonlinearSolver

Hi :)
I'm building a new NonlinearFormintegrator for nonlinear material modeling. Due to the difficulty in producing the analytical Jacobian, I want to try the PetscNonlinearSolver to perform the matrix-free solution. I had a look at the petsc/ex10p, which can be activated to be matrix-free with -snes_mf or  -snes_mf_operator. However, ex10p  already has the analytical Jacobian built in the AssembleElementGrad().
By using the matrix-free method, do I still need to explicitly build the GetGradient() of the operator ? How should I construct the AssembleElementGrad() of the new NonlinearFormintegrator, as I try to use the matrix-free Newton-Krylov to avoid deriving the analytical Jacobian ?
Thanks !
Bright

By using the matrix-free method, do I still need to explicitly build the GetGradient() of the operator ?
Yes. If you need the matrix for preconditioning, you need to assemble it.

You can setup petsc to compute jacobian-vector products on the fly using finite differences and coloring, but this option is not available in combination with mfem right now.

By using the matrix-free method, do I still need to explicitly build the GetGradient() of the operator ?

Yes. If you need the matrix for preconditioning, you need to assemble it.
You can setup petsc to compute jacobian-vector products on the fly using finite differences and coloring, but this option is not available in combination with mfem right now.

Many thanks for your quick reply !
By using  -snes_mf_operator, PETSc can evaluate the correct the Jacobian even with incorrect or incomplete analytic Jacobian (https://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf 5.12). Therefore I wonder if I can perform matrix-free solution to my nonlinear modeling in MFEM without knowing the exact analytic Jacobian ?
I heard the finite difference Jacobian -snes_fd is very slow, so the matrix-free method  is closer to what I want.
Thanks
Bright

I think snes_mf will use JFNK and it will not use GetGradient() as preconditioner. So for simple problems which has weak nonlinearity, it may be enough. I believe in example 10, the case of rc_ex10p_mf should demonstrate that.
In general, you need to provide the preconditioner for JFNK through GetGradient(). You can start with providing a simple approximation to the Jacobian matrix through GetGradient(), like a block diagonal matrix that only keeps the diagonal small matrices. The case of rc_ex10p_mf demonstrates how to use it.
Yes, snes_fd can be very slow. The normal way to get it working is through snes_fd_color. But it looks like this option is not available in mfem yet.

You get the matrix-vector-product of the Jacobian with -snes_mf_operator, but this will not allow the preconditioning with matrix based solvers. Jacobi, Krylov methods and the like will work fine regardless.
This works out of the box with MFEM by just setting the option on the command line (e.g. ex10). The gradient that is implemented only uses (if desired) the assembled gradient for the PC.

Yes, snes_fd can be very slow. The normal way to get it working is through snes_fd_color. But it looks like this option is not available in mfem yet.

The limiting factor in MFEM is that we don't provide a DM object for PETSc from which coloring of matrix could be determined.

Thanks for the comments.
It seems even for the matrix-free method, I'll need to produce an approximate analytic Jacobian  matrix in the GetGradient(), with or without the preconditioner.  Am I understanding it correctly ?
As for the approximated analytic Jacobian, how does the degree of approximation matter ? such as some terms missing in the Jacobian equations.
thanks

We can't answer the the question without knowing the details of the equations you are solving. In coupled equations it might be OK to just provide diagonal blocks for the approximation, although it might not be enough in strongly nonlinear cases.

@brightzhang91 If you use -snes_mf, then you do not need to provide a Jacobian via the GetGradient method.
In this case, the Krylov method will be without preconditioner.  Instead, if you use -snes_mf_operator you need to provide a Jacobian matrix to be used to setup the preconditioner. The mat-vec within the Krylov method will be matrix free, but you will be able to construct a preconditioner using whatever operator you provide via GetGradient
@jandrej Regarding the coloring: well, the interface from MFEM to PETSc is completely algebraic and specifying mesh information is somewhat out-of-synch with the MFEM approach, since GetGradient is already designed to give access back to the user to provide the Jacobian. On the other hand, users can typecast the PetscNonlinearSolver to SNES, get the DM out of it, and, using private petsc headers, set the dm->ops->getcoloring callback. Cumbersome, I know, but PETSc is designed to be extensible. I can add a specific interface in DMSHELL (like DMShellSetColoring) if you need it

This issue has not been active for more than two weeks, so I will close it now. Feel free to reopen if you still have questions.


Deprecation warning by moment.js

Deprecation warning: value provided is not in a recognized ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non ISO date formats are discouraged and will be removed in an upcoming major release. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.
Arguments:
[0] _isAMomentObject: true, _isUTC: false, _useUTC: false, _l: undefined, _i:<PHONE_NUMBER>0000.000000+000, _f: undefined, _strict: undefined, _locale: [object Object]

It's worth noting that to install the patched version by @s-a you must use the command npm install s-a/node-wmi.git##fix-moment-deprecation-warning because his fix branch is named #fix-moment-deprecation-warning (with a #)...why tho?
Kinda wish the author of node-wmi would accept your pull request though.

Sorry for the hash. :O) It seems @Zinglish does not watch this repo...

Lmmfao, that's quite amusing that he doesn't watch his own repo. Maybe he didn't expect it to become somewhat popular...
Anyway, @s-a thanks for the fix :)

Maybe he do not want to maintain it anymore.
Since this issue is open a few month we should think about to make a hard fork together and republish on npm?

Thank you!

Still getting a deprecation warning by moment.js:
> wmi.Query({
...   class: 'Win32_BIOS'
... }, function(err, bios) {
...   console.log(bios);
... });
Query {
  _params:
   { class: 'Win32_BIOS',
     host: 'localhost',
     namespace: 'root\\CIMV2',
     password: undefined,
     props: undefined,
     username: undefined,
     where: undefined } }
> Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged and will be removed in an upcoming major release. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.
Arguments:
[0] _isAMomentObject: true, _isUTC: false, _useUTC: false, _l: undefined, _i:<PHONE_NUMBER>0000.000000+000, _f: undefined, _strict: undefined, _locale: [object Object]
Error
    at Function.createFromInputFallback (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:320:94)
    at configFromString (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:2178:11)
    at configFromInput (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:2547:9)
    at prepareConfig (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:2530:9)
    at createFromConfig (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:2497:40)
    at createLocalOrUTC (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:2584:12)
    at createLocal (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:2588:12)
    at hooks (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\moment\moment.js:16:25)
    at typeValue (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\node-wmi\lib\Query.js:208:15)
    at extractProperty (D:\Users\VertigoRay\Git\desktop\PhoneHome\node_modules\node-wmi\lib\Query.js:232:13)
[ { BuildNumber: undefined,
    Caption: '0601',
    CodeSet: undefined,
    ...

It seems new version was not published to NPM. 😮

Is it just because a new release hasn't been made? Need this and #6 to be published to NPM.
CC: @jgrusling @jpgrusling


Full-fledged SQLAlchemy support

For using software that uses SQLAlchemy as backend (e.g. https://github.com/airbnb/caravel), full-fledged SQLAlchemy support is needed.
To developers: Are there plans for further development of this repo?

I'm interested in helping out where I can, but I no longer use BQ in my day to day.
Do you know what else is needed for this dialect to help you get to where you want?

hey @Froskekongen did you managed to use bq with caravel?

I think that having basic functionality to run queries is the obvious first step. I am unfamiliar with how this is done in sqlalchemy, though - could anyone share pointers on where this functionality can be 'filled in'? That way I could help out by implementing some basic capability

does anyone know of a good python odbc driver for BigQuery? could probably think about incorporating that and the dialect would reflect patterns in the official dialects.

Afaik there are two: simba and cdata. They don't seem to be open source.
Given that the synthesis of SQL seems to be working, is it not a case that the main functionality required is the ability to submit the queries and return the results? In general this is not super difficult so I think at least a subset of the functionality could be straightforward to implement.

Yeah I haven't looked into it.  Does Big Query even support regular database connections? every client I used had you submit a job and then poll for results afterwards.

Hey,
Not really - streaming queries are the closest in that they are expected to
be serviced when received (rather than batched). Even so the concept of a
connection would be much weaker, as it would apply basically to creating an
object that can then later be used to authenticate with Google. However,
each query is essentially independent of all others; BigQuery does not
support per-session state as provided by e.g. postgres.
WRT to your question, it is correct - it is necessary to poll BigQuery for
results, and if the result set is large, it becomes necessary to retrieve
individual pages. Of course it would be possible to abstract this away,
only returning when the results are available. The problem is of course
that this would be slow.
I think that the match between BigQuery and a full fledged database is
definitely not perfect and it may be that only a subset of functionality is
implementable (e.g. using BigQuery as an ORM sounds exceptionally painful).
Nevertheless, it would seem a useful thing to have and it may provide
inspiration for other sql-like systems that are nevertheless not
database-like (e.g. connections per-se are not supported; batched/polled
behaviour needs to be abstracted, etc).
On Mon, Aug 15, 2016 at 2:21 PM, Conrad<EMAIL_ADDRESS>wrote:

Yeah I haven't looked into it. Does Big Query even support regular
database connections? every client I used had you submit a job and then
poll for results afterwards.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cpdean/sqlalchemy-bigquery/issues/6#issuecomment-239799803,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOruCQcfJLO1kCadDTU1IvF_Bt-RJbhks5qgGfPgaJpZM4H-0uO
.

--
Raul Landa | Data Scientist

Another issue that may be relevant is that it may be a good idea to check whether the current SQL query rendering is compatible with the 'standard' BigQuery SQL dialect, rather than the 'legacy' - this is clarified in  the docs. This is particularly important for queries operating over wildcard tables.

Something I'm particularly concerned about is how to deal with the JOIN EACH and GROUP EACH BY resource hints you apply in your query.
Maybe a ton of of engine work could be done so that the metadata object holds not only information about your schemas but also the sizes that a given join/groupby needs to operate over to intelligently add them but that's something that would absolutely have to happen if connections were implemented.
If this feature wasn't present, users would run into errors on anything larger than trivial sizes and would lose all value in this dialect abstracting things away from them. I'd rather it just fit into the toolset as a means of generating sql so that other more mature libraries handle the bq client stuff.

Hey,
BigQuery gets normal analytic functions when using standard SQL. This means
that 'EACH', is no longer needed, conventional 'OVER' and 'PARTITION BY'
will suffice:
https://cloud.google.com/bigquery/sql-reference/query-syntax#analytic-functions
So this is not a problem anymore! And this of course means that we can
re-use more of the previous parser constructs.
Thanks for your responses,
Raul
On Wed, Aug 17, 2016 at 11:40 PM, Conrad<EMAIL_ADDRESS>wrote:

Something I'm particularly concerned about is how to deal with the JOIN
EACH and GROUP EACH BY resource hints you apply in your query.
Maybe a ton of of engine work could be done so that the metadata object
holds not only information about your schemas but also the sizes that a
given join/groupby needs to operate over to intelligently add them but
that's something that would absolutely have to happen if connections were
implemented.
If this feature wasn't present, users would run into errors on anything
larger than trivial sizes and would lose all value in this dialect
abstracting things away from them. I'd rather it just fit into the toolset
as a means of generating sql so that other more mature libraries handle the
bq client stuff.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cpdean/sqlalchemy-bigquery/issues/6#issuecomment-240571310,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOruDC3m5wY3udvY2aIG-4XGi4mZuZ3ks5qg43tgaJpZM4H-0uO
.

--
Raul Landa | Data Scientist

@Froskekongen, @rlanda
I'm not able to work on this as I do not use BigQuery anymore, nor do I have accounts for it.  I feel like there may be tricky things that come up in supporting this, such as (and remember I'm speaking about what I know about BQ from a year and a half ago) large result sets not being supported without first saving those results to cloudstorage and downloading the files from there.
As a user, if I was using something that promised to be a seamless sql-database connector for a thing that holds data, and it would just not return data and instead tell me where to download it myself, I would be pretty pissed.
If a SQLAlchemy engine for BigQuery is something people want, these caveats either have to be built into it or it needs to not pretend that queries can be used seamlessly with large result sets and build some stuff into it that'll guide users with informative error messages.
I want to provide the support that I can, but this will be limited to helping out with offline testing and publishing merged work to PyPI.  I cannot test SQA engine features, and even if I got an account temporarily, the quality of SQA engine support will suffer since I don't use BQ -- I won't be properly dog-fooding the lib.
Is anyone interested in working on this? Does anyone need this feature anymore?

Hey,
Whether you need to export to cloud storage or not is immaterial, as the
library could abstract this away. It is not necessary, by the way.
WRT your involvement, I understand. I am working on this on my own. If and
when I have something to release maybe you could help as you suggested
testing and with PyPI. I think the amount of code that could be reused from
the present library is negligible.
Thanks,
Raul
On Sat, Sep 17, 2016 at 11:58 PM, Conrad<EMAIL_ADDRESS>wrote:

@Froskekongen https://github.com/Froskekongen, @rlanda
https://github.com/rlanda
I'm not able to work on this as I do not use BigQuery anymore, nor do I
have accounts for it. I feel like there may be tricky things that come up
in supporting this, such as (and remember I'm speaking about what I know
about BQ from a year and a half ago) large result sets not being supported
without first saving those results to cloudstorage and downloading the
files from there.
As a user, if I was using something that promised to be a seamless
sql-database connector for a thing that holds data, and it would just not
return data and instead tell me where to download it myself, I would be
pretty pissed.
If a SQLAlchemy engine for BigQuery is something people want, these
caveats either have to be built into it or it needs to not pretend that
queries can be used seamlessly with large result sets and build some stuff
into it that'll guide users with informative error messages.
I want to provide the support that I can, but this will be limited to
helping out with offline testing and publishing merged work to PyPI. I
cannot test SQA engine features, and even if I got an account temporarily,
the quality of SQA engine support will suffer since I don't use BQ -- I
won't be properly dog-fooding the lib.
Is anyone interested in working on this? Does anyone need this feature
anymore?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cpdean/sqlalchemy-bigquery/issues/6#issuecomment-247813238,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOruLum5Rqyxg0T5KNvv4RQwEiE3OLNks5qrHCTgaJpZM4H-0uO
.

--
Raul Landa | Data Scientist


Design for empty screens (transactions, addresses etc)

Atomic Swap (active offers) https://zpl.io/VQ1BnX9

Atomic Swap (my offers) https://zpl.io/add0MJp

Atomic Swap (transactions) https://zpl.io/VQPDqGk

Address book https://zpl.io/aXOkDL8

No contacts in address book https://zpl.io/aRndNnv

Transaction https://zpl.io/bWmYp97

No in progress (sent/received) transactions https://zpl.io/Vk40YP8

Notifications https://zpl.io/aXOkDwx

UTXO https://zpl.io/2GDokMY

Please, change the text for empty Atomic swaps - My offers screen as in the design:
https://zpl.io/add0MJp
Actual:

Change the text for empty Atomic swaps - Transactions screen as in the design:
https://zpl.io/VQPDqGk
Actual:

Checked.


Object detection question

Hi!
Sorry for raising an issue, I found it as a way of reaching out to you.
I am very impressed by your solutions for many image segmentation competitions, where participants are provided with image and ground-truth labels.
However I wanted to ask about your approach in cases when ground truth is only point, not mask (for example https://www.datasciencechallenge.org/challenges/1/safe-passage competition).
Could you reveal your appropach to such a problem? How to prepare masks in such a case, what kind of networks to use?
Kind regards,
Tomasz

Probably, if you would reach me out on the LinkedIn it would be faster.
There are 2 main approaches:

Label bounding boxes yourself for the train set and use these labels with Faster RCNN, YOLO, SSD, etc (That is what I did in SafePassage)
Assign gaussian to the dot and use segmentation approach. The peak of the predicted heat map will correspond to the centers of the objects.

I hope this helps.


Fixing "Uncaught exception bad bigarray kind"

Hello,
I have got the error message with "Uncaught exception bad bigarray kind" and followed the instructions from "https://gist.github.com/pch/aa1c9c4ec8522a11193b".
However, the command "git checkout 924387b brew --prefix/Library/Formula/objective-caml.rb" responded with the error message:
error: pathspec '924387b' did not match any file(s) known to git.
error: pathspec '/usr/local/Library/Formula/objective-caml.rb' did not match any file(s) known to git.
What can I do for "Uncaught exception bad bigarray kind" error?
Thanks!

That gist looks like it's unrelated to this project.
If you want to use this vagrant plugin, follow the instructions in the Usage section of this project's README (first install unison on both your VM Host and Guest, following the instructions for either Linux or OSX, then vagrant plugin install vagrant-unison2

The gist is provided in the README for troubleshooting.
And I have managed to give osx/guest os the same version of ocaml.
It's good now, thank you!


Very large numbers make sorting very slow

I have an issue that was opened for tablesorter stating that sorting was very slow with very large numbers. Specifically this made-up value:
"MySnmp<PHONE_NUMBER>234567891234567891234567891234567891234567891234567891234567891234567891234567891234567891234567891234567891234567891234567"

When I tested it, I found that the date matching regex was the bottleneck, so I added a "quick" regex date check* before the big one and it substantially increased the sort speed. Here is a demo of the difference, and the code that was added:
// quick date compare, look for am, pm, any 4 digit number, #:#, #/# or #-#
qdre = /([a|p]m|\b\d{4}\b|\d{1,4}[\/\-\:]\d)/,

and
// numeric, hex or date detection
xD = parseInt(x.match(hre), 16) || (xN.length !== 1 && x.match(qdre) && x.match(dre) && Date.parse(x)),
yD = parseInt(y.match(hre), 16) || xD && y.match(qdre) && y.match(dre) && Date.parse(y) || null,

* I'm not the best with regex, so I'm sure the "qdre" regex can be improved.

All fixed! Thanks!


Questions about Merge tag submission

The Merge tag '[tagName]' into develop commit of this repository will have an extra useless SSH SIGNATURE section starting from v8.18. I tried running git merge v8.21 --no-ff in the terminal and found that it would enter the interactive interface and the prompt was as follows. This part of the content would not appear after submission.
# -----BEGIN SSH SIGNATURE-----
# sth......
# -----END SSH SIGNATURE-----
# sth......
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.

Maybe there is some configuration error in your merge process?

I haven't noticed the problem yet. Maybe I'm using git flow release start/finish to merge the develop and master branches automatically.

Before I configure the gpg.ssh.allowedSignersFile, finishing a release branch will append the following content
# -----BEGIN SSH SIGNATURE-----
# ....
# -----END SSH SIGNATURE-----
# gpg verification failed.

After configured, it will be
# -----BEGIN SSH SIGNATURE-----
# ....
# -----END SSH SIGNATURE-----
# Good "git" signature for [EMAIL] with ED25519 key [KEY]

I'll try to remove this from message

When I using Fork to do the same job, that finishing a git-flow release branch with gpg signing enabled, it also appends the commit message……

Looks like it's a issue in git-flow


class cast exception with expandable item

Details

[3.2.8] Used library version
[27.1.1 ] Used support library version
[3.1.4 ] Used tooling / Android Studio version

Checklist

[V ] Searched for similar issues
[ V] Checked out the sample application
[ V] Read the README
[ V] Checked out the CHANGELOG
[ V] Checked out the MIGRATION GUIDE

About this issue
i am implementing an expandable list with selection but when deselecting the last item (not the item in the last index, but the last item to be deselected) i get class cast exception saying that the sub item class cannot be cast into the parent class. the exception is only thrown when deselecting the last item, for example:

list containing 10 items
selecting items 1, 2, 3 - fine
deselect item 1 and 3 - fine
deselect item 2 - class cast exception

this is the parent
class EpisodeAdapterItem(episode: EpisodeEntity)
    : ModelAbstractExpandableItem<EpisodeEntity, EpisodeAdapterItem, EpisodeAdapterItem.ViewHolder, EpisodeAdapterSubItem>(episode)

this is the sub item
class EpisodeAdapterSubItem(private val descriptionHtml: String)
    : AbstractExpandableItem<EpisodeAdapterItem, EpisodeAdapterSubItem.ViewHolder, EpisodeAdapterSubItem>()

after a little investigating i found that in the library class SelectExtension, the method public void deselect(Item item, int position, @Nullable Iterator<Integer> entries)  is called twice (not sure sure why twice): once with item as the parent (everything is fine), and the second time with item as the subitem.
the exception is thrown in the line mSelectionListener.onSelectionChanged(item, false); because the selection listener is expecting item to be of the parent type, however the second time this method is trying to pass the sub item type which causes the exception.
why is the method deselect called twice once with parent and once with sub item? is this a bug? am i doing something wrong?

potential duplicate of: https://github.com/mikepenz/FastAdapter/issues/703

doesnt look like it... my problem does not relate to search/filtering and setting item animator to null does not resolve the issue.
however maybe the issue you mentioned in your comment on onViewDetachedFromWindow() method somehow causes both #703 and my issue... any news on this since July 24th (the date of your comment)? are you still working on it?
can you please confirm whether my class definitions for parent and sub item are ok?
both the parent and sub item have different type ids, but i am only setting identifier for the parent (which is the id of the model), whereas in the sub item class i am not doing anything related to identifier. could this be related to the crash somehow?
if it helps, the item is not expanded when deselecting it

@mikepenz any news? this is an app breaking bug for me...

@or-dvir I am so sorry. was on vacation in iceland for a week and had no chance to look at anything.
I am still hoping that I can resolve the other ticket I linked you. just in case could you please create a sample app which showcases the problem or a branch to showcase the problem in the sample app, so I can hopefullly have a quicker look at it on the weekend?

no worries. nothing to apologize for. i was just wondering whats going on...
by all means, go for the linked ticket first. it might be related.
about a sample app, i will do my best but i dont think it will be within the next week as i have some other projects i must finish.

do you use the abstract expandable item for this?
could it be that the reference of parent to childs might be not done properly?

other than that I'd definitely need some sample to try to debug this. otherwise it's searching around in the dark :( sorry.

my class definitions are as above. as far as i can tell they are fine... correct me if im wrong.
ill try to make a small sample app today

@mikepenz please see this small sample app
as mention above, the crash occurs when deselecting the last item (no the item in the last index, but the last item to be deselected)
steps to reproduce the crash:
1- select any item
2- deselect that item

Ok now after seeing the problem, this one is actually rather easy described and solved.
So the problem is that you have a FastAdapter<EpisodeAdapterItem>. That would mean it only accepts this type of items. Adding a sub child adapter holding other items might be allowed (due to the fact that generics are not really helpful to prevent such a usage), but it will crash during runtime.
Instead you need to use a type for the FastAdapter which is valid across all child items you will hold in the adapter.
It might be that your implementation has a combined parent type, or it might be ModelAbstractItem, or AbstractItem, but one thing which is always true is the IItem interface.
So if you adjust the FastAdapter to be FastAdapter<*> it will allow all kind of items and it will not lead to the crash.

To be a bit more specific.
The FastAdapter is the global managing instance which wraps like an umbrella around all child adapters, thus the FastAdapter must accept all child items you might add.
If you have a FastAdapter<*> it does not matter how many different ItemAdapter, ModelAdapter with all sorts of different items you add.
but if you have a FastAdapter<FancyItem> it will only be able to handle adapters with this item type (ItemAdapter<FancyItem> or any item which extends FancyItem basically ItemAdapter<T extends FancyItem>)
Here's your code adjusted to make it work. (this btw. might not be the best thing, there are possibly other variants to have for example FastAdapter<IItem<*, *>> but having FastAdapter<*> worked during testing :D)
class ActivityMain : AppCompatActivity() {
    private lateinit var mAdapterEpisodes: ModelAdapter<EpisodeEntity, EpisodeAdapterItem>
    private lateinit var mAdapterRv: FastAdapter<*>
    private lateinit var mActionModeHelper: ActionModeHelper<*>

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        rv.apply {
            addItemDecoration(DividerItemDecoration(this@ActivityMain, DividerItemDecoration.VERTICAL))
            layoutManager = LinearLayoutManager(this@ActivityMain, RecyclerView.VERTICAL, false)
            setupAdapter()
            adapter = mAdapterRv
        }

        mAdapterEpisodes.add(
                EpisodeEntity("item 1"),
                EpisodeEntity("item 2"),
                EpisodeEntity("item 3"),
                EpisodeEntity("item 4"),
                EpisodeEntity("item 5"),
                EpisodeEntity("item 6"),
                EpisodeEntity("item 7"),
                EpisodeEntity("item 8"),
                EpisodeEntity("item 9"),
                EpisodeEntity("item 10")
        )
    }

    private fun setupAdapter() {
        mAdapterEpisodes = ModelAdapter { episode ->
            val subItem = listOf(EpisodeAdapterSubItem("${episode.text} sub item"))
            val finalItem = EpisodeAdapterItem(episode)
            finalItem.withSubItems(subItem)

            finalItem
        }

        mAdapterRv = FastAdapter<IItem<*, *>>().addAdapter(0, mAdapterEpisodes as IAdapter<IItem<*, *>>)
                .withSelectable(true)
                .withSelectOnLongClick(true)
                .withMultiSelect(true)
                .withOnPreClickListener { v, adapter, item, position ->
                    if (item is EpisodeAdapterItem) {
                        //if actionMode is active, it means the the users' clicks
                        //should select another item and NOT expand it
                        item.autoExpand = !mActionModeHelper.isActive
                    }
                    mActionModeHelper.onClick(item) ?: false
                }
                .withOnLongClickListener { v, adapter, item, position ->
                    //this is just so selecting an item will not trigger "onClick()"
                    true
                }
                .withOnPreLongClickListener { v, adapter, item, position ->
                    //if we have no actionMode we do not consume the event
                    mActionModeHelper.onLongClick(this, position) != null
                }
                .withSelectionListener { item, selected ->

                    //NOTE:
                    //the "selected" parameter is the state of the item AFTER
                    //the selection/de-selection

                    val numSelected = mAdapterRv.selected.size

                    //when selecting our first item on the list,
                    //collapse all expanded items.
                    //this will make it easier for the user to select more items without having
                    //to scroll past the expanded items (which may be long)
                    if (numSelected == 1)
                        mAdapterRv.collapseAll()

                    //NOTE:
                    //this if-else statement makes it so that
                    //once an item is selected, selecting other items could be done via
                    //a regular click (tap).
                    //once all items are de-selected, re-enable long click
                    //for selecting items

                    if (selected)
                        mAdapterRv.withSelectOnLongClick(false)
                    else if (numSelected == 0)
                        mAdapterRv.withSelectOnLongClick(true)
                }

        (mAdapterRv as FastAdapter<IItem<*, *>>?)!!.addExtension(ExpandableExtension<IItem<*, *>>()
                .withOnlyOneExpandedItem(true))

        mActionModeHelper = ActionModeHelper(mAdapterRv,
                R.menu.context_menu,
                object : ActionMode.Callback {
                    override fun onActionItemClicked(mode: ActionMode,
                                                     item: MenuItem): Boolean {
                        mode.finish()
                        return true
                    }

                    override fun onCreateActionMode(mode: ActionMode,
                                                    menu: Menu): Boolean {
                        return true
                    }

                    override fun onPrepareActionMode(mode: ActionMode?,
                                                     menu: Menu?): Boolean {
                        return false
                    }

                    override fun onDestroyActionMode(mode: ActionMode?) {
                        /*do nothing*/
                    }
                })
    }
}

thank you mike for this great explanation!
after reading it i have a few follow up questions if i may:

in the class SelectExtension, the method public void deselect(Item item, int position, @Nullable Iterator<Integer> entries) is called twice. once for sub item and once for parent item. i assume this is because both of those can be selectable. however, i specifically set my subitem to not be selectable. so why is this method still called? i dont think this method should be called on items which are not selectable as this will trigger the selection listener for items which are not selectable (which in my case caused the crash) and could probably lead to other bugs.
why is this only happening for the last item? i would expect it to be the same logic for every item that is deselected. is this by design? a bug?
simply using FastAdapter<*> doesnt seem right... not very object oriented. i would imagine i should use the "highest" object which is shared for both the parent and the child. am i correct? do you have an official recommendation on which type to use?
3.1. i believe that using FastAdapter<*> would also lead to future problems
please add some explanation of this (FastAdapter need to be of type shared for parent and sub item) in the readme file

1.) making an item not selectable will only prevent the user not selecting it, you can very easily still select it programatically. the programmatic selecting/deselecting is always valid.
2.) in the sample you provided it happened for all items
3.) exactly. so if you have a common type of your items you can use that. in the end the IItem is always the main thing true for all. so I believe (not sure if that's 100% right for kotlin) * would just mean that you can add all elements which implement IItem (as this type is defined in the FastAdapter generic definition.
4.) the readme does not go up to type safe usage. it uses a FastAdapter without a type. so it is not needed there. The sample showcases it in the multi type samples :)

but as this is open source. I am happy to take a PR to add this into the FAQ section

ah. i understand.
i dont know how many tests you did, but it only crashes when deselecting the last item. for example if i have items 1, 2 and 3 selected, deselecting items 2 and 3 works perfectly fine but deselecting item 1 causes a crash. so why doesnt it crash when deselecting items 3 and 2?
in the sample app (ExpandableSampleActivity) parent, subItem, subSubitem, and subSubSubItems are all of the same type (SimpleSubExpandableItem)

SOLUTION
the "trick" (as mentioned by mike) is to make the parent and sub items the same type (which in my case is AbstractExpandableItem).
for convenience, i defined this wrapper class to save me from typing <*,*,*> all the time
abstract class MyExpandableItem<VH: RecyclerView.ViewHolder>
    : AbstractExpandableItem<EpisodeAdapterItem, VH, EpisodeAdapterSubItem>()

this wrapper class is not mandatory and may change according to your own project.

then i went ahead and changed the definition of my parent and sub items
//old
class EpisodeAdapterItem(episode: EpisodeEntity)
    : ModelAbstractExpandableItem<EpisodeEntity, EpisodeAdapterItem, EpisodeAdapterItem.ViewHolder, EpisodeAdapterSubItem>(episode)
//new
class EpisodeAdapterItem(private val model: EpisodeEntity)
    : MyExpandableItem<EpisodeAdapterItem.ViewHolder>()

//old
class EpisodeAdapterSubItem(private val descriptionHtml: String)
    : AbstractExpandableItem<EpisodeAdapterItem, EpisodeAdapterSubItem.ViewHolder, EpisodeAdapterSubItem>()
//new
class EpisodeAdapterSubItem(private val descriptionHtml: String)
    : MyExpandableItem<EpisodeAdapterSubItem.ViewHolder>()

now in my Activity which holds the adapter i made the following changes:
//old line
private lateinit var mAdapterEpisodes: ModelAdapter<EpisodeEntity, EpisodeAdapterItem>
//new line
private lateinit var mAdapterEpisodes: ModelAdapter<EpisodeEntity, MyExpandableItem<*>>

//old line
private lateinit var mAdapterRv: FastAdapter<EpisodeAdapterItem>
//new line
private lateinit var mAdapterRv: FastAdapter<MyExpandableItem<*>>

in case you use ActionModeHelper, don't forget to make the same change there.
depending on your implementation, this may (and probably will) require some similar changes in other parts of your code


[WIP] Fix get_tx_set() for rectangular tx_sizes > 32x32

Also fix uv_tx_type decision for intra mode block.

Coverage increased (+0.4%) to 83.462% when pulling 9041b2b9cc5ed503fb933cdbb79b6cbfb60cb21e on ycho:fix_get_tx_set_1 into 6237e6b34fe157c13f1ddd2f0a77029c0666f049 on xiph:master.

Wow, look at this.
There is unexpected gain of -0.0373%, which I have no idea where it comes from!!

PSNR
PSNR Cb
PSNR Cr
PSNR HVS
SSIM
MS SSIM
CIEDE 2000

-0.0373
-0.2968
-0.2811
-0.1200
-0.0071
-0.0768
-0.1019

AWCY

@ycho You seem to disallow IDTX for 32x32 inter

@fbossen You mean the change inside write_tx_blocks()? That function is called for intra mode block only?

@fbossen Oh, the first line of change in get_tx_set()? Yes, it seems so.... let me check.

@ycho Yes, the value returned by get_tx_set() is different for 32x32 inter blocks

Confirm that fix works!
No change in AWCY for five frames.

PSNR
PSNR Cb
PSNR Cr
PSNR HVS
SSIM
MS SSIM
CIEDE 2000

0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000

AWCY
Full sequence AWCY result will be available soon.


Bug fix

Fixed bug in TwoObservations and printTwoObservations. Degrees of freedom used in F-test was supposed to be determined by max- and min- variance.
Also added "stringr" to install packages (wouldn't work without on windows).

Nice :smile:


Comment previews do not work

How to reproduce:

Find a submission
Type in a comment
Hit the "Preview" tab
Look at all the JS errors

We should probably add a test.

While we are at it - can we add realtime previews for markdown? Not necessary but I figured this would be a good place to mention.
Vue.js will make it really easy to have live markdown preview (which I think is better for staff who aren't aware how markdown makes their comments look?)
Last I checked, the vue.js source is also smaller than our existing markdown JS library.
All the JS we need

  <textarea v-model="input" debounce="300"></textarea>

new Vue({
  el: '#editor',
  data: {
    input: '# hello'
  },
  filters: {
    marked: marked
  }
})

Result: https://jsfiddle.net/yyx990803/oe7axeab/?utm_source=website&utm_medium=embed&utm_campaign=oe7axeab

Update: I tried integrating Vue - it wasn't as simple as I hoped. So we'll just avoid databinding for now.
8dacaa0 resolves this issue for now.  It only affected the student view.

Seems like a hacky fix. I'd like to keep this open so I can look into it later

Closing in favor of making a different issue


Fix regressions from performance refactor, more tests

Don't render page text overlay when none of the lines have text.
This fixes a bug with lazily loaded annotation texts in combination with the recent performance optimizations that resulted in text not being rendered.
Don't show text until first transformation has been applied
This prevents glitchy text display during page changes where there would be unscaled OCR text filling the screen for a few frames.
Make sure the text parse always includes page dimensions.
With the recent performance work, it has become mandatory that text parses contain their page's width and height. This commit adds a new source for those values in ALTO documents and adds a generic fallback heuristic to determine a rough page size from the coordinates of the text lines on the page.
Fix bug when left page in book view didn't have text

Also unifies OCR parsing to use querySelector instead of XPath for both hOCR and ALTO, as well as a rudimentary test suite for the PageTextDisplay component.

Codecov Report

Merging #16 into master will increase coverage by 8.68%.
The diff coverage is 54.83%.

@@            Coverage Diff             @@
##           master      #16      +/-   ##
==========================================
+ Coverage   62.17%   70.86%   +8.68%
==========================================
  Files          10       10
  Lines         423      429       +6
  Branches      130      135       +5
==========================================
+ Hits          263      304      +41
+ Misses        134      103      -31
+ Partials       26       22       -4

Impacted Files
Coverage Δ

src/components/MiradorTextOverlay.js
0.00% <0.00%> (ø)

src/lib/ocrFormats.js
90.13% <75.00%> (-3.29%)
:arrow_down:

src/components/PageTextDisplay.js
95.83% <100.00%> (+95.83%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72f90ee...a1bfd60. Read the comment docs.


Code workflow

Added configuration for all the project requirent dependency tasks

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: deepashreed sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


MLE error on problem 106

The code in '106 Construct Binary Tree from Preorder and Inorder Traversal.py' causes a MLE error now. It may be better to use integer than list in recursion.

Hi, thanks. Can you raise a pull request?

Thanks


nukeeper update ... finds 5 possible updates, nukeeper repo ... finds 4

I have a very strange one here... I'm running NuKeeper on a TeamCity build agent and have the issue that's it's not finding the package update that I expect.
I've set up a PAT so that NuKeeper can open a PR on our Azure DevOps account, and everything works fine on my machine. Now that I've moved the exact same code to the build agent I'm no longer seeing package updates from our private NuGet feed. To make sure I'm not going mad I've put in an extra step to run nukeeper update rather than nukeeper repo and, sure enough, the update command finds the package just fine!
For reference, here are the commands I'm running in TeamCity:
Step 4
~\.dotnet\tools\nukeeper.exe update --include ^XXXXXXXX. --change minor --age 0

Step 5
~\.dotnet\tools\nukeeper.exe repo https://XXXXXXXX.visualstudio.com/Marketplace/_git/%system.teamcity.projectName% %NuKeeper.AzureDevOps.AccessToken% --include ^XXXXXXXX. --change minor --age 0

Output:
***snip***
[17:39:17][Step 4/5] XXXXXXXX.ErrorReporting, XXXXXXXX.ProductCatalogue.Search.Algolia.Service, microsoft.applicationinsights, Microsoft.Configuration.ConfigurationBuilders.Azure, Microsoft.Configuration.ConfigurationBuilders.Base, Newtonsoft.Json, NServiceBus, NServiceBus.Host, NServiceBus.Persistence.Sql, NServiceBus.Persistence.Sql.MSBuild, NServiceBus.SqlServer, Octopack
[17:39:18][Step 4/5] Selected update of package Newtonsoft.Json to version 11.0.2, but version 12.0.1 is also available. Allowing Minor version updates.
[17:39:18][Step 4/5] Selected update of package NServiceBus to version 6.5.4, but version 7.1.6 is also available. Allowing Minor version updates.
[17:39:18][Step 4/5] Selected update of package NServiceBus.Host to version 7.0.2, but version 8.0.0 is also available. Allowing Minor version updates.
[17:39:18][Step 4/5] Selected update of package NServiceBus.Persistence.Sql to version 3.2.0, but version 4.4.0 is also available. Allowing Minor version updates.
[17:39:18][Step 4/5] Selected update of package NServiceBus.Persistence.Sql.MsBuild to version 3.2.0, but version 4.3.0 is also available. Allowing Minor version updates.
[17:39:18][Step 4/5] Selected update of package NServiceBus.SqlServer to version 3.1.4, but version 4.1.0 is also available. Allowing Minor version updates.
[17:39:18][Step 4/5] Found 5 possible updates
[17:39:18][Step 4/5] XXXXXXXX.ProductCatalogue.Search.Algolia.Service from 2.6.201 to 2.12.234 in Messaging\Messaging.csproj
[17:39:18][Step 4/5] Microsoft.Configuration.ConfigurationBuilders.Base from 1.0.1 to 1.0.2 in Messaging\Messaging.csproj
[17:39:18][Step 4/5] NServiceBus from 6.4.4 to 6.5.4 in Messaging\Messaging.csproj
[17:39:18][Step 4/5] NServiceBus.Host from 7.0.0 to 7.0.2 in Messaging\Messaging.csproj
[17:39:18][Step 4/5] NServiceBus.Persistence.Sql from 3.1.0 to 3.2.0 in Messaging\Messaging.csproj
[17:39:18][Step 4/5]
[17:39:18][Step 4/5] Found 5 package updates
[17:39:18][Step 4/5] NServiceBus to 6.5.4 from 6.4.4 in 1 place since 2 months ago.
[17:39:18][Step 4/5] NServiceBus.Host to 7.0.2 from 7.0.0 in 1 place since 1 year and 5 months ago.
[17:39:18][Step 4/5] NServiceBus.Persistence.Sql to 3.2.0 from 3.1.0 in 1 place since 5 months ago.
[17:39:18][Step 4/5] Microsoft.Configuration.ConfigurationBuilders.Base to 1.0.2 from 1.0.1 in 1 place since 5 months ago.
[17:39:18][Step 4/5] XXXXXXXX.ProductCatalogue.Search.Algolia.Service to 2.12.234 from 2.6.201 in 1 place since 5 hours ago.
[17:39:18][Step 4/5] Filtered by Include '^XXXXXXXX.' from 5 to 1
[17:39:18][Step 4/5] Selection of package updates: 5 candidates, filtered to 1
[17:39:18][Step 4/5] Nuget restore on C:\BuildAgent\work\92b3febe3a07ddce ProductCatalogue.Search.AlgoliaFeeder.Messaging.sln
[17:39:21][Step 4/5] Updating XXXXXXXX.ProductCatalogue.Search.Algolia.Service to 2.12.234 from 2.6.201 in 1 place since 5 hours ago.
[17:39:21][Step 4/5] Nuget restore on C:\BuildAgent\work\92b3febe3a07ddce\Messaging Messaging.csproj
[17:39:28][Step 4/5] Process exited with code 0

***snip***
[17:39:28][Step 5/5] Matched uri 'https://XXXXXXXX.visualstudio.com/' to collaboration platform 'AzureDevOps'
[17:39:28][Step 5/5] User name missing from profile, falling back to .gitconfig
[17:39:29][Step 5/5] Using upstream fork as push, for project Marketplace at https://XXXXXXXX.visualstudio.com/Marketplace/_git/XXXXXXXX.ProductCatalogue.Search.AlgoliaFeeder.Messaging/
[17:39:29][Step 5/5] Git clone https://XXXXXXXX.visualstudio.com/Marketplace/_git/XXXXXXXX.ProductCatalogue.Search.AlgoliaFeeder.Messaging/ to C:\BuildAgent\temp\buildTmp\NuKeeper\7fed03ce4df447a4921495cb19f353ef
[17:39:29][Step 5/5] Found 12 packages in use, 12 distinct, in 1 projects.
[17:39:29][Step 5/5] XXXXXXXX.ErrorReporting, XXXXXXXX.ProductCatalogue.Search.Algolia.Service, microsoft.applicationinsights, Microsoft.Configuration.ConfigurationBuilders.Azure, Microsoft.Configuration.ConfigurationBuilders.Base, Newtonsoft.Json, NServiceBus, NServiceBus.Host, NServiceBus.Persistence.Sql, NServiceBus.Persistence.Sql.MSBuild, NServiceBus.SqlServer, Octopack
[17:39:30][Step 5/5] Selected update of package Newtonsoft.Json to version 11.0.2, but version 12.0.1 is also available. Allowing Minor version updates.
[17:39:30][Step 5/5] Selected update of package NServiceBus to version 6.5.4, but version 7.1.6 is also available. Allowing Minor version updates.
[17:39:30][Step 5/5] Selected update of package NServiceBus.Host to version 7.0.2, but version 8.0.0 is also available. Allowing Minor version updates.
[17:39:30][Step 5/5] Selected update of package NServiceBus.Persistence.Sql to version 3.2.0, but version 4.4.0 is also available. Allowing Minor version updates.
[17:39:30][Step 5/5] Selected update of package NServiceBus.Persistence.Sql.MsBuild to version 3.2.0, but version 4.3.0 is also available. Allowing Minor version updates.
[17:39:30][Step 5/5] Selected update of package NServiceBus.SqlServer to version 3.1.4, but version 4.1.0 is also available. Allowing Minor version updates.
[17:39:30][Step 5/5] Found 4 possible updates
[17:39:30][Step 5/5] Microsoft.Configuration.ConfigurationBuilders.Base from 1.0.1 to 1.0.2 in Messaging\Messaging.csproj
[17:39:30][Step 5/5] NServiceBus from 6.4.4 to 6.5.4 in Messaging\Messaging.csproj
[17:39:30][Step 5/5] NServiceBus.Host from 7.0.0 to 7.0.2 in Messaging\Messaging.csproj
[17:39:30][Step 5/5] NServiceBus.Persistence.Sql from 3.1.0 to 3.2.0 in Messaging\Messaging.csproj
[17:39:30][Step 5/5]
[17:39:30][Step 5/5] Found 4 package updates
[17:39:30][Step 5/5] Microsoft.Configuration.ConfigurationBuilders.Base to 1.0.2 from 1.0.1 in 1 place since 5 months ago.
[17:39:30][Step 5/5] NServiceBus to 6.5.4 from 6.4.4 in 1 place since 3 months ago.
[17:39:30][Step 5/5] NServiceBus.Host to 7.0.2 from 7.0.0 in 1 place since 1 year and 5 months ago.
[17:39:30][Step 5/5] NServiceBus.Persistence.Sql to 3.2.0 from 3.1.0 in 1 place since 5 months ago.
[17:39:30][Step 5/5] Filtered by Include '^XXXXXXXX.' from 4 to 0
[17:39:30][Step 5/5] Selection of package updates: 4 candidates, filtered to 0
[17:39:30][Step 5/5] No updates can be applied. Exiting.
[17:39:30][Step 5/5] Process exited with code 0

The interesting lines are:
[17:39:18][Step 4/5] Found 5 package updates

vs
[17:39:30][Step 5/5] Found 4 package updates

Can anyone think of a reason I'm seeing this? My first thought was around package sources, but I can't for the life of me figure out why everything would be fine with update but not with repo.

Okay, I have debugged the NuKeeper source and have figured out why this is happening... it turns out that when you do an update NuKeeper runs in the current directory, when you run repo it creates a new temp directory and runs there...
I'd love to know why this is the case because I can't find that fact documented anywhere.
It's breaking my use case because the nuget.config with the authenticated sources in is generated on the fly by the build agent in the working directory, but instead of running in the working directory NuKeeper does a fresh checkout elsewhere and runs there...
How can I configure NuKeeper to run "in place" but still create the branch and pull request?

Hey,
As discussed in Gitter, and for future people that run into the same problem:
You can run : nukeeper repo . <token> to run against the current directory
The other option is to move the credentials to the machine nuget.config


CSS Output is not applied in pattern library

Issue description:
I'm just starting to play with block patterns and realized that none of my Kirki CSS is used in the pattern preview when inserting a pattern, either in the sidebar or the "Explore" modal.
Version used:
4.0.24
Using theme_mods or options?
options
To reproduce, click the + at the top left of the block editor, click Patterns tab, and you'll see that none of your styles are loaded fonts, colors, etc. or anything that that would be applied to default content. After inserting the pattern the styles are there.
I'm thinking since Kirki is not registering block editor assets for these styles, they are ignored in the pattern loader.

Weird finding... if I put anything in the Additional CSS field, then the Kirki CSS works. No idea how that's related. This is what I added, then the pattern has all of my Customizer output:


Questionable license

How can foxish be licensed under MIT when its source code contains lines like the following?
https://github.com/davidhampgonsalves/foxish/blob/e008ce3c7d088536d9dcef93b20685c79b741659/subscribe.html#L3-L5

Poorly apparently. The license was added after the fact and I didn't remember that it does contain that code from Googles RSS reader. So the Foxish code is MIT and the Google code is BSD.
If anyone wants alternate licensing let me know.


Maintain error handling

Error handling as we are doing miscellaneous implementation for start-up.

[x] BabyDeerAGI/TaskCreation

https://github.com/miurla/babyagi-ui/pull/104


Multi-statement transaction support (persistent database connection)

I'd like to enhance the driver implementation to allow some drivers to opt-in to adding the ability to provide a persistent connection to the database.
When SQLPad runs a query today, a database connection is established, the query is run, and the database connection is then closed.
This enhancement would allow reuse of a long-running connection, so that things could be done like creating a temp table in one query execution, then referencing it in another. Or perhaps a transaction is opened, some commands are run, and then the transaction is rolled back or committed as desired.
This would not be supported for all drivers, and would need to be added on a case-by-case basis.
This work would involve some sort of additional UI component in the toolbar allowing the user to connect/disconnect the connection as desired.

This has been implemented for Postgres, SQLite, and unixodbc as multi-statement transaction support. This feature is optional and opt-in. Drivers can be updated to this implementation as desired.


Improvements to C++ image api

What
Follow-up to image refactors.

Fixes #7040
rerun::Collection more thoroughly and don't want to do that currently
Based on https://github.com/rerun-io/rerun/pull/7055

Rust & C++ image apis now take elements first
Add some constructors to Image in favor of from_ methods
Remove direct construction from std::vector - before this caused issues and now this caused expensive copies (see also #7040)
Improve overall method documentation

special care of pointing out borrow vs ownership

Add more tests

Checklist

[x] address remaining todos
[ ] main ci
[x] I have read and agree to Contributor Guide and the Code of Conduct
[x] I've included a screenshot or gif (if applicable)
[x] I have tested the web demo (if applicable):

Using examples from latest main build: rerun.io/viewer
Using full set of examples from nightly build: rerun.io/viewer

[x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
[x] If applicable, add a new check to the release checklist!
[x] If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

PR Build Summary
Recent benchmark results
Wasm size tracking

To run all checks from main, comment on the PR with @rerun-bot full-check.

@rerun-bot full-check


[Form Validation] Dropdown not being validated, even in demo

Steps
Opening: https://semantic-ui.com/behaviors/form.html
Click on the first example (Tell us more about yourself) of "Usage" on send.
Expected Result
All fields should be red including the "Gender".
Actual Result
All fields are red except of "Gender" and "Password".
Version
2.4
Testcase
https://semantic-ui.com/behaviors/form.html

I prepared a fix for the community fork Fomantic-UI here
https://github.com/fomantic/Fomantic-UI/pull/2211

Can I work on this issue? Waiting for a reply. Thank you.


Adds bioconda recipe conversion integration test

All bioconda recipes (as of 2024-03-27) has been copied into this project for the purposes of integration testing the conversion work with rattler-build
This will be a great way to test both the recipe conversion AND determine if the other integration tests can scale to a few thousand recipes
Hopefully the bioconda folks won't mind me doing this...

UPDATE: Half of the bioconda recipes are currently not being evaluated. I have raised #16 to figure out why. For now, the files are included but the tests are disabled.

Instead of adding all these files to this repository (making it huge), maybe you could dynamically fetch them or use a git submodule?

I want the integration tests to be snapshots and stable. In principle, the tests should only fail IFF there is an issue with the conversion work or rattler-build. The success metrics per test are expected to increase over time as compatibility improves.
Fetching individual files would mean a ton of HTTP requests are made per run at this scale AND I'd have to mark each commit hash I'd want to fetch to ensure the files do not change between tests. That doesn't sound very manageable at this scale.
Submodules have the same fetch issue don't they? Not to mention I'd have to fetch the whole feedstock repo at once and not just the one file I am interested in? In my testing a few thousand recipes only take a few dozen megabytes of disk space, whereas a few thousand feedstock repositories take up potentially gigabytes of disk, and it's potentially unbounded.

I've also considered putting the integration test files in a separate repo to checkout in the workflow, but that effectively has the same network fetch issue, albeit a bit easier to manage.

On top of all this, I generally try to avoid the network in my automated testing. It adds a layer of uncertainty and another point of failure outside of the domain of the work being tested.

I understand where you are coming from. 👍
I do feel like this adds a lot of bloat to the repository. I would personally use a submodule which in my eyes is at the same level harmful as the git repository itself in terms of netwerk access. But I do understand that you want to have more control over the files you include.
But disregard my comment, having these integration tests is awesome. :)

No I'm glad you brought it up, I have had some of these thoughts/conversations with others internally.
It sucks to have thousands of files in a repo but I don't think I've seen/can think of an alternative that manages the pros/cons better than this.

@schuylermartin45 This is a no-go, let's use a git submodule for this instead of copying over those files.

Let's talk on early next week. I have some other ideas that I think are best discussed over the phone.

I'm going to be closing this PR and it's sister draft PR. We'll be moving the integration test files into it's own separate repo to checkout/cache in the CI workflow. So that'll be 1 potential network call over having to manage thousands, with no need to mess with submodules.
In the mean time, I will be cleaning up the branch from the draft PR and re-submit it as a new PR with the new scripts, but no test files.


better error message for cleanDirectBuffer

https://tachyon.atlassian.net/browse/TACHYON-1442
Only log the error message when the buffer is supposed to have a non-null Cleaner. Otherwise when the buffer is empty, the Cleaner is actually set as null during initialization.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/7471/
Test PASSed.

Thanks for improving the error logging! Could you add a comment in the source code on when the cleaner is expected to be null?

thanks for the patch! @yupeng9 has a good point in adding some more comment. After that, I will LGTM this PR.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/7477/
Test PASSed.

LGTM


Lint Potential Recursive Hook Loops Errors

Specific Demand
The hooks with dependencies should have a compile time check for the dependancy being changed inside the hook in order to not have recursive loops and system crashes in runtime.
For example this code should produce a compile time error:
`fn EditableHeader<'a>(cx:Scope, val: String, onedit: EventHandler<'a,Option>, globally_editable: bool) -> Element {
let locally_editable = use_state(cx, || {globally_editable.to_owned()});

let _ = use_memo(cx,(globally_editable, locally_editable), |(globally_editable,_)| {
    locally_editable.set(globally_editable.to_owned()); //error: this hook changes the value of its own dependancy!!!
});

//irrelevant code below
`
Implement Suggestion
I do not know how to implement this as I am just learning how to use the framework and do not understand how it works. but i imagine you can probably hardcode any hooks with dependancys in them to produce an error if a UseState.set() or UseRef.with_mut() is used on any of the dependancies inside them. This is a simple user error and this feature probably shouldn`t be a high priority.

This would be great to add to dioxus-check! We could hard code some hooks with dependancies initially. Once a more complete linting system that has more complete access to the source code like clippy, we could potentially detect this error for custom hooks as well.
We could also detect a similar issue that happens when you write to a hook within the main body of a component/hook:
let locally_editable = use_state(cx, || {globally_editable.to_owned()});
// If you unconditionally write to locally_editable, it will cause an infinite loop
locally_editable.write();

This would be great to add to dioxus-check! We could hard code some hooks with dependancies initially. Once a more complete linting system that has more complete access to the source code like clippy, we could potentially detect this error for custom hooks as well.
We could also detect a similar issue that happens when you write to a hook within the main body of a component/hook:
let locally_editable = use_state(cx, || 0);
// If you unconditionally write to locally_editable, it will cause an infinite loop
locally_editable.set(1);

I think you could try and modify the visit_expr_call() function`s match statement to have an explicit Node::HookFn(_) arm. Then inside that arm you can conditionally check if a  use_ref write() or with_mut() and/or use_state.set() is used and then return a new issue type called something like UnconditionalDependencyMutation or InfinitelyRecursiveHookError

With signals, we can warn at runtime if a signal writes to a reactive context that subscribes to it


"Run Cell" codelens is not showing above marked cells in .py files

Environment data

VS Code version: 1.51.1
Extension version (available under the Extensions sidebar):
--    Python: v2020.11.358366026
--    Jupyter: v2020.11.358541065
OS and version: macOS Catalina 10.15.6
Python version (& distribution if applicable, e.g. Anaconda): 3.8.6
IPython version: 7.19.0
Value of the python.languageServer setting: Jedi

Expected behaviour
The CodeLens
Run Cell | Run Cells Above | Debug Cell
above each cell marked by
#%%
should be showing but is not. It was working perfectly until a few days ago. When I opened by laptop this morning, the codelens were not showing anymore above code cells.
Steps to reproduce:

Create a python file test.py
Create a code cell

#%% This is a cell
msg = 'Hello world'
print(msg)

Additional note:
The "Jupyter: Enable Cell Code Lens" setting is enabled in my VSCode settings.

I figured it out. It was due to the setting jupyter.codeLenses.
For anyone who has similar issues, open your settings.json file, and replace
"jupyter.codeLenses": "python.datascience.runcell,  python.datascience.runallcellsabove",
with
"jupyter.codeLenses": "jupyter.runcell,  jupyter.runallcellsabove",
Apparently, the first one used to work and was replaced with the second one, but the settings were not automatically modified to take this change into account.


Stackoverflow error when serializing class with Parant / Child structure

This code results in a Stackoverflow error
public class Person
    {
        public string Name { get; set; }
        public Person Parent { get; set; }

        public List<Person> Children { get; set; } = new List<Person>();
    }

var person = new Person{Name = "John"};
var person2 = new Person{Name = "Alice", Parent = person};
person.Children.Add(person2);

var jsonString = JsonSerializer.ToJsonString(person); //Stackoverflow

Is there any way to make this work? I only needs to be able to serialize an object to a string, for logging purpose, I don't need to be able to desialize it again.

Wouldn't [IgnoreDataMember] on property Parent solve the issue?

Yes and No
It would stop the Stackoverflow error, but I would lose the info about the Parent.
Other json serializers solves this by a reference key
https://github.com/gregsdennis/Manatee.Json/wiki/Serialization#maintaining-references
Anothers solves this by being able to set a limit for recursion

Policy of Utf8Json, don't implement original format(like Union, Typeless, Cyclic-Reference).
https://github.com/neuecc/Utf8Json#which-serializer-should-be-used

@neucc, what you mean by "original format"? Json doesn't support cyclic-references at all and an attempt to serialize object with cyclic-references to Json is definitely bug in user code. But StackOverflowException is non-recoverable and cause failure of entire application - not the best way to inform the user about his mistake. Why you don't want to avoid this exception and throw another before overflow happens? Or why not make behavior configurable?


Ternary Search

🚀 Algorithm
Ternary Search
(Add both recursive and iterative approach)
Languages Supported :

C :
CPP :
Python :
Java :
JavaScript :
GO :
C# :

Will be happy to contribute if assigned in python language :)

Please Assign Me This. I Can Do It in CPP

I would like to work on this issue in Java.

I would work on this issue in C

@basharfarid77 Assigned in Python.
@CormacKrum Assigned in C++.
@parjanyaacoder Assigned in Java.
@salonishah01 Assigned in C.

@atarax665, I will implement it in Javascript.

@atarax665 @iamrajiv @plazzy99 Can you assign me the solution of the problem in C-sharp


Feature Request: Support MongoDB version  4.0

Hi,
According to the official document, MongoDB supports 3.2 or 3.4 for both Single User and Master User.
I would like to use MongoDB 4.0 to build DocumentDB,
So, it would be nice if the lambda to be rotated could also support version 4.0.

Thank you for your feedback. We have noted this as a feature request.

Is there any estimated time for this? It's been a while.

I just tested the current script for the Single rotation scenario and it works for DocumentDB with MongoDB 4.0 compatibility (docdb 4.0.0)!
I am using pymongo version 3.12.3 (BTW, after a lot of testing,  the pymongo version used when set the rotation using the AWS Web Console by the current AWS is 3.2), but there is an error with pymongo 4+, this is what AWS Lambda shows using that newer version:
[ERROR] TypeError: 'Collection' object is not callable. If you meant to call the 'authenticate' method on a 'Database' object it is failing because no such method exists.
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 77, in lambda_handler
    set_secret(service_client, arn, token)
  File "/var/task/lambda_function.py", line 158, in set_secret
    conn = get_connection(pending_dict)
  File "/var/task/lambda_function.py", line 304, in get_connection
    conn = connect_and_authenticate(secret_dict, port, dbname, use_ssl)
  File "/var/task/lambda_function.py", line 377, in connect_and_authenticate
    db.authenticate(secret_dict['username'], secret_dict['password'])
  File "/var/task/pymongo/collection.py", line 3164, in __call__
    "exists." % self.__name

From this, maybe we should change the title or open a new ticket asking for the support of pymongo 4+

Closing this issue since DocumentDB with MongoDB 4.0 compatibility works with the 3.x recommended version of pymongo. If you have a need for pymongo 4, please open a separate issue indicating the need for this newer library.


[Run] Interact with plugin hints

Summary of the Pull Request
Interact with plugin hints with keyboard and mouse.

PR Checklist

[x] Closes: #30330
[ ] Communication: I've discussed this with core contributors already. If work hasn't been agreed, this work might be rejected
[ ] Tests: Added/updated and all pass
[ ] Localization: All end user facing strings can be localized
[ ] Dev docs: Added/updated
[ ] New binaries: Added on the required places

[ ] JSON for signing for new binaries
[ ] WXS for installer for new binaries and localization folder
[ ] YML for CI pipeline for new test projects
[ ] YML for signed pipeline

[ ] Documentation updated: If checked, please file a pull request on our docs repo and link it here: #xxx

Detailed Description of the Pull Request / Additional comments

Fixed a bug where plugins were not refreshed when changing setting between all/non-global
Allow plugin hints navigation with Up/Down arrows
Allow plugin hints navigation with Alt+Tab/Alt+Shift+Tab
Type the action keyword for the selected plugin with mouse click
Type the action keyword for the selected plugin with Enter key

Validation Steps Performed

Tested plugin hints refresh when the setting is changed
Tested action keyword typing with mouse/keyword
Tested plugin hints navigation with mouse/keyboard
Verified that Up/Down arrows and Alt+Tab/Alt+Shift+Tab are working as expected when results are displayed

Implements #30599 too.


Added support for RHEL8 in params.

RHEL 7 and 8 param settings are compatible (i mean as far as I've experienced so far  ;) )  This PR simply updates the params to account for RHEL 8.  Note: I did not do any adjustments to tests or anything else beyond params -- if you need me to do that just let me know.  Thanks!

Sorry I had not seen this PR before. This was merged in #257

Not a problem at all!  =)   Thanks!


"region `iram0_0_seg' overflowed" (when optimizations are off) but IRAM seems to be nearly not used (IDFGH-5133)

Environment

Development Kit:      none
Kit version (for WroverKit/PicoKit/DevKitC):
Module or chip used:  ESP32-WROOM-32
IDF version (run git describe --tags to find it):
v4.4-dev-960-gcf457d412
Build System:         CMake
Compiler version (run xtensa-esp32-elf-gcc --version to find it):
xtensa-esp32-elf-gcc (crosstool-NG esp-2020r3) 8.4.0
Operating System:     Windows
(Windows only) environment type: [MSYS2 mingw32|ESP Command Prompt|Plain Command Prompt|PowerShell].
Using an IDE?: Yes, eclipse with esp-idf plugin
Power Supply:         external 3.3V

Problem Description
A project cannot be build when optimization is turned off (really off -O0) because of an overflowed IRAM segment. With optimization set to debug the project builds and runs and debugging works.
Expected Behavior
The program will fit into IRAM since it is not full.
Actual Behavior
Linker sees a filled up IRAM segment.
Steps to reproduce
Impossible to reproduce - Hello world project does work with full optimization and I am not allowed to publish the project I work on.
Debug Logs
c:/espressif_esp_idf_tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: [...].elf section `.iram0.text' will not fit in region `iram0_0_seg'
c:/espressif_esp_idf_tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: IRAM0 segment data does not fit.
c:/espressif_esp_idf_tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: region `iram0_0_seg' overflowed by 18144 bytes

Other items if possible
Application size analysis:

How do I find the cause of this problem?

@TobiasBoeschMiele I suspect that this is a bug in the Eclipse plugin rather than a bug in the linker. "126kB available" doesn't look right. Could you please run idf.py size from the command line and paste its output as well here? Alternatively, if you can upload the .map file or email it to me (ivan at %companyname%.com), that would help us diagnose the issue quicker.

Could you please run idf.py size from the command line

I tried, but I think this command needs a working build. When the optimization is off, the build will not be successful and this command fails with the build error. Should I share the commands output when optimization is not turned off?

idf.py size with optimization set to Debug -Og

Total sizes:
 DRAM .data size:   16196 bytes
 DRAM .bss  size:   21624 bytes
Used static DRAM:   37820 bytes (  86760 available, 30.4% used)
Used static IRAM:  108763 bytes (  22309 available, 83.0% used)
      Flash code:  522083 bytes
    Flash rodata:  133172 bytes
Total image size:~ 780214 bytes (.bin may be padded larger)

It seems IRAM is already filled up to more than 80 %. That means I have to somehow reduce the IRAM size.

This is the component based analysis of size:
(Real names of some components were replaced by [numer])
Per-archive contributions to ELF file:
            Archive File DRAM .data & .bss & other   IRAM   D/IRAM Flash code & rodata   Total
                 libbt.a        164  12803       0    487        0     243869    74459  331782
           libbtdm_app.a        773   2573       0  27818        0      69835     4551  105550
                  libc.a          4      0       0      0        0      80624     4734   85362
                libphy.a       1436    715       0   7790        0      32263        0   42204
             libdriver.a       1040    248       0   5740        0      23387    10705   41120
           libfreertos.a       3112   1044       0  17186        0        370     4190   25902
          libnvs_flash.a          0     24       0      0        0      14196     2924   17144
          libspi_flash.a       1538    294       0   9574        0       1962     1802   15170
         libesp_system.a        245    206       0   3080        0       6075     3885   13491
                libhal.a        136      0       0   6521        0       4846     1935   13438
            libcoexist.a       1804     33       0   3707        0       4190      325   10059
          libapp_trace.a       1570   1977       0   6442        0          0        0    9989
               libheap.a       1965      8       0   5227        0       1408      777    9385
     libesp_hw_support.a        176     45       0   4058        0       3995      878    9152
                lib[1].a         40    654       0    406        0       5481      938    7519
         libesp_common.a          0      0       0      0        0         53     6957    7010
                libgcc.a          8     12       0    104        0       5543      872    6539
                libvfs.a        308     48       0      0        0       5640      517    6513
                libsoc.a          0      0       0    285        0        520     4145    4950
                lib[2].a        113     52       0      0        0       2322     1813    4300
             libstdc++.a          8     20       0      0        0       2517     1044    3589
          libesp_timer.a         32     24       0   1500        0       1180      743    3479
           libesp_wifi.a         92     27       0    242        0       1732     1336    3429
             libnewlib.a        190    272       0   1021        0        667      151    2301
                librtc.a          0      4       0   2227        0          0        0    2231
              libefuse.a         64      4       0      0        0       1230      807    2105
 libbootloader_support.a          0      0       0    965        0        594        0    1559
                liblog.a          8    272       0    262        0        855      144    1541
        libesp_ringbuf.a          0      0       0    905        0          0      538    1443
             libxtensa.a       1024      0       0     16        0        239       35    1314
            libpthread.a         16     12       0    239        0        535      354    1156
              libesp32.a          8     40       0    406        0        220      260     934
         libapp_update.a          1     12       0    194        0        175      453     835
            libesp_ipc.a          0     56       0    188        0        464       89     797
                lib[3].a          0     40       0      0        0        713        0     753
             libxt_hal.a          0      0       0    515        0          0       32     547
                lib[4].a          0      0       0      0        0        139       42     181
            libesp_rom.a          0      0       0    112        0          0        0     112
                   (exe)          0      0       0      3        0          3       12      18
                libcxx.a          0      0       0      0        0         11        0      11
             libesp_pm.a          0      0       0      0        0          8        0       8
               libcore.a          0      0       0      0        0          0        0       0
            libesp_eth.a          0      0       0      0        0          0        0       0
          libesp_event.a          0      0       0      0        0          0        0       0
          libesp_netif.a          0      0       0      0        0          0        0       0
               liblwip.a          0      0       0      0        0          0        0       0
         libmbedcrypto.a          0      0       0      0        0          0        0       0
            libmbedtls.a          0      0       0      0        0          0        0       0
           libmbedx509.a          0      0       0      0        0          0        0       0
               libmesh.a          0      0       0      0        0          0        0       0
           libnet80211.a          0      0       0      0        0          0        0       0
                 libpp.a          0      0       0      0        0          0        0       0
      libtcpip_adapter.a          0      0       0      0        0          0        0       0
     libwpa_supplicant.a          0      0       0      0        0          0        0       0

When the optimization is off, the build will not be successful and this command fails with the build error. Should I share the commands output when optimization is not turned off?

You're right, idf.py size is indeed not working when the build fails... However the .map file should still be generated by the linker, so you can try to run the following after the failed idf.py build:
python $IDF_PATH/tools/idf_size.py build/your_app_name.map

If that works, please also run
python $IDF_PATH/tools/idf_size.py build/your_app_name.map --json

and post the output you get — it's the JSON output that is consumed by Eclipse.
With regards to reducing IRAM usage, you can try enabling CONFIG_FREERTOS_PLACE_FUNCTIONS_INTO_FLASH — it places FreeRTOS function which aren't used in ISRs into flash instead of IRAM.

Note: I used %IDF_PATH% on windows.
somewhere>python %IDF_PATH%/tools/idf_size.py build/[appname].map
Total sizes:
 DRAM .data size:   19268 bytes
 DRAM .bss  size:   21656 bytes
Used static DRAM:   40924 bytes (  83656 available, 32.8% used)
Used static IRAM:    1027 bytes ( 130045 available, 0.8% used)
      Flash code:  715291 bytes
    Flash rodata:  160148 bytes
Total image size:~ 895734 bytes (.bin may be padded larger)

while the build fails with:
c:/espressif_esp_idf_tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: [appname].elf section `.iram0.text' will not fit in region `iram0_0_seg'
c:/espressif_esp_idf_tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: IRAM0 segment data does not fit.
c:/espressif_esp_idf_tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: region `iram0_0_seg' overflowed by 18144 bytes

This looks very similar to the eclipse plugin calculations in the initial comment

Alright, so it's an idf_size.py issue then. I think that it simply doesn't consider the possibility of an overflow yet, hence the calculation is incorrect. We will fix this, thank you for all the details.

You're welcome.
I think solving the overflow issue (which I think will persist after fixing the idf.size.py issue) has to wait for fixing the idf.size.py issue?

This issue persists (veryfied). I cannot reopen it.

@TobiasBoeschMiele could you please clarify which issue you are seeing? The issue that IRAM segment overflows at -O0 level, or that idf.py size doesn't work when overflow happens? The linked commit should have fixed the latter issue.

I made a mistake and did not update before testing the issue. I will test this again right now.

On latest master we've added a kconfig option for using parts of SRAM1 as IRAM in https://github.com/espressif/esp-idf/commit/5cbd311ecf0b381a0711ef8a96980a67b30a135f which will help with this issue.
We've also focused on reducing IRAM usage/making placement optional for several components. For an full overview of all such option see Optimizing-iram-usage. All new options for saving IRAM we add will be listed in this chapter.


Changing "name" field in configuration.yaml has no effect in HA

I have a RGBWW strip setup that's working great with this HA integration.  I went into the configuration.yaml file today to change the "name" of one of my devices and restarted HA -- but this had no effect in HA; the previous name of the light persisted, and I can't see how to get HA to acknowledge the change.
I changed the name from "master_shower" to "master_mirror" but HA still reflects the "light.master_shower" naming convention.  Normally I'd just rename the entity in HA, but I wanted to rename it in the artnet_led configuration so I could use the  WiFi easynode that I had previously set up for one set of LED strips for another, and free up the name for use with a second easynode.
Appreciate any guidance
light:
  - platform: artnet_led
    host: <IP_ADDRESS>                   # IP of Art-Net Node
    max_fps: 40                           # Max 40 per second
    refresh_every: 0                      # Resend values if no fades are running every x seconds, 0 disables automatic refresh
    universes:                            # Support for multiple universes
      0:                                  # Nr of Universe (see configuration of your Art-Net Node)
        devices:
          - channel: 1                    # first channel of dmx dimmer
            name: master_mirror           # name
            type: rgbww                   # type
            transition: 0                 # default duration of fades in sec. Will be overridden by Transition sent from HA
            output_correction: quadratic  # optional: quadratic, cubic or quadruple. Apply different dimming curves to the output. Default is None which means linear dimming
            channel_size: 16bit           # width of the channel sent to DMX device, default "8bit", "16bit", "24bit" and "32bit" available.

I've played around with it and haven't yet found a way to implement this without creating excess garbage (disabled/restored entities). For now, please just use the GUI to rename Entity ID's.