failed to store statistics: timeout

...
2015/11/06 00:31:11 InfluxDB starting, version <IP_ADDRESS>, branch 0.9.4, commit da1b59e7d7764d36786253b2db13b97f42ed4e1d
2015/11/06 00:31:11 Go version go1.5, GOMAXPROCS set to 4
2015/11/06 00:31:11 Using configuration at: /config/config.toml
[metastore] 2015/11/06 00:31:11 Using data dir: /data/meta
[metastore] 2015/11/06 00:31:11 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[localhost:8088]
[metastore] 2015/11/06 00:31:11 Node at localhost:8088 [Follower]
[metastore] 2015/11/06 00:31:13 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[monitor] 2015/11/06 00:31:13 Starting monitor system
[monitor] 2015/11/06 00:31:13 'build' registered for diagnostics monitoring
[monitor] 2015/11/06 00:31:13 'runtime' registered for diagnostics monitoring
[monitor] 2015/11/06 00:31:13 'network' registered for diagnostics monitoring
[monitor] 2015/11/06 00:31:13 'system' registered for diagnostics monitoring
[store] 2015/11/06 00:31:13 Using data dir: /data/db
[monitor] 2015/11/06 00:31:13 Storing statistics in database '_internal' retention policy '', at interval 10s
[wal] 2015/11/06 00:31:13 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 20971520 partition size threshold
[wal] 2015/11/06 00:31:13 WAL writing to /data/wal/_internal/monitor/1
[metastore] 2015/11/06 00:31:14 Updated node id=1 hostname=localhost:8088
[monitor] 2015/11/06 00:31:28 failed to store statistics: timeout
[monitor] 2015/11/06 00:31:38 failed to store statistics: timeout
[monitor] 2015/11/06 00:31:48 failed to store statistics: timeout
[monitor] 2015/11/06 00:31:58 failed to store statistics: timeout
2015/11/06 00:31:11 InfluxDB starting, version <IP_ADDRESS>, branch 0.9.4, commit da1b59e7d7764d36786253b2db13b97f42ed4e1d
2015/11/06 00:31:11 Go version go1.5, GOMAXPROCS set to 4
2015/11/06 00:31:11 Using configuration at: /config/config.toml
...

I'm using influxdb <IP_ADDRESS> (https://github.com/tutumcloud/influxdb) inside Kubernetes cluster, data is persistent to one of the nodes. From a fresh start (data is empty) it run just fine, after a few hundreds MB the error above occurs and influxdb hangs (not crash, but cannot access to it, admin or driver). Restart doesn't help, the error still there, unless I clear all  the data.
Thanks for any help!

Try doubling these timeouts:
https://github.com/influxdb/influxdb/blob/master/etc/config.sample.toml#L126

What kind of load are you putting on the system? What are the resources underneath the system?

@otoolep Thanks I'll try that. Anyway, it seems working when I removed the resource limits for the container. Before it was set to 2GB of RAM. At least I can restart it and been running for 1 hours now, keep watching.

I got one question: what's is the behavior of influxdb when a write is timeout?

In a single-node system it probably means nothing was written. Your code should backoff and retry if your client code receives. For internals stats, it's not that serious.
Generally it's a sign the system is too busy, which it can be particularly when starting up.

I have ran into this issue on 0.9.5 nightly:
My InfluxDb was under high load for an hour or two, then OOM happened.
When I restarted the database and tried to reinitialize, I got the following:
query] 2015/11/10 20:50:48 DROP DATABASE analytics
[monitor] 2015/11/10 20:50:57 failed to store statistics: timeout
[monitor] 2015/11/10 20:51:08 failed to store statistics: timeout
[monitor] 2015/11/10 20:51:17 failed to store statistics: timeout
[monitor] 2015/11/10 20:51:27 failed to store statistics: timeout
[monitor] 2015/11/10 20:51:37 failed to store statistics: timeout
[monitor] 2015/11/10 20:51:47 failed to store statistics: timeout
[monitor] 2015/11/10 20:51:57 failed to store statistics: timeout
[monitor] 2015/11/10 20:52:07 failed to store statistics: timeout
[monitor] 2015/11/10 20:52:17 failed to store statistics: timeout
[monitor] 2015/11/10 20:52:27 failed to store statistics: timeout
[monitor] 2015/11/10 20:52:37 failed to store statistics: timeout
[monitor] 2015/11/10 20:52:47 failed to store statistics: timeout
[monitor] 2015/11/10 20:52:57 failed to store statistics: timeout
[monitor] 2015/11/10 20:53:07 failed to store statistics: timeout
[monitor] 2015/11/10 20:53:17 failed to store statistics: timeout
[monitor] 2015/11/10 20:53:27 failed to store statistics: timeout
[monitor] 2015/11/10 20:53:38 failed to store statistics: timeout
[monitor] 2015/11/10 20:53:47 failed to store statistics: timeout

P.S: single-node setup, not clustered.

@otoolep sounds like an _internal write issue, perhaps. Any workarounds?

This is most likely not specific to the internal database, the system simply couldn't accept any more load.
Did the issue ever resolve @mtzaurus ?

Today I had it again, it was hanging for 20+ minutes, so I just killed to process.
What I can say for sure, is that it's again DATABASE DROP - related.
[query] 2015/11/11 11:47:36 DROP DATABASE analytics
[monitor] 2015/11/11 11:47:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:47:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:48:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:48:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:48:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:48:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:48:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:48:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:49:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:49:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:49:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:49:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:49:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:49:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:50:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:50:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:50:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:50:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:50:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:50:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:51:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:51:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:51:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:51:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:51:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:51:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:52:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:52:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:52:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:52:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:52:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:52:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:53:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:53:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:53:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:53:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:53:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:53:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:54:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:54:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:54:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:54:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:54:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:54:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:55:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:55:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:55:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:55:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:55:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:55:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:56:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:56:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:56:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:56:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:56:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:56:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:57:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:57:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:57:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:57:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:57:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:57:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:58:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:58:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:58:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:58:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:58:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:58:51 failed to store statistics: timeout
[monitor] 2015/11/11 11:59:01 failed to store statistics: timeout
[monitor] 2015/11/11 11:59:11 failed to store statistics: timeout
[monitor] 2015/11/11 11:59:21 failed to store statistics: timeout
[monitor] 2015/11/11 11:59:31 failed to store statistics: timeout
[monitor] 2015/11/11 11:59:41 failed to store statistics: timeout
[monitor] 2015/11/11 11:59:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:00:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:00:11 failed to store statistics: timeout
[retention] 2015/11/11 12:00:16 retention policy enforcement check commencing
[retention] 2015/11/11 12:00:16 retention policy shard deletion check commencing
[monitor] 2015/11/11 12:00:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:00:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:00:41 failed to store statistics: timeout
[monitor] 2015/11/11 12:00:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:01:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:01:11 failed to store statistics: timeout
[monitor] 2015/11/11 12:01:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:01:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:01:41 failed to store statistics: timeout
[monitor] 2015/11/11 12:01:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:02:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:02:11 failed to store statistics: timeout
[monitor] 2015/11/11 12:02:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:02:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:02:41 failed to store statistics: timeout
[monitor] 2015/11/11 12:02:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:03:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:03:11 failed to store statistics: timeout
[monitor] 2015/11/11 12:03:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:03:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:03:41 failed to store statistics: timeout
[monitor] 2015/11/11 12:03:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:04:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:04:11 failed to store statistics: timeout
[monitor] 2015/11/11 12:04:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:04:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:04:41 failed to store statistics: timeout
[monitor] 2015/11/11 12:04:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:05:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:05:11 failed to store statistics: timeout
[monitor] 2015/11/11 12:05:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:05:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:05:41 failed to store statistics: timeout
[monitor] 2015/11/11 12:05:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:06:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:06:11 failed to store statistics: timeout
[monitor] 2015/11/11 12:06:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:06:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:06:41 failed to store statistics: timeout
[monitor] 2015/11/11 12:06:51 failed to store statistics: timeout
[monitor] 2015/11/11 12:07:01 failed to store statistics: timeout
[monitor] 2015/11/11 12:07:11 failed to store statistics: timeout
[monitor] 2015/11/11 12:07:21 failed to store statistics: timeout
[monitor] 2015/11/11 12:07:31 failed to store statistics: timeout
[monitor] 2015/11/11 12:07:41 failed to store statistics: timeout

I'm seeing this on Influx <IP_ADDRESS>.  When I hop on the box I don't see any obvious system contention, the load is low, CPU idle, and there is plenty of memory and disk free.  InfluxDB appears to be wedged and not storing or querying data.  I see the following in the logs, query and http logs are off as they cause the log to grow to multiple gigs:
{"log":"[wal] 2015/12/02 21:15:28 Flush due to idle. Flushing 51 series with 51 points and 1947 bytes from partition 1\n","stream":"stderr","time":"2015-12-02T21:15:28.305797744Z"}
{"log":"[wal] 2015/12/02 21:15:28 write to index of partition 1 took 14.9136ms\n","stream":"stderr","time":"2015-12-02T21:15:28.321011441Z"}
{"log":"[wal] 2015/12/02 21:15:38 Flush due to idle. Flushing 51 series with 51 points and 1947 bytes from partition 1\n","stream":"stderr","time":"2015-12-02T21:15:38.32964811Z"}
{"log":"[wal] 2015/12/02 21:15:38 write to index of partition 1 took 16.21831ms\n","stream":"stderr","time":"2015-12-02T21:15:38.346211589Z"}
{"log":"[monitor] 2015/12/02 21:15:48 failed to store statistics: timeout\n","stream":"stderr","time":"2015-12-02T21:15:48.287640874Z"}
{"log":"[monitor] 2015/12/02 21:15:58 failed to store statistics: timeout\n","stream":"stderr","time":"2015-12-02T21:15:58.286771226Z"}
{"log":"[monitor] 2015/12/02 21:16:08 failed to store statistics: timeout\n","stream":"stderr","time":"2015-12-02T21:16:08.288266584Z"}
{"log":"[monitor] 2015/12/02 21:16:18 failed to store statistics: timeout\n","stream":"stderr","time":"2015-12-02T21:16:18.288594376Z"}
{"log":"[monitor] 2015/12/02 21:16:28 failed to store statistics: timeout\n","stream":"stderr","time":"2015-12-02T21:16:28.28714024Z"}

Restarting InfluxDB resolves it, though it takes some time to restart while logging about writing from WAL.  I am running InfluxDB in a container as non-root, but I'm not applying any limits.
# ps awwux | grep '[0-9] /usr/bin/influx[d]'
904       9823 50.9 11.5 3992448 468408 ?      Sl   00:13  15:54 /usr/bin/influxd -config=/etc/influxdb/influxdb.conf -hostname=<IP_ADDRESS>:8088
# cat /proc/9823/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             524288               1048576              processes
Max open files            524288               1048576              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       31449                31449                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Which resources can cause this?

Is this duplicate of 4121?

@CVTJNII looks like it

Does anyone know if this issue been fixed and if so where the relevant patch / fix details are ?

Any updates on this?


fix: pruning by bloom filters for dictionary columns

Which issue does this PR close?

Closes #13574 .
Rationale for this change

Currently row group pruning by Bloom filters is unaware of Dictionary typed scalar values (the literal part of predicate) while checking for values being contained in filter. This PR adds limited support for them  -- now Bloom filter pruning is able to handle majority of Dictionary scalars (with the exception of Decimal type).
What changes are included in this PR?

Part of BloomFilterStatistics responsible for checking scalar value against Sbbf is moved into helper function with additional support of Dictionary() and LargeUtf8/LargeBinary type.
Are these changes tested?

There are tests for supported datatypes checking pruning and matching (to ensure that there won't be any excessive pruning) cases.
Are there any user-facing changes?

No

The reason for not including all already supported ParquetExecBuilder::build` is not able to produce pruning predicate for Dictionary(numeric/binary) datatypes, so these queries won't normally reach bloom filter/statistics pruning.

Is this true? So Dictionary columns are incompatible with predicate pruning based on stats as well?

Looks very nice to me thank you @korowa !

Is this true? So Dictionary columns are incompatible with predicate pruning based on stats as well?

No, it's not, it was my mistake -- it works fine if literal is explicitly casted to column type.

I've added more tests along with types support, and updated PR description.
There are two issues I'm going to debug and create tickets or fix after getting more understanding of what's wrong there (they don't affect current implementation):

Decimal type -- there is something wrong with both statistics pruning and bloom filters (currently working on a reproducer)
Writing Dictionary(Float) panics in arrow writer, so there is no test for Float there

Thank you all for working on this. DataFusion is restoring my belief in open source one interaction at a time.


Expressions - another attempt

I'd just like to make another case for the expressions syntax, and to see if a simpler version of the proposal I'd previous put together might be acceptable
The current syntax isn't sufficient for new libraries
Great Tables (a new project by Posit) has recently introduced native Polars support - the way they did it really caught my attention. They didn't repeat their pandas support but with Polars functions - instead, they really make expressions a main focus: https://posit-dev.github.io/great-tables/blog/polars-styling/ . The whole thing's worth reading, but I really want to draw your attention to

As it turns out, polars expressions make styling tables very straightforward. The same polars code that you would use to select or filter combines with Great Tables to highlight, circle, or bolden text.
In this post, I’ll show how Great Tables uses polars expressions to make delightful tables, like the one below.

I was expecting this to happen, and I expect it'll happen a whole load more. If new libraries lean in to the expressions syntax, then the Standard will be dead on arrival.
If we want to promote the Standard, we need to keep up with the times. This requires extra work, but so does anything worthwhile.
The current rules break method chaining
Let's take the following:

join lineitem and supplier on 'a' (left join)
we only keep rows where column 'a' plus column 'b' is greater than 0
double the value of column 'a' and only keep that and column 'd'

You might expect to be able to do this with:
(
    lineitem.join(supplier, on="a", how="left")
    .filter((lineitem.col("a") + lineitem.col("b")) > 0)
    .assign(lineitem.col("a") * 2)
    .select("a", "d")
)

However, it will raise, because lineitem.col('a') was derived from a different dataframe than lineitem.join(supplier, on='a', how='left'), and that's not allowed. (yes, I'm aware that you can workaround this with temporary variables, but my point is: method chaining is very popular among dataframe users and devs - are we sure we don't want to support it?).
With expressions, though, there's no issue:
(
    lineitem.join(supplier, on="a", how="left")
    .filter((pdx.col("a") + pdx.col("b")) > 0)
    .select(pdx.col("a") * 2, pdx.col("d"))
)

You also don't need the extra assign statement
It's not a zero-cost abstraction
The current syntax is also not a zero-cost abstraction on Polars - trying to use only two objects (Column, DataFrame) to represent four (Series, Expr, DataFrame, LazyFrame) means that the resulting code isn't going to be as efficient as it could be:
df: pl.DataFrame
df.filter((df['a']+df['b'])>0)

is less efficient than
df: pl.DataFrame
df.filter((pl.col('a')+pl.col('b'))>0)

and the current API, in the persisted case, resolves to the first one. I don't see a way of this unfortunately.
Telling people "you can use the standard if you want, but it'll be more efficient to use the Polars API directly" is a recipe for people just using Polars and forgetting about the Standard. I'm calling it.
The way forwards
We don't necessarily need to separate DataFrame from LazyFrame. But I'm once again making the case for Expr being separate from Column.
@shwina @kkraus14 if I made a simpler version of this summer's proposal, would you be open to reconsidering this? I'm tagging you two specifically because, as far as I remember, everyone else was positive about it.
Alternatives
We need to do something here, I don't want my name on standard which is just a "pandas minus"

I agree with you: column expressions feel very natural to use here - although I maintain they are seriously limiting when you want to operate with columns across tables.
I also agree that the direction the standard is evolving is making it increasingly difficult and awkward to use.
I don't wish to be the sole voice of dissent here. If the rest of the team feels column expressions are the way to go, I'm on board. I just observe that the way things are going is going, the standard is focused now more on making it easy and efficient for lazy APIs like polars to adopt it, and increasingly more difficult for eager APIs like cuDF to do so. I'm really just hoping we can borrow/steal most of what pandas builds here to support the standard.

I don't agree that this part is true though:

You might expect to be able to do this with
(
lineitem.join(supplier, on="a", how="left")
.filter((lineitem.col("a") + lineitem.col("b")) > 0)
.assign(lineitem.col("a") * 2)
.select("a", "d")
)

In pandas, I'd use something like .pipe() - which does allow method chaining here.

A Standard which doesn't allow for maximally efficient Polars code will end up being just another abandoned github project which nobody will use
If this means we have to put it some extra work, then we put in some extra work

I'm really just hoping we can borrow/steal most of what pandas builds here to support the standard.

Assuming you implement all the Column methods, then this will be relatively straightforward. You'd just need to add some definitions which defer to the respective Column method, e.g.
    def __add__(self, other: Expr | Any) -> Expr:
        if isinstance(other, Expr):
            return Expr(lambda df: self.call(df) + other.call(df))
        return Expr(lambda df: self.call(df) + other)

I agree that using namespace level column expressions as opposed to dataframe specific column expressions for interacting with intermediate tables is a much cleaner experience. My concern has always been that it's an implementation change for many libraries and we had previously pushed back against doing that where the goal was to be compatible with all relevant DataFrame libraries as they stood.
The gaps for these expressions that I'm aware of are that the Pandas implementation is a WIP, Dask doesn't support them, and cuDF doesn't support them.

Telling people "you can use the standard if you want, but it'll be more efficient to use the Polars API directly" is a recipe for people just using Polars and forgetting about the Standard. I'm calling it.

A Standard which doesn't allow for maximally efficient Polars code will end up being just another abandoned github project which nobody will use

I 100% disagree with these statements and they're quite frustrating to read. Polars is one dataframe project out of many. You could make the same argument for every dataframe library that is trying to be supported by this standard and it's not possible to have maximally efficient code for every library due to API and design differences between them. Some food for thought from pypi stats:

Polars is downloaded ~3.4MM times a month
Dask is downloaded ~5.7MM times a month
PySpark is downloaded ~28.8MM times a month
Pandas is downloaded ~149.7MM times a month

I'm sorry to have frustrated you
Let's go over which projects have shown interest in the Standard:

scikit-learn: Polars and pandas are currently the only two dataframe libraries they support in set_output
skrub: primarily interested in Polars support
scikit-lego: primarily interested in Polars support
temporian: primarily interested in Polars support
buckeroo: primarily interested in Polars support

I'm not aware of libraries showing interest in the Standard without being primarily based on interest in Polars. Counterexamples are welcome, happy to admit I'm wrong here.
The standard being a zero-cost abstraction on all major dataframe libraries including Polars is a necessary condition for adoption. I'm not interested in authoring a "pandas minus" standard.
Seeing as you bring download stats up, you may be interested in looking at these numbers over time:

The standard being a zero-cost abstraction on all major dataframe libraries including Polars is a necessary condition for adoption.

Based on the discussion and iteration thus far I'm not sure if this is possible. There's fundamental design differences between the different dataframe libraries (lazy vs eager, cpu vs gpu, supported APIs, etc.) that may be impossible to make zero-cost for all libraries.

I'm not interested in authoring a "pandas minus" standard.

Neither am I, but I am interested in authoring something that as many dataframe libraries as possible, both current and future, can adopt as easily as possible.

Let's go over which projects have shown interest in the Standard:

scikit-learn: Polars and pandas are currently the only two dataframe libraries they support in set_output
skrub: primarily interested in Polars support
scikit-lego: primarily interested in Polars support
temporian: primarily interested in Polars support
buckeroo: primarily interested in Polars support

scikit-learn: https://github.com/scikit-learn/scikit-learn/issues/25896

"Support other dataframes like polars and pyarrow not just pandas"

skrub: https://github.com/skrub-data/skrub/discussions/719

"However, with the rise of polars, duckdb, and many more, we have to wonder which modules to support."

scikit-lego: https://github.com/koaning/scikit-lego/pull/597

I can't dig up the Discord conversation, but seems like you drove this work yourself?

temporian:

Can't find anything in the repo related to the dataframe standard or polars, only a notebook comparing their performance to Polars and Pandas: https://github.com/tryolabs/temporian-examples/blob/main/examples/grouped_simple_moving_average.ipynb

buckeroo: https://github.com/paddymul/buckaroo/issues/75

Nothing related to adoption of the standard or even a desire to adopt the standard other than using the work here as inspiration for a JavaScript dataframe API. Also seems like you drive this work yourself based on the comment from the author saying they connected with you.

I don't think anyone is arguing that Polars is a major dataframe library and we should strive to make sure that the work we do in designing a new API is as friendly to it as possible, but from the public discussions it seems like it's always pointing towards wanting to support dataframe libraries more generally and the places where it's primarily Polars has been after connecting with you in Discord or elsewhere?
If we don't wish to more or less treat different dataframe libraries on equal footing then I don't see how my continued involvement in this work is an effective use of my time or efforts. I'll happily step away from this effort and evaluate it when a specification is published at some point in the future.

If you want efficient Polars code, you write expressions, see the user guide
Expressions detract nothing from pandas/cudf and enable things for polars (and probably pyspark too)
If we truly want to treat different dataframe libraries on equal footing, we introduce expressions as a distinct concept from columns

Hi all, this is an important discussion. I'll add a few thought on this now. We should aim to have a higher-bandwidth conversation about it this week I think.
Some of my thoughts on the topic and conversation above:

No one wants to simply copy pandas or a "pandas minus", that's been clear from the start (all the way back to gh-4).
The download stats are probably more distracting than helpful. There are a number of widely used libraries with different APIs and internals; some are currently more popular, some have more momentum, etc. I think we'd like a design that can work well for as many of them as possible.
Regarding overhead, the statements above mostly seem a bit too absolute. This is the situation I believe:

(a) on the one hand it's clearly not possible to be strictly "maximally efficient" or "zero overhead", since anywhere a new layer is introduced to bridge differences in APIs or semantics, there will always be at least some extra method call or dispatch overhead.
(b) on the other hand, if the overhead is not minor at the application level, but leads to different performance scaling compared to the native performance of the implementing library, or makes certain code multiple times slower by preventing the library from optimizations that it would otherwise be able to do, then that clearly is a problem.
(c) we need something that is a clear win for dataframe-consuming code authors. If see a single code path with only negligible/minor performance overhead, and they can avoid two or more separate code paths to support multiple dataframe libraries - great. If they'd vote with their feet to implement separate code paths anyway, then we've done something not so useful.

My interpretation of some of the discussion above is that @MarcoGorelli is worried that the current design is situation 3(b) for Polars (and that not being the case for any library if we'd go with expressions), while @kkraus14 is emphasizing point 3(a).

I also agree that the direction the standard is evolving is making it increasingly difficult and awkward to use.

That is a concern indeed, the rules for what can and cannot be done with columns seem to get more awkward as we learn more.

I agree that using namespace level column expressions as opposed to dataframe specific column expressions for interacting with intermediate tables is a much cleaner experience. My concern has always been that it's an implementation change for many libraries and we had previously pushed back against doing that where the goal was to be compatible with all relevant DataFrame libraries as they stood.

This is the important part in a nutshell I think. It seems everyone is in agreement that expressions are a nicer design (glossing over the details here). It's the implications for libraries that are the concern. The question I have is how real a problem that really is, given that:

It's been clear for quite a while that this API is not going to end up as the main user-facing API in the main namespace of each library (that would either mean large bc-breaking changes or be a "pandas minus", and those have been ruled out).
If it's a separate API/namespace, I think Marco is making the argument that it's actually pretty easy to implement as thin wrappers around the current API, and he's willing to do a lot of the work (partially already done during prototyping).

Re the "compatible with all relevant DataFrame libraries", that's not quite accurate, since there is no common denominator that can be targeted. "easy to implement for all relevant libraries" is more the point I think.

The gaps for these expressions that I'm aware of are that the Pandas implementation is a WIP, Dask doesn't support them, and cuDF doesn't support them.

I think "the Pandas implementation" doesn't exist; @jorisvandenbossche did a prototype demo last year, and @MarcoGorelli did some prototyping more recently. However, I'd also say that that's probably a chicken-and-egg type of problem - it will never exist if it's not worked on because it doesn't yet exist. The more interesting questions are whether pandas & other libraries would add it, and how much work it would be. Both are relevant questions; even if it's easy to implement, it's possible maintainers of a library would react like "expressions are a concept that make our library harder to understand without having any added value, and hence we don't like it". On the other hand, if it does lead to a nicer API or new performance optimization opportunities, or something else like that, it's a much better story. So that's probably something to get a wider range of opinions on.

Right - new week, new discussions. Let's try to move things forwards rather than giving up 💪 Thanks Ralf for your comments.
First, let's clear up what we're talking about. An expression is a function from a dataframe to a column. I've heard people here refer to them as "lazy columns", but I don't think that's right. We wouldn't call
def add_one(v: int) -> int:
    return v+1

a lazy integer. It's a function. Likewise, an expression is a function. A Callable[[DataFrame], Column].
What's the effort involved?
Expr class
Here's an example of how to define it:
def col(self, column_name: str) -> Expr:
    return Expr.from_column_name(column_name)

class Expr:
    def __init__(self, call: Callable[[DataFrame], Column]) -> None:
        self.call = call

    @classmethod
    def from_column_name(cls: type[Expr], column_name: str) -> Expr:
        return cls(
            lambda df: Column(
                df.dataframe.loc[:, column_name],
                api_version=df._api_version,
            ),
        )

Then, for each method you want to define, you just defer to the Column implementation. For example, for __gt__:
    def __ge__(self, other: Expr | AnyScalar) -> Expr:
        if isinstance(other, Expr):
            return Expr(lambda df: self.call(df).__ge__(other.call(df)))
        return Expr(lambda df: self.call(df).__ge__(other))

It's a bit laborious, but nothing hard. It can be done in an afternoon whilst listening to a good podcast.
Expr-consuming function
Let's use select as an example. Suppose you currently have it as:
    def select(self, *columns: str) -> DataFrame:
        if not columns:
            msg = "must select at least one column"
            raise ValueError(msg)
        new_cols = [
            self.dataframe.loc[:, col]
            for col in columns
        ]
        df = pd.concat(new_cols, axis=1, copy=False)
        return self._from_dataframe(df)

Here's the change you'd need to make in order for it to take expressions (instead of just strings):
-    def select(self, *columns: str) -> DataFrame:
+    def select(self, *columns: str | Expr) -> DataFrame:
         if not columns:
             msg = "must select at least one column"
             raise ValueError(msg)
         new_cols = [
-            self.dataframe.loc[:, col]
+            col.call(self).column if isinstance(col, Expr) else self.dataframe.loc[:, col]
             for col in columns
         ]
         df = pd.concat(new_cols, axis=1, copy=False)

Note that this is a very simplified version of Polars expressions (which can have multiple roots and multiple outputs). I'm not suggesting that everyone has to reimplement Polars. I'm only suggesting a simplified definition of "expression" which, if limited to the standard's API, is pretty simple to implement. There are some subtleties to discuss, but I'm purposefully leaving them out of the present discussion.
The purpose of this post is to check whether people would at least be open to discussing it. Based on comments above, the answer appears "no", but the benefits of what I'm suggest seem clear enough to me that I won't be deterred.
Simplifications
The current design is pretty awkward because we're trying to squash different concepts into just Column.
Personally, I'd be happy to get rid of a lot of the current awkwardness, such as "you can't compare columns which originated from different dataframes", if we could just separate the concepts.
It would be a lot simpler to teach then. If it's simpler to teach, it's more appealing to developers, and ends up benefitting the ecosystem more.

Hi all,
I've decided to leave the project. This isn't a decision taken lightly, I've been considering it for some time - after yesterday's call, I slept on it, and finally have resolved to move on. Unfortunately I'm past my tipping point
This is nothing personal. I'm just physically and mentally unable to continue.
I wish you all the best and hope we can stay on good terms


Is the package really needed?

Flutter already provides State Restoration.
http://flutter.dev/go/state-restoration-design
What features this package has over built-in state restoration mechanism? If this package has some features that's missing in Flutter's one, then mentioning it in your docs should help package users.
Great stuff though 👍

flutter team still working on state-restoration,i think this library can help to deal with part of process death.

@sarbagyastha , Simply this will be useful for flutter user that user version <1.22 which do not support state restoration.

@sarbagyastha , Simply this will be useful for flutter users that use version <1.22 which do not support state restoration.

Hello, I am quite new to flatter, dart and BLoC but ate some of those concepts a litte. Found your project @salim-lachdhaf and was about to ask the same question as in this issue.
As I see your response I have two other questions. Would be more than happy to get answers from You :)

Why flutter users should stay on flutter < 1.22? Is that point still releavant in 2022?
What are cons & pros to use yours not hydrated_bloc, isnt it preserving the state - have you considered wrapping it or do something more agnostic with use of other packages usage?

Happy day I wish you :)

@salim-lachdhaf  is this alive as you not answered for 2 months 🙂


bluebird to Promise in stack

There was a mistake in naming convention as we were returning from naming bluebird-ish promise's just 'bluebird' to Promise once again.
So sorry.

I know I should have not merge it by myself but there was an open issue for this fix so I wanted to fix it ASAP.
Sorry if that will disturb you.

np, thanks.
published


Non-blocking token request method

I have a project in which the ESP32 can eventually be configured on a Wi-Fi network without internet access (and it's fine for my application);
But, when using Firebase-ESP-Client, the tokenProcessingTask() will hangs the execution of the entire application because it has a while() that infinitely tries to obtain the token.
My suggestion is to add an option like "config.abort_connection_if_token_fail = true" and "config.token_request_attempts = 5" (examples only), to allow use cases like mine to continue working (even if it means being without Firebase services available).
I will modify for my own use, but the official solution may help others.

It is about http request-response process which of course it is blocking for a while which is actually done once at a time.
The process utilized the states in a predetermined condition loop instead of infinite loop as you may feel that is why the callback is presented.
The time out is not necessary as long as you can control the execution of Firebase.ready().
FYi, this is not async operation library, the time out was done internally with TCP client and you can change it (see basic example comment).

The Async Firebase Client library is under development.


Wrong direction of focus navigation.

Describe the bug
I have a situation where I need to display the button only after some loading logic. And after I implemented a simple piece of code I observed weird behavior.
The first button that I need to show appears in the correct place conditionally, but the focus direction is not correct.
In the video, you can see how the first button is accessible through navigation by buttons on the remote.
Assumption
I guess this happens when spatialNavigator.registerNode, gets children from DOM where there is no first button at that moment. After the button appears it registers it as a latest element which we could see on focus behavior. The button is accessible as the last right (in my case) element.
To Reproduce
You need to conditionally show SpatialNavigationNode.
<SpatialNavigationView direction="horizontal">
  {!isLoading && (
    <SpatialNavigationFocusableView>
      {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >First button</Text>
      )}
    </SpatialNavigationFocusableView>
  )}

  <SpatialNavigationFocusableView>
    {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >Second button</Text>
      )}
  </SpatialNavigationFocusableView>

  <SpatialNavigationFocusableView>
    {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >Third button</Text>
      )}
  </SpatialNavigationFocusableView>
</SpatialNavigationView>

Expected behavior
The focus direction works correctly, and the first button (in my case) will be accessible not as the last right but as the first left focus element.
Comment
Saw a comment in the code:
react-tv-space-navigation/src/spatial-navigation/components/Node.tsx:118

/*
* We don't re-register in LRUD on each render, because LRUD does not allow updating the nodes.
* Therefore, the SpatialNavigator Node callbacks are registered at 1st render but can change (ie. if props change) afterwards.
* Since we want the functions to always be up to date, we use a reference to them.
*/

As far as I understand, the LRUD library doesn't support updating, and to update navigation, we need to update the whole navigation with all children which could be an expensive operation.
In case this behavior is really a bug I'm ready to help and contribute to fixing it, but maybe I need some guidance.
Screenshots

Version and OS

Library version: 3.3.0
React Native version: 0.73.6
OS: Android API 34

Additional context
No additional context.

Found a workaround using the index property from the LRUD library.
LRUD /docs/usage.md#index
Need to add a new prop to SpatialNode called index in /src/spatial-navigation/components/Node.tsx and update spatialNavigator.registerNode to something like that:
spatialNavigator.registerNode(id, {
  parent: parentId,
  index,
  isFocusable,
  onBlur: () => {
    currentOnBlur.current?.();
    setIsFocused(false);
  },
  onFocus: () => {
    currentOnFocus.current?.();
    setIsFocused(true);
  },
  onSelect: () => currentOnSelect.current?.(),
  orientation,
  isIndexAlign: alignInGrid,
  indexRange,
  onActive: () => setIsActive(true),
  onInactive: () => setIsActive(false),
});

After this modifications we'll be able to pass index prop to SpatialNode like this:
<SpatialNavigationView direction="horizontal">
  {!isLoading && (
    <SpatialNavigationFocusableView index={0} alignInGrid={true}>
      {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >First button</Text>
      )}
    </SpatialNavigationFocusableView>
  )}

  <SpatialNavigationFocusableView index={1} alignInGrid={true}>
    {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >Second button</Text>
      )}
  </SpatialNavigationFocusableView>

  <SpatialNavigationFocusableView index={2} alignInGrid={true}>
    {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >Third button</Text>
      )}
  </SpatialNavigationFocusableView>
</SpatialNavigationView>

It works perfectly.
But as we need isIndexAlign set to true, that this feature will work I can assume that will somehow conflict with alignInGrid logic.

Hey! Sorry that I didn't answer quicker 😁
This is actually a very classic issue with the lib. While the index will work, it's quite annoying and not very scalable.
The best solution (in my opinion), is to add SpatialNavigationNode that are not conditionally rendered.
Your component inside can be conditionally rendered (and can have its own Nodes/FocusableViews).
This is all documented here 😊
https://github.com/bamlab/react-tv-space-navigation/blob/main/docs/pitfalls.md
Here's the fix for you:
<SpatialNavigationView direction="horizontal">
+ <SpatialNavigationNode>
  {!isLoading && (
    <SpatialNavigationFocusableView>
      {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >First button</Text>
      )}
    </SpatialNavigationFocusableView>
  )}
+ </SpatialNavigationNode>

  <SpatialNavigationFocusableView>
    {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >Second button</Text>
      )}
  </SpatialNavigationFocusableView>

  <SpatialNavigationFocusableView>
    {({ isFocused }) => (
        <Text
          style={{
            color: 'white',
            backgroundColor: isFocused ? 'tomato' : 'green',
          }}
        >Third button</Text>
      )}
  </SpatialNavigationFocusableView>
</SpatialNavigationView>

This is one very faulty abstraction with the library but I don't know how to make it clearer (or solve it completely) 😭
I wish we could just do it the natural way without the trick of fake nodes.

Also, the proposed workaround works perfectly, but assume that you not only want to show loading but to show node only on success response. In this case, we'll have an empty Node in case of failure response.

Also, the proposed workaround works perfectly, but assume that you not only want to show loading but to show node only on success response. In this case, we'll have an empty Node in case of a failure response.

Yes, which is totally fine... Because it is non focusable 😁
You can try out this case, it works perfectly 😊 we've been using this pattern a lot without any problems.

I added an example in the app to showcase the solution in a more visible way: https://github.com/bamlab/react-tv-space-navigation/pull/121


SUGGESTION : Add some get started tutorial

the project as is is confusing to get used to, a simple doc / tutorial would be appreciated

I know and there are other things to be done as well. Time is all I need, as I am swamped with my current commercial project.
Thanks for the feedback anyway...


Spanner: Example how to decode ARRAY[STRUCT]

I have tried to decode it with
var arr []*Struct

But it won't work.

Got it working.


IMDB Box Office Prediction

Good Morning, I'm a GSSOC'22 participant,
May I know details about the project please?

Afternoon @dona-samir , you have to do box office revenue collection prediction, you have to do ML to present the best model. Please make sure to go through with all processes(data preprocessing to modes)

@dona-samir comment if you want to work on this issue.

I am very interested to contribute in this issue, Would you please provide me with the dataset, and the details required?

https://www.kaggle.com/c/box-office-revenue-prediction
@dona-samir you can use this dataset

I am very interested to contribute in this issue, please assign this repo so that  I can start working ASAP


[Sema] Improve diagnoses on args to generic members

In both figuring out candidate closeness and in diagnosing generic parameter errors, if the parameter is a GenericTypeParamType, get its decl’s archetype to perform archetype substitutability checking upon.
This extends https://github.com/apple/swift/pull/1089 to apply to generic member methods to act the same as with the previous generic free functions.
Added tests, all tests pass.

Very nice!


Make Content-Type json validation less rigid so it works with things like application/manifest+json (Fixes #983)

What is it?

[ ] Feature / enhancement
[x] Bug
[ ] Docs / tests

Description
There are Content-Types that are variations of application/json, like application/manifest+json, which is needed to output a route like /manifest.webmanifest.ts. This PR make it less rigid so it works with those variations.
Use cases and why

One use case

Using routes/manifest.webmanifest.ts needs to print  application/manifest+json.

Another use case

application/ld+json and other variations.
Checklist:

[x] My code follows the developer guidelines of this project
[x] I have performed a self-review of my own code
[x] I have made corresponding changes to the documentation
[x] Added new tests to cover the fix / functionality

This doesn't fix it, my bad

great idea, thanks!


misc: Expose all KV commands in TiKV-Details Grafana Dashboard

Signed-off-by: Breezewish<EMAIL_ADDRESS>
What problem does this PR solve?
Problem Summary:
We frequently meet issues related to KV commands other than currently exposed ones, like KvGet, KvBatchGet, etc. Now the detail of these KV commands cannot be viewed directly.
What is changed and how it works?
What's Changed:
Expose all KV commands.
Related changes

Need to cherry-pick to the release branch

Commands like KvGet and KvBatchGet is only frequently used in v4.0+, so cherry-picking to 4.0 is enough.
Check List
Tests

Manual test (add detailed scripts or steps below)

Release note
Expose details of all KV commands in Grafana dashboard.

/run-all-tests

/test

LGTM

/merge

/test

/test

/test


fix(open in codesandbox button): optimize for long sandoxes

This introduces a breaking change due to useCodeSandboxLink is no longer used to generate the URL. From now on, the BareCodeSandboxButton is responsible for managing the URL or the form that will create a new sandbox in the CodeSandbox.
In the previous approach, all the parameters to create a sandbox in the CodeSandbox were attached directly in the URL, making it easily reach the request too long (401). That's why we deprecated the hook in favor of using a whole new component, which is responsible for either using an anchor (for short sandboxes) or a form-POST (for long sandboxes).
The BareCodeSandboxButton is a basic component that doesn't carry any style, so if you want a ready-to-use component, use the OpenInCodeSandboxButton instead, which contains the CodeSandbox logo and the same functionality.

Migration guide:
import {
--  useCodeSandboxLink,
++  BareCodeSandboxButton
}
const CustomOpenInCSB = () => {
--  const url = useCodeSandboxLink();
--  return <a href={url}Open in CodeSandbox
++  return <BareCodeSandboxButton>pen in CodeSandbox</BareCodeSandboxButton>
   );
};

CSB-519

It seems the export for useCodeSandboxLink in 'sandpack-react/src/hooks/index.ts' needs to be removed

Hi @danilowoz, I just stumbled on this limitation (401 on long requests) and wanted to open an issue but than noticed this PR :)
Is this fix planned for a coming release?


Guard against Document without editions

Some Document instances only have superseded editions. We should ignore these for the purposes of republishing.

👍


Customize the look and feel

Hi,
How can I customise the controls to match my color scheme?
How can I make the material theme look like that of bootstrap theme wrt to the font size and control size?
Regards.

Also the size in general is very big. I tried to juggle a bit with CSS to decrease the size, but the date picker went worse than before...
Is there a recommended way to change the size?


Update markdown.ts to remove replace of &lt and &gt with < and >.

By removing the replace this the React output is correct and markdown containing < or > renders correctly.
Addresses https://github.com/geekris1/vite-plugin-react-markdown/issues/2#issue-1832121224

Keen for this to go! Will make things easier...

I need more tests, and I will finish it tomorrow, Beijing time.

I have fixed it, and you can pass the latest version 0.2.10.
https://github.com/geekris1/vite-plugin-react-markdown/commit/2267d95b99454b816952c3588c7c8b811884ad37


Unsafe cast is wrong

Steps to Reproduce (for bugs)
val l : List<String> = mutableListOf()
val l2  = l as MutableList<String>

Expected Behavior
No issue detected
Current Behavior
Detects unsafe cast
Your Environment

Version used: 1.0.0.RC8
Link to your project: https://github.com/loewenfels/dep-graph-releaser/blob/master/dep-graph-releaser-gui/src/main/kotlin/ch.loewenfels.depgraph/gui/actions/Releaser.kt#L162

There are more issues like the above, following a link to the sonarcloud.io project:
https://sonarcloud.io/project/issues?id=depgraph&resolved=false&rules=external_detekt%3AUnsafeCast

Ah... I see, I was expecting a different behaviour for the UnsafeCastRule but the reporting is actually as intended. I would consider to rename the rule, unsafe cast sounds like casting a type parameter to something we are not sure of.


FDroid

Please add the Android app to FDroid which is an alternative APK repository to Google Play.
Many Android devices do not have access to Google Play Store and choose to remove Google Play Services for privacy. Please allow any Android users to download the app and receives updates from FDroid.

Is there a way to add the APK here in the repo?

Let me look into it. @MalteKiefer

Thanks

@MalteKiefer sorry but we discussed this and the resources we have (minimal) and the risk don't line up well enough yet.  Please reach out later

the resources we have (minimal) and the risk don't line up well enough yet

@kcw-grunt what notable extra resources does it require to publish APKs? And what are the risks that you talk about?

Please, add some possibility to install the APK.  Or at least provide MD5 sums of an APK's. Because the absence of official way to install the APK without google play implies that the people download the APK from such ressources like apkpure.com
And this is much more insecure than even f-droid repo.


SkyLightStateless not working

My SkyLightStatelessComponent is  not showing even when the isVisible prop is set to true.
When debugging with react dev tools i could see that the isVisible prop of SkyLightStatelesscomponent changes but its isVisible state is not set accordingly.
My code
<SkyLightStateless isVisible={this.props.page_details.user_popup_requested} dialogStyles={myBigGreenDialog} onCloseClicked={() => {this.closePopUp()}}>
          <Loader loaded={this.props.page_details.popup_user_loaded}/>
          <If condition={this.props.page_details.popup_user_loaded}>
	        <UserPopupCard />
	  </If>
 </SkyLightStateless>

(https://cloud.githubusercontent.com/assets/15194796/22596077/27c6b2f0-ea50-11e6-9cfe-7ce3c689d82c.png)

SkyLightStateless also throwing these warnings in the console.
Warning: Unsupported style property margin-top. Did you mean marginTop? Check the render method of SkyLightStateless.
Warning: Unsupported style property margin-left. Did you mean marginLeft? Check the render method of SkyLightStateless.
Warning: Unsupported style property margin-bottom. Did you mean marginBottom? Check the render method of SkyLightStateless.
Warning: Unsupported style property padding-bottom. Did you mean paddingBottom? Check the render method of SkyLightStateless.


Judgement calls that need further UX research:

Is "decode" too techy of a word? I like it personally

=====
Is it too "surprising" to be launched into a keypad when you go to receive? I know it saves me a click but I keep being surprised by it for some reason idk
=====
should we put the settings button in the top bar?

=====
is it helpful to show this syncing indicator? not really obvious the what or the why of it... especially if you don't have anything pending. I like it but that's because I know what it's doing under the hood

=====
are default amount buttons useful outside of rapid testing? I know one reference we had was default zap amounts. could potentially customize...

Is it obvious that these are balances and not the amount you're about to pay?


[python] prevent overtainting dictLiterals

In #4583, I didn't account for how Python dictionary literals are lowered as 3 address code. Specifically, the problem is at inCall.assignment here:
https://github.com/joernio/joern/blob/ff9c08f62539eba4194eee5c0a96f5d28d586ab7/dataflowengineoss/src/main/scala/io/joern/dataflowengineoss/package.scala#L19
In an expression such as print(1, {'x': 10}), 1.inCall.assignment wouldn't be empty - it would contain the assignments that build {'x': 10}. The intent behind inCall.assignment was to filter the surrounding call by assignment. So, introduced a new CallTraversal isAssignment to do just that and used it here instead.
I'm a bit concerned about introducing a new API for CallTraversal. Any thoughts about that?
Resolves #4605

I'm a bit concerned about introducing a new API for CallTraversal. Any thoughts about that?

👍 from me on that. I'd use that API. (Unlike the existing inAssignment/assignment, for the same reasons why you had to move away from them here.) Only comment from me is whether it shouldn't filter by all the fullnames from https://github.com/joernio/joern/blob/15e241d3174ecba9e977a399793c9c6a1249d819/semanticcpg/src/main/scala/io/shiftleft/semanticcpg/language/operatorextension/package.scala#L24


Better Error Messages for Strings in Resize

Added Error Message "Invalid width [width].  Cannot be string."
Added Error Message "Invalid height [height].  Cannot be string."
This will make it easier for developers to identify when they have mis-cast their variables before sending them into sharp.resize()

Thank you! How about we consolidate this improvement with the existing Error and make it more positive, taking more of a "here's what you should do" approach rather than "here's one possible thing that won't work".
I'm thinking something along the lines of:
- Invalid height (1 to ' + this.constructor.maximum.height + ') ' + height
+ Expected integral Number for height between 1 and ' + this.constructor.maximum.height + ', received ' + height + ' (' + typeof height +')'

Coverage increased (+0.003%) to 99.493% when pulling b4de52f1435ab9418c0d8149afbb8c3942b10da5 on spikeon:patch-1 into 7231d92d1fff5216ba5b45792716f13066be0da0 on lovell:master.

Placing the term "Integral number" doesn't tell the user that they have sent the wrong data-type.  Perhaps instead of "Cannot be a string" and validating to see if it's a string, it should say: "Must be an Integral Number" if any non numeric datatype is sent into the function?
So, I'm all for the enhancement to the existing error, but there should also be an error specific to receiving an incorrect data type.

Perhaps ... it should say: "Must be an Integral Number" if any non numeric datatype is sent into the function?

Agreed, this sounds like what I was thinking too.
How about we externalise the responsibility for creating the Error and its message to a separate, generic function, something like:
const invalidParameterError = function (name, expectedDescription, actual) {
  return new Error(
    `Expected ${expectedDescription} for ${name} but received ${actual} of type ${typeof actual}`
  );
};

which could then be used by the resize() function via something like:
throw invalidParameterError('height', `integral Number between 1 and ${maximum.height}`, height);

That... looks just about perfect.  It tells the developer specifically what's wrong, as well as exactly what's expected.

I've added this as part of commit 70a3d4f - thanks for your work on this.


[BUG] Fix GraalVM issue with Application Insights for Spring native

Reproducer: https://github.com/Azure-Samples/java-native-telemetry
Version:
<dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter-monitor</artifactId>
    <version>1.0.0-beta.5</version>
</dependency>

--initialize-at-build-time=com.azure.core.util.logging.ClientLogger has to be added for GraalVM native compilation

The error with class initialization message:
com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
 com.azure.core.util.logging.ClientLogger was unintentionally initialized at build time. com.azure.core.util.Configuration caused initialization of this class with the following trace:
        at com.azure.core.util.logging.ClientLogger.<clinit>(ClientLogger.java:61)
        at com.azure.core.util.Configuration.<clinit>(Configuration.java:258)

com.azure.core.util.Configuration is initialized at build time: https://github.com/Azure/azure-sdk-for-java/blob/33b5a31328e84f3177293a3906bc829f696ab059/sdk/core/azure-core/src/main/resources/META-INF/native-image/com.azure/azure-core/native-image.properties#L21
ClientLogger is then initialized by this static field: https://github.com/Azure/azure-sdk-for-java/blob/33b5a31328e84f3177293a3906bc829f696ab059/sdk/core/azure-core/src/main/java/com/azure/core/util/Configuration.java#L258


Execute Job API by name

As a Network Automation Engineer, want to execute Nautobot Jobs in the API, referenced by their name, so that I can use the most natural identifier, instead of having to know (or first obtain) the job's UUID, or class path.
I will know this is done when the API and its schema allow accessing and executing a job by reference of the job's name alone. This should now be achievable since Job.name is unique: /api/extras/jobs/{name}/
The 2.0 schema those these routes:
Private Zenhub Image
1.6 shows these routes, notably with class path being deprecated:
Private Zenhub Image

Hey team! Please add your planning poker estimate with Zenhub @glennmatthews @gsnider2195 @HanlinMiao @jathanism @timizuoebideri1

For reference: https://www.django-rest-framework.org/api-guide/generic-views/#attributes


Support for the nodejs_package annotation

Summary of the change
The main.js generated file is modified to allow the initialisation of the thingmodel in an exported function. The most visible changes are the builder.append converted to main.append.
Then, the terminate code is put in a function terminate() so it can be exported or called in the SIGINT handler.
The SIGINT handler is modified to check whether it can run (it can't on web browsers with browersify) and it is present only when the nodejs_package annotation is not present.
Finally, if the configuration contains the nodejs_package flag/annotation, an object is returned with an object of instances and the terminate function.
This should work okay.

About the whitespaces changes, I don't even know if I pushed tabs instead of spaces or vis-versa. This is annoying.


Installation error

Hi,
Thank you for your contribution with SViper, we are trying to apply it to polish structural variants called using ONT in humans.
I'm encountering an error trying to install SViper in a slurm/lustre cluster system. Note that my GCC version (6.3.0) is not the recommended, but I do not have access to another. Could that be the cause of the problem? I'd welcome any input to try to fix the issue.
Attached is the full error report:
[aserres@login1 build]$ cmake ../
-- The CXX compiler identification is GNU 6.3.0
-- Check for working CXX compiler: /apps/GCC/6.3.0/bin/c++
-- Check for working CXX compiler: /apps/GCC/6.3.0/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
source dir: /scratch/devel/aserres/bin/SViper
-- These dependencies were found:
OPENMP     TRUE      -fopenmp
ZLIB       TRUE      1.2.3
SEQAN      TRUE      2.4.0
-- The requirements were met.
-- Configuring done
-- Generating done
-- Build files have been written to: /scratch/devel/aserres/bin/SViper/build
[aserres@login1 build]$ make sviper
Scanning dependencies of target sviper
[ 50%] Building CXX object CMakeFiles/sviper.dir/src/sviper.cpp.o
In file included from /scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/basic.h:101:0,
from /scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/bam_io.h:46,
from /scratch/devel/aserres/bin/SViper/include/sviper/basics.h:9,
from /scratch/devel/aserres/bin/SViper/include/sviper/auxiliary.h:2,
from /scratch/devel/aserres/bin/SViper/include/sviper/io.h:7,
from /scratch/devel/aserres/bin/SViper/src/sviper.cpp:1:
/scratch/devel/aserres/bin/SViper/include/sviper/sviper.h: In function ‘bool sviper::polish_variant(sviper::Variant&, sviper::input_output_information&)’:
/scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/basic/basic_tangle.h:43:15: error: ‘viewRecords’ is not a member of ‘seqan’
#define seqan seqan
^
/scratch/devel/aserres/bin/SViper/include/sviper/sviper.h:221:9: note: in expansion of macro ‘seqan’
seqan::viewRecords(long_reads, long_read_bam, info.long_read_bai, rID_long, var_ref_pos_sub50, var_ref_pos_add50);
^
/scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/basic/basic_tangle.h:43:15: error: ‘viewRecords’ is not a member of ‘seqan’
#define seqan seqan
^
/scratch/devel/aserres/bin/SViper/include/sviper/sviper.h:223:9: note: in expansion of macro ‘seqan’
seqan::viewRecords(long_reads, long_read_bam, info.long_read_bai, rID_long, var_ref_pos_end_sub50, var_ref_pos_end_add50);
^
/scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/basic/basic_tangle.h:43:15: error: ‘viewRecords’ is not a member of ‘seqan’
#define seqan seqan
^
/scratch/devel/aserres/bin/SViper/include/sviper/sviper.h:379:17: note: in expansion of macro ‘seqan’
seqan::viewRecords(short_reads, short_read_bam, info.short_read_bai, rID_short, ref_region_start, e);
^
/scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/basic/basic_tangle.h:43:15: error: ‘viewRecords’ is not a member of ‘seqan’
#define seqan seqan
^
/scratch/devel/aserres/bin/SViper/include/sviper/sviper.h:385:17: note: in expansion of macro ‘seqan’
seqan::viewRecords(tmp_short_reads, short_read_bam, info.short_read_bai, rID_short, s, ref_region_end);
^
/scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/basic/basic_tangle.h:43:15: error: ‘viewRecords’ is not a member of ‘seqan’
#define seqan seqan
^
/scratch/devel/aserres/bin/SViper/include/sviper/sviper.h:392:17: note: in expansion of macro ‘seqan’
seqan::viewRecords(short_reads, short_read_bam, info.short_read_bai, rID_short, ref_region_start, ref_region_end);
^
/scratch/devel/aserres/bin/SViper/extern/seqan/include/seqan/basic/basic_tangle.h:43:15: error: ‘ConvexGaps’ is not a member of ‘seqan’
#define seqan seqan
^
/scratch/devel/aserres/bin/SViper/include/sviper/sviper.h:460:28: note: in expansion of macro ‘seqan’
seqan::ConvexGaps());
^
make[3]: *** [CMakeFiles/sviper.dir/src/sviper.cpp.o] Error 1
make[2]: *** [CMakeFiles/sviper.dir/all] Error 2
make[1]: *** [CMakeFiles/sviper.dir/rule] Error 2
make: *** [sviper] Error 2

Hi, I'm having the same installation problem as you. If you have solved the problem, can you tell me how to do it? Thanks!

Hi @aserres and @Anzy-99,
I'm very sorry to get back to you so late but I was on maternity leave this year until a few weeks ago.
Let's trouble shoot.
First: When cloning SViper, did you use the git option --recurse-submodules?
If not or if you are not sure. Go to the SViper directory and type
git submodule update --init

Remove everything in the build directory and try using cmake and make again.

For me works these way:
git clone https://github.com/smehringer/SViper.git
micromamba install -c bioconda seqan
micromamba install -c conda-forge cxx-compiler
micromamba install -c anaconda cmake
cd SViper
git submodule update --init
mkdir build
cd build
cmake ..
make sviper


load admin first

Fixes https://github.com/inventree/InvenTree/issues/3355
Fixes https://github.com/inventree/InvenTree/issues/3483

Placing 'django.contrib.admin' at the start of INSTALLED_APPS seems to address this issue, although it is a very unsatisfying result. Still no idea why the simpleactionplugin causes the admin page to error out.
@matmair any thoughts as to negative consequences of putting 'django.contrib.admin' at the start of the list?

@SchrodingersGat nothing negativ comes to mind. Plugins still work so should be fine.


Top-right corner not filled.

Hello and thanks for the repo. When changing the size of images from 320x240 to 128x128 or 256x256 in local_test.bash (I didn't make any other modifications to the code) some images appear with the top-right corner not filled (cf. attached example). I don't see this problem with images of size 320x240.
Do you have any idea what might be causing this ?
Thank you.

By changing the aspect ration, extra image space in top and bottom is revealed. The unfilled area is simple the edge of the photo backdrop and black void behind it (nothing else exists in the scene). If you want to render at a different aspect ratio, I would recommend moving the camera closer/zooming in. It can be done by modifying the basic scene file.
Alternatively, you could render at the same aspect ration (4:3) and just cut off left and right sides from images after the rendering is done. That is similar to what happens with ClevrTex and CLEVR benchmarks during data loading.
For purple objects, see Issue 3.

Thank you!


Library VS Framework

The docs refer to this as a library.  Here's where I see the distinction:
A library is something that your code calls. Your code is in charge of when and how the library code will be called. The library has little-to-no say over the structure of your code.
A framework is something that you put your code into. The framework is in charge of when and how your code will be called. The framework strongly influences the structure of your code.

Fixed!


expand comment text area

🙏
On Fri, 25 Nov 2022, 3:33 pm Feruz M, @.***> wrote:

Merged #1124 https://github.com/ecency/ecency-vision/pull/1124 into
development.
—
Reply to this email directly, view it on GitHub
https://github.com/ecency/ecency-vision/pull/1124#event-7893635571, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AVAB2UTZRZQXHP3IWCQ4J5TWKDE4FANCNFSM6AAAAAASLAM5KA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Add plugin ID as command parameter or as config within the Resources/store folder

If I wan't to use the plugin uploader for multiple plugins I have to edit the PLUGIN_ID in the .env file every time I swtich the plugin. I think the .env file is not the correct place for it and it would make more sense if I can add the plugin ID as command parameter.
Or better to a config file within the Resources/store folder. So I can add the plugin ID once and then it's versioned and read out automatically.

Hey,
the reason for the env is, that we use that in the Travis CI. So its updated on any git change.
Maybe we could add a global argument to override it?

I have decided to fork the project and adjust the code for my needs. I made a "plugin:init" command with plugin path and plugin ID as parameter. This command will download the resources to the store folder of the plugin. And then I have added the plugin ID to the store.json. So all other commands only have the plugin path as parameter and they read the plugin ID from the store.json and not from the .env.

Can you create a PR? That both works? That env and the entry in the store.json?


feat(make): improved test report colorizing

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (b2df9ef) to head (7655e2b).

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #28   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           27        27
  Lines          658       658
=========================================
  Hits           658       658

Flag
Coverage Δ

gi
100.00% <ø> (ø)

optional
100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

coverage: 100.0%. remained the same
when pulling 7655e2bc5031423a2a8690e0db63181d5ad17bfd on feature/make
into b2df9ef81b065e60618376880920fa05bcae33b1 on main.


feat(project): add ssai ads for vod

Description

Add support for server side ads. When "adDeliveryMethod" is set to "ssai" and when "adConfig" is set, then we modify manifest URL and return ads injected at the server side.
Some refactoring of the sources function.

Steps completed:

According to our definition of done, I have completed the following steps:

[ ] Acceptance criteria met
[ ] Unit tests added
[ ] Docs updated (including config and env variables)
[ ] Translations added
[ ] UX tested
[ ] Browsers / platforms tested
[ ] Rebased & ready to merge without conflicts
[ ] Reviewed own code

@ChristiaanScheermeijer yeah, I think it is a good idea.

I will create a ticket in the back-log.


Users of ChangeNotifier should dispatch event of object creation in constructor.

Prerequisite: https://github.com/flutter/flutter/pull/133060

Google testing failure is ok to ignore, because the missing method is merged to GitHub, but not to G3 yet:
ERROR: third_party/dart/flutter/lib/src/widgets/navigator.dart

3361: maybeDispatchObjectCreation(); Undefined method. #undefined_method

ERROR: third_party/dart/flutter/lib/src/widgets/selectable_region.dart

1560: maybeDispatchObjectCreation(); Undefined method. #undefined_method

ERROR: third_party/dart/flutter/lib/src/widgets/widget_inspector.dart

2901: maybeDispatchObjectCreation(); Undefined method. #undefined_method

google3:///[third_party/dart/flutter/lib/src/widgets/navigator.dart:3361](https://cs.corp.google.com/piper///depot/google3/third_party/dart/flutter/lib/src/widgets/navigator.dart?l=3361&ws=tap-presubmit-server/94134956&snapshot=5):7: Error: The method 'maybeDispatchObjectCreation' isn't defined for the class '_History'.
 - '_History' is from 'package:flutter/src/widgets/navigator.dart' ('google3:///third_party/dart/flutter/lib/src/widgets/navigator.dart').
Try correcting the name to the name of an existing method, or defining a method named 'maybeDispatchObjectCreation'.
      maybeDispatchObjectCreation();
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
google3:///[third_party/dart/flutter/lib/src/widgets/selectable_region.dart:1560](https://cs.corp.google.com/piper///depot/google3/third_party/dart/flutter/lib/src/widgets/selectable_region.dart?l=1560&ws=tap-presubmit-server/94134956&snapshot=5):7: Error: The method 'maybeDispatchObjectCreation' isn't defined for the class 'MultiSelectableSelectionContainerDelegate'.
 - 'MultiSelectableSelectionContainerDelegate' is from 'package:flutter/src/widgets/selectable_region.dart' ('google3:///third_party/dart/flutter/lib/src/widgets/selectable_region.dart').
Try correcting the name to the name of an existing method, or defining a method named 'maybeDispatchObjectCreation'.
      maybeDispatchObjectCreation();
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
google3:///[third_party/dart/flutter/lib/src/widgets/widget_inspector.dart:2901](https://cs.corp.google.com/piper///depot/google3/third_party/dart/flutter/lib/src/widgets/widget_inspector.dart?l=2901&ws=tap-presubmit-server/94134956&snapshot=5):7: Error: The method 'maybeDispatchObjectCreation' isn't defined for the class 'InspectorSelection'.
 - 'InspectorSelection' is from 'package:flutter/src/widgets/widget_inspector.dart' ('google3:///third_party/dart/flutter/lib/src/widgets/widget_inspector.dart').
Try correcting the name to the name of an existing method, or defining a method named 'maybeDispatchObjectCreation'.
      maybeDispatchObjectCreation();
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^

Google testing failure can be ignored, because g3 did not catch up on prerequisite PR yet:
ERROR: third_party/dart/flutter/lib/src/widgets/navigator.dart

3361: maybeDispatchObjectCreation(); Undefined method. #undefined_method

ERROR: third_party/dart/flutter/lib/src/widgets/selectable_region.dart

1560: maybeDispatchObjectCreation(); Undefined method. #undefined_method

ERROR: third_party/dart/flutter/lib/src/widgets/widget_inspector.dart

2901: maybeDispatchObjectCreation(); Undefined method. #undefined_method

google3:///[third_party/dart/flutter/lib/src/widgets/navigator.dart:3361](https://cs.corp.google.com/piper///depot/google3/third_party/dart/flutter/lib/src/widgets/navigator.dart?l=3361&ws=tap-presubmit-server/94538208&snapshot=5):7: Error: The method 'maybeDispatchObjectCreation' isn't defined for the class '_History'.
 - '_History' is from 'package:flutter/src/widgets/navigator.dart' ('google3:///third_party/dart/flutter/lib/src/widgets/navigator.dart').
Try correcting the name to the name of an existing method, or defining a method named 'maybeDispatchObjectCreation'.
      maybeDispatchObjectCreation();
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
google3:///[third_party/dart/flutter/lib/src/widgets/selectable_region.dart:1560](https://cs.corp.google.com/piper///depot/google3/third_party/dart/flutter/lib/src/widgets/selectable_region.dart?l=1560&ws=tap-presubmit-server/94538208&snapshot=5):7: Error: The method 'maybeDispatchObjectCreation' isn't defined for the class 'MultiSelectableSelectionContainerDelegate'.
 - 'MultiSelectableSelectionContainerDelegate' is from 'package:flutter/src/widgets/selectable_region.dart' ('google3:///third_party/dart/flutter/lib/src/widgets/selectable_region.dart').
Try correcting the name to the name of an existing method, or defining a method named 'maybeDispatchObjectCreation'.
      maybeDispatchObjectCreation();
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
google3:///[third_party/dart/flutter/lib/src/widgets/widget_inspector.dart:2901](https://cs.corp.google.com/piper///depot/google3/third_party/dart/flutter/lib/src/widgets/widget_inspector.dart?l=2901&ws=tap-presubmit-server/94538208&snapshot=5):7: Error: The method 'maybeDispatchObjectCreation' isn't defined for the class 'InspectorSelection'.
 - 'InspectorSelection' is from 'package:flutter/src/widgets/widget_inspector.dart' ('google3:///third_party/dart/flutter/lib/src/widgets/widget_inspector.dart').
Try correcting the name to the name of an existing method, or defining a method named 'maybeDispatchObjectCreation'.
      maybeDispatchObjectCreation();
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^


Publish newer version with 'prop-types' changes

Is there anything holding back a new release? Would be good to get the PropTypes changes (https://github.com/dogfessional/react-swipeable/commit/c52c66fd867b96fe1c6a0a19ea8be242fe753010) into npm.

was hoping to get some feedback on a few things, and finding the time to merge in PRs, test, tag, publish.  should be later this week with v4
tracking in #66

Great thank you!


Substance Disk Usage Warning

A lot of problems with substance have occur for me when it hits the max disk usage. I think it might be because I actively code on so many projects. By the time you max out fixing things is hard (usually only noticing when sync fails) . You might have already just finished setup a machine and database then realise you need to do it all again on a new VM because you can't resize the hard drives (or at least the format my VirtualBox chose).
I'd appreciate (would have saved my days):

[ ] A low disk space warning that you had to actively dismiss on machine start.
[ ] It would also be good if when doing a substance switch did a rough 500mb estimate per container and said "This project might use too much data"
[ ] A recommendation for a course of action would be nice
[ ] A line in the docs where you can choose the HD size like you can choose CPU and memory

Interesting suggestions ; I don't think all of those are in the realm of substance's actual job but we can certainly add a few checks for disk space when running commands.
The tricky part is going to be to make those agnostic to the VM's os.
-b

A line in the docs where you can choose the HD size like you can choose CPU and memory

A low disk space warning that you had to actively dismiss on machine start.

I think at least those two would be pretty nice additions that remain in the scope of the project.

See #32


Local development server exits with "UnknownError: 404"

Expected behavior
Following instructions for quickstart and local development, I should be able to run npm start and load the index route at http://localhost:3333.
Description of bug
Running npm start seems to bootstrap the local dev server without error:
$ npm start

><EMAIL_ADDRESS>start /Users/ian/src/git/impressiver/begin-functions-app
> NODE_ENV=testing ARC_LOCAL=1 npx sandbox

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-west-2
         AWS_PROFILE xxx

                Node 8.11.3
                 npm 6.3.0
                .arc 3.2.19

✓ @tables created in local database
✓ src/shared copied to lambda node_modules/@architect/shared
✓ .arc copied to lambda<EMAIL_ADDRESS>✓ @events pub/sub ready on local event bus

 @html
 get / ................................. get-index
 get /echo/:str ........................ get-echo-000str

 ✓ Started HTTP "server" @  http://localhost:3333

But as soon as I load http://localhost:3333 in a browser, the process exits w/ the following stack trace:
{ UnknownError: 404
    at Request.extractError (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
  message: '404',
  code: 'UnknownError',
  statusCode: 404,
  time: 2018-08-11T21:04:49.771Z,
  requestId: undefined,
  retryable: false,
  retryDelay: 28.650255248456446 }
/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:31
            throw err;
            ^

UnknownError: 404
    at Request.extractError (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>start: `NODE_ENV=testing ARC_LOCAL=1 npx sandbox`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.

Steps to reproduce
Detailed steps to reproduce the behavior:

Follow step-by-step directions for local setup:
Note: I used git:// instead of https://
git clone git://github.com/impressiver/begin-functions-app.git
cd begin-functions-app
npm install
npx hydrate
npm start
Open browser to http://localhost:3333, Chrome shows "This site can't be reached"
Console shows the process has exited, dumping the above stack trace

Desktop

OS: macOS 10.14 Beta (18A353d)
Browser: Chrome 68.0.3440.106 (Official Build) (64-bit)
Node: v8.11.3
npm: 6.3.0

Additional context or notes
I use nvm to install/use different versions of node.
I have aws-sdk installed globally and configured w/ my aws credentials, if that matters.

Full issue from before for reference:
Expected behavior
Following instructions for quickstart and local development, I should be able to run npm start and load the index route at http://localhost:3333.
Description of bug
Running npm start seems to bootstrap the local dev server without error:
$ npm start

><EMAIL_ADDRESS>start /Users/ian/src/git/impressiver/begin-functions-app
> NODE_ENV=testing ARC_LOCAL=1 npx sandbox

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-west-2
         AWS_PROFILE xxx

                Node 8.11.3
                 npm 6.3.0
                .arc 3.2.19

✓ @tables created in local database
✓ src/shared copied to lambda node_modules/@architect/shared
✓ .arc copied to lambda<EMAIL_ADDRESS>✓ @events pub/sub ready on local event bus

 @html
 get / ................................. get-index
 get /echo/:str ........................ get-echo-000str

 ✓ Started HTTP "server" @  http://localhost:3333

But as soon as I load http://localhost:3333 in a browser, the process exits w/ the following stack trace:
{ UnknownError: 404
    at Request.extractError (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
  message: '404',
  code: 'UnknownError',
  statusCode: 404,
  time: 2018-08-11T21:04:49.771Z,
  requestId: undefined,
  retryable: false,
  retryDelay: 28.650255248456446 }
/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:31
            throw err;
            ^

UnknownError: 404
    at Request.extractError (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/ian/src/git/impressiver/begin-functions-app/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>start: `NODE_ENV=testing ARC_LOCAL=1 npx sandbox`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.

Steps to reproduce
Detailed steps to reproduce the behavior:

Follow step-by-step directions for local setup:
Note: I used git:// instead of https://
git clone git://github.com/impressiver/begin-functions-app.git
cd begin-functions-app
npm install
npx hydrate
npm start
Open browser to http://localhost:3333, Chrome shows "This site can't be reached"
Console shows the process has exited, dumping the above stack trace

Desktop

OS: macOS 10.14 Beta (18A353d)
Browser: Chrome 68.0.3440.106 (Official Build) (64-bit)
Node: v8.11.3
npm: 6.3.0

Additional context or notes
I use nvm to install/use different versions of node.
I have aws-sdk installed globally and configured w/ my aws credentials, if that matters.

Thanks, this is a very well reported issue.
The things I'd normally want to look into (such as our recent update addressing #4) don't seem to apply here.
Since you mentioned you've got local AWS creds, can we try something to see if it's potentially related / conflicting?
I'd have you just do a simple mv ~/.aws/credentials ~/.aws/credentials-hold and try that repro flow again and see what happens. (Move it back to the original filename when you're done with that, of course.)

also: can you try downgrading npm npm i<EMAIL_ADDRESS>-g and re-running npm i && npx hydrate && npm start

No luck pinning npm 6.1.0 or removing aws credentials. What version of node are you guys running?
$ mv ~/.aws/credentials{,.bak}
$ rm -rf node_modules/
$ npm i<EMAIL_ADDRESS>-g
...
+<EMAIL_ADDRESS>updated 1 package in 15.115s

$ npm i

><EMAIL_ADDRESS>install /Users/ian/src/git/impressiver/begin-functions-app/node_modules/leveldown
> prebuild-install || node-gyp rebuild

npm WARN<EMAIL_ADDRESS>No repository field.
npm WARN<EMAIL_ADDRESS>No license field.

added 382 packages from 348 contributors and audited 777 packages in 10.818s
found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

$ npx hydrate

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-west-2
         AWS_PROFILE xxx

                Node 8.11.3
                 npm 6.1.0
                .arc 3.2.19

✓ Success 1646ms

$ npm start

><EMAIL_ADDRESS>start /Users/ian/src/git/impressiver/begin-functions-app
> NODE_ENV=testing ARC_LOCAL=1 npx sandbox

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-west-2
         AWS_PROFILE xxx

                Node 8.11.3
                 npm 6.1.0
                .arc 3.2.19

✓ @tables created in local database
✓ src/shared copied to lambda node_modules/@architect/shared
✓ .arc copied to lambda<EMAIL_ADDRESS>✓ @events pub/sub ready on local event bus

 @html
 get / ................................. get-index
 get /echo/:str ........................ get-echo-000str

 ✓ Started HTTP "server" @  http://localhost:3333
{ UnknownError: 404
... same stack trace and exit ...

AWS cli shows no default credentials loaded:
$ aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [us-east-1]:
Default output format [json]:
$ ls ~/.aws/
aws-keychain.list  config  credentials.bak  keys

I’m looking into the architect docs, they expect node 8.10.x to match aws. Let me try that, I’ve seen weirder than this on minor version mismatches.

No diff...
$ npm start

><EMAIL_ADDRESS>start /Users/ian/src/git/impressiver/begin-functions-app
> NODE_ENV=testing ARC_LOCAL=1 npx sandbox

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-west-2
         AWS_PROFILE xxx

                Node 8.10.0
                 npm 6.1.0
                .arc 3.2.19

✓ @tables created in local database
✓ src/shared copied to lambda node_modules/@architect/shared
✓ .arc copied to lambda<EMAIL_ADDRESS>✓ @events pub/sub ready on local event bus

 @html
 get / ................................. get-index
 get /echo/:str ........................ get-echo-000str

 ✓ Started HTTP "server" @  http://localhost:3333
{ UnknownError: 404
...

Super weird, what is your NODE_ENV?

Sorry / can see its in the output. Long week! 😴 Just cloned your repo and gave it a go / here's my output:
#! npm i && npx hydrate && npm start

><EMAIL_ADDRESS>install /Users/brianleroux/Desktop/impressiver/node_modules/leveldown
> prebuild-install || node-gyp rebuild

npm WARN<EMAIL_ADDRESS>No repository field.
npm WARN<EMAIL_ADDRESS>No license field.

added 382 packages from 348 contributors and audited 777 packages in 15.136s
found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-east-1
         AWS_PROFILE smallwins

                Node 8.11.2
                 npm 6.1.0
                .arc 3.2.19

✓ Success 5831ms

><EMAIL_ADDRESS>start /Users/brianleroux/Desktop/impressiver
> NODE_ENV=testing ARC_LOCAL=1 npx sandbox

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-east-1
         AWS_PROFILE smallwins

                Node 8.11.2
                 npm 6.1.0
                .arc 3.2.19

✓ @tables created in local database
✓ src/shared copied to lambda node_modules/@architect/shared
✓ .arc copied to lambda<EMAIL_ADDRESS>✓ @events pub/sub ready on local event bus

 @html
 get / ................................. get-index
 get /echo/:str ........................ get-echo-000str

 ✓ Started HTTP "server" @  http://localhost:3333

I have an AWS_PROFILE set here. So I tried again without it:
#! unset AWS_PROFILE && npm start

><EMAIL_ADDRESS>start /Users/brianleroux/Desktop/impressiver
> NODE_ENV=testing ARC_LOCAL=1 npx sandbox

        ARC_APP_NAME dictum-n8q
          AWS_REGION us-east-1
         AWS_PROFILE xxx

                Node 8.11.2
                 npm 6.1.0
                .arc 3.2.19

✓ @tables created in local database
✓ src/shared copied to lambda node_modules/@architect/shared
✓ .arc copied to lambda<EMAIL_ADDRESS>✓ @events pub/sub ready on local event bus

 @html
 get / ................................. get-index
 get /echo/:str ........................ get-echo-000str

 ✓ Started HTTP "server" @  http://localhost:3333

Which also worked! Hate to be superstitious but maybe try installing Node 8.11.2 and npm 6.1.0 ? Also what version of the aws-sdk do you have?

closing for inactivity but pls feel free to reopen


Fix to set_status in TransSys

Fixes a bug when two property have the same name, i.e. either the same expression or the same user-specified name.
In that case, assuming it proved both, Kind 2 would say it manages to prove one but never terminate.

I might have overridden this with my last pull request #389. I'll check.


FirebaseVisionBarcodeDetectorOptions not detect PDF-417

I am making an application that reads barcode PDF-417 I use FirebaseVisionBarcodeDetectorOptions to pass the Formats ship, but I don't get any results, I put another format and it works. Previously I used the Barcode API of mobile Vision and it worked without problems.
Is there a bug or failure with the PDF-417?

@jamacado the best thing to do would be to write in to Firebase support (https://firebase.google.com/support) and provide:

The exact code you're running (as little as possible)
An example of an image with PDF-417 code in it that does not scan

They should be able to help!

@jamacado it would be also helpful to indicate which formats you specified could work.