unable to install

Hi guys,
so when I run the command "pip install -U socketIO-client" on CMD, here is what I get;

C:\WINDOWS\system32>pip install -U socketIO-client
Collecting socketIO-client
Using cached https://files.pythonhosted.org/packages/12/d4/abeb2596c2f16276c66910362b27d04b8d2cf12a746dcccf1d00de3f691b/socketIO-client-0.7.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "c:\users\revaz\appdata\local\programs\python\python36-32\lib\site-packages\setuptools_init_.py", line 12, in
from setuptools.extension import Extension
File "c:\users\revaz\appdata\local\programs\python\python36-32\lib\site-packages\setuptools\extension.py", line 7, in
from setuptools.dist import _get_unpatched
File "c:\users\revaz\appdata\local\programs\python\python36-32\lib\site-packages\setuptools\dist.py", line 16, in
import pkg_resources
File "c:\users\revaz\appdata\local\programs\python\python36-32\lib\site-packages\pkg_resources.py", line 1479, in
register_loader_type(importlib_bootstrap.SourceFileLoader, DefaultProvider)
AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in C:\Users\Revaz\AppData\Local\Temp\pip-install-pa2fkzge\socketIO-client\

nevermind, got this fixed

Hello  @frostyworks
I'm facing the same issue. It would really help if you could tell me how you fixed it.
Thank you.

Did you replace socketIO-Client with python-socketio, or is that something to be installed before?
Can this be done with pip3?


does gblinear have eta parameter?

I use Python package.
the code is:
num_round= 2900

param = { 'booster': 'gblinear', 'lambda': 0.5, 'alpha': 0.5, 'nthread':5,
         # 'eta': 0.3
        }

dtrain=xgb.DMatrix(X_train,label=y_train)
dtest=xgb.DMatrix(X_test)

watchlist  = [(dtrain,'train')]

start_time = time.time()
bst = xgb.train(param, dtrain, num_round, watchlist)

When I don't specify eta prameter, I get big values -> inf -> nan of train-rmse.
When I specify eta prameter, the train error becomes normal.
another related issue is #950
Is there is eta(learning rate) for gblinear booster?
is it's default value is 0.3?
If so, I thing it's better explicitly doc this behavior.
Currently the doc says eta parameter is for tree booster.
Thanks
By the way. another issue is the nthread parameter sometimes doesn't work.
XGBoost use all cores even specify nthread when use gblinear booster.
Should I explicitly set OMP_NUM_THREADS variable?
Thanks again

closed for no response.

There is indeed learning rate in gblinear. But usually there is no need to set this parameter as long as the result converge. Smaller learning rate indeed help convert when you have skewed dataset.

Thanks for reply, I get it.


[12.1] ISPN-13578 Workaround for Elytron not using supplied providers

https://issues.redhat.com/browse/ISPN-13578

Thanks @tristantarrant


media buttons not working on my logitech keyboard

my play/pause, stop and skip buttons do not work for the player. these buttons work perfectly in other players (such as vlc, winamp).

I have tested this with the only logitech keyboard I have (a K260) and it works perfectly fine.  Can you please confirm which keyboard you have?

it is a logitech wave 2.4 cordless keyboard running setpopoint 6.67.83 and driver 5.90.41

Can you use something like Auto Hotkey to see what key is actually being received the computer

hi. i think it has something to do with the (stupid) way setpoint handles the media keys. see the following links:
http://tomse-blog.blogspot.co.za/2014/11/logitech-setpoint-and-media-keys-not.html
http://www.mediamonkey.com/support/index.php?/Knowledgebase/Article/View/49/2/multimedia-keyboard-buttons-dont-work
are you able to work out what line i need to add to my players.ini file?

I have no idea mate, try the first fix on that page, or use autohotkey
This is confirmed as not being my fault / problem.  Good luck :smile_cat:

I just wanted to chime in here since I was having the same problem.  I installed UberOptions and set all three of the media keys to "Other:Generic Button ".  Play/Pause, Next, and Previous all work regardless of if GPMDP has focus or not.  I suggest installing UberOptions, just be aware that you will have to set up all your settings in SetPoint again.

thank you so much!!! this has solved my problem 100% :)


adds minutes for 2020-10-05 meeting

Closes #146

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


auto-refresh on environment change

auto refresh env if files or settings change

fixed in npm lib

fixed in npm lib


Kube prometheus 0.6 is using an incompatible version of kube mixin based on k8s compatibility

The release-0.6 branch of kube-prometheus has a dependency on kubernetes-mixin's release-0.5  branch. The 0.5 branch of kubernetes-mixin is said to be incomatible with k8s 1.18 as given here. But the release-0.6 branch of kube-prometheus is said to be compatible with 1.18. according to here.
So, one of the 3 correction should be made

kube prom release-0.6 should be made incompatible with k8s 1.18
kube prom release-0.6 should be updated to have a dependency on kube mixin release-0.4 branch
kube mixin release-0.5 should be made compatible with k8s 1.18

I have not done any tests as I am yet to deploy but since I came across this while going through the docs as a preparation for my deployment, I thought I would create an issue to track this.

I believe we should move release-0.6 from 1.18 to 1.19 column in the compatibility matrix.

Do you want me to create a couple of PRs to master and release-0.6 with this doc change?

I would want to know what is not compatible in mixin release branch with k8s 1.18 first. As our tests pass, but as said on slack it could be an alerting rule or recording rule' metric that changed. Otherwise sgtm

Yeah @lilic , even I think that the best outcome would be if kube-mixin 0.5 is compatible with k8s 1.18. But if that's not possible due to some incompatible alerting or recording rule, I am fine with using the 0.5 release of kube prometheus for now.

The discussion comes from: https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/480#discussion_r462981529
To quote @s-urbaniak who cut the release on the exact reason:

technically it still works with 1.18. so my suggestion to me would be: despite us knowing it works for 1.18 we only guarantee >=1.19 as we'll only fix 0.5 for 1.19 issues.

I'd say, let's do the same here. wdyt?

The compatibility matrix should reflect what we know for sure and what we can guarantee. Currently, we can guarantee* that release-0.6 branch will work with k8s 1.19. For every other k8s version, we can say it works, but there are no guarantees it will in the future. Considering this I would prefer if we move compatibility of release-0.6 to k8s 1.19 and not invest time into finding what and if 1.18 works.
*Guarantee comes from downstream usage in OpenShift and CI tests.

I just ran into this since I upgraded to release-0.6 (thought that was the newest I could upgrade to). Some dashboards like for the kubelet are using metrics like kubelet_running_pods (changed from kubelet_running_pod_count in k8s 1.19), which caused some panels to show "N/A". I'll be upgrading to 1.19 soon anyway, but yeah I agree, leaning towards not including k8s versions in the compatibility matrix if they don't have these guarantees (or marking them differently) would be helpful. At least, to whatever extent is possible with these guarantees.

Closing as release-0.6 is not compatible with k8s 1.18. If anyone would like to work on making it compatible then we are open to PRs :)


Support training Moses in the docker container

It'd be great to be able to train the MT inside the docker container, and that should be possible with up-to-date docker now. I heard. @DewiBrynJones said so, at least :-)

On the case. Stay tuned...

@tornoz

Latest commits on the 'dockerization' branch enables this.
To quickly verify your pull/clone, delete all Moses related containers and images. Then do:
 $ make

which builds the Dockerfile.
Then simply run to get a running container and a command prompt:
 $ make run
 root@798d82cad57a:~/moses-smt# python moses.py fetchcorpus -e Meddalwedd

This will fetch en-cy bitexts that techiaith in Bangor have prepared from the localizations of Firefox, WordPress, LibreOffice etc. from a URL hardcoded in the scripts. 'Meddalwedd'=Software
Then train.....
 root@798d82cad57a:~/moses-smt# python moses.py train -e Meddalwedd -n 8 -s en -t cy

and start a server...
 root@798d82cad57a:~/moses-smt# python moses.py start -e Meddalwedd -s en -t cy

Then in a browser on the same machine go to:
http://localhost:8008
A demo page should appear. Enter
"An error occurred while connecting to the server"
in the box on the left. You should get
"Digwyddodd gwall wrth geisio cysylltu i'r gweinydd"
in the box on the right.
Mapped volumes are required in order to train with your own bitext data. Coming next...


[3.3] Add drop-check for index creation in cluster

Scope & Purpose
Backport of https://github.com/arangodb/arangodb/pull/9219
Add check in coordinator index creation process to verify that the index has not been dropped between polling attempts. Previously, this could result in the creation request hanging and timing out instead of recognizing that a simultaneous drop request had been made.

[x] Bug-Fix for a released version (did you remember to port this to all relevant release branches?)
[x] The behavior in this PR can be (and was) manually tested (support / qa / customers can test it)

Related Information

[x] There is an internal planning ticket: https://github.com/arangodb/release-3.5/issues/101

Testing & Verification
http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/4802/
Documentation

[x] Added a Changelog Entry

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/4806/

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/4848/


View Command Feature

Resolves #83

Codecov Report

Merging #98 (805899d) into master (c8b9cce) will decrease coverage by 1.46%.
The diff coverage is 16.66%.

@@             Coverage Diff              @@
##             master      #98      +/-   ##
============================================
- Coverage     70.68%   69.22%   -1.47%
  Complexity      475      475
============================================
  Files            76       78       +2
  Lines          1535     1566      +31
  Branches        179      182       +3
============================================
- Hits           1085     1084       -1
- Misses          391      421      +30
- Partials         59       61       +2

Impacted Files
Coverage Δ

src/main/java/seedu/intern/logic/LogicManager.java
71.42% <0.00%> (-3.58%)
:arrow_down:

.../java/seedu/intern/logic/commands/ViewCommand.java
0.00% <0.00%> (ø)

...seedu/intern/logic/parser/InternWatcherParser.java
88.88% <0.00%> (-11.12%)
:arrow_down:

...a/seedu/intern/logic/parser/ViewCommandParser.java
0.00% <0.00%> (ø)

src/main/java/seedu/intern/model/Model.java
100.00% <ø> (ø)

src/main/java/seedu/intern/model/ModelManager.java
90.38% <0.00%> (-9.62%)
:arrow_down:

src/main/java/seedu/intern/ui/MainWindow.java
0.00% <0.00%> (ø)

...ava/seedu/intern/logic/commands/CommandResult.java
78.94% <80.00%> (-9.29%)
:arrow_down:

.../java/seedu/intern/logic/commands/ExitCommand.java
100.00% <100.00%> (ø)

.../java/seedu/intern/logic/commands/HelpCommand.java
100.00% <100.00%> (ø)

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8b9cce...805899d. Read the comment docs.


Use separate sessions

This PR implements separate sessions per query to the database in query_tx, just like in MIRA.
I recreated the transaction error by running the web service locally and submitting large amounts of requests from two separate terminals to the service and verified that the same action did not reproduce the error after the update.
Left to do
The second method in the client, create_tx, that also does transactions outside a session context is used for indexing and I have not yet tested that to feel confident enough to replace its current implementation.

Converted to draft for the time being since we might want to replace the current implementation for create_tx as well

@kkaris yes, please add this for create_tx as well. That's not used much in practice but we should do it for consistency.


[FLINK-35935][table] Fix RTAS not supporting LIMIT

What is the purpose of the change
When using SqlReplaceTableAsConverter to convert SqlNode,  SqlReplaceTableAsConverter wrongly used the original query asQuerySqlNode. Actually it should use the validated SqlNode.
Brief change log

Use validated SqlNode when converting RTAS
Add full tests

Verifying this change
Many tests (including UT and IT) have been added to verify this pr.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented?

Hi, @lincoln-lil . The reason is: The plan will be optimized to source -> exchange (single) -> limit. Theoretically, if the source outputs data with multi parallelism, it could result in variations in the query results, leading to unstable tests. To mitigate this risk, I am enforcing single parallelism in this case.

@xuyangzhong So at least we can use a sort clause for batch tests to get the stable result?

@lincoln-lil You're right. I have replaced all limit with sort limit in IT cases.


A controller is an instantiable public class

I understand that from the library POV a non-public class is non-instantiable.  However, when I look at my code, I can instantiate the controller class all right, so I cannot see the problem.  Also, there is no warning from the library when a controller class is non-public, so there is no way to discover the problem at run time.  I hope this small amendment will do no harm.

I think the problem with this change is that just because a controller isn't marked public (which would be strange outside of just making the actions unavailable for some reason in one spot) it doesn't change the definition of what a controller is. By adding the word "public" here, it's conditionally defining the object. If something like this were done, it seems like the language should be more along the lines of '... usually public ...'.
cc: @Rick-Anderson @serpent5


Modal: Add support for .modal-dialog-scrollable

It'd be nice (and pretty easy) to add support for the .modal-dialog-scrollable class:
https://getbootstrap.com/docs/4.4/components/modal/#scrolling-long-content
Right now, it isn't possible to use this feature without resorting to hacks.
(I might submit a pull request for this when I have time – this issue is just a reminder for myself, or someone else, to do that.)

Please add support for this. Much needed!

I would be fine if someone wants to add that. I'll gladly review the code and merge it in. I, however, am doing active development on this project, so this will have to come from the community.

@Morgul - this tripped me up because in the 3.0.7 version the dist/ui-bootstrap.js version is still at 3.0.6. The 3.0.7 ones have the version in the filename, but it was unexpected for them to not match, so by including the file without the version in the name this functionality was missing.


[SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

What changes were proposed in this pull request?
Non-partitioned data source table and partitioned data source table behave inconsistently:
build/sbt clean package
export SPARK_PREPEND_CLASSES=true
./bin/spark-shell

-- non-partitioned data source table
spark.sql("CREATE TABLE t1 USING PARQUET AS SELECT 1 AS id, 2 as c2")
spark.sql("CREATE TABLE t2 USING PARQUET AS SELECT 1 AS id, 2 as c2")
spark.sql("EXPLAIN COST SELECT * FROM t1 JOIN t2 ON t1.c2 = t2.c2").show(false)

|== Optimized Logical Plan ==
Join Inner, (c2#7 = c2#9), Statistics(sizeInBytes=364.6 KiB)
:- Filter isnotnull(c2#7), Statistics(sizeInBytes=611.0 B)
:  +- Relation[id#6,c2#7] parquet, Statistics(sizeInBytes=611.0 B)
+- Filter isnotnull(c2#9), Statistics(sizeInBytes=611.0 B)
   +- Relation[id#8,c2#9] parquet, Statistics(sizeInBytes=611.0 B)

== Physical Plan ==
*(2) BroadcastHashJoin [c2#7], [c2#9], Inner, BuildRight
:- *(2) Project [id#6, c2#7]
:  +- *(2) Filter isnotnull(c2#7)
:     +- *(2) FileScan parquet default.t1[id#6,c2#7] Batched: true, DataFilters: [isnotnull(c2#7)], Format: Parquet, Location: InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(c2)], ReadSchema: struct<id:int,c2:int>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[1, int, true] as bigint)))
   +- *(1) Project [id#8, c2#9]
      +- *(1) Filter isnotnull(c2#9)
         +- *(1) FileScan parquet default.t2[id#8,c2#9] Batched: true, DataFilters: [isnotnull(c2#9)], Format: Parquet, Location: InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/t2], PartitionFilters: [], PushedFilters: [IsNotNull(c2)], ReadSchema: struct<id:int,c2:int>

-- partitioned data source table
spark.sql("CREATE TABLE t3(id int, c2 int) USING parquet PARTITIONED BY(id)")
spark.sql("CREATE TABLE t4(id int, c2 int) USING parquet PARTITIONED BY(id)")
spark.sql("INSERT INTO TABLE t3 PARTITION(id=1) SELECT 2")
spark.sql("INSERT INTO TABLE t4 PARTITION(id=1) SELECT 2")
spark.sql("EXPLAIN COST SELECT * FROM t3 JOIN t4 ON t3.c2 = t4.c2").show(false)

|== Optimized Logical Plan ==
Join Inner, (c2#17 = c2#22), Statistics(sizeInBytes=8.51E+37 B)
:- Filter isnotnull(c2#17), Statistics(sizeInBytes=8.0 EiB)
:  +- Relation[c2#17,id#18] parquet, Statistics(sizeInBytes=8.0 EiB)
+- Filter isnotnull(c2#22), Statistics(sizeInBytes=8.0 EiB)
   +- Relation[c2#22,id#23] parquet, Statistics(sizeInBytes=8.0 EiB)

== Physical Plan ==
*(5) SortMergeJoin [c2#17], [c2#22], Inner
:- *(2) Sort [c2#17 ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(c2#17, 200)
:     +- *(1) Project [c2#17, id#18]
:        +- *(1) Filter isnotnull(c2#17)
:           +- *(1) FileScan parquet default.t3[c2#17,id#18] Batched: true, DataFilters: [isnotnull(c2#17)], Format: Parquet, Location: CatalogFileIndex[file:/root/opensource/apache-spark/spark-warehouse/t3], PartitionCount: 1, PartitionFilters: [], PushedFilters: [IsNotNull(c2)], ReadSchema: struct<c2:int>
+- *(4) Sort [c2#22 ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(c2#22, 200)
      +- *(3) Project [c2#22, id#23]
         +- *(3) Filter isnotnull(c2#22)
            +- *(3) FileScan parquet default.t4[c2#22,id#23] Batched: true, DataFilters: [isnotnull(c2#22)], Format: Parquet, Location: CatalogFileIndex[file:/root/opensource/apache-spark/spark-warehouse/t4], PartitionCount: 1, PartitionFilters: [], PushedFilters: [IsNotNull(c2)], ReadSchema: struct<c2:int>

The reason is that sizeInBytes is calculated by different way when the table statistics are not available:

Non-partitioned data source table:
SizeInBytesOnlyStatsPlanVisitor.default() -> LogicalRelation.computeStats() -> HadoopFsRelation.sizeInBytes() -> PartitioningAwareFileIndex.sizeInBytes()
PartitioningAwareFileIndex.sizeInBytes() is calculated by allFiles().map(_.getLen).sum if table statistics are not available.

Partitioned data source table:
SizeInBytesOnlyStatsPlanVisitor.default() -> LogicalRelation.computeStats() -> CatalogFileIndex.sizeInBytes
CatalogFileIndex.sizeInBytes is spark.sql.defaultSizeInBytes if table statistics are not available.

This pr moves DetermineTableStats from HiveStrategies to DataSourceStrategy and add
support for recalculating data source table statistics if it statistics are not available.
How was this patch tested?
unit tests

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11064/
Test PASSed.

Test build #105808 has started for PR 24715 at commit dd5a125.

Test build #105808 has finished for PR 24715 at commit dd5a125.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class DetermineDataSourceTableStats(session: SparkSession) extends Rule[LogicalPlan]

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105808/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12193/
Test PASSed.

Test build #106995 has started for PR 24715 at commit 70d3557.

Test build #106995 has finished for PR 24715 at commit 70d3557.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan]

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106995/
Test FAILed.

Test build #107010 has started for PR 24715 at commit dd5f356.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12204/
Test PASSed.

Test build #107010 has finished for PR 24715 at commit dd5f356.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107010/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12227/
Test PASSed.

Test build #107034 has started for PR 24715 at commit 22dd26e.

Test build #107034 has finished for PR 24715 at commit 22dd26e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107034/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12228/
Test PASSed.

Test build #107035 has started for PR 24715 at commit 22dd26e.

Test build #107035 has finished for PR 24715 at commit 22dd26e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107035/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12265/
Test PASSed.

Test build #107080 has started for PR 24715 at commit 7748a32.

Test build #107080 has finished for PR 24715 at commit 7748a32.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107080/
Test PASSed.

The idea LGTM, can you rebase this PR?

I did some benchmark.
Prepare data:
spark.range(100000000).repartition(10000).write.saveAsTable("test_non_partition_10000")
spark.range(100000000).repartition(300000).write.saveAsTable("test_non_partition_300000")
spark.range(100000000).selectExpr("id", "id % 5000 as c2", "id as c3").repartition(org.apache.spark.sql.functions.col("c2")).write.partitionBy("c2").saveAsTable("test_partition_5000")
spark.range(100000000).selectExpr("id", "id % 10000 as c2", "id as c3").repartition(org.apache.spark.sql.functions.col("c2")).write.partitionBy("c2").saveAsTable("test_partition_10000")

Add these lines to LogicalRelation.computeStats:
val time1 = System.currentTimeMillis()
val relationSize = relation.sizeInBytes
val time2 = System.currentTimeMillis()
val fallBackToHdfsSize = CommandUtils.getSizeInBytesFallBackToHdfs(relation.sqlContext.sparkSession, catalogTable.get)
val time3 = System.currentTimeMillis()
// scalastyle:off
println(s"Get size from relation: $relationSize, time: ${time2 - time1}")
println(s"Get size fall back to HDFS: $fallBackToHdfsSize, time: ${time3 - time2}")
// scalastyle:on

Non-partitioned table benchmark result:
scala> spark.sql("explain cost select * from test_non_partition_10000 limit 1").show
Get size from relation: 576588171, time: 22
Get size fall back to HDFS: 576588171, time: 41
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

scala> spark.sql("explain cost select * from test_non_partition_10000 limit 1").show
Get size from relation: 576588171, time: 3
Get size fall back to HDFS: 576588171, time: 28
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

scala>

scala> spark.sql("explain cost select * from test_non_partition_300000 limit 1").show
Get size from relation: 706507984, time: 135
Get size fall back to HDFS: 706507984, time: 2038
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

scala> spark.sql("explain cost select * from test_non_partition_300000 limit 1").show
Get size from relation: 706507984, time: 168
Get size fall back to HDFS: 706507984, time: 3629
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

Partitioned table benchmark result:
scala> spark.sql("explain cost select * from test_partition_5000 limit 1").show
Get size from relation:<PHONE_NUMBER>854775807, time: 0
Get size fall back to HDFS:<PHONE_NUMBER>, time: 46
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

scala> spark.sql("explain cost select * from test_partition_10000 limit 1").show
Get size from relation:<PHONE_NUMBER>854775807, time: 0
Get size fall back to HDFS:<PHONE_NUMBER>, time: 43
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

Partitioned table with spark.sql.hive.manageFilesourcePartitions=false (set it by --conf) benchmark result:
scala> spark.sql("set spark.sql.hive.manageFilesourcePartitions").show
+--------------------+-----+
|                 key|value|
+--------------------+-----+
|spark.sql.hive.ma...|false|
+--------------------+-----+

scala> spark.sql("explain cost select * from test_partition_5000 limit 1").show
Get size from relation:<PHONE_NUMBER>, time: 3
Get size fall back to HDFS:<PHONE_NUMBER>, time: 45
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

scala> spark.sql("explain cost select * from test_partition_10000 limit 1").show
Get size from relation:<PHONE_NUMBER>, time: 865
Get size fall back to HDFS:<PHONE_NUMBER>, time: 69
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

@wangyum do you mean CommandUtils.getSizeInBytesFallBackToHdfs is very slow if there are many files?

@wangyum do you mean CommandUtils.getSizeInBytesFallBackToHdfs is very slow if there are many files?

CommandUtils.getSizeInBytesFallBackToHdfs is not very slow.
I have no idea why PartitioningAwareFileIndex.sizeInBytes is faster than CommandUtils.getSizeInBytesFallBackToHdfs.
It may be related to the cluster load, I plan to switch to an  idle cluster to test tomorrow.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14241/
Test PASSed.

Test build #109172 has started for PR 24715 at commit 001e1e1.

I switched to an idle cluster, PartitioningAwareFileIndex.sizeInBytes is basically still faster than CommandUtils.getSizeInBytesFallBackToHdfs:
https://issues.apache.org/jira/browse/SPARK-25474?focusedCommentId=16908660&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16908660

let's discuss the perf problem later, and focus on the bug fix first.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14255/
Test PASSed.

Test build #109185 has started for PR 24715 at commit 7d22c15.

Test build #109172 has finished for PR 24715 at commit 001e1e1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #109185 has finished for PR 24715 at commit 7d22c15.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109185/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109172/
Test FAILed.

Test build #109195 has started for PR 24715 at commit cae3b14.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14266/
Test PASSed.

Test build #109195 has finished for PR 24715 at commit cae3b14.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109195/
Test FAILed.

retest this please

Test build #109208 has started for PR 24715 at commit cae3b14.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14278/
Test PASSed.

Test build #109208 has finished for PR 24715 at commit cae3b14.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109208/
Test PASSed.

If we do 3 times computes statistics like,
sql(s"EXPLAIN COST SELECT * FROM spark_25474 '")
sql(s"EXPLAIN COST SELECT * FROM spark_25474 '")
sql(s"EXPLAIN COST SELECT * FROM spark_25474 '")

Then all the 3 times it falls back to HDFS or only the first time, as the table statistics hasn't changed? Then shouldn't we update the table statistics after it computes first time, and next time onwards we can directly gets from the catalog table?.

Then shouldn't we update the table statistics after it computes first time, and next time onwards we can directly gets from the catalog table

fallBackToHDFS is a runtime config, which means users can change it at any time. If we update the table statistics with the HDFS size, we need a way to invalidate the table stats if fallBackToHDFS is set to false later. This seems over-complicated to me.

Actually, can we really fallback to HDFS size for partitioned tables? The partitions can be external (the partition directory is outside of the table directory), we can only get the file size by getting all the partition information from the metastore, which is very expensive.
Maybe this is the reason why fallBackToHDFS is false by default.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14382/
Test PASSed.

Test build #109320 has started for PR 24715 at commit 1fce508.

Test build #109320 has finished for PR 24715 at commit 1fce508.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109320/
Test FAILed.

retest this please

Test build #109322 has started for PR 24715 at commit 1fce508.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14389/
Test PASSed.

Test build #109322 has finished for PR 24715 at commit 1fce508.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109322/
Test PASSed.

Test build #109413 has started for PR 24715 at commit 1e1378d.

Does anyone has an answer for https://github.com/apache/spark/pull/24715#issuecomment-522414331 ?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14477/
Test PASSed.

@cloud-fan I added a test for external partitioned table: https://github.com/apache/spark/pull/24715/files#diff-8c27508821958acbe016862c9ab2f25fR754-R779

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14479/
Test PASSed.

Test build #109415 has started for PR 24715 at commit cc32b48.

Test build #109413 has finished for PR 24715 at commit 1e1378d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan]

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109413/
Test PASSed.

Test build #109415 has finished for PR 24715 at commit cc32b48.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109415/
Test PASSed.

scala> // enable fallBackToHdfs

scala> spark.sql("set spark.sql.statistics.fallBackToHdfs=true")
res4: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("EXPLAIN COST SELECT * FROM t3").show(false)
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|plan                                                                                |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|== Optimized Logical Plan ==
Relation[c2#1,id#2] parquet, Statistics(sizeInBytes=8.0 EiB)

Why is sizeInBytes still 8 EiB, even after enabling the fallback configuration?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14694/
Test PASSed.

Test build #109645 has started for PR 24715 at commit 3b0c234.

Test build #109645 has finished for PR 24715 at commit 3b0c234.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109645/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14696/
Test PASSed.

Test build #109647 has started for PR 24715 at commit d23fd47.

Test build #109647 has finished for PR 24715 at commit d23fd47.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109647/
Test PASSed.

retest this please

thanks, merging to master!


[Feature] Provide configuration via config files

Is your feature request related to a problem? Please describe.
Users have requested the ability to set configuration using config files on their file system (instead of using env vars for secrets such as a GITHUB_TOKEN). These files could be encrypted at rest to ensure secrets are not readable outside of the application
Describe the solution you'd like
support for config files on windows, linux, mac to allow a github token to be provided via a config file - instead of an env var.
the order of preference should be env var (if set), otherwise, default to file system for configuration.
Additional context
I'm open to discussing how folks would like to implement this - feedback and suggestions welcome in the comments.

This is actually kind of possible already? Since you've already using decouple it appeared to work for free the first time I tried it in editable mode(?)
Was just reviewing #128 though and it didn't appear to work the way I had used the .env file before.
At least in my testing I was able to work around it by throwing the env vars in a script (though, they're still env vars).
an approach that I've taken in other applications is to provide either an env var/command line arg to a --config=/path/to/file or something like that.

thanks for the feedback; the above approach sounds good to me 👍
using either an env var or --config= command-line argument to supply the path to the settings.ini or .env: https://pypi.org/project/python-decouple/#where-the-settings-data-are-stored

I've added a PATH_TO_CONFIG env var to provide configuration outside of module/repository and adds PATH_TO_CONFIG to the config command:

adding --config= will require additional work to refactor how configuration is currently initialised and retrieved. for now PATH_TO_CONFIG provides enough functionality to provide the behaviour listed above.


DPOTrainer incompatible with Falcon models and gradient checkpointing

Description
Running the dpo.py example with gradient_checkpointing=True produces different errors for different Falcon models (1B and 7B). Note that this happens both with and/or without DeepSpeed ZeRO-{2,3}, which suggests a deeper problem with the modelling code 🙀.
To trigger the error, first activate gradient checkpointing in the training arguments of dpo.py:
    training_args = TrainingArguments(
        per_device_train_batch_size=script_args.per_device_train_batch_size,
        max_steps=script_args.max_steps,
        remove_unused_columns=False,
        gradient_accumulation_steps=script_args.gradient_accumulation_steps,
        learning_rate=script_args.learning_rate,
        evaluation_strategy="steps",
        logging_first_step=True,
        logging_steps=10,  # match results in blog post
        eval_steps=500,
        output_dir="./test",
        optim="rmsprop",
        warmup_steps=150,
        report_to=script_args.report_to,
        bf16=True,
+        gradient_checkpointing=True,
    )

Then run one of the commands provided below:

Stack trace for https://huggingface.co/tiiuae/falcon-rw-1b
Command to reproduce
# DDP
ACCELERATE_LOG_LEVEL=info TRANSFORMERS_VERBOSITY=info accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero2.yaml examples/scripts/dpo.py --model_name_or_path tiiuae/falcon-rw-1b

# ZeRO-2
ACCELERATE_LOG_LEVEL=info TRANSFORMERS_VERBOSITY=info accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero2.yaml examples/scripts/dpo.py --model_name_or_path tiiuae/falcon-rw-1b

File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 1892, in _inner_training_loop
  tr_loss_step = self.training_step(model, inputs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 2776, in training_step
  loss = self.compute_loss(model, inputs)
File "/fsx/lewis/git/trl/trl/trainer/dpo_trainer.py", line 500, in compute_loss
  loss, metrics = self.get_batch_metrics(model, inputs, train_eval="train")
File "/fsx/lewis/git/trl/trl/trainer/dpo_trainer.py", line 451, in get_batch_metrics
  ) = self.concatenated_forward(model, batch)
File "/fsx/lewis/git/trl/trl/trainer/dpo_trainer.py", line 417, in concatenated_forward
  all_logits = model(
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  ret_val = func(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1735, in forward
  loss = self.module(*inputs, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 1303, in forward
  transformer_outputs = self.transformer(
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 1177, in forward
  outputs = torch.utils.checkpoint.checkpoint(
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
  return CheckpointFunction.apply(function, preserve, *args)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
  return super().apply(*args, **kwargs)  # type: ignore[misc]
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 107, in forward
  outputs = run_function(*args)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 1173, in custom_forward
  return module(*inputs, use_cache=use_cache, output_attentions=output_attentions)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 787, in forward
  attn_outputs = self.self_attention(
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 527, in forward
  attention_probs = attention_probs * head_mask
RuntimeError: The size of tensor a (203) must match the size of tensor b (8) at non-singleton dimension 2

Stack trace for https://huggingface.co/tiiuae/falcon-7b
Command to reproduce
ACCELERATE_LOG_LEVEL=info TRANSFORMERS_VERBOSITY=info accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero3.yaml examples/scripts/dpo.py --model_name_or_path tiiuae/falcon-7b

Error
Traceback (most recent call last):
File "/fsx/lewis/git/trl/examples/scripts/dpo.py", line 173, in <module>
  dpo_trainer.train()
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 1591, in train
  return inner_training_loop(
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 1892, in _inner_training_loop
  tr_loss_step = self.training_step(model, inputs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 2776, in training_step
  loss = self.compute_loss(model, inputs)
File "/fsx/lewis/git/trl/trl/trainer/dpo_trainer.py", line 500, in compute_loss
  loss, metrics = self.get_batch_metrics(model, inputs, train_eval="train")
File "/fsx/lewis/git/trl/trl/trainer/dpo_trainer.py", line 451, in get_batch_metrics
  ) = self.concatenated_forward(model, batch)
File "/fsx/lewis/git/trl/trl/trainer/dpo_trainer.py", line 417, in concatenated_forward
  all_logits = model(
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  ret_val = func(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1735, in forward
  loss = self.module(*inputs, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
  result = forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 1303, in forward
  transformer_outputs = self.transformer(
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
  result = forward_call(*args, **kwargs)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 1213, in forward
  presents = self._convert_cache_to_standard_format(presents, batch_size)
File "/fsx/lewis/miniconda/envs/trl/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 947, in _convert_cache_to_standard_format
  batch_size_times_num_heads, kv_length, head_dim = past_key_value[0][0].shape
IndexError: tuple index out of range

Does this also happen in the transformers.Trainer? Could be a deeper issue maybe? cc @younesbelkada

This is indeed a problem in the modeling code of Falcon, will have a look in the next days


Negedge-triggered SRAMs

We currently have SeqMem as a primitive for posedge-triggered SRAM; I would like to get people's thoughts on adding a corresponding primitive for negedge-clock-triggered SRAM.
This request may require negedge registers too, since it is common to flop the output of SRAM immediately. Perhaps optionally rolling the output flops into the NegSeqMem will be decent approach? Example: SeqMem(n, w, latency=1, negedge=true).

Type of issue

[ ] Bug report
[x] Feature request
[ ] Other enhancement

What is the use case for changing the behavior?

Very clever designs can shave off a pipeline stage by having some logic trigger on the negative clock edge. For example, accessing an SRAM that is indexed by a hash function:
Posedge logic pipeline:
Cycle1: compute hash; send idx to SRAM
Cycle2: access SRAM; flop output because no time is left after cycle-long access.
Cycle3: do stuff with data

Negedge logic:
Cycle1-hi: compute hash; send idx to SRAM
Cycle1-lo: access SRAM; flop output after cycle-long access
Cycle2-lo: do stuff with data

Impact

[ ] no functional change
[x] API addition (no impact on existing code)
[ ] API modification
[x] unknown

Development Phase

[x] request
[ ] proposal

Other information

Here is the current workflow to effect this behavior:

create a module with asynchronous-read memory
Pad out with a register to effect the 1-cycle latency
Blackbox/manually drop in neg-edge SRAM when performing actual ASIC design

I'd like to believe we can model negedge triggered memories in a way that doesn't blow up our ecosystem, like chisel-testers or FIRRTL compiler passes, but still provide a comfortable path to optimal ASIC designs. The closer the Chisel-generated behavioral module is to what the blackboxed implementation would be, the lower the design risk will be.

What's wrong with just inverting the clock?
Having a latency parameter might be nice for technology mapping though I'm not sure often they're used.

Inverting the clock could work given an annotation to inform the CAD tools that clk and ~clk are related clocks, and to emit posedge ~clk instead as negedge clk.  Perhaps surprisingly, the tools have trouble with this.
Adding NegSeqMem is a slippery slope, since we'll also need to add NegReg, NegQueue, ...; we should define a general solution.

I'm 100% onboard, but I think the related discussion is going to be around extending FIRRTL's IR to have some information about clock edges (and updating the Verilog emitter to consume this). Note: this would be a FIRRTL specification update.

Yeah. I think expanding the Firrtl IR to handle negedge clocking is inevitable.


Could not log "sql.active_record" event.

In https://ide.c9.io/wamelias/msm-debug, visiting  a director's show page results in:
Could not log "sql.active_record" event. ArgumentError: wrong number of arguments (given 2, expected 1) ["/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/log_subscriber.rb:45:in `type_casted_binds'", "/usr/local/rvm/gems/ruby-2.3.4/bundler/gems/draft_log-e37849266cfd/lib/draft_log/active_record_log_subscriber.rb:21:in `sql'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/subscriber.rb:99:in `finish'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/log_subscriber.rb:83:in `finish'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications/fanout.rb:102:in `finish'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications/fanout.rb:46:in `block in finish'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications/fanout.rb:46:in `each'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications/fanout.rb:46:in `finish'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:42:in `finish_with_state'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:27:in `instrument'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:604:in `log'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:208:in `exec_query'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:375:in `select_prepared'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:40:in `select_all'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `block in select_all'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:118:in `block in cache_sql'", "/usr/local/rvm/rubies/ruby-2.3.4/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:104:in `cache_sql'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/querying.rb:39:in `find_by_sql'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/statement_cache.rb:107:in `execute'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/core.rb:187:in `find'", "/home/ubuntu/workspace/app/controllers/directors_controller.rb:9:in `show'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/abstract_controller/base.rb:186:in `process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal/rendering.rb:30:in `process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/abstract_controller/callbacks.rb:20:in `block in process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:131:in `run_callbacks'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/abstract_controller/callbacks.rb:19:in `process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal/rescue.rb:20:in `process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications.rb:166:in `block in instrument'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/notifications.rb:166:in `instrument'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal/instrumentation.rb:30:in `process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/railties/controller_runtime.rb:22:in `process_action'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/abstract_controller/base.rb:124:in `process'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionview-5.1.6/lib/action_view/rendering.rb:30:in `process'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal.rb:189:in `dispatch'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_controller/metal.rb:253:in `dispatch'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:31:in `serve'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `each'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `serve'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:844:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/meta_request-0.6.0/lib/meta_request/middlewares/app_request_handler.rb:13:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/meta_request-0.6.0/lib/meta_request/middlewares/meta_request_handler.rb:13:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'", "/usr/local/rvm/gems/ruby-2.3.4/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/cookies.rb:613:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activerecord-5.1.6/lib/active_record/migration.rb:556:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:97:in `run_callbacks'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/callbacks.rb:24:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/executor.rb:12:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/bundler/gems/dev_toolbar-f9745615dc92/lib/dev_toolbar/middleware.rb:10:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/better_errors-2.4.0/lib/better_errors/middleware.rb:59:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-contrib-2.0.1/lib/rack/contrib/response_headers.rb:17:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/meta_request-0.6.0/lib/meta_request/middlewares/headers.rb:16:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/web-console-3.6.2/lib/web_console/middleware.rb:135:in `call_app'", "/usr/local/rvm/gems/ruby-2.3.4/gems/web-console-3.6.2/lib/web_console/middleware.rb:22:in `block in call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/web-console-3.6.2/lib/web_console/middleware.rb:20:in `catch'", "/usr/local/rvm/gems/ruby-2.3.4/gems/web-console-3.6.2/lib/web_console/middleware.rb:20:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/railties-5.1.6/lib/rails/rack/logger.rb:36:in `call_app'", "/usr/local/rvm/gems/ruby-2.3.4/gems/railties-5.1.6/lib/rails/rack/logger.rb:24:in `block in call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/tagged_logging.rb:69:in `block in tagged'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/tagged_logging.rb:26:in `tagged'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/tagged_logging.rb:69:in `tagged'", "/usr/local/rvm/gems/ruby-2.3.4/gems/railties-5.1.6/lib/rails/rack/logger.rb:24:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/request_id.rb:25:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/activesupport-5.1.6/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/executor.rb:12:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/railties-5.1.6/lib/rails/engine.rb:522:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/puma-3.11.4/lib/puma/configuration.rb:225:in `call'", "/usr/local/rvm/gems/ruby-2.3.4/gems/puma-3.11.4/lib/puma/server.rb:632:in `handle_request'", "/usr/local/rvm/gems/ruby-2.3.4/gems/puma-3.11.4/lib/puma/server.rb:446:in `process_client'", "/usr/local/rvm/gems/ruby-2.3.4/gems/puma-3.11.4/lib/puma/server.rb:306:in `block in run'", "/usr/local/rvm/gems/ruby-2.3.4/gems/puma-3.11.4/lib/puma/thread_pool.rb:120:in `block in spawn_thread'"]

We received a request at 09:20PM on Friday, May 11! Someone wants to
  GET /directors/1

The route told me to use the "DirectorsController" and "show" action.

I found these inputs in flexible path segments:
{
    :id_to_display => "1"
}

The final params hash containing all inputs looks like this:
  # params
{
    "id_to_display" => "1"
}

The "DirectorsController#show" action defined these instance variables:
{
    "@director" => "#<Director:0x007fdd650c7d90>"
}

The "DirectorsController#show" action told me to use
  "app/views/director_templates/show_details.html.erb"
to format the response.

Completed 200 OK in 264ms (Views: 228.4ms | ActiveRecord: 2.5ms)

========================================

Any ideas?

@raghubetina Need write access to the above c9 instance.

The error is due to different rails version. Can we lock the draft_log gem to be used in Rails <=5.1.4 .

@murugan-r This is high priority — in MSM Associations, the extra log output is causing the characters#index action to freeze Cloud9 workspaces.
We're going to be upgrading all projects to 5.2 next quarter anyway, so we need to sort this out.

@raghubetina @arjunvenkat Fixed the issue.


init module

This module should take paths to genotype data files and construct a poseidon package from them. For the POSEIDON.yml and the .janno file useful dummies can be constructed.
The interface could look like this:
trident init \
  --genoFile my_project.geno --snpFile my_project.snp --indFile my_project.ind \
  -n new_package_name \
  -o path/to/new_package_name

Done.


ML 6.11 : Implement a Multi Layered Perceptron model on XOR function with 3 inputs (D)

Description
📌  Issues for Week 6
Welcome to 'ML' Team, good to see you here
:arrow_forward: This issue will helps readers in acquiring all the knowledge that one needs to know about ----
Implement a Multi Layered Perceptron model on XOR function with 3 inputs.
🟣  Condition : The model must fit the function optimally (least/no-errors).
🟣  Help/Hint : There are three features taking either 0 or 1 and one label vector taking either 0 or 1 as per the truth table of an XOR gate. Therefore use 1 perceptron function thrice for three perceptrons taking each of the 3 inputs.

:red_circle: To get assigned to this issue, add your Batch Numbers mentioned in the spreadsheet of "Machine Learning", the approach one would follow and choice you prefer (Documentation, Audio, Video). You can go with all three or any number of options you're interested to work on.
✔ Domain : Machine Learning
:red_circle::yellow_circle: Points to Note :

The issues will be assigned on a first come first serve basis, 1 Issue == 1 PR.

"Issue Title" and "PR Title should be the same. Include issue number along with it.

Changes should be made inside the Machine Learning/Machine_Learning/Dimensionality_Reduction_&_Neural_Networks Branch.

Follow Contributing Guidelines & Code of Conduct before start Contributing.

This issue is only for 'GWOC' contributors of 'Machine Learning' domain.

:white_check_mark: To be Mentioned while taking the issue :

Full name
Batch Number
GitHub Profile Link
Which type of Contribution you want to make :

[ ] Documentation

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎
Domain
Machine Learning
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

I will push it tomorrow, I was very sick


Pretty high system load on RPI 4

Description
Hi, I'm wondering what can make a system load of about 2.5-3 on an RPI. I mean, it is more or less some GStreamer stuff and USB 3 transfer, inference is running in the Coral TPU.
I'm using a USB camera as input (/dev/video1). PI is not overclocked, 2GB RAM
What could cause this high load, any ideas?
Other than that the inference is running fine.
Click to expand!
Issue Type
No response
Operating System
Linux
Coral Device
USB Accelerator
Other Devices
Rapsberry Pi 4
Programming Language
Python 3.9
Relevant Log Output
No response

Are having the same system load for uncompiled tflite model?
It could be due to the ML model running on EdgeTPU..Please check the power consumption section on page 5 for more details at: https://coral.ai/static/files/Coral-Accelerator-Module-datasheet.pdf

Could you please give me a hint, how I could make use of the uncompiled tflite model?

Strange. New day, new attempt. Now system load is 1.11, which would be ok

Could you please give me a hint, how I could make use of the uncompiled tflite model?

Please download the ssd_mobilenet_v2_coco_quant_postprocess.tflite from test_data repo to the models directory and update the model name at: https://github.com/google-coral/example-object-tracker/blob/master/gstreamer/detect.py#L147

While having you here may I ask you, if you are aware of any Coral based inference solution, which delivers 6DOF poses? I mean not only bounding boxes but also translation and rotation?

I have responded to this one at: https://github.com/google-coral/project-posenet/issues/89.. Thanks!!

OK, system load is higher with uncompiled tflite model, between 2.5 and 3, while the original model is about 2. 6 fps inference rate.

OK, system load is higher with uncompiled tflite model, between 2.5 and 3, while the original model is about 2. 6 fps inference rate.

Then the high system load is not due to ML model running on TPU. Please try the CPU usage monitoring tools to analyze what is causing this system load: https://linuxhint.com/raspberry-pi-cpu-usage-monitoring/ and check with rpi forum for any additional questions at: https://forums.raspberrypi.com/viewtopic.php?t=283330.. Thanks!

May I ask one last additional question? It is about the ID. I'm aware of the fact, that it can't really be a unique ID since it is detection, not recognition: Once you leave the FOV and return you'll get another ID. But is my assumption correct, that the ID is something not coming from the inference results (like bounding boxes), but is something "post-processed"?

The ID is coming from inference results and it would be mapped to bounding boxes.

Oh? Could you please point me to where the ID is derived? I was using the model with an older Coral application. I'm a bit lost where to find the ID.
This is how I'm deriving the bounding boxes, classes and scores:
            boxes = self.interpreter.get_tensor(self.output_details[0]["index"])[0]
            classes = self.interpreter.get_tensor(self.output_details[1]["index"])[0]
            scores = self.interpreter.get_tensor(self.output_details[2]["index"])[0]

Where to find the ID?

object_name = self.labels[int(classes[i])]

The ID is coming from classes and it would be mapped to labels (for ex ID 2 would be car).

I mean this ID

It basically comes from "trackID" out of trdata. I can't figure out, where this trdata finally comes from. A GStreamer userCallback... Hmmm..
https://github.com/google-coral/example-object-tracker/blob/df20497f328ee48578e766e817a41b1e4f18a376/gstreamer/detect.py#L201
What is this "motTracker"?

That "tracker.update(detections)"  - is it a "Sort" function? Does that "Sort" object finally provide the ID?

OK, after having made a deep dive into the code I confirm my own statement. The trackID is not coming from the inference, like bbox and score. It is resulting from a post process "sort", which assigns that "unique" id as long as the object is in view.

Today I ran it on a 32 bit Raspbian Buster. Runs with system load around 1. And it achieves 30 fps. I need to double check this, but my above attempts have been made with Bullseye Lite 64 bit.


TestCafe problem with decompression buffer from RequestLogger

What is your Scenario?
I want to recive request body from RequestLogger.
I use several ways:

Native automation = true, RequestLogger with stringifyResponseBody = ture
Native automation = true,  RequestLogger with stringifyResponseBody = false and I decode with zlib.brotliDecompressSync(buffer)
Native automation = false - RequestLogger with stringifyResponseBody = false and I decode with zlib.brotliDecompressSync(buffer)
Native automation = false - RequestLogger with stringifyResponseBody = true

What is the Current behavior?
1st way - works OK.
2nd way - I got a Decompression failed error
3rd way - Decompression works fine
4th way - don't works, response body looks like: `�bu�L'!u§F♦��♥]���@↔j♥�t�n��U��♦����i��r�UM4T�D��▼z�؁ǆ@�$x����RBt◄
��2�1�L[�Pk��bG��q�@����CZ�♦|ǈ
��@�▲)F�*�GV‼7�I��PN0�����↓��ȉn�1duo_�8�O�����R��
H��/��E�↕y���5♣Q3�¶�������� �K�J♦�wbQ�►�@&��3�→#Ug��☻�▬�W♠♦�S6�ܧ�b��;寄�{�8��A�fv��М����`
What is the Expected behavior?
I expect to my decopression code zlib.brotliDecompressSync(buffer) works when I use NativeAutomation.
(but RequestLogger with stringifyResponseBody = ture on nonNative Automation mode should works too)
I run my test on chrome (with Native Automation) and firefox (wchith is nonChromimum so it's run in nonNative Automation mode) there is no way to run my tests correctly.
There is a way to run mytests correctly on chrome or Filrefox but not on both of this.
Workaround is to run chrome in onNative Automation mode.
What is your public website URL? (or attach your complete example)
I reproduce this on https://reqres.in/
What is your TestCafe test code?
import zlib from 'zlib';
import { RequestLogger, t } from 'testcafe';

const logger = RequestLogger({ url: /https:\/\/reqres.in\/api\/users/, method: 'get' }, {
    logResponseBody: true,
    logResponseHeaders: true,
});

const getBodyFromRequest = (request) => {
    const reqHeaderContentEncoding = request.response?.headers?.['content-encoding'];
    const isBrotliEncoding = reqHeaderContentEncoding === 'br';
    const buffer = request.response?.body;

    if (!buffer) {
        return {};
    }

    if (isBrotliEncoding) {
        const decoded = zlib.brotliDecompressSync(buffer);

        return JSON.parse(decoded.toString('utf-8'));
    }

    return JSON.parse(buffer.toString('utf-8'));
};

fixture('Test decode')
    .page('https://reqres.in/')
    .requestHooks(logger);

test('testDecodeBuffer', async () => {
    await t.expect(logger.contains(record => record.response.statusCode === 200)).ok();

    const requestBody = getBodyFromRequest(logger.requests[0]);

    /*
    ... test code
   */
});

Your complete configuration file
No response
Your complete test report
No response
Screenshots
No response
Steps to Reproduce

Receive request from website using RequestLogger (request should be compressed with br)
Decode request response body with node function zlib.brotli DecompresS(buffer)

TestCafe version
3.1.0
Node.js version
18.17.0
Command-line arguments
testcafe chrome tests/decode.js
Browser name(s) and version(s)
chrome 115.0.5790.110, Firefox 115.0
Platform(s) and version(s)
Windows
Other
No response

Hi @JakubMaz,
It's the difference between the NativeAutomation mode and the legacy proxy mode.
You don't need to manually decompress data with the zlib.brotliDecompressSync method in NativeAutomation mode.
Google Chrome API, which is used in NativeAutomation mode, automatically decodes request data.
And you need to decode the response manually in legacy proxy mode.
In your example, the target request has gzip encoding. I modified the shared example to make it work completely.
import zlib from 'zlib';
import { RequestLogger, t } from 'testcafe';

const logger = RequestLogger({ url: /https:\/\/reqres.in\/api\/users/, method: 'get' }, {
    logResponseBody: true,
    logResponseHeaders: true,
    stringifyResponseBody: false,
});

const getBodyFromRequest = (request, isNativeAutomation) => {
    const reqHeaderContentEncoding = request.response?.headers?.['content-encoding'];
    const isGZipEncoding = reqHeaderContentEncoding === 'gzip';
    const buffer = request.response?.body;

    if (!buffer) {
        return {};
    }

    if (isNativeAutomation)
        return JSON.parse(buffer.toString('utf-8'));

    if (isGZipEncoding) {
        const decoded = zlib.gunzipSync(buffer);

        return JSON.parse(decoded.toString('utf-8'));
    }

    return decoded.toString('utf-8');
};

fixture('Test decode')
    .page('https://reqres.in/')
    .requestHooks(logger);

test('testDecodeBuffer', async () => {
    // setup before test run
    const isNativeAutomation = !!process.env.NATIVE_AUTOMATION === 'true';
    await t.expect(logger.contains(record => record.response.statusCode === 200)).ok({ timeout: 20_000});

    const requestBody = getBodyFromRequest(logger.requests[0], isNativeAutomation);
    console.log('requestBody', requestBody);
    /*
    ... test code
    */
});

Hi @JakubMaz,
We already have a PR  with this capability. It will be published in the nearest release after the merge.


AttributeError: Cannot find field 'polygons' in the given Instances!

The dataset script will leverage the 'polys' in the given annotation files and formulate the 'polygons' field of label instances. Maybe you should check the dataset annotation you are using.


Remove $ from readme commands

It's not part of command
Allows easier copy

i see the logic, but then we should be consistent. further down in the readme, we have the command line to run the tests. can you please adjust that as well?

done

thanks


How to create handler per 'successful_payment' webhook?

On the last stage of Payment API (Checkout) Telegram sends the webhook 'successful_payment' request.
I didn't find info how to create system command per that requests.
Also i can't override GenericmessageCommand.
Please help!

After debug Telegram.php found the solution:
<?php

declare(strict_types=1);

namespace Longman\TelegramBot\Commands\SystemCommands;

use Longman\TelegramBot\Commands\SystemCommand;
use Longman\TelegramBot\Conversation;
use Longman\TelegramBot\Entities\ServerResponse;
use Longman\TelegramBot\Exception\TelegramException;
use Longman\TelegramBot\Request;

class SuccessfulpaymentCommand extends SystemCommand
{
    /**
     * @var string
     */
    protected $name = 'successfulpayment';
    /**
     * @var string
     */
    protected $usage = '/successfulpayment';

    /**
     * @var string
     */
    protected $description = 'Handle successful payment';

    /**
     * Command execute method
     *
     * @return ServerResponse
     * @throws TelegramException
     */
    public function execute() : ServerResponse
    {
        $payment = $this->getMessage()->getSuccessfulPayment();

        error_log("Payment success #1");

        if($payment) {
            error_log("Payment success: {$payment->getProviderPaymentChargeId()} {$payment->getTotalAmount()} {$payment->getCurrency()}");
            return Request::emptyResponse();
        }

        return Request::emptyResponse();
    }

}


obscure @-moz-document selector

This selector:
@-moz-document url-prefix(http://) {

Throws this error:
(Line 455: Invalid CSS after "		input": expected ")", was "[type=submit].b...")

The line comes from Zurb Foundation, foundation.css, line 454.
It is fixed easily by quoting the http:// part:
@-moz-document url-prefix("http://") {

However I don't know if this breaks something in some version of Gecko, therefore I thought it might need to be fixed.

@tixz what version of Sass are you using?
From the spec, using double-quotes are safe in this context. @nex3 I think it's strange that we're parsing the @-moz-document directive. But if we are going to, it seems like we'll need to make url-prefix and quote like we do for url.

I'm using Sass 3.2.0.alpha.76 (Bleeding Edge)

@chriseppstein We aren't currently parsing @-moz-document; this is hitting the directive fallback code.
I think we should actually support the @-moz-document syntax, as we do for ::-moz-any and other vendor-specific syntaxes.

@nex3 I thought we didn't parse unknown directives and just consumed everything to the first ; or { as a string.

@chriseppstein That heuristic would fail when encountering quoted strings or other grammar edge cases. The fallback behavior for an unknown directive is to try to parse the tokens after it as a selector or, failing that, as an expression.


Feature flag directive

AC

[ ] Should be able to show/hide a feature in a template using a directive
[ ] Should be based on feature flags configuration


Imporve shebang detection for osascript when language is specified with -l argument

Part 2 of #234

Decrease in coverage is because for this tests are in latest lingust version


Add Debian Stretch

Add Debian Stretch Support

Waiting for Puppetlabs to release puppet-agent packages for stretch...

I thought we did? http://apt.puppet.com/dists/stretch/

Ah, not agent yet 😢

I've generated Facter 2.x fact sets for stretch in #47 (no 3.x facts yet as no puppet-agent packages are available for stretch yet)

@rodjek I think you mean #46.

D'oh, thanks @ekohl!

#46 was merged.


Weird space at end of profile table view

Update on this. I don't know the exact height, if you do, update the static height there.

Update: Will be fixed in Issue #200

Same issue that I just assigned you two so I am closing this


Try to switch field in the Forms.Layout representation by making it higher order type

If we consider "MultiField" widgets which should be well typed - I mean contain specific and well structured access to its input and result we can think of switching in Forms.Layout.Layout from:
data Layout
  = Section ...
  | Field { field ∷ field, input ∷ input, result ∷ result }

into
data Layout field input result
  = Section ...
  | Field (field input result)

I hope that by requiring BiFoldable, BiTraversable and Bifunctor instances on this field we can provide sensible layout and form related implementations of the validation process, but not only for single value fields but also for such multifield widgets.

Another question which can be raised is - could we work with Variants in such a case and provide a preset of fields and widgets (like simple text input, multitext input) but allow extending this preset with custom types too but still provide the above instances implementation for our contrib subset.

We are using fully polymorphic field now without exposing input or result.


chore!: txsim to use signer

Overview
txsim now uses the Signer class. This means txsim no longer requires RPC endpoints, relying solely on gRPC
Checklist

[ ] New and updated code has appropriate documentation
[ ] New and updated code has new and/or updated testing
[ ] Required CI checks are passing
[ ] Visual proof for any user facing features like CLI or documentation updates
[ ] Linked issues closed with keywords

It looks like the test workflow fails b/c a test is attempting to use the --rpc-endpoints flag which is removed in this PR. See here.

Codecov Report

Merging #2326 (2169dfd) into main (c4886d3) will increase coverage by 0.33%.
Report is 1 commits behind head on main.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2326      +/-   ##
==========================================
+ Coverage   20.24%   20.57%   +0.33%
==========================================
  Files         132      131       -1
  Lines       15508    15258     -250
==========================================
  Hits         3140     3140
+ Misses      12067    11817     -250
  Partials      301      301

Files Changed
Coverage Δ

pkg/user/signer.go
0.00% <0.00%> (ø)

test/cmd/txsim/cli.go
0.00% <0.00%> (ø)

test/txsim/account.go
0.00% <0.00%> (ø)

test/txsim/run.go
0.00% <0.00%> (ø)

test/txsim/send.go
0.00% <0.00%> (ø)

test/txsim/stake.go
0.00% <0.00%> (ø)


Tile icon downsizing loops several times

Could be optimized in future, but it effects only one icon, handled two times - so, feel free to propose a better solution :-)


Reference Microsoft.NET.Sdk.Razor in projects with Razor files

@pranavkm - thank you for looking into this!


More documentation to explain how to use or learn more about a chart

The README provides an example using Redis but there is no guidance on why this command's properties are necessary to deploy this Chart:
bin/helm deploy --name test --properties "workers=2" gs://kubernetes-charts-testing/redis-2.0.0.tgz

Clearly, the deployment contains 2 slaves (aka "workers") but how would I infer this?

This relates to an old version of Helm, but we should still make sure we address it in the current version


How to make my bot go offline?

Want to ask because I want to make my bot go offline to show that it is on maintenance

try closing your command prompt.
unless you want a specific message as the bot's status.


Improvement in UI of footer

Describe the bug
I would like to improve the footer in the website. as it is not differentiating from rest of the website.
Please assign me under Hacktoberfest'23.
Screenshots

Desktop (please complete the following information):

OS: Windows 11
Browser Chrome

i want to work on this issue can you please assign me this issue.

Nope, it doesn't need changing.


V6 Using Paper overrides style of unrelated components

Steps to reproduce
Link to live example: https://codesandbox.io/p/devbox/peaceful-estrela-forked-94sr78
Current behavior
App bar isn't blue
Expected behavior
App bar is blue
Context
When there is a paper component in the page, the paper css conflict
Your environment

npx @mui/envinfo
  Don't forget to mention which browser you used.
  Output from `npx @mui/envinfo` goes here.

Search keywords: v6 paper

I was able to reproduce the issue. The moment a Paper component is imported into the project, the AppBar (which uses Paper under the hood) becomes white. I'm transferring the issue to the Pigment CSs repo.


Remove lsp4j

Replaces lsp4j with a Clojure implementation. The full list of dependencies is now clojure, core.async, cheshire, and camel-snake-kebab.
Also merges the lsp4clj-server and lsp4clj-protocol projects into one top-level project, lsp4clj.
clojure-lsp has an accompanying PR.
@Cyrik, as stated in the CHANGELOG, other users of lsp4clj (you, I think) are encouraged to upgrade. You can follow that clojure-lsp PR as a guide on the changes that'll need to be made. They're fairly extensive, but I believe will make lsp4clj a more stable platform for LSP development. When you're upgrading if you'd like a little direction, I'd be happy to chat. Perhaps we can turn our discussion into edits to the Usage section in the README.
Notable changes:

As described in #8, servers can choose which requests and notifications they respond to by implementing the lsp4clj.server/receive-request and lsp4clj.server/receive-notification multimethods. (In contrast to the protocols based approach we had before.) This gives servers much more flexibility. They can implement new parts of the spec at their leisure, or according to the needs of their language. They can also choose when and how to implement asynchronicity. The tradeoff is that much of the spec checking and logging moves back to clojure-lsp and other servers. The servers also have to know and use the names of the LSP requests/notifications. Personally, I find clojure-lsp.server code to be more readable now, even though it has a few more responsibilities.
Input maps are all coerced to kebab-case. The server receives whatever the client sent, which should follow the LSP spec.
Output specs still exist, but do only very lightly conformation. Of what conformation remains, most of it converts keywords to enum values. There are a few places where more complex conformation happens. Honestly, I'd like to remove almost all conformation. I think it will be easier if you can look up what to output based on the LSP spec, not the LSP spec, plus a bunch of reasoning about coercion.
It is much less code to initialize a server. None of it is Java interop.
We have a path to working on #1. We need code that converts a socket into a pair of input and output channels. Otherwise, much of the code can stay the same.

Fixes #8.

Thanks for the feedback @snoe! I've made or responded to your suggested changes. Let me know if there's anything else you'd like to see changed.


404 page while accessing the webapi using json format

Hi,
I am trying to check the webapi with json as below and it is giving me 404 error.
cat command.json :

{
"api_request": {
"command": "CheckVersion",
"option1": "dummy1",
"value": "1.2.3",
"api_key": "oMPIgICN9KXsDhGgxfh"
}
}
curl -k --data-ascii @command.json https://<IP_ADDRESS>/versions /search.json?api_key=oMPIgICN9KXsDhGgxfh

TestCaseDB - (404)

The page you requested cannot be found.

        If you continue to experience issues with this page, please contact your system administrator.

Copyright © 2012 MSJ IT Services. All rights reserved.

This is intended behavior. You do not want the test only items available in the production environment.
You need to start the rails server in development or test mode and make sure the test/development environment and databases have been properly set up.
I believe to run in development or test mode you must run the setup script without the -p flag to create the databases and then for the gems run the rake command with the proper RAILS_ENV.
script/setup
rake assets:precompile RAILS_ENV=development

You may also have to restart the rails server.
passenger start -a <IP_ADDRESS> -p 3000 -d -e development

Something different may have to be done now with the updated version of rails but I'm not sure.

@joffems, can we have a simple automation script using v2 webapi ?

@kirantpatil can you be more specific? What language and what are you trying to do? The v2 API provides very detailed error messages. Are you not able to even send and receive a request? You van even test it with a GUI tool like API kitchen.

@agramian, I need a simple script to as shown in http://news.testcasedb.com/wp-content/uploads/2012/05/automation_example1.pdf  for testing GUI. I have no idea of how to access them and come up with a automated script.
I will check with API kitchen.
Thanks.

I would like to use ruby to automate.

First of all @kirantpatil let's go back to your first example.
If you look at https://github.com/msjit/testcasedb/blob/master/spec/webapi/versions_api_spec.rb you will find example usage of the versions endpoint.
Your code should be revised as follows and it should work.
contents of command.json:
{
"version": "1.2.3",
"api_key": "oMPIgICN9KXsDhGgxfh"
}

curl command to TestCaseDB v2 web api
curl -k --data-ascii @command.json https://<IP_ADDRESS>/api/versions/search.json

@agramian, yes now it worked. Thanks a lot.


Invalid paths in go generate directive

Problem statement
Invalid path in go generate directive in generated/restapi/configure*.go file
Steps to reproduce

Run swagger generate
swagger generate server -f ./swagger.yaml -t ./generated --exclude-main
Go to generated/restapi/configure*.go
There I have comment

//go:generate swagger generate server --target ../generated --name ExampleSwaggerServer --spec ../swagger.yaml --exclude-main

Which is not right, because swagger.yaml and generated directory are 2 levels up, but not one.
If you try to run go generate you will get an error couldn't find a swagger spec
So comment should be
//go:generate swagger generate server --target ../../generated --name ExampleSwaggerServer --spec ../../swagger.yaml --exclude-main

Environment
swagger version: v0.17.2
commit: 800a3052f0305ab54980f961bd4f1e8b3e5599fe
go version: 1.10.3
OS:   Ubuntu

That's right.
This statement is defined here in templates:
https://github.com/go-swagger/go-swagger/blob/be966a22e91309b7ed8d57d20e737ab5b3861cbd/generator/templates/server/configureapi.gotmpl#L28
The generate_xxx.go is actually put in {{ ServerPackage }} relative to {{ TargetPath }}


botania version requirement seems a little wrong (see image)

https://imgur.com/dCp1raD

They say a picture speaks a thousand words lol

I didn’t know anyone would ever use the snapshot version :D will change that soon


Philips Mifare 1 S50 tag compatible?

Hello I got this reader kit on Ebay:
http://www.ebay.com/itm/371104303227
The reader has no response to the keychain tag but responds to the card.  The description says the tag has the following IC:  "Philips Mifare 1 S50"  Should the library work with this tag?
Thanks,
Stephen

It should work, recheck wiring and try not to use a protoboard,  always use a direct wire soldering.
Good.luck!
Enviado desde BlueMail
En 22 ago. 2016 3:06, en 3:06, stevekramer99<EMAIL_ADDRESS>escribió:

Hello I got this reader kit on Ebay:
http://www.ebay.com/itm/371104303227
The reader has no response to the keychain tag but responds to the
card.  The description says the tag has the following IC:  "Philips
Mifare 1 S50"  Should the library work with this tag?
Thanks,
Stephen
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/miguelbalboa/rfid/issues/233

The reader works fine with the card, but does not respond at all to the
keychain tag.  I don't touch the wiring between reading them just scan each
one sequentially.  There is no response to the keychain fob.  It must be
faulty, please send a replacement correctly working tag.
On Aug 22, 2016 6:44 AM, "SimbadMarino"<EMAIL_ADDRESS>wrote:

It should work, recheck wiring and try not to use a protoboard,  always
use a direct wire soldering.
Good.luck!
Enviado desde BlueMail
En 22 ago. 2016 3:06, en 3:06, stevekramer99<EMAIL_ADDRESS>escribió:

Hello I got this reader kit on Ebay:
http://www.ebay.com/itm/371104303227
The reader has no response to the keychain tag but responds to the
card. The description says the tag has the following IC: "Philips
Mifare 1 S50" Should the library work with this tag?
Thanks,
Stephen
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/miguelbalboa/rfid/issues/233

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/miguelbalboa/rfid/issues/233#issuecomment-241376755,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AETnkvQyfYdpIUayhXyQjM95sogWRYznks5qiX2FgaJpZM4JpmqM
.

I AM SORRY, I SENT THIS TO THE WRONG PERSON!  I was supposed to send this
to the ebay seller!
Thanks for your help.
On Aug 22, 2016 10:51 AM, "Stephen Kramer"<EMAIL_ADDRESS>wrote:

The reader works fine with the card, but does not respond at all to the
keychain tag.  I don't touch the wiring between reading them just scan each
one sequentially.  There is no response to the keychain fob.  It must be
faulty, please send a replacement correctly working tag.
On Aug 22, 2016 6:44 AM, "SimbadMarino"<EMAIL_ADDRESS>wrote:

It should work, recheck wiring and try not to use a protoboard,  always
use a direct wire soldering.
Good.luck!
Enviado desde BlueMail
En 22 ago. 2016 3:06, en 3:06, stevekramer99<EMAIL_ADDRESS>escribió:

Hello I got this reader kit on Ebay:
http://www.ebay.com/itm/371104303227
The reader has no response to the keychain tag but responds to the
card. The description says the tag has the following IC: "Philips
Mifare 1 S50" Should the library work with this tag?
Thanks,
Stephen
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/miguelbalboa/rfid/issues/233

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/miguelbalboa/rfid/issues/233#issuecomment-241376755,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AETnkvQyfYdpIUayhXyQjM95sogWRYznks5qiX2FgaJpZM4JpmqM
.

@stevekramer99 Did it work with the Philips Mifare 1 S50 ?


[FEATURE REQUEST] Show unapproved, uncleared and unreconciled count instead of white dot

I'd like to see at a glance which accounts have unapproved, uncleared and unreconciled transactions, and how many there are.
We already get a white dot to the left of the account when there are unapproved transactions, what if this indicator told us how many are unapproved, then once all transactions are approved, it showed us how many are uncleared, then when all are cleared it showed us how many are unreconciled, finally rewarding us with an "all reconciled" indicator (or disappearing altogether).
I'd replace the white dot that indicates there are unapproved transactions with a white number showing the count, this decrements as items are approved
When there are no more unapproved, but there are uncleared, change the indicator to a gray circle containing a white number of uncleared (which visually echoes the uncleared indicator for each transaction in the account), this decrements as items are cleared.
When there are no unapproved or uncleared transactions, change the indicator to a green unlocked padlock with a white number of unreconciled transactions.
When everything is approved, cleared and reconciled in an account change to a solid green locked padlock. or perhaps remove the indicator all together.
You could also make each stage optional:
"replace white dot unapproved indicator with white number of unapproved, disappear when all items are approved"
"white number of unapproved, then gray number of uncleared, disappear when all items are approved and cleared"
"white number of unapproved, then gray number of uncleared, green c when all items are approved and cleared"
"white unapproved, gray uncleared, green unreconciled, disappear when all reconciled"
"white unapproved, gray uncleared, green unreconciled, green padlock when all reconciled"

I added a feature request tracker to our trello here: https://trello.com/c/PR5kvC5G


apl.interactive.plot is not working anywhere

Hello. Tryed to use interactive plot in python and in colab, both don't work.
AttributeError                            Traceback (most recent call last)
 in <cell line: 1>()
----> 1 apl.interactive.plot([0, 1, 2, 3, 2, 1, 0])
2 frames
/usr/local/lib/python3.10/dist-packages/bokeh/core/has_props.py in _raise_attribute_error_with_matches(self, name, properties)
366             matches, text = sorted(properties), "possible"
367
--> 368         raise AttributeError(f"unexpected attribute {name!r} to {self.class.name}, {text} attributes are {nice_join(matches)}")
369
370     def str(self) -> str:
AttributeError: unexpected attribute 'plot_width' to figure, similar attributes are outer_width, width or min_width

Thank you so much for reporting the issue and sorry for the inconvenience.
I guess it is probably due to an upgrade of a library that depends on it.
I will attempt to fix it this weekend and will mention it to you.

oh thank you! I am very very excited to try it out, but can I pip install the old library as a workarround?

Actually i also want an image explorer that turns the image into sinewaves. e.g. if there is a circle image, 2 sinewaves are plotting the circle shape (one goes up, other goes down. and than both go to the same point, representing circle).
i'd like to open a png or svg image and view it's soundplot.

Some simple demo interactive html can be found at the followings:

https://a11y-ds-intro.hassaku-labs.com/chapter-3/audio-plot-lib.html
https://hassaku.github.io/DS-and-ML-with-screen-reader/

I would be happy to help.

@hassaku The html code is very big and difficult to understand. I thaught it would all be 1 small js library that makes all work with sliders and graphs and i thaught it would be that easy as this:
<script src="path/to/audioplotlib.js"></script>
<script>
apl=new AudioPlotLib("id of the <div>", x=[-5,-4,-3,...], y=[...]);
</script>

and it would automatically add the unmute button, make a slider and draw the graph in canvas for not blind people!
well, i'm just sharing idea.

@denizsincar29
Sorry it took so long to fix, I hope #4 has resolved the error reported in this Issue.
Please try it when you have time.
And I'm sorry that this library is not very simple, as it is very dependent on the Bokeh functionality.

well, it's working fine


Immediate calling of functions

In #331 support for immediate indexing, accessing and calling functions was added, however, it is still not possible to do things like:
fn -> int { 2 }()
// or
fn -> int { 2 }'

Should this be fixed? Because it is still possible to call the functions if they are wrapped in parenthesis like this:
(fn -> int { 2 })()

I think it's a cool construct to support, especially if we already support the wrapped version.


demo is broken :'( (readme link)

will fix soon, please be patient

fixed

502 again!

thanks for feedback. this time it is because of running out of memory. we are investigating why it happened