Warnings from static analysis with clippy

Hello,
I have been reading through the Ketos code which has been a great learning experience -- thanks.
I noticed that the Rust static analyser called "clippy" is now available on Rust stable:
rustup component add clippy-preview

More information on installing is given in the github repository.
I ran clippy on the Ketos code and it made lots of suggestions that I would mostly consider to be useful. I have attached the report clippy.log
Would you be open to people making PRs to make the changes suggested by clippy? Or do you think that the changes suggested  by clippy are not helpful?

I looked over the clippy warnings and fixed a few non-trivial issues myself.
I would be happy to accept a PR that addresses the remaining warnings.

I think that #46 may fix this.


[WIP] server: fix fips_mode stat

Signed-off-by: Ravindra Akella<EMAIL_ADDRESS>
For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md
Commit Message: Fix fips_mode stat  by using a static variable to check if the ssl version is fips compliant or not.
Additional Description: Originally added as part of https://github.com/envoyproxy/envoy/pull/14719
Risk Level: Low
Testing: Updated unit tests
Docs Changes: None. Already documented
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

@raakella Are you able to iterate locally to get basic tests passing with bazel test //test/...? That might be a bit faster than trying to iterate over CI, at least for the default build.

/wait

@jmarantz Locally it has been working. Seems like something is broken when these test are run in CI. Seems like https://github.com/envoyproxy/envoy/blob/main/source/common/version/version.cc#L41 is evaluating to true from tests but from the actual code, it is evaluating to false.
I tried locally with "--define boringssl=fips" but that did not work it. A

/wait

windows failure was a flake in pulling build image.
/retest

@ggreenway @lizan Please merge this.


The fastly compute serve command errors out when using versions > 0.8.9

Version
Fastly CLI version v10.0.0 (daf4824e)
Built with go version go1.20.4 linux/amd64
Viceroy version: viceroy 0.5.0

What happened
Updated our computed edge service to use fastly 0.9.2. This service is utilizing the rust sdk. Ran the command
fastly compute serve

This is the error log I am receiving
✓ Verifying fastly.toml
✓ Identifying package name
✓ Identifying toolchain
✓ Running [scripts.build]
✓ Creating package archive

SUCCESS: Built package (pkg/users-api-proxy.tar.gz)

✓ Running local server

INFO: Listening on http://<IP_ADDRESS>:7676

INFO: Command output:
--------------------------------------------------------------------------------
2023-05-17T15:03:36.343726Z ERROR unknown import: `fastly_http_resp::http_keepalive_mode_set` has not been defined
--------------------------------------------------------------------------------

Here is the Cargo.toml for reference
[package]
name = "name"
version = "0.1.0"
authors = []
edition = "2021"
# Remove this line if you want to be able to publish this crate as open source on crates.io.
# Otherwise, `publish = false` prevents an accidental `cargo publish` from revealing private source.
publish = false

[profile.release]
debug = 1

[dependencies]
fastly = "^0.9.2"
log = "^0.4.17"
log-fastly = "^0.8.7"
jwt-simple = "0.10"
serde = "1.0.160"
regex = "1"

[dev-dependencies]
wasm-bindgen-test = "0.3.0"

Thanks @Integralist!


Suggestion for multi-use

Some suggestions voor a alternative faster non-external copy-paste solution?
Let me know and i will work out the other pages ;)

That's all for now ;)

@Kelper0N I've just published a completely refactored set of error pages with a new layout, bootstrap 5 and improved ease of customization. I'm going to close this.


Drop configure AuthorizationMessageHandler guidance

Fixes #18778
The original example came in from the Preview 5 blog post at https://devblogs.microsoft.com/aspnet/blazor-webassembly-3-2-0-preview-5-release-now-available/. It's at the end of the Attach tokens to outgoing requests section, except that we changed the registration from AddSingleton to AddTransient later. IIRC, I tested it back in those days, and it worked back then.

That code snippet is wrong. Likely because you are not using HttpClientFactory. In that case you need to create an HttpClientHandler instance and assign it as the innerhandler of the AuthorizationMessageHandler manually.

☝ Didn't work for me ... no doubt due to my not implementing it exactly right. The error is that the AuthorizationMessageHandler isn't configured ... but of course that was the goal of my code. Clearly, I botched it.
In any case, you said earlier that the custom AuthorizationMessageHandler class is the preferred approach. I tested that, and it works great. :tada: Therefore, I punt 🏈 here and drop the errant section. If you want to keep that section, give me the code example to make it work.
btw ... The change on Line 56 is a passing nit. We need to start code fencing ALL C# for localization. The MT is translating these. Yikes!
Thanks @paulguz! :rocket: ... sorry we didn't get this sooner. You're the first one to report a problem with that example/approach.

Tried again this morning ... no 🎲🎲.
The custom AuthorizationMessageHandler class works well.
Side ❓ tho: For the section on using a custom AuthorizationMessageHandler, should we add a note for MSAL users (AddMsalAuthentication) that one doesn't need to specify the ProviderOptions.DefaultAccessTokenScopes?

This PR will be re-opened after the ToC updates are complete.


Added my name to guestbook.md

Thank you very much for your course.

Thank you! I’m so happy you enjoyed it!


lexer.d: do short integer literal optimization

The idea is to have a special fast track for integer literals that are one digit long.

I think we should only pull optimization code if there's measurements showing it's improving speed.

Fair 'nuff. Lexing time for 1 digit numbers dropped by 41%, lexing time increased for 2 digit numbers by 3%. I think this makes it a clear win.

microbenchmark is unconvincing, but passons


Add enable toggle?

This is handy to have if files are huge, causing Sublime to bog down with highlighting.  A better approach might be to back off on highlighting if it takes over a certain threshold of time, or something... but this was "good enough" for my needs :)

It looks good; I'm just wondering if, like you said, we should take some other approach to this. What do you think?

There's two approaches I was thinking of:
1-Dynamically backing off the selection_delay (and ramping it back up again after some period).
2-Only searching in the currently visible text.
My gut says #1 would require a fair amount of tweaking to make it "feel good".  It also wouldn't fix the underlying "search the whole file and be slow" problem with large files.
#2 isn't entirely solvable given the Sublime API right now (see this post ).  However, I guess you could have a heartbeat thread that watches for changes in view.visible_region, and do a pend_highlight_occurences if it's changed?
Regardless, both take a sec to implement, and it feels like those could easily be added in addition to a "master enable switch" feature, which is useful now ;).

I rebased this and pushed it. There was another change that makes it so that we don't search files over 4MB or something, but now this'll be in there too in case people want it.


PaginationService : FontSize

Hi,
To use Unicode's Private Use Area,
I need to modify the size of caracters because the default value is 0.
Can you create a function to modify this variable ?
PaginationCalculator.java

@lesbleu unless these values are not matching on the server, I believe you would also need to update them on the clients. Reading Mojira bug reports has me believing that specifying them in a server resource pack wouldn't be enough.
Yes it's an issue, one we could probably solve by recreating these values if a serverside resource pack was specified, but it's a lot of work.
The Unicode Private Use Area is super useful though, I'd love to see some richer chat interfaces because of it.
I just feel that including this in the pagination calculator, with no other utilities however is probably too small of a Usecase.
Would you be happy just specifying these values yourself (easy but not a nice API)? Have the server calculate them from a resource pack instead (hard, current method relies on client side only code, would have to be re-implemented and kept up to date on the server)?
Another option to consider, is in the 2 years since this file was introduced, running sponge on the client has become a possibility, maybe we should just expose the ability to read
https://github.com/SpongePowered/SpongeCommon/blob/bleeding/src/main/resources/org/spongepowered/common/service/pagination/font-sizes.json
From an external location, generatable by users using Sponge on the client, and promote the code within to a proper command?

I'm going to close this. The PaginationService in API-8 is replaceable - and I would suggest those wanting extra functions like this should create their own service.


added !removefriend command

This command will basically remove a friend (theoretically).
dont kill me ok

No? Then it removes "steamID64" steamid64 isn't defined.

dis better


Refactor json and xml mods into typespec crate

Actually, given something @JeffreyRichter said in a Go review the other day, I should pull these back up to typespec_client_core and we can move them down the dependency stack later if and when needed. We can't pull them up post-GA.


When adding new exportables (env vars, aliases, etc.), parametrize the ID type with generics

Found a TODO comment on line 16 of file crates/nu-protocol/src/engine/engine_state.rs

Duplicated, closing it.


fix(CI): disable cache

see travis-ci/travis-ci#10113
disabling cache ain't ideal since it adds about 1 min per build, but seems the only way to keep running with node 6 for now.
no-cache
with cache

superseded by #16


Parse relationships of patients

Relationships of patients are listed under the travel_hx field (travel history). Create a way to extract the data as accurately as possible.
Sample data
(HOUSEHOLD MEMBER OF PH9) HUSBAND OF PH30
(HOUSEHOLD MEMBER OF PH9) NIECE OF PH9
(HOUSEHOLD OF PH9) WIFE OF PH28
(HOUSEHOLD OF PH9) WIFE OF PH9
BROTHER OF PH52
CONTACT OF PH141
CONTACT OF PH143
CONTACT OF PH157
CONTACT OF PH158
CONTACT OF PH164
CONTACT OF PH169 AND PH170
CONTACT OF PH170 AND PH176
CONTACT OF PH173
CONTACT OF PH35
CONTACT OF PH42, PH43
CONTACT OF PH43
CONTACT OF PH43
CONTACT OF PH49 AND PH52
DAUGHTER OF PH134, SISTER OF PH51
HUSBAND OF PH166
HUSBAND OF PH17
HUSBAND OF PH35
HUSBAND OF PH6
HUSBAND OF PH90
MOTHER OF PH135 AND PH51
NO TRAVEL HISTORY; CONTACT OF FN2
RELATIVE OF PH41, PH44, PH87
SISTER OF PH41 AND PH44; RELATIVE OF PH112
SISTER OF PH41 AND PH87; RELATIVE OF PH112
SON OF PH134, BROTHER OF PH135
SON OF PH169 AND PH176
TAIWAN; HUSBAND OF PH19
USA; SISTER OF PH44 AND PH87; RELATIVE OF PH112
UNITED KINGDOM; SISTER OF PH49
WIFE IS SISTER OF PH12; CLOSE CONTACT OF PH35
WIFE OF PH16
WIFE OF PH165
WIFE OF PH18
WIFE OF PH32
WIFE OF PH34
WIFE OF PH5
WITH EXPOSURE HISTORY TO PH5
WITH EXPOSURE HISTORY TO PH9
WITH EXPOSURE HISTORY TO A KNOWN COVID-19 CASE
WITH EXPOSURE HISTORY TO A KNOWN COVID-19 CASE
WITH EXPOSURE HISTORY TO A KNOWN COVID-19 CASE
WITH EXPOSURE TO PH21
WITH EXPOSURE TO PH21
WITH EXPOSURE TO PH21 AND PH67
WITH EXPOSURE TO KNOWN COVID-19 CASE
WITH EXPOSURE TO KNOWN COVID-19 CASE
WITH EXPOSURE TO KNOWN COVID-19 CASE
YES (CHINA); HUSBAND OF PH1
YES (CHINA); WIFE OF PH2
YES (UAE)
YES (USA, KOREA); IN SAME HOUSEHOLD WITH PH27-30; CONTACT OF PH31

Dropping this feature. New cases no longer have contact tracing data.


objective-c cannot jump to definition, I sure vscode support it by built-in.

Please see https://github.com/microsoft/vscode/issues/94735

See https://github.com/Microsoft/vscode-cpptools/issues/2533
AFAIK this is is the home of the built-in extension that provides VS Code's default C and C++ support.

And the built-in objective-can extension only implements syntax colouring
https://github.com/microsoft/vscode/blob/master/extensions/objective-c/package.json

/duplicate #94735 #94867 please do not open more duplicates

https://github.com/microsoft/vscode/blob/master/extensions/objective-c/build/update-grammars.js

Doesn't this file already show that defined jumps are supported?
I don't understand why you shut it down every time you don't answer a question.
If I wrote it in the closed answer, you wouldn't see it.

/duplicate #94735 #94867 please do not open more duplicates

No, that is GitHub telling you about code navigation in the files in GH.

Finally, thank you for your patience.


Restoring model fails: InputStream.available() = 0

Issue Description
I am trying to load a model from a stream i received from a REST Api. Every time i try to load it the method https://github.com/eclipse/deeplearning4j/blob/master/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/util/ModelSerializer.java#L956-L971 throws an error as the available() function on the input stream always returns 0.
The method is also annotated with a comment referencing another issue: https://github.com/eclipse/deeplearning4j/issues/4887. In the issue thread there is a commit commenting out the method. It looks like this change has been reverted by the migration to this repository.

@PossessedC0bra  do you have the model? That sounds like a corrupt zip file.

Yes i do have it. It is clean though, i have already checked that.
Funny thing is when running the application locally it works perfectly fine. Also when attaching a remote debugger and placing a breakpoint at the location, where inputStream.available() is called prevents the error from ever happening. As soon as i remove the breakpoint the remote application runs into the same InputStream.available = 0 error every single time.
Might this have something to do with latency over the network or something?

I restore model from Spring GridFS and have the same problem with InputStream.available() = 0

Sorry just doing clean up here. There wasn't really a lot to go on here. I'm guessing this wasn't a bug. For future issues like this please post on the forums: https://community.konduit.ai - if we don't respond please feel free to ping (not excessively of course :) ) - thanks!


NIXPKGS_ALLOW_UNFREE=1 test fails for n8n

basically
devenv-run-tests --only n8n

and

tried also

devenv.yaml: allowUnfree: true
.envrc NIXPKGS_ALLOW_UNFREE=1 use devenv

still same error

devenv.yaml: allowUnfree: true

This option should work. The devenv.yaml should be placed in the same directory as the devenv.nix for the test/example.

i'll give it a shot

that worked! thanks


logonWithSessionToken does not set userId

Hey! Wondering if there's a good way here to resume a logged in user session -- I was planning to just set a token + id in NSUserDefaults, but logging back in with the session token doesn't seem to set the userId (which I need), so I figured what I should do. (e.g. I could just set a singleton, but I'd rather not if possible)

Actually I can probably PR this -- seems like a few line fix.

ignore me had an old version


Ability to set start and end dates for incidents

I'd like to be able to retrospectively load incidents into OneUptime, either via UI or at least the API. Often the actual downtime will not correspond directly to the incident time.
Describe the solution you'd like
I'd like the ability to set a start and end date to an incident when creating or editing.
Describe alternatives you've considered
I can't find a way to do this - alternative is probably using another tool since it's pretty important for my use case.

+1


Add linear scipy projected regridder

This PR adds the linear scheme of the regridder that uses scipy.interpolate.griddata.
I have also added some more tests for the nearest-neighbour scheme as source cubes with aux_factories weren't being tested, as highlighted by @pp-mo in #2263
Note the test data doesn't have much variation in the data within the layers themselves (between the layers there's quite a difference, hence the large values for the standard deviation which is calculated for the whole cube), so I was getting similar values for mean and standard deviation, regardless of whether I used the linear or nearest (see the test TestProjectedUnstructured.test_use_default_rtol, which I do intend to remove, I just added it to prove what I had found).
I have solved this problem by allowing the tolerance to be set but I don't think this is necessarily a great solution. Another option would be to calculate the mean and standard deviation for a level.

Hi @lbdreyer
given out current status, I would appreciate you targeting this against 1.12.x, rather than master
this has just had #2275 merged, so the chance of failure due to http retry is reduced for 1.12.x PRs
thank you
mark

i'm pretty happy with the code and testing.
please may we have a what's new entry?

@marqh I have added a whatsnew entry which applies to the ProjectedUnstructured nearest and linear schemes.
I have also removed the test I used to show that testing with a low tolerance was giving the same results regardless of whether the linear or nearest scheme was used.
I have instead replaced this with testing against the whole cube (with a high tolerance), as well as a single layer.
(see the most recent commit)

this looks good to go now, thanks @lbdreyer


chore: build on Ubuntu 12.04 Docker base images

We recently discovered that users not running latest GNU/Linux
distributions were having problems when Etcher tried to load native
modules. We used to build packages on Ubuntu 16.04, which ships with
GLIBC 2.23.
In order to solve that issue, we moved our build Docker containers to
Debian Jessie, which ships with GLIBC 2.19, however we noticed that
users running even older Linux distributions were still having the same
issue, so this commit switched the Docker containers to Ubuntu 12.04,
which ships with GLIBC 2.15.
See: https://github.com/resin-io/etcher/pull/1319
Signed-off-by: Juan Cruz Viotti<EMAIL_ADDRESS>

Rather than committing near-identical changes to 3 separate files, each time Dockerfile.template gets modified, I guess we could remove the arch-specific Dockerfiles from github, and have run-command.sh create them on demand?

Yeah, could be. We're going to introduce ARM Dockerfiles soon, which look like will need some different steps, so I wonder if we should maintain them separate, rather than introducing a complex template mechanism.

Fair enough, I'll wait to see what happens with the ARM versions then before making any drastic changes ;)


Parallel search

Improve searching bar not separate API, DB but do both

Improve searching bar not separate API, DB but do both

Now it works well. Let's close this.


Support SchemaRegistryClient to handle cluster URLs with url selectors

SchemaRegistryClient should be able to take clusterUrl which contains sequence of schemaregistry urls separated by , and it should be able to handle failover of the target schema registry instances.

PR #33 was raised and merged to address this issue.


Fix for soa optimizations

Fix for SOA optimizations(https://github.com/intel/daal/pull/422) to be able to build examples with pedantic mode(https://github.com/intel/daal/pull/435).

make test


Add method to format error

@benblack86 @DanSimon @aliyakamercan @jlbelmonte @amotamed
My stab at https://github.com/tumblr/colossus/issues/414

Codecov Report

Merging #589 into develop will decrease coverage by 0.01%.
The diff coverage is 75%.

@@             Coverage Diff             @@
##           develop     #589      +/-   ##
===========================================
- Coverage    84.94%   84.93%   -0.02%
===========================================
  Files           99       99
  Lines         4431     4433       +2
  Branches       378      352      -26
===========================================
+ Hits          3764     3765       +1
- Misses         667      668       +1

Impacted Files
Coverage Δ

...rc/main/scala/colossus/service/ServiceServer.scala
92.15% <75%> (-0.85%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3853de...7f199d6. Read the comment docs.

Just trying to think how it would be overriden:
def formatError(error: Throwable): String = {
  error match {
    case DroppedReplyException => "DroppedReplyException"
    case _ => // get stack trace
  }
}

The problem is that it is a bit of work to extract the stack trace since that functionality is contained within the logger. Maybe it would be more useful to be able to indicate whether an error should be logged. There could be three levels:

don't log
log name only
log name and stack trace

It would be great if this stuff can be configurable (like the metrics) so by default the DroppedReplyException would only log name, but everything else would be name and stack trace.

I like the general idea.  What do you guys think about having formatError also take in the failed request as a parameter, and then using the returned string as the entire error message?
I also agree with what @benblack86 said.  Having an ADT like:
sealed trait LogAction
case object DoNotLog extends LogAction
case class Log(message: String, includeStackTrace: Boolean) extends LogAction

or something similar seems like a good idea.  Also #414 does mention doing something for exceptions like DroppedReplyException.  I'm ok if that's not addressed in this PR, but we should then create another issue for it.

bump. will add stuff in conf soon

Looks good so far.

bump @DanSimon

Might be wrong, but I don't think the RequestFormatter is documented. Can you add/update the docs.

bump

bump


Adding operator tests for 4.15 and 4.17

Description

Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library

Related Tickets & Documents

Related Issue #
Closes #

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] PR has passed all pre-merge test jobs.
[ ] If it is a core feature, I have added thorough tests.

Testing

Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

/override "Red Hat Konflux / ols-enterprise-contract / bundle "
/override "Red Hat Konflux / ols-enterprise-contract / test-bundle"
/override "Red Hat Konflux / ols-operator-e2e-tests-417 / bundle"
/override "Red Hat Konflux / ols-operator-e2e-tests-417 / test-bundle"

/override "Red Hat Konflux / ols-enterprise-contract / bundle"

/override "Red Hat Konflux / ols-enterprise-contract / test-bundle"
/override "Red Hat Konflux / ols-operator-e2e-tests-417 / bundle"
/override "Red Hat Konflux / ols-operator-e2e-tests-417 / test-bundle"


[SPARK-21428] Turn IsolatedClientLoader off while using builtin Hive jars for reusing CliSessionState

What changes were proposed in this pull request?
Set isolated to false while using builtin hive jars and SessionState.get returns a CliSessionState instance.
How was this patch tested?
1 Unit Tests
2 Manually verified: hive.exec.strachdir was only created once because of reusing cliSessionState
➜  spark git:(SPARK-21428) ✗ bin/spark-sql --conf spark.sql.hive.metastore.jars=builtin

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/07/16 23:59:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/16 23:59:27 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
17/07/16 23:59:27 INFO ObjectStore: ObjectStore, initialize called
17/07/16 23:59:28 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
17/07/16 23:59:28 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
17/07/16 23:59:29 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
17/07/16 23:59:30 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/07/16 23:59:30 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/07/16 23:59:31 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/07/16 23:59:31 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/07/16 23:59:31 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
17/07/16 23:59:31 INFO ObjectStore: Initialized ObjectStore
17/07/16 23:59:31 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/07/16 23:59:31 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/07/16 23:59:32 INFO HiveMetaStore: Added admin role in metastore
17/07/16 23:59:32 INFO HiveMetaStore: Added public role in metastore
17/07/16 23:59:32 INFO HiveMetaStore: No user is added in admin role, since config is empty
17/07/16 23:59:32 INFO HiveMetaStore: 0: get_all_databases
17/07/16 23:59:32 INFO audit: ugi=Kent	ip=unknown-ip-addr	cmd=get_all_databases
17/07/16 23:59:32 INFO HiveMetaStore: 0: get_functions: db=default pat=*
17/07/16 23:59:32 INFO audit: ugi=Kent	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*
17/07/16 23:59:32 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
17/07/16 23:59:32 INFO SessionState: Created local directory: /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/beea7261-221a-4711-89e8-8b12a9d37370_resources
17/07/16 23:59:32 INFO SessionState: Created HDFS directory: /tmp/hive/Kent/beea7261-221a-4711-89e8-8b12a9d37370
17/07/16 23:59:32 INFO SessionState: Created local directory: /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/Kent/beea7261-221a-4711-89e8-8b12a9d37370
17/07/16 23:59:32 INFO SessionState: Created HDFS directory: /tmp/hive/Kent/beea7261-221a-4711-89e8-8b12a9d37370/_tmp_space.db
17/07/16 23:59:32 INFO SparkContext: Running Spark version 2.3.0-SNAPSHOT
17/07/16 23:59:32 INFO SparkContext: Submitted application: SparkSQL::<IP_ADDRESS>
17/07/16 23:59:32 INFO SecurityManager: Changing view acls to: Kent
17/07/16 23:59:32 INFO SecurityManager: Changing modify acls to: Kent
17/07/16 23:59:32 INFO SecurityManager: Changing view acls groups to:
17/07/16 23:59:32 INFO SecurityManager: Changing modify acls groups to:
17/07/16 23:59:32 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(Kent); groups with view permissions: Set(); users  with modify permissions: Set(Kent); groups with modify permissions: Set()
17/07/16 23:59:33 INFO Utils: Successfully started service 'sparkDriver' on port 51889.
17/07/16 23:59:33 INFO SparkEnv: Registering MapOutputTracker
17/07/16 23:59:33 INFO SparkEnv: Registering BlockManagerMaster
17/07/16 23:59:33 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/07/16 23:59:33 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/07/16 23:59:33 INFO DiskBlockManager: Created local directory at /private/var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/blockmgr-9cfae28a-01e9-4c73-a1f1-f76fa52fc7a5
17/07/16 23:59:33 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
17/07/16 23:59:33 INFO SparkEnv: Registering OutputCommitCoordinator
17/07/16 23:59:33 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/07/16 23:59:33 INFO SparkUI: Bound SparkUI to <IP_ADDRESS>, and started at http://<IP_ADDRESS>:4040
17/07/16 23:59:33 INFO Executor: Starting executor ID driver on host localhost
17/07/16 23:59:33 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51890.
17/07/16 23:59:33 INFO NettyBlockTransferService: Server created on <IP_ADDRESS>:51890
17/07/16 23:59:33 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/07/16 23:59:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, <IP_ADDRESS>, 51890, None)
17/07/16 23:59:33 INFO BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:51890 with 366.3 MB RAM, BlockManagerId(driver, <IP_ADDRESS>, 51890, None)
17/07/16 23:59:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, <IP_ADDRESS>, 51890, None)
17/07/16 23:59:33 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, <IP_ADDRESS>, 51890, None)
17/07/16 23:59:34 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/Kent/Documents/spark/spark-warehouse').
17/07/16 23:59:34 INFO SharedState: Warehouse path is 'file:/Users/Kent/Documents/spark/spark-warehouse'.
17/07/16 23:59:34 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
17/07/16 23:59:34 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
17/07/16 23:59:34 INFO HiveMetaStore: 0: get_database: default
17/07/16 23:59:34 INFO audit: ugi=Kent	ip=unknown-ip-addr	cmd=get_database: default
17/07/16 23:59:34 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
17/07/16 23:59:34 INFO HiveMetaStore: 0: get_database: global_temp
17/07/16 23:59:34 INFO audit: ugi=Kent	ip=unknown-ip-addr	cmd=get_database: global_temp
17/07/16 23:59:34 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
17/07/16 23:59:34 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
17/07/16 23:59:34 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
spark-sql>

cc @cloud-fan @gatorsmile

Can one of the admins verify this patch?

ping @gatorsmile could you help to review this？

OK to test

ok to test

Test build #80442 has started for PR 18648 at commit 6c0bf70.

Test build #80442 has finished for PR 18648 at commit 6c0bf70.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80442/
Test FAILed.

retest this please

Test build #80448 has started for PR 18648 at commit 51fac11.

Test build #80448 has finished for PR 18648 at commit 51fac11.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80448/
Test PASSed.

Test build #80458 has started for PR 18648 at commit c50a32f.

Test build #80460 has started for PR 18648 at commit c6ed2d7.

Test build #80458 has finished for PR 18648 at commit c50a32f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80458/
Test PASSed.

Merged build finished. Test PASSed.

Test build #80460 has finished for PR 18648 at commit c6ed2d7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80460/
Test PASSed.

ping @jiangxb1987 @cloud-fan anymore suggestions？

ping @jiangxb1987 @cloud-fan again

LGTM

ping @cloud-fan would you take another look？

@jiangxb1987 Could this pr be merged？

@cloud-fan

LGTM, merging to master!

The description is not clear, at least I get understood after diving into the code changes.


tour_mode_choice throws runtime error for some tours

Runtime error in tour_mode_choice_simulate running the full zone dataset with 12000 HH sample and random seed of 0
settings.yaml:
preload_3d_skims: True
households_sample_size: 12000

simulation.py:
def set_random_seed():
    np.random.seed(0)

orca.add_injectable("set_random_seed", set_random_seed)

Stack trace on error:
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 2411, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1802, in run
    launch(file, globals, locals)  # execute the script
  File "/Users/jeff.doyle/work/activitysim/sandbox/simulation.py", line 104, in <module>
    orca.run(['tour_mode_choice_simulate'])
  File "/Users/jeff.doyle/anaconda/envs/asim/lib/python2.7/site-packages/orca/orca.py", line 1876, in run
    step()
  File "/Users/jeff.doyle/anaconda/envs/asim/lib/python2.7/site-packages/orca/orca.py", line 780, in __call__
    return self._func(**kwargs)
  File "/Users/jeff.doyle/work/activitysim/activitysim/defaults/models/mode.py", line 195, in tour_mode_choice_simulate
    cache_skim_key_values=cache_skim_key_values)
  File "/Users/jeff.doyle/work/activitysim/activitysim/defaults/models/mode.py", line 130, in _mode_choice_simulate
    locals_d=locals_d)
  File "/Users/jeff.doyle/work/activitysim/activitysim/activitysim.py", line 216, in simple_simulate
    choices = make_choices(probs)
  File "/Users/jeff.doyle/work/activitysim/activitysim/mnl.py", line 67, in make_choices
    return pd.Series(choices, index=probs.index)
  File "/Users/jeff.doyle/anaconda/envs/asim/lib/python2.7/site-packages/pandas/core/series.py", line 228, in __init__
    data = SingleBlockManager(data, index, fastpath=True)
  File "/Users/jeff.doyle/anaconda/envs/asim/lib/python2.7/site-packages/pandas/core/internals.py", line 3752, in __init__
    fastpath=True)
  File "/Users/jeff.doyle/anaconda/envs/asim/lib/python2.7/site-packages/pandas/core/internals.py", line 2461, in make_block
    return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)
  File "/Users/jeff.doyle/anaconda/envs/asim/lib/python2.7/site-packages/pandas/core/internals.py", line 84, in __init__
    len(self.mgr_locs)))
ValueError: Wrong number of items passed 1277, placement implies 1278

This si caused because one tour ((19970) has zero probabilities for all modes because all NaN utilities
probs.loc[19970,].sum(axis=0) = 1.8e-299

Alternative
BIKE              1.000000e-300
DRIVEALONEFREE    1.000000e-300
DRIVEALONEPAY     1.000000e-300
DRIVE_COM         1.000000e-300
DRIVE_EXP         1.000000e-300
DRIVE_HVY         1.000000e-300
DRIVE_LOC         1.000000e-300
DRIVE_LRF         1.000000e-300
SHARED2FREE       1.000000e-300
SHARED2PAY        1.000000e-300
SHARED3FREE       1.000000e-300
SHARED3PAY        1.000000e-300
WALK              1.000000e-300
WALK_COM          1.000000e-300
WALK_EXP          1.000000e-300
WALK_HVY          1.000000e-300
WALK_LOC          1.000000e-300
WALK_LRF          1.000000e-300
Name: 19970, dtype: float64

utilities.loc[19970,]

Alternative
BIKE              NaN
DRIVEALONEFREE    NaN
DRIVEALONEPAY     NaN
DRIVE_COM         NaN
DRIVE_EXP         NaN
DRIVE_HVY         NaN
DRIVE_LOC         NaN
DRIVE_LRF         NaN
SHARED2FREE       NaN
SHARED2PAY        NaN
SHARED3FREE       NaN
SHARED3PAY        NaN
WALK              NaN
WALK_COM          NaN
WALK_EXP          NaN
WALK_HVY          NaN
WALK_LOC          NaN
WALK_LRF          NaN
Name: 19970, dtype: object

choosers.loc[19970,]

destination                       1439
destination_in_cbd                 NaN
person_id                       820171
tour_departure_and_duration         36
tour_num                             1
tour_type                       school
start                                6
end                                 23
duration                            17

make_choices in mnl.py probably should check for probs not summing to 1 or being NaNs?
def make_choices(probs):
    """
    Make choices for each chooser from among a set of alternatives.

    Parameters
    ----------
    probs : pandas.DataFrame
        Rows for choosers and columns for the alternatives from which they
        are choosing. Values are expected to be valid probabilities across
        each row, e.g. they should sum to 1.

    """

Tour 19970 was assigned destination TAZ 1439 which has a dest_density_index Nan because land_use.household_density and land_use.employment_density in that TAZ are both 0 so density_index is undefined.
@orca.column("land_use")
def density_index(land_use):
    return (land_use.household_density * land_use.employment_density) / \
        (land_use.household_density + land_use.employment_density)

Thus the 10 rows in tour_mode_choice.csv that use dest_density_index eval to NaN for this tour
Thus the Utilities computed by the cross product with spec are likewise Nan when
utilities = model_design.dot(spec)
utilities.loc[19970,]

Alternative
BIKE              NaN
DRIVEALONEFREE    NaN
DRIVEALONEPAY     NaN
DRIVE_COM         NaN
DRIVE_EXP         NaN
DRIVE_HVY         NaN
DRIVE_LOC         NaN
DRIVE_LRF         NaN
SHARED2FREE       NaN
SHARED2PAY        NaN
SHARED3FREE       NaN
SHARED3PAY        NaN
WALK              NaN
WALK_COM          NaN
WALK_EXP          NaN
WALK_HVY          NaN
WALK_LOC          NaN
WALK_LRF          NaN
Name: 19970, dtype: object

Which makes the probabilities for tour 19970 all NaN
Which causes the groupby at line 66 of make_choices in mnl.py to come up one row short:
choices = (s.iat[0] for _, s in pd.Series(cols).groupby(rows))
And so the choices array has one less row than probs.index so the attempr to create a series fails:
return pd.Series(choices, index=probs.index)
ValueError: Wrong number of items passed 1277, placement implies 1278

Tour model shouldn't choose destination TAZ where employment_density is 0?
But this is a school tour, so maybe it is a legitimate destination?
Here is the dest TAZ 1439 land use:
DISTRICT                  34
SD                        34
COUNTY                     9
TOTHH                      0
HHPOP                      0
TOTPOP                  4855
EMPRES                     0
SFDU                       0
MFDU                       0
HHINCQ1                    0
HHINCQ2                    0
HHINCQ3                    0
HHINCQ4                    0
TOTACRE                   26
RESACRE                    0
CIACRE                    13
SHPOP62P                0.05
TOTEMP                     0
AGE0004                    5
AGE0519                   98
AGE2044                 3598
AGE4564                 1050
AGE65P                   104
RETEMPN                    0
FPSEMPN                    0
HEREMPN                    0
OTHEMPN                    0
AGREMPN                    0
MWTEMPN                    0
PRKCST                     0
OPRKCST                    0
area_type                  4
HSENROLL                   0
COLLFTE                    0
COLLPTE                    0
TOPOLOGY                   3
TERMINAL              7.3102
ZERO                       0
hhlds                      0
sftaz                   1439
gqpop                   4855
employment_density         0
total_acres               26
total_households           0
density_index            NaN
county_name            Marin
household_density          0
county_id                  9
total_employment           0
Name: 1439, dtype: object

0 households, population 4855, high school and college population is 0, but group quarters population GQPOP is 4855 - I'm guessing this is either a barracks, a monastery, or a prison?
So probably shouldn't be the destination of a school tour?

San Quentin State Prison?

This is a very good catch.  There are a few issues/observations here:

This is a destination with no school enrollment, why it was picked as a school tour destination?  Probably due to the shadow pricing mechanism, used to enforce consistency between school location choices and enrollment inputs, is not in place in the school location choice model yet?

Employment density (population and dwelling unit densities too in the SANDAG case) is used as a proxy to represent urban forms which affect the likelihood of choosing non-motorized modes (bike and walk).  Employment density not only affects tour/trip mode choice for work purpose, but also affects trip/tour mode choices for other purposes, including school trips.

More importantly, the all NaN probability situation could also happen in other places, such as stop location choices per the SANDAG experience.  We put temporary fixes to default choices to an alternative we think is relatively reasonable to get by.  However, it would be great to have a more sounding solution to the no alternative is available cases like this.

@toliwaga suggestion of checking the sum of probability equal to 1 is a good way to detect (not solve) problems like this.

Crash fixed in #82 but leaving open to remind us to add check for probs not summing to 1 or being NaNs

props currently end up not summing to 1 rather frequently. Specifically, when all the utilities are large negative numbers.
this happens for instance in non_mandatory_scheduling call to vectorize_tour_scheduling when all utilities are between -990 and -999
When this happens, all the exp(utils) are zero and so the conversion to probabilities zero-divides when it does this:
    prob_min = 1e-300
    prob_max = np.inf

    utils_arr = np.exp(utils.as_matrix().astype('float'))
    arr_sum = utils_arr.sum(axis=1)

    if np.isinf(arr_sum).any():
        raise RuntimeError('utilities have infinite values')

    np.clip(utils_arr, prob_min, prob_max, out=utils_arr)
    np.divide(
        utils_arr, arr_sum.reshape(len(utils_arr), 1),
        out=utils_arr)
    utils_arr[np.isnan(utils_arr)] = prob_min

    np.clip(utils_arr, prob_min, prob_max, out=utils_arr)

Moving the computation of arr_sum to after the clip makes mathematical sense and appears to solve the problem.
Anyone see any obvious reason not to make this change?
Note: this wasn't getting caught because numpy.divide() wasn't throwing a zero-divide error. This is apparently a pandas feature.
Running this:
import numpy as np
print "np.geterr()", np.geterr()

import pandas as pd
print "np.geterr()", np.geterr()

Yields this:
np.geterr() without pandas: {'over': 'warn', 'divide': 'warn', 'invalid': 'warn', 'under': 'ignore'}
np.geterr() with pandas: {'over': 'ignore', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'}

Why probabilities are not summing up to 1 frequently?  If this happens only when the util has large negative values like -999, it is kind of understandable.  If this also happens to util with values not 'too' negative (such as -200), then it is a significant issue, because it is likely the -200 is a legit choice util, but then the math engine can't get the probabilities computed right.  Is this a precision issue in Python when it come to mathematic computation?

For the clipping, what if we have a case like this:
Two choices available.  Choice 2 is meant to be blocked, so the utility is assigned a value -999.
before clipping: util1=-100, util2=-999
after clipping: util1=-100, util2=-300 (if I understand the clipping correctly)
Now, both choices will have a valid probability computed for them.  But then the Monte Carlo could well pick choice 2, which was really meant to be a non-valid choice.

The existing CT-RAMP has the same issue of probabilities not summing up to 1.  One odd case we had was something like this:
Choice 1 is assigned with a probability 0.95, choice 2, 3, and 4 are blocked with -999 utils.  In Monte Carlo, if the generated random number <0.95, then it is ok and alternative 1 is chosen.  But, there are times the random number is >0.95 and no alternative can be chosen and caused model bombing.

The large negative values are like -998, -997 not -200

In general, every choice should have at least one alternative available.  If not, then the problem is usually due to a data issue (often in the skims) in combination with an error in expression definitions.  Ultimately we need to log when a choice cannot be made and exit the simulation.
Also, it looks like we need to review/audit the expressions to ensure we don't get any -998, -997s since these are probably the result of errors in the expression definitions.  In CT-RAMP, once a -999 was encountered, the rest of the expressions in the UEC were skipped and the alternative was turned off.  We need to think some more about how best to incorporate the treatment of NAs in the system.

It seems to me there are two different issues:

probabilities not summing up to 1
all choice alternatives are blocked with larger negative utils

For issue 2, as @bstabler mentioned, it is caused by data, we need to log the error with informative info so the users know where to check data.
For issue 1, a lot of times it happens when one or more of the alternative is blocked with large negative utils. We might be able remove the blocked alternatives from the alternative set, only leave valid alternatives in the set, and it might help to resolve sum not 1 issues.

Regarding mode choice nesting coefficients, we also have to be careful about how they are applied, alogit style:
Unest = ln(sum(exp(U))
or utility style (McFadden)
Unest = 1/lambda * sum (exp(lambda * U))
and whether there are any "dummy" nodes to correct for the problems of alogit style

as @wusun2 points out, the probabilities are not summing to 1 since the alternatives with -999 are being included in the calculation of probability.  What we need to do is specify a global NA value (such as -999) and then if an expression returns that value for an alternative, then we remove that alternative before calculating probabilities.   So we have two issues to fix and I'm going to create new issues for them so we can have a more specific discussion about just those issues.

I'm going to close this issue now since we have a complete run. I'm sure we'll run into additional issues, but we'll make those individual issues instead.


RETRY_AFTER_MS_HEADER not getting properly "honored"

Considering following code,

In the context of this code, if the RETRY_AFTER_MS_HEADER is found in the response headers, the function will sleep for that much time but then the function will resume from the point where asyncio.sleep was called, not from the start of the function or the start of the while loop.
Thus, in cases when the RETRY_AFTER_MS_HEADER  duration would be greater than MAX_RETRY_SECONDS, which is very much possible, considering how much less the value of  MAX_RETRY_SECONDS (5s) currently is, the condition of the while loop will be false after waking from the thread sleep as the while loop is considering the stale time value capture from the 1st incident of throttling, hence the while loop will not continue, effectively meaning the request will not be retried.
Suggestions:

The time check logic in while loop should not be based on the stale value and should update considering the time difference of thread sleep from RETRY_AFTER_MS_HEADER  sleep condition.
Value for MAX_RETRY_SECONDS should not be hardcoded and be made configurable and decided by user. See #36
Documenting about MAX_RETRY_SECONDS and letting the users know how it works. See #37

Hey @sushaanttb,
Have a look at this PR and the discussion within. You will see some more info there about what is going on with different retry settings, and how MAX_RETRY_SECONDS effects the outputs. That PR has not been merged and it doesn't look like it will, so if you want the behaviour you suggest, I would suggest you use my fork which fixes this issue and uses a more suitable value of 60 seconds for MAX_RETRY_SECONDS.


chore: 🤖 convert DS build to tsup, add explicit vanilla export

Swapped Rollup out for TSup
Notes:

I wasn't able to get the nested structure to work, so right now it's just bundling
Due to renaming limitations in ESBuild, styles.css is now index.css, but the import in the app should be the same(?) since I updated the exports on the package.json
Due to renaming and configuration limitations (and not wanting to write another custom plugin), the vanilla export is created from a new vanilla entry point. This allows for complete control over what is exported, but is a more manual process. Maybe there's a future way for this file to be generated based on a glob
You'll see chunks in the dist. But they appear to be valid sharing of code / types between the exports without mixing anything into vanilla that we don't want

I don't know the ramifications of this, but currently the dist/index.js file and dist/vanilla.js file are exporting the same things (with dist/index.js additional ones from same chunk file):

dist/vanilla.js

dist/index.js

I tried doing an experiment with running three separate tsup configs one for each of our intended outputs, but was unable to get it working (or more accurately, matching the current output of the dist to a degree). If we wanted to hyper optimize the bundle output we may need to slightly restructure the package structure a smidge.


Fix Archlinux Install Command

Description
This is a fix for the wrong install command on archlinux
By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

I can't accept the PR without DCO. Do you mind amending your commit with DCO as described here? Or, if you don't mind I can submit it for you.

Beat you to it. Thank you for pointing this out and submitting a fix so quick.


Database Migration

I've been trying to migrate the tables into the database (PostgreSQL) right after cloning and yarn install using both typeorm and sequelize however when migrating with typeorm i get this
error Command "typeorm" not found.
and with sequelize using this command
npx sequelize db:migrate
i get this
Sequelize CLI [Node: 14.15.0, CLI: 5.5.1, ORM: 5.21.2]

Loaded configuration file "src\config\database.js".

After i installed typeorm and created the ormconfig.json as follows
{
    "type": "postgres",
    "host": "localhost",
    "port": 5432,
    "username": "postgres",
    "password": "",
    "database": "goeat"
 }

and rerun yarn typeorm migration:run with this response
yarn run v1.22.10
$ D:\Oness\code\PersonalProjects\goeat-api\node_modules\.bin\typeorm migration:run
Done in 3.44s.

but when i check my db, no tables were created
EDIT:
package version = 1.0.0
nodejs version = v14.15.0

Hello @on3ss
Thanks for creating this issue!
My apologies, the reason for the error error Command "typeorm" not found. is that typeorm is not being used in this application... I've used it often in other projects and accidentally paste it here while I was updating the docs.
The migrations should be run with sequelize-cli
I've fixed the migrations intruction at b2e7dc1625966e8fbb382352c7c794e3f96c472e

I'll close it since the issue was with the docs and I've already fixed at b2e7dc1 but let me know if you have any other problem to run the migrations


ros_industrial_cmake_boilerplate: 0.2.10-1 in 'noetic/distribution.yaml' [bloom]

Increasing version of package(s) in repository ros_industrial_cmake_boilerplate to 0.2.10-1:

upstream repository: https://github.com/ros-industrial/ros_industrial_cmake_boilerplate.git
release repository: https://github.com/ros-industrial-release/ros_industrial_cmake_boilerplate-release.git
distro file: noetic/distribution.yaml
bloom version: 0.10.7
previous version for package: 0.2.9-2

ros_industrial_cmake_boilerplate
* Improve target_clang_tidy to support individual options over single argument list
* Break out individual function from configure_package
* Contributors: G.A. vd. Hoorn, Levi Armstrong

@Levi-Armstrong I may have rushed the order. Can I assume #30103 supersedes this PR?

That is correct.


Turn off WPS warnings that require code refactoring

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3350
Originally created by @aaronsteers on 2022-03-25 18:41:08

From https://wemake-python-stylegui.de/en/latest/

Welcome to the strictest and most opinionated Python linter ever.

As came up today in !2581, the aggressiveness of the WPS errors are costing us time and discouraging community contributions. I suggest we disable WPS or at least a large number of the WPS rules, at least until we've caught up on the other more traditional lint issues which are also in the codebase.
This class of lint failures might be acceptable if tested just against new contributions, but since every file touched is getting retested, the situation is not really tenable for new community contributions as is.
Also important in consideration is that WPS-related lint errors like code complexity and number of lines allowed in a "try" block will require refactoring and that refactoring requires robust testing. It is safer and more stable to leave these lines of code alone than to refactor them and take the unnecessary risk of breaking existing aspects of the existing codebase.

View 4 previous comments from the original issue on GitLab


[WIP] Upgrade spark integration version to 2.3.2

Upgrade spark integration version to 2.3.2
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@jackylk @chenliang613 @sujith71955 please review.

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/615/

Thanks for raising the PR, It will better if you can add the description about the changes in this PR.

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8876/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/807/

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/619/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8880/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/811/

@zzcclp Please check and fix the tests

@ravipesala can you help me to check why these three test cases fail? It's about the decimal precision.

@ravipesala I know how to fix and will fix the tests ASAP.

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/837/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/643/

@sujith71955 @chenliang613 @ravipesala @jackylk this pr is ready, please review, thanks.

@chenliang613 I think it is not possible to add lazy variable without breaking spark 2.2 integration. Since this class is small, I think it is ok to have a separate one for spark 2.3 integration

I think it is not possible to add lazy variable without breaking spark 2.2 integration. Since this class is small, I think it is ok to have a separate one for spark 2.3 integration

@sujith71955 any suggestion for this?

LGTM

LGTM

LGTM  Even though SPARK-PR#21815 changes is a workaround solution and in future  its subjected to change. Need a caution here. Can you please add this PR reference in the modified code of the carbon  for future reference.
Thanks for your effort.

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/656/

LGTM

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8920/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/852/

LGTM

the failed test case is not related to this pr, right?

I think better to re-trigger once

retest this please

Seems to be impact of this PR , please recheck once

ok, but it doesn't add lazy for spak 2.2.

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/855/


[SR SQL Planner Summer Camp] Support RoutineLoadResume Statement in new Parser and Analyzer

What type of PR is this：

[ ] feature

Which issues of this PR fixes ：

Fixes https://github.com/StarRocks/starrocks/issues/7949
Problem Summary(Required) ：

[FE PR Coverage Check]
:disappointed: fail : 51 / 122 (41.80%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/parser/AstBuilder.java
0
32
00.00%
[1156, 1157, 1158, 1160, 1161, 1162, 1164, 1169, 1170, 1171, 1173, 1174, 1175, 1177, 1182, 1183, 1184, 1185, 1187, 1188, 1189, 1191, 1192, 1193, 1195, 1196, 1197, 1198, 1200, 1201, 1202, 1204]

:large_blue_circle:
com/starrocks/analysis/FunctionalExprProvider.java
0
7
00.00%
[145, 147, 148, 149, 151, 153, 154]

:large_blue_circle:
com/starrocks/sql/analyzer/PrivilegeChecker.java
0
4
00.00%
[834, 835, 836, 838]

:large_blue_circle:
com/starrocks/qe/ShowExecutor.java
0
1
00.00%
[936]

:large_blue_circle:
com/starrocks/sql/ast/AstVisitor.java
1
3
33.33%
[315, 319]

:large_blue_circle:
com/starrocks/sql/analyzer/ResumeRoutineLoadAnalyzer.java
7
13
53.85%
[13, 18, 19, 20, 22, 23]

:large_blue_circle:
com/starrocks/sql/analyzer/PauseRoutineLoadAnalyzer.java
7
13
53.85%
[13, 18, 19, 20, 22, 23]

:large_blue_circle:
com/starrocks/analysis/ShowRoutineLoadStmt.java
14
25
56.00%
[133, 139, 140, 142, 177, 184, 187, 188, 189, 192, 228]

:large_blue_circle:
com/starrocks/analysis/ResumeRoutineLoadStmt.java
7
8
87.50%
[74]

:large_blue_circle:
com/starrocks/analysis/PauseRoutineLoadStmt.java
7
8
87.50%
[58]

:large_blue_circle:
com/starrocks/sql/analyzer/Analyzer.java
4
4
100.00%
[]

:large_blue_circle:
com/starrocks/sql/analyzer/ShowStmtAnalyzer.java
4
4
100.00%
[]

run starrocks_admit_test

seems some repetitive changes with https://github.com/StarRocks/starrocks/pull/8985.

https://github.com/StarRocks/starrocks/pull/8985
contains  RoutineLoadResume  , so this pr can close
thanks


feat(model): add gradient checkpointing to GPTJ and Llama

Why this PR
Implement Gradient Checkpointing for Lightning Fabric training pipeline
Changes

add Gradient Checkpointing option for GPTJ and LLama

To test gradient checkpointing
!torchrun --standalone --nproc_per_node=1 src/model/scripts/lighting_training/train.py --model_name llama --vocab_size 32000 --checkpoint 1

1 is for model
2 is for self attentions only

Related Issues
Close #116
Checklist

[x] PR should be in the Naming convention
[x] Assign yourself in to Assigneees
[x] Tag related issues
[x] Constants name should be ALL_CAPITAL, function name should be snake_case, and class name should be CamelCase
[x] complex function/algorithm should have Docstring
[x] 1 PR should not have more than 200 lines changes (Exception for test files). If more than that please open multiple PRs
[x] At least PR reviewer must come from the task's team (model, eval, data)

Member

This PR is too long. You should separate in to 2 PR. 1 PR is for GPTJ and 1 PR is for Llama.
can I highlight line that I edit only because It model from huggingfacec?

This PR is too long. You should separate in to 2 PR. 1 PR is for GPTJ and 1 PR is for Llama.

Can I highlight line that I edit/extent because base is from huggingface.


httpfast support

Is your feature request related to a problem? Please describe.
Some web frameworks use https://github.dev/valyala/fasthttp rather than net/http like fiber. The probe can not those built with fasthttp now. Do you have a plan to support it?
Additional context
Here is the error when trying to instrument fasthttp:
{"time":"2024-12-03T12:55:42.272278819Z","level":"ERROR","source":{"function":"main.main","file":"/home/vagrant/opentelemetry-go-instrumentation-oss/cli/main.go","line":119},"msg":"failed to create instrumentation","error":"could not find function offsets for instrumenter"}

We are currently working on stabilizing our framework to make it easier to add support for any new library (https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1105). So, while we aren't planning to add this library right now, we can in the future once that stabilization work is done.


add: prepare sends

Expose the unsigned transaction, for allowing bridging using fireblocks wallet
Motivation
Enable partners to bridge using fireblocks
Solution
Add prepareSend and prepareSendNative that expose the unsigned tx
PR Checklist

[x] Added Tests (no additional code, just splits up existing functions)
[x] Updated Documentation
[x] Updated CHANGELOG.md for the appropriate package

is there any fireblocks ethers provider? 🤔 It seems like these should be
solved via a lower-level integration, rather than at the sdk
On Thu, May 12, 2022, 19:32 Erin Hales @.***> wrote:

@ErinHales https://github.com/ErinHales requested review from
@nomad-xyz/protocol on: #280
https://github.com/nomad-xyz/monorepo/pull/280 add: prepare sends as a
code owner.
—
Reply to this email directly, view it on GitHub
https://github.com/nomad-xyz/monorepo/pull/280#event-6601943741, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACNN4MO6OPEN47CDZIGXEBLVJWIKFANCNFSM5VZYH7QA
.
You are receiving this because your review was requested.Message ID:
@.***>

is there any fireblocks ethers provider? 🤔 It seems like these should be solved via a lower-level integration, rather than at the sdk

Short answer, no, long answer, it's complicated.


not getting all article links after refreshing again the code?

hello everyone
I am scrapping a website to get the latest tab news. I followed the code and at first it worked perfectly but after code cleaning and running again it returned nothing at all?
Here is the code:
Initial Tab Link Scraping and then pass the link to build newspaper object for getting the articles links:
# function to extract HTML document from given URL
def getHTMLdocument(url):
    # request for HTML document of given url
    response = requests.get(url)
    response = response.text
    # response will be provided in JSON format
    return response

# soup
# create soap object
def b_soup(html_document):
    soup = BeautifulSoup(html_document, 'html.parser')
    # soup.prettify()
    return soup

def scrap_links(link):
    urls = []
    # extract all links from the url
    news_paper = newspaper.build(link)
    for article in news_paper.articles:
        urls.append(article.url)
    return URLs

url_to_scrape = 'https://www.thenews.com.pk/'
# create document
html_document = getHTMLdocument(url_to_scrape)
soup = b_soup(html_document)
# find all the required tags links with "href"
# attribute starting with "https://"
links = []
latest = soup.find_all('a',attrs={'href': re.compile("latest-stories")})[0]
links.append(world.get('href'))
latest_news = links[0]
# Scrap the articles links using newsarticles
latest_urls = scrap_links(latest_news)

It returns me:
https://www.thenews.com.pk/latest-stories
[]

Is there any limit that you can't refresh the links for scraping or something else wrong?
Thank you

The function scrap_links() is incorrect.  Build isn't used that way.  Reference my overview document on how to use build.

Thank you @johnbumgarner for your reply solved as stated:
I Used below chunk of code and it return the links of all articles.
def scrap_links(link):
    USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
    config = Config()
    config.browser_user_agent = USER_AGENT
    config.request_timeout = 10
    article_urls = set()
    marketwatch = newspaper.build(
        link, config=config, memoize_articles=False, language='en')
    for sub_article in marketwatch.articles:
        article = Article(sub_article.url, config=config,
                          memoize_articles=False, language='en')
        article.download()
        article.parse()
        if article.url not in article_urls:
            article_urls.add(article.url)
    return article_urls

You're welcome @Aliktk.  Happy coding.


App Download

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):
Fix #40
Test URLs:

Before: https://main--icicidirect--aemsites.hlx.live/sidekick/blocks/app-download

After: https://app-download--icicidirect--aemsites.hlx.live/sidekick/blocks/app-download

[ ] New Blocks introduced in this PR
If yes, please provide details below

Block Name
Documentation

app-download
doc-link

I have changed mob view from this

to this on purpose as original one look distored to me because of diff height img side by side

@ravverma psi check ?

@ravverma psi check ?

not sure why it is failing not getting any thing related to block in report.

See what I found on the original site ^ ^

See what I found on the original site ^ ^

still viewing same

may be geo location updating time ?

@ravverma I can see page itself passed the PSI, but library failed.

yes all static content. Structure from the block gives straight way to created desired experience, let me try to pull that from section itself.

Ok yes. Maybe check if plain columns block can help here. If not, we can have a new block.

yes all static content. Structure from the block gives straight way to created desired experience, let me try to pull that from section itself.

Ok yes. Maybe check if plain columns block can help here. If not, we can have a new block.

Thought of that but it will be a still a block. able to do by section only

order will not matter here.

Thought of that but it will be a still a block. able to do by section only

If you are anyway ending up with section level app-download style , then lets do it as a block only.

Closing this


Allowing matching a filterchain when applying EnvoyPatchPolicy

EnvoyPatchPolicy utilities JSON Patch for applying patches to XDS resources, but JSON Patch lacks the ability to select a specific member in an array based on conditions other than its index.
For example: assuming there're two filter chains in a listener, there's no way that we could target the second filter chain based on its serverName www.bar.com when applying a patch.
We won't be able to know the index of the targeted filter chain without inspecting the generated Envoy configuration, which is impossible when we want to use a controller to generate the patch. We could possibly create that patch manually, but it's cumbersome and unreliable, and the index might change with updates to the EG releases.
      - address:
          socketAddress:
            address: <IP_ADDRESS>
            portValue: 10443
        filterChains:
        - filterChainMatch:
            serverNames:
            - www.foo.com
          filters:
          - name: envoy.filters.network.http_connection_manager
            typedConfig:
              ......
        - filterChainMatch:
            serverNames:
            - www.bar.com
          filters:
          - name: envoy.filters.network.http_connection_manager
            typedConfig:
              ......

thanks for surfacing this limitation where today with the JSONPatch approach, the patch producer cannot always proactively create a patch and be sure it will eventually patch, instead must reactively patch to be sure of a successful outcome.
there are 3 ways to tackle this

dont do anything, and document this limitation
if JSONPath gets standardized and is accepted in JSONPatch as well as implemented in libraries such as https://github.com/evanphx/json-patch, https://datatracker.ietf.org/wg/jsonpath/about/ , we can achieve this
we can define our own match criteria, but requests for extending these matches is hard to maintain and support

Close this as we don't need it anymore. We can reopen if anyone else asks for this feature.

Hi @zhaohuabing,
I would like to restart the discussion on that topic.
I stumbled upon several limitations, which are not yet addressed or merged etc.
I think EnvoyPatchPolicy is a great feature to interfere directly with the envoy config.
Theoretically I would have been able to get around the limitations I stumbled upon. But in reality the EnvoyPatchPolicy is not able to select a specific member in an array based on conditions other than its index.
As I said its a great feature, but array element selection is essential.
Can we move away from JSON Patch or provide an additional option?
And maybe use something like that:
https://github.com/itchyny/gojq?tab=readme-ov-file#usage-as-a-library
This would allow to manipulate the json doc like this:
json:
{
    "a": "123",
    "b": "456",
    "c": [
        {
            "id": "50",
            "value": "test"
        },
        {
            "id": "31",
            "value": "xyz"
        }
    ]
}

jq expression:
(.c[] | select(.id == "31") | .value) = {x: 1, y:2}

result:
{
    "a": "123",
    "b": "456",
    "c": [
        {
            "id": "50",
            "value": "test"
        },
        {
            "id": "31",
            "value": {
                "x": 1,
                "y": 2
            }
        }
    ]
}

full command to reproduce:
echo '{"a":"123","b":"456", "c": [{"id":"50", "value":"test"}, {"id":"31", "value":"xyz"}]}' | gojq -r '(.c[] | select(.id == "31") | .value) = {x: 1, y:2}'

@denniskniep Thanks for proposing this solution. It looks promising.
@arkodg How do you think about supporting jq style within the EnvoyPatchPolicy?

Essentially what we discussed on eg community meeting:

There are objections because of potential missing backwards combability of jq expressions (cause its not coupled with a standard)

To bring up an additionally point: that lib is released under MIT License, is that fine for eg project?
cc: @zhaohuabing, @arkodg, @guydc

From my point of view it would be simply another type: jq how you can execute an EnvoyPatch in addition to type: JsonPatch.
We can document, that jq is following no standard and that it could break something with a new release, if the syntax change.

thanks for bringing it up in the community meeting today @denniskniep
the core problem this GH issue is trying to address -  is being able to search/find the right object within a field that is a list and modify it
would ideally prefer if we used a JSON/YAML standard here

Hi,
lets discuss a different approach utilizing JsonPath:

Add an additional property jsonPath to EnvoyPatchPolicy.spec.jsonPatches.operation. (Either path or jsonPath must be set)
Use an existing JsonPath lib and resolve JsonPath Expression to exact locations in json document
Convert the locations to JsonPointer syntax
Execute for each JsonPointer the normal JsonPatch operation

I did a small PoC here: https://github.com/denniskniep/json-patch-by-path-demo
Disclaimer: JsonPathLocation to JsonPointer conversion is not really seriously implemented in that PoC...
The core logic is:
func Patch(req PatchRequest) (string, error) {
	jObj, err := oj.ParseString(req.JsonDoc)
	if err != nil {
		return "", errors.Wrap(err, "Error during parsing json")
	}

	jPath, err := jp.ParseString(req.JsonPath)
	if err != nil {
		return "", errors.Wrap(err, "Error during parsing jpath")
	}

	locations := jPath.Locate(jObj, 0)

	jsonDoc := req.JsonDoc
	for _, l := range locations {

		resolvedJsonPath := l.String()
		jsonPointer := convertToJsonPointer(resolvedJsonPath)

		patch, err := createJsonPatch(req.Operation, jsonPointer, req.Value)
		if err != nil {
			return "", err
		}

		modified, err := patch.Apply([]byte(jsonDoc))
		if err != nil {
			return "", errors.Wrap(err, "Error during patch apply")
		}

		jsonDoc = string(modified)
	}

	return jsonDoc, nil
}

In this PoC I used the following JsonPath lib: https://github.com/ohler55/ojg
What do you think @zhaohuabing @arkodg ? Is that conceptionally something we can aim for?

hey @denniskniep thanks for digging deeper, I'd be a +1 for this approach wdyt @envoyproxy/gateway-maintainers

@denniskniep brilliant. +1

thanks for bringing this up in the community meeting today @denniskniep, there were no objections for this approach yet, please feel free to continue with this.
A good suggestion made by @guydc was to look into which library kuberenetes was using for JsonPath, and comparing that with the one suggested above

Seems to be that kubectl is using its own implementation from k8s client-go lib
Client-Go Lib parser is called from the get command  here
I dont think that it makes sense to include the entire client-go lib only for that json path. Furthermore it has no implementation to resolve the path to location strings. That would make the jsonPath to Pointer conversion even harder.
Therefore my vote for https://github.com/ohler55/ojg
(BTW: This lib has a pretty good coverage of the rfc9535 see here)
Once we agreed on the lib, I can start the implementation

https://cburgmer.github.io/json-path-comparison/ is great view, please go ahead with https://github.com/ohler55/ojg

@arkodg can we add this again to milestone v1.1.0-rc1 ?

@zhaohuabing  is it possible to share the EnvoyPatchPolicy CR file which you are using to allow your domain. I am stuck up with an issue on envoy-gateway i hope it might help me

@meektechie Not sure that I got what you meant :-) What EnvoyPatchPolicy are you looking for?

Thanks for the response. I am trying to configure SNI for listener and found this can be achieved through EnvoyPatchPolicy. But unable to get the sample for it. With that approach i hope we might setup ssl certificate for the explicit subdomains. So requesting to share that.
Also, is it possible to refer certificate from the secret instead of filename/path
  common_tls_context:
    tls_certificates:
    - certificate_chain: { filename: "example_com_cert.pem" }
      private_key: { filename: "example_com_key.pem" }


Reaching out...

Not sure if there is a better place to put this message — I work on DuckDuckGo and would love to chat about the integration with the maintainer (@PalmerAL). I like what you're doing and would love to see how we can further help you out. Drop me a line at jordan (at) duckduckgo .com

Thanks! I've sent an email.


giving error after running main.py

C:\Users\kapse\Downloads\chess\chess.com-bot-main\src>python main.py
Traceback (most recent call last):
File "C:\Users\kapse\Downloads\chess\chess.com-bot-main\src\main.py", line 5, in
from pywinauto import application
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto_init_.py", line 89, in
from . import findwindows
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto\findwindows.py", line 42, in
from . import controls
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto\controls_init_.py", line 36, in
from . import uiawrapper # register "uia" back-end (at the end of uiawrapper module)
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto\controls\uiawrapper.py", line 47, in
from ..uia_defines import IUIA
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto\uia_defines.py", line 181, in
pattern_ids = _build_pattern_ids_dic()
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto\uia_defines.py", line 169, in _build_pattern_ids_dic
if hasattr(IUIA().ui_automation_client, cls_name):
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto\uia_defines.py", line 50, in call
cls._instances[cls] = super(_Singleton, cls).call(*args, **kwargs)
File "C:\Users\kapse\AppData\Roaming\Python\Python310\site-packages\pywinauto\uia_defines.py", line 63, in init
self.ui_automation_client.CUIAutomation().IPersist_GetClassID(),
AttributeError: module 'comtypes.gen.UIAutomationClient' has no attribute 'CUIAutomation'

cmd logs

Hi, thank you for reporting this issue. I will review and get back to you.👍

Hi, I was able to run this locally on my machine fine with Python 3.10. Googling this error, it looks to be related to the comtypes sub module. Could you please install comtypes again via pip3 install comtypes or pip install comtypes.
Thank you!


Not using CI correctly

CI is not a vps.  You should stop right now


[WASM, Windows] Mouse position incorrect with screen DPI 150% (UI interaction broken)

Set monitor to 150% DPI in Windows 10
Open https://comfyengine.org/examples/
Click blood_canvas example
Click letter "f"

Expected: blood under "f"
Actual: blood offset lower right

Note: this breaks UI interaction like button clicks as well due to mouse pos issues.

Does this only happen for you on the web, or also on desktop?

Web only

I can see a high DPI issue on Windows now.
150% DPI offsets the egui window incorrectly. It's supposed to be centered below:

Windows bug should be fixed now, WASM isn't yet.

Seems this issue is fixed in WASM too but not on the website. I wonder if there's something wrong with the website that further stretches the canvas or something that causes this, but it doesn't seem to happen when running https://github.com/darthdeus/comfy/tree/master/demos/egui-scaling through wasm-server-runner


Release @google-cloud/pubsub v0.26.1

This pull request was generated using releasetool.

@JustinBeckwith how do version TypeScript changes? Part of my PR removed a a duplicate interface with a different name. I know a number of the definitions were incorrect, but do we treat something like that as a patch or breaking change?

You know... I don't know what the right answer is here.  @ofrobots and the @googleapis/node-team - any guidance on how to treat breaking TypeScript changes with regards to semver?

@callmehiphop yeah - you're right.  This should be a breaking change, and a minor version bump.  We're gettin really good at those with this lib lol. Can you cut a 0.27 release today?


codecov activation

think this will work @jasmainak ?

ok tests are passing.......let's see if this works :-)


Specify exact versions to be installed via npm in dependency list

https://github.com/airbnb/superset/blob/master/superset/assets/package.json#L62
uses a range of semver versions to pick up latest compatible versions, this can potentially silently upgrade dependencies for those compiling from source.
It would be better if we can specify exact versions for these dependencies.

will accept a PR pinning the npm deps

Notice: this issue has been closed because it has been inactive for 375 days. Feel free to comment and request for this issue to be reopened.


Don't hang when batch calls error

A few problems with the batch code error handling can lead to a hang:

[x] Actually return an error instead of nil
[x] Properly manage the waitgroup when a batch call errors
[x] Make sure we display errors instead of just a 0 count
[x] Tests for this scenario

/cc #593

First problem:
return nil, Errorf(nil, "Invalid status for %s %s: %d", req.Method, req.URL, res.StatusCode)

Would actually return a nil error, so the error could never be reported.
Second problem: Even with properly reporting the error, the waitgroup was never decremented, leading to the actual hang.
Also added: error output for fetch, which had none.

 This tweak passes:
diff --git a/lfs/errors.go b/lfs/errors.go
index c66a8c4..e79bbf7 100644
--- a/lfs/errors.go
+++ b/lfs/errors.go
@@ -1,10 +1,13 @@
 package lfs

 import (
+       "errors"
        "fmt"
        "runtime"
 )

+var genericError = errors.New("Git LFS client error")
+
 type WrappedError struct {
        Err     error
        Message string
@@ -19,7 +22,7 @@ func Error(err error) *WrappedError {

 func Errorf(err error, format string, args ...interface{}) *WrappedError {
        if err == nil {
-               return nil
+               err = genericError
        }

        e := &WrappedError{

Now, since this specific case doesn't have an error to wrap, maybe it should just create one?
Error(fmt.Errorf("Invalid status for %s %s: %d", req.Method, req.URL, res.StatusCode))

The WrappedError struct will use that message for both the internal and user-visible error message.
Also, this WrappedError jank is not the best. It may make sense to look into errgo instead.

Also, this WrappedError jank is not the best. It may make sense to look into errgo instead. Definitely not for this PR :)

I feel the same, we could definitely smooth things out a bit.

I added the genericError and used that for now. I've got a couple more steps I can take as part of #595 to go toward cleaning up some error shenanigans.


[rest] GA in docs!

API changes have been detected in azure-cosmos. You can review API changes here

API changes have been detected in azure-storage-file-share. You can review API changes here
API changes
-             permanent_delete: bool = ...,
-             set_immutability_policy: bool = ...,

API changes have been detected in azure-storage-queue. You can review API changes here
API changes
-             permanent_delete: bool = ...,
-             set_immutability_policy: bool = ...,


DOC: the @traceable decorator docs example not working

Issue with current documentation:
The script below, taken from the documentation page does not trace anything to my langsmith project. I have created a new api key but nothing seems to work.

import os
from datetime import datetime
from typing import List, Optional, Tuple

import openai

openai.api_key = os.environ['OPENAI_API_KEY']
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_ENDPOINT"] = "https://api.smith.langchain.com"
os.environ["LANGCHAIN_API_KEY"] = ""
os.environ["LANGCHAIN_PROJECT"] = ""

from langsmith.run_helpers import traceable

# We will label this function as an 'llm' call to better organize it
@traceable(run_type="llm")
def call_openai(
    messages: List[dict], model: str = "gpt-3.5-turbo", temperature: float = 0.0
) -> str:
    return openai.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
    )

# The 'chain' run_type can correspond to any function and is the most common
@traceable(run_type="chain")
def argument_generator(query: str, additional_description: str = "") -> str:
    return (
        call_openai(
            [
                {
                    "role": "system",
                    "content": f"You are a debater making an argument on a topic."
                    f"{additional_description}"
                    f" The current time is {datetime.now()}",
                },
                {"role": "user", "content": f"The discussion topic is {query}"},
            ]
        )
        .choices[0]
        .message.content
    )

@traceable(run_type="chain")
def critic(argument: str) -> str:
    return (
        call_openai(
            [
                {
                    "role": "system",
                    "content": f"You are a critic."
                    "\nWhat unresolved questions or criticism do you have after reading the following argument?"
                    "Provide a concise summary of your feedback.",
                },
                {"role": "system", "content": argument},
            ]
        )
        .choices[0]
        .message.content
    )

@traceable(run_type="chain")
def refiner(
    query: str, additional_description: str, current_arg: str, criticism: str
) -> str:
    return (
        call_openai(
            [
                {
                    "role": "system",
                    "content": f"You are a debater making an argument on a topic."
                    f"{additional_description}"
                    f" The current time is {datetime.now()}",
                },
                {"role": "user", "content": f"The discussion topic is {query}"},
                {"role": "assistant", "content": current_arg},
                {"role": "user", "content": criticism},
                {
                    "role": "system",
                    "content": "Please generate a new argument that incorporates the feedback from the user.",
                },
            ]
        )
        .choices[0]
        .message.content
    )

@traceable(run_type="chain")
def argument_chain(query: str, additional_description: str = "") -> str:
    argument = argument_generator(query, additional_description)
    criticism = critic(argument)
    return refiner(query, additional_description, argument, criticism)

result = argument_chain(
    "Whether sunshine is good for you.",
    additional_description="Provide a concise, few sentence argument on why sunshine is good for you.",
)
print(result)

Idea or request for content:
No response

The platform is being slow right now as we make an infra upgrade. Thanks for your patience. Have  the traces populated yet?

It doesn't seem like langfuse or langsmith are production ready. Do these issues happen often?

LangSmith is still in beta. We do have thousands of prod users at present, but we have to complete these infra upgrades before we can start making the type of SLAs I think you're hinting at (outside of our enterprise offerings).

LangSmith is still in beta. We do have thousands of prod users at present, but we have to complete these infra upgrades before we can start making the type of SLAs I think you're hinting at (outside of our enterprise offerings).
As for LangFuse, I can't comment. They're not affiliated with LangChain but seem to have drawn a lot of inspiration from what we have built

Oh sorry I meant 'langchain' not 'langfuse'. I just corrected the comment.

Hm ok - rumorz looks great btw! congrats - lmk if you have any other pointers or questions. The last ~2 weeks have been the most flakey LangSmith has been since we launched due to a confluence of factors so hopefully won't get this bad again

Hm ok - rumorz looks great btw! congrats - lmk if you have any other pointers or questions. The last ~2 weeks have been the most flakey LangSmith has been since we launched due to a confluence of factors so hopefully won't get this bad again. On the LangChain side we have split out the core library and are further separating out ecosystem packages to make the core API more clear, but if there's anything you'd like us to improve, we're open to feedback

Thank you! I haven't launched rumorz yet but getting ready for launch as we speak. This is mainly for my LLM library https://github.com/zozoheir/tinyllm  - > Which I am using for building Rumorz

Well in any case, I'll do my best to fix issues if they'd help you. In the meanwhile I'm going to close out this ticket since those queueing were resolved


Support for index templates

To solve this problem, we need to take into account the component template from 7.x
https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-component-template.html

no, after 7.8
https://www.elastic.co/guide/en/elasticsearch/reference/7.x/release-notes-7.8.0.html


box2dLight.Light.dispose issue

I have a problem when i want to removeAll from my RayHandler class, its something with the buffer. I dont know if its something with my code, or something with box2dlight class, thats the exception:
java.lang.IllegalArgumentException: buffer not allocated with newUnsafeByteBuffer or already disposed
at com.badlogic.gdx.utils.BufferUtils.disposeUnsafeByteBuffer(BufferUtils.java:507)
at com.badlogic.gdx.graphics.glutils.VertexArray.dispose(VertexArray.java:65)
at com.badlogic.gdx.graphics.Mesh.dispose(Mesh.java:507)
at box2dLight.Light.dispose(Light.java:232)
at box2dLight.RayHandler.removeAll(RayHandler.java:403)
I have that a RayHandler and i only set this things:
RayHandler rayHandler;
//I create and set some properties:
rayHandler = new RayHandler(lightWorld);
rayHandler.setBlur(false);
rayHandler.setAmbientLight(new Color(1f,1f,1f,1f));
RayHandler.setGammaCorrection(true);
RayHandler.useDiffuseLight(true);
// I removeAll in some cases:
rayHandler.removeAll();

Sorry for late reply. This looks like (99%) the double call of dispose method for one light instance in your code.
Also, please post such questions on forum if you are unsure before doing it here - it's a bugtracker and not the help site :) P.S. Issue could be reopened if you are absolutely sure it's the box2dlights issue, not your code.

Hi again. Sorry to dont be sure, but know i am. I see al the code for days to be sure and the only thing i do is that line of rayHandler:
// I removeAll in some cases:
rayHandler.removeAll();
I think that the problem is like you say: "From some light disable, it dispose twice", but i see our code and dont see another dispose, only the removeAll from the rayHandler. So there is a way that the garbage collector dispose some PointLight but not remove from the list of light disable in the RayHandler? or there is a way that some PointLight can be save twice in the list of lights disable or active, or one instance of the same PointLight in both, of RayHandler?
We have a render method that has all the PoingLights in map disable, but the ones that are nearby the LocalPlayer set active, render, and after set disable again, and the other render the same. Is it okey?
I cant eliminate that problem, i read all issues similar but dont answer from my problem

This actually might happen if you are disabling/enabling lights and calling removeAll() in different threads. Is your issue stably reproduces in some scenarios or it occures rarely and randomly?

rinold you save as man! thanks a lot, it was that, we have diferents threads working with disabling/enabling and dispose, we separete correctly and now works!!!
We love working with open source, and to see that we recognize that, take a look to our project:
http://littlewaronline.com/
Thanks again, and we allways be mention that lib part of libgdx!


Unit tests for click based CLI interface

Fixes #761
I have added the Unit test for command ci currently.
And is the issue have to be solved in a similar way for all other commands?

Hello @vipulchhabra99! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file cli/test/test_cmd_ci.py:

Line 5:1: E302 expected 2 blank lines, found 1
Line 6:1: W191 indentation contains tabs
Line 6:1: W293 blank line contains whitespace
Line 7:1: W191 indentation contains tabs
Line 7:2: E117 over-indented
Line 8:1: W191 indentation contains tabs
Line 8:3: E117 over-indented
Line 9:1: W191 indentation contains tabs
Line 9:29: E231 missing whitespace after ','
Line 10:31: W292 no newline at end of file

hi @vipulchhabra99 thanks a lot. Yes, that's the idea: port all the commands from the shell scripts to python and delete all of them once all of them are ported.

hi @ivotron, I thought of creating a single file containing all the test cases is this fine?

Yeah, that'd be ideal

@ivotron, I think some of the has clashed with other tests and due to which the tests are failing. Am I guessing correct?

@vipulchhabra99 sorry for the confusion. The goal is to port the tests from shell scripts to python, that is, we won't be running popper run --wfile ci/cli.workflow from the test, but instead implement those tests in Python.

Hi @ivotron , I have one more question in porting them to python do I need to run the commands through os module or I have to pass the arguments of commands written in shell scripts to run function?

Hi @ivotron , I have one more question in porting them to python do I need to run the commands through os module or I have to pass the arguments of commands written in shell scripts to run function(Through Click testing)?

the latter (through click testing)

ok, thanks @ivotron then I believe I would have to do in a separate file for each command.

@ivotron , I have updated the file by partially completing skip tests. Could you please review it once so that If I am progressing in right direction, I can continue for other parts

Yeah great @ivotron , I have understood the points just one last question. Can I do in parts and get it reviewed so that It becomes easy for me as well as you to review it completely?

Yeah great @ivotron , I have understood the points just one last question. Can I do in parts and get it reviewed so that It becomes easy for me as well as you to review it completely?

whatever works best for you. Either is fine on my end

Also, can I create the common file using the os module to execute the commands?

this is similar to the test scripts. That is, the functionality contained in the ci/test/common file needs to be ported to Python.

HI @ivotron, Sorry for the delays I have written the common file and the siginit function is still pending, I would add it soon and for some of the part, I have used the os module for them which I am planning to replace with the appropriate module. Please review the common file once.

@vipulchhabra99, now that #779 and #781 have been merged, we'll need to rebase master for this PR?

Ok, sure @ivotron I will rebase the branch, I will get back to you soon with new updates.

thanks @vipulchhabra99! couple of observations:

we need to add the functionality of ci/test/common file and the init_test_repo function to the tests. See for example the ci/test/skip script to see how it's making use of them.
as part of the above, we need to support the POPPER_TEST_MODE environment variable.
you might want to create a super class that does the above so there's no need to be repeating this code.
use file-based fixtures (inside a cli/test/fixtures folder) instead of inlining the content of workflows.

the last step above is motivated by the following. We will soon support a YAML syntax that's similar to HCL (#773), so it would be ideal if we can abstract the syntax from the tests. For example, we can have cli/test/fixtures/skip.workflow and cli/test/fixtures/skip.yml files, and let the test look for the appropriate file based on an environment variable (e.g. POPPER_TEST_FORMAT, selecting .workflow by default).
let me know if anything from above is not clear.
thanks!

In this like you mentioned for POPPER_TEST_FORMAT do it needs to accept it as an argument during the running of the tests?

hi @vipulchhabra99, we merged a major refactoring of the code base that also affected unit tests. Can you please rebase master? Or it might be easier to start all over with a new PR.
In short, it is not necessary to have POPPER_TEST_MODE or POPPER_TEST_FORMAT environment variables anymore. Take a look at the travis.yml file to see how tests are getting invoked. Also, you can take a look at the parser tests to get an idea of how to make use of file-based fixtures. Also, it is OK to only use YAML for your tests, no need to write them both in HCL and YAML, as the differences in the syntax are already tested by the parser tests. In other words, for the click unit tests, we only care that the CLI can be invoked as it is described in the docstring help.
please let me know if you have any question

hi @vipulchhabra99, we merged a major refactoring of the code base that also affected unit tests. Can you please rebase master? Or it might be easier to start all over with a new PR.
In short, it is not necessary to have POPPER_TEST_MODE or POPPER_TEST_FORMAT environment variables anymore. Take a look at the travis.yml file to see how tests are getting invoked. Also, you can take a look at the parser tests to get an idea of how to make use of file-based fixtures. Also, it is OK to only use YAML for your tests, no need to write them both in HCL and YAML, as the differences in the syntax are already tested by the parser tests. In other words, for the click unit tests, we only care that the CLI can be invoked as it is described in the docstring help.
Also, for the click tests, it is not necessary to create a git repository, we can only create workflow files and scripts associated to the workflows. The tests that check for support for .git folders has been consolidated into the test_scm.py file.
please let me know if you have any question

Thanks @ivotron for clearing the doubts, Yes I have rebased the branch after which I start getting the error that I mentioned in the communication channel. I even tried today on one VM but I got a similar error. But I believe that It would be great to give this issue a fresh start since a lot of the features have changed. So I can close this pr and start from scratch.


Problem with "/npc tphere" command

that when we select 1 npc and call it with the "/ mpet tphere" command, all other npc's come to us.

Hello,
Thanks for the report! this issue will be resolved in the next release.

thx :+1:


Working on removing site.baseurl

Please do not merge this until it has been reviewed by @mateoclarke and @luqmaan.
Edits made in this change.

removed baseurl in the _config.yml
removed {{ site.baseurl }}, {{site.baseurl}}, and site.baseurl from the entire codebase
added a new scope to the _config.yml for posts, which basically sets a permalink property. This way, each blog post will have the url pattern http://localhost:4000/blog/:year/:month/:date/blog-post-title

:dancer:

I couldn't find anything wrong :smile:
Great job @techtolentino