[FLINK-1789] [core] [runtime] [java-api] Allow adding of URLs to the usercode class loader

See [FLINK-1789].
I was a little bit unsure in JobWithJars because it is not guaranteed that the classpath is correct locally, maybe someone can have a look at it.

I didn't spot anything. But, the travis build didn't pass ;)

It builds now ;)

:clap:

This does not work if the user uses classes that are not available on the local machine since you don't add the additional class path entries in JobWithJars.buildUserCodeClassLoader(). Correct?

@aljoscha I'm sorry, I cannot follow? Can you elaborate?
The idea is to add support for proper folders next to jars when opening an execution environment. Since folders cannot be handled the same way as jars during execution (i.e., serialize and ship them around by the blob manager), the assumption for folder paths is that they are accessible from all agents in the distributed runtime (JobManager + TaskManagers), e.g., via shared NFS folders.

Yes, this is true, but the way it is implemented, the folders are not always added to the class loader. Maybe I'm wrong here, but JobWithJars.getUserCodeClassLoader and JobWithJars.buildUserCodeClassLoader don't add the URLs to the ClassLoader that they create.

@aljoscha Yes, this is what I also mentioned in my PR description. If we are using a RemoteEnvironment, it is unlikely that the classpath that is available on the cluster is also valid on the local machine. The user has to add the libraries to his local project manually.

OK, @StephanEwen, any thoughts on this? Should we allow that the local user code class loader in the client potentially doesn't have the same jars available as the workers?

I think that keeping the classpaths in sync is really crucial. What is the problem in exposing this to the client as well?
Can we make the following assumptions for the use case where we need the global class path:

The URL is either a file path that points to a directory accessible to all nodes (NFS or so) and the client runs in the cluster as well.
The URL is an HTTP URL or so that points to a file server that serves the classes to work in non-shared directory settings.

Some other remark: This makes the code quite dirty on a lot of parts, since we have always Files and URLs in Client, executors, ... Can we not store everything as URLs (rather than having Strings for jar files and URLs extra). Files can be expressed as URLs anyways as well...

Yes, this would make things a lot cleaner. @twalthr what do you think?

It is not a problem in exposing this to the client as well, I was just unsure if it is necessary. I will add to the client as well to be in sync.
Yes, it makes sense to use URLs instead of  URLs and Files. But the user facing APIs remain Strings right?

Yes, user-facing remains strings which are file paths.

I have implemented your comments and rebased to the current master. The build succeeded except for a KafkaITCase, but I think that has nothing to do with my code.

Oh .. failing {{KafkaITCase}}s are something I'm very interested in ;)
Sadly I can not see the root cause of the issue :( But the issue is unrelated to your change.

Any comments for this PR?

I was originally thinking to merge this after #681 , but I may have to merge this by itself to get it into the release.

I have rebased the PR and added commits of @aalexandrov that introduce a CLI option --classpath.

@twalthr @rmetzger is there a chance to include this PR in the 10.0 release?

Do you mean the 0.10-milestone-1 release? or the final 0.10 release?
Fabian is managing the 0.10-milestone-1 release, as far as I know he has not started with the release activities, so we could still include it, given that we have consensus to merge the PR

@rmetzger the final 10.0, this and the Scala 2.11 compatibility are the two pending issues that make the current Emma master incompatible with vanilla Flink.

I rebased the PR two times. So before I will rebase it a third time, it would be great if we finally find a consensus.

I agree. Lets wait until https://github.com/apache/flink/pull/858 is merged. As far as I understood Stephan, this PR depends on #858.

https://github.com/apache/flink/pull/858 is merged, so we can try to get consensus here

-1
I was not able to get this feature working as expected, with the following job:
public class URLClassloader {

	public static void main(String[] args) throws Exception {
		URL about = URLClassloader.class.getResource("/about.md");

		System.out.println("about = "+about);

		final StreamExecutionEnvironment see = StreamExecutionEnvironment.getExecutionEnvironment();
		see.addSource(new SourceFunction<Object>() {
			@Override
			public void run(SourceContext<Object> ctx) throws Exception {
				URL about = this.getClass().getResource("/about.md");
				System.out.println("about = "+about);
			}
			@Override
			public void cancel() {}
		});

		see.execute();
	}
}

invoking it like this
./bin/flink run -c com.dataartisans.URLClassloader -C file:///home/robert/flink-workdir/flink-irc-logs.github.io/ /home/robert/flink-workdir/debug-project-0.10/target/debug-project-0.10-1.0-SNAPSHOT.jar
lead to the following output
./bin/flink run -c com.dataartisans.URLClassloader -C file:///home/robert/flink-workdir/flink-irc-logs.github.io/ /home/robert/flink-workdir/debug-project-0.10/target/debug-project-0.10-1.0-SNAPSHOT.jar
about = file:/home/robert/flink-workdir/flink-irc-logs.github.io/about.md
09/23/2015 11:38:33	Job execution switched to status RUNNING.
09/23/2015 11:38:33	Custom Source(1/1) switched to SCHEDULED
09/23/2015 11:38:33	Custom Source(1/1) switched to DEPLOYING
09/23/2015 11:38:33	Custom Source(1/1) switched to RUNNING
09/23/2015 11:38:33	Custom Source(1/1) switched to FINISHED
09/23/2015 11:38:33	Job execution switched to status FINISHED.
robert@robert-da ~/incubator-flink/build-target (git)-[CustomUrlsToUserClassloader] % cat log/flink-robert-jobmanager-0-robert-da.out
about = null
about = null
about = null
about = null
about = null

So it seems that the client is using the correct classpath, and its able to load the file. The JobManager/TaskManager (I've started flink with start-local.sh) doesn't seem to be able to load the file.

I just had a look at this pull request. Why don't we treat the JARs in the given folder like the the other jar files that are submitted to the cluster via the BlobManager as part of the job submission? This assumes that the Client can access the JARs but that's IMHO a fair assumption to make. If the Client can't access one of the provided JAR files, we can fail with an appropriate error message.

Do we have to have a separate list of classpaths in addition to the user classloader or can we have a user classloader which contains all the JARs that a user has provided (either via the user jar or via an extra folder)?

@mxm The jars in which folder? The main motivation point of the pull request is to add the option to add a classpaths where one can generate code at runtime.

@mxm Think of it that the URLs are http URLs to an http server that serves dynamically generated classes...

@aalexandrov Concerning the folders, you wrote:

The idea is to add support for proper folders next to jars when opening an execution environment. Since folders cannot be handled the same way as jars during execution (i.e., serialize and ship them around by the blob manager), the assumption for folder paths is that they are accessible from all agents in the distributed runtime (JobManager + TaskManagers), e.g., via shared NFS folders.

@StephanEwen Why can't the "dynamically generated" class files not be shipped during deployment of the job and thereafter be available through the flink user code classloader?
I didn't see any hint in the JIRA or in this pull request that the classes should be generated at runtime. So either I don't understand this feature correctly or the feature description in the JIRA is not correct.

I've read through the JIRA conversation again. We recently had a discussion on the mailing list that we would like to have more self-explanatory JIRA descriptions. I think this is a classic case where this is simply not met. @twalthr Could you please update the JIRA/Pull request description and clearly state your problem definition and use case.
These things aside, I think this is a pretty neat idea. Could you rebase again to the current master? It also needs to have some more test exposure. It seems like the URLs are not properly passed for streaming applications. I might be able to help you with rebasing or tests. Just let me know how you want to proceed and where you are currently. Thank you.

Just to defent @twalthr and @aalexandrov here a bit: The JIRA and PR were both created before we asked for more extensive descriptions.

@rmetzger Your test program runs now. I forgot the support for packaged programs. There are quite a lot possiblilities to run a job ;)

@mxm I updated the JIRA description with the most important comments of the discussion.

Thanks for updating the JIRA description and the pull request! I'll try to review the pull request later today.

This looks good to merge. However, there are a lot of merge conflicts. Could you rebase to the master again? We're are thinking about creating a release candidate for 0.10 until the end of next week.

It took me one working day to rebase it (actually I had to touch almost every line again), but now it is mergable. @mxm maybe you could have look at it again.

.... and it has conflicts again :(

Conflicts solved again ;)

Thanks a lot @twalthr for rebasing the pull request. There is an issue with the Scalashell test. Not sure why it only occurs in some builds.

@mxm The issue is probably related to Scala 2.10, since the two passing builds have Dscala-2.11. I'll investigate tonight if I can get a hold on the PR commits (GitHub is somewhat slow at the moment)

I think the NullPointerException was due to a wrong array to list conversion. I don't know why this is related to the Scala version.

It could be that the test is not executed for Scala 2.11.

The build succeeded. How do we proceed? It would be very nice if we finally merge this.

I went through the pull request again and it looks good.
+1 for merging

Do we need another committers +1? It touches very sensitive parts...

I think @StephanEwen wanted to merge it as well. @rmetzger concerns should be out of the way with the new version. I would merge this by the end of the day.

Looks good to merge from my side.
The only uncertainty remaining is whether we need the static factory methods on the ExecutionEnvironment. Adding these global classpaths seems a very specific case, in which we can probably expect people to manually create a RemoteEnvironment. That way we don't over load the ExecutionEnvironment with too many factory methods (we have a tendency to get API bloat).

+1 for removing the factory methods from ExecutionEnvironment. Most users won't even think about using this feature and be rather confused by the additional methods. Let's also keep the old constructor of the RemoteEnvironment in addition to the new one.

@twalthr Could you make the changes and update the PR? Thank you.

@mxm done ;)

@twalthr Doesn't seem to compile :)

Merged. Congrats :)

Thanks, @mxm ;)


cmedb database question

Any idea how to clear the database for a new engagement?

Is there any reason not to just revert to checked out HEAD commit? It would wipe out any local changes:
got reset --hard HEAD
If you made any local changes you would lose them, but if you haven't it should also now be easy fetch new changes. Tthis repo is actively maintained and new stuff is added and fixed all the time.

@bucky67gto you can just delete the ~/.cme directory, it will be re-created on runtime. I definitely agree there should be a better way of doing this though.

Or alternatively just delete the smb.db file in the ~/.cme directory ;)

Hello,
Thanks for the issue, it is now fixed on the last public release of CrackMapExec https://github.com/mpgn/CrackMapExec v6.0.0
Regards,
mpgn


PR: Add Unit Tests for ObjectAsPrimitiveConverter

This PR adds a bunch of unit tests for ObjectAsPrimitiveConverter to make sure it handles various JSON scenarios correctly. Here’s what’s covered:

Basic Types: Checks null, true, false, strings, and numbers to confirm they’re converted right.
Array and Object Parsing: Tests JSON arrays as List<object> and JSON objects as either ExpandoObject or Dictionary, depending on settings.
Special Number Handling: Adds a test for unknown numbers that should return a JsonElement when configured.
Nested Structures: Ensures nested JSON arrays and objects are parsed accurately.
Edge Cases:

Empty arrays and objects should return empty lists/dictionaries.
Invalid JSON should throw a JsonException.
Tests for special property names with unusual characters or whitespace.

Boundary Values: Tests big numbers to make sure we handle edge cases like max Int32 and Double values correctly.

Looks like coverage is still below the allowed threshold.

@JasonBoggsAtHMSdotCom
yes, Im still working on it.... checking the lines and the implementation, there some places in the code that I cant reach, maybe will never execute

have either of you viewed the build warnings?  would clearing them up improve the codecoverage?  to see these build warnings, view 'files changed' in this PR and scroll past the file that is changed to see the warnings in question

Thank you @RenanCarlosPereira and @asulwer for your work on this. Is there any chance we can get this released and pushed to Nuget ASAP please?

@JasonBoggsAtHMSdotCom i am pushing an update later today


Clarify how to use the library with Laravel and Lumen

Thanks for this library and your blog post about it - it's super a useful, and I feel like a really valuable feature to have!
I used this with Lumen, but I've also slightly improved the instructions on how to use it with Laravel.

@timrogers Thanks!


Small fixes

Addressed #45 and upped Rennet droprate to 100%

Can you store the value in a variable rather than querying it twice


chore(deps): update infima npm dependency to version 0.2.0-alpha.44

Pre-flight checklist

[x] I have read the Contributing Guidelines on pull requests.
[x] If this is a code change: I have written unit tests and/or added dogfooding pages to fully verify the new behavior.
[x] If this is a new API or substantial change: the PR has an accompanying issue (closes #0000) and the maintainers have approved on my working plan.

Test links
Deploy preview: https://deploy-preview-_____--docusaurus-2.netlify.app/
Related issues/PRs
addresses https://github.com/facebookincubator/infima/issues/306

thanks


Release v0.24.1

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:

NONE

🚀
/lgtm

🚀
/approve


 change does not respect encoding attribute as mentioned in the documentation

Environment
Liquibase Version: 4.3.1
Liquibase Integration & Version: Java 1.8 (custom Gradle plugin)
Operating System Type & Version: Windows 10
Description
The documentation (https://docs.liquibase.com/change-types/community/sql-file.html) describes a very useful attribute for the <sqlFile> change: encoding - Specifies encoding used in the file defined in the path attribute.
This attribute is not processed by the <sqlFile> change implementation and thus does not work.
Steps To Reproduce

Create a changelog with a changeset containing a <sqlFile> change that refers to some SQL file with UTF-8 encoded special characters.
Set encoding attribute to UTF-8 (which is an example value directly from the XSD schema)
Generate SQL with Liquibase while using a default file encoding for the JVM other than UTF-8.

Actual Behavior
The generated SQL contains wrongly rendered characters.
Expected/Desired Behavior
The generated SQL contains characters as given in the provided SQL file referred by the <sqlFile> change.

I think it's really simple to fix. In liquibase.util.StreamUtil.readStreamAsString(InputStream) use StreamUtil.readStreamAsString(sqlStream, getEncoding()) instead of StreamUtil.readStreamAsString(sqlStream).

Hi @ckulenkampff Thanks for writing up this issue and the suggestion for the fix.  We will add this to the list of issues to process.  If you want to create a PR with your fix that may speed up the process.

Just to save others some work; I used the following workarounds:
For the CLI, one can simply set the system property file.encoding to the encoding of the SQL files (they must all have the same encoding).
When using the Java API, one can use the following:
Scope.child(Scope.Attr.fileEncoding, Charset.forName(fileEncoding), () -> liquiBase.update(new Contexts()));

this issue totally blocks changeSets with different encoding in the sql-files. which is usual practice for multilingual projects. 'ImpactLow' - I'm not sure... :)
and 'the fix' is recorded in source of 3.x versions already - just see diff 3.9.x/4.0.x

Hi @EGlaznev Thanks for the update. We will add this to the list of issues to process.
In the meantime, you can take the artifact from https://github.com/liquibase/liquibase/pull/1941 as an interim fix and use that to get past your blocked changeSets.


SecurityPlugin: Add primitiveGetUntrustedUserDirectory

This change allows it, for instance, to enable the preference #startInUntrustedDirectory in Squeak.
Please explicitly review whether I have rebuilt the bundles correctly. To me, it looks as if the change in the SoundGenerationPlugin has been forgotten to be built via e3673571294da2cf40adf5265ae0106f087cca65.

With https://github.com/squeak-smalltalk/squeak-filesystem/pull/4, also Squeak-FileSystem works in SqueakJS. 🎉
(BTW, it was very nice to get this project running directly from gitpod.io :D)

Hi @codefrau, could this be merged? :)

Hi – yes, something like this could be merged, and I'm sorry for not replying sooner.
I do have a few issues with this PR:

pull requests should not modify the dist/ dir – I will update the bundles when I make a new release. Apps that need unreleased features should use squeak.js directly (which will import everything else in turn) or build their own bundle.
SecurityManager default untrustedUserDirectory already answers '/SqueakJS' – why do we need this?
If we decide to add more SecurityPlugin functions I'd add at the very least both untrustedUserDirectory and trustedUserDirectory prims, to answer distinct directories. trustedUserDirectory must never be a subdirectory of (or the same as) untrustedUserDirectory.
Preferably we'd actually implement SecurityPlugin properly, enforcing file/socket/image write checks.

@codefrau Sorry again for the long delay!

SecurityManager default untrustedUserDirectory already answers '/SqueakJS' – why do we need this?

This comes from the fallback path in SecurityManager default untrustedUserDirectory, which falls back to FileDirectory default pathName. FileSystem, however, invokes the primitive manually. But I don't know whether it has to. I would also be open to patching FileSystem to use the SecurityManager class instead. This might be simpler than providing a proper implementation of the SecurityPlugin. What do you prefer? Right now I don't have the capacities (nor a personal motivation ^^) to implement a more sophisticated interface in SqueakJS's SecurityPlugin. I just thought it was a low hanging fruit to complete this PR ...

I merged without your dist changes. It doesn't hurt to have another fake prim, after all.


Update import dock when editing a resource from the inspector

If you need to change the import settings of a resource you are editing, its very boring to have to find said file in the FS dock first.
With this change, once you select the resource to edit it, it will update the import dock to reflect that file.
not sure if this is totally clean the way it is now, or if we should reset the import dock when something different is selected in the scene tree. Not very relevant, but..... WDYT? :)

Would solve #8139.

oh, i think it's in the wrong place, can you move it to EditorNode::_edit_current ? otherwise it won't be updated when moving with the forward and back arrows


Gem Load Error is: uninitialized constant Uglifier::ExecJS

i was create new rails app found that issue,

rails new appname

i was cancel install gem from rails new appname because network seem have some issue
so when run bundle install appearance i typed ctrl-c
and cd appname directory run bundle install,  after all the gem is installed

rails s

/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:94:in `rescue in block (2 levels) in require': There was an error while trying to load the gem 'uglifier'. (Bundler::GemRequireError)
Gem Load Error is: uninitialized constant Uglifier::ExecJS
Backtrace for gem load error is:
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/uglifier-3.2.0/lib/uglifier.rb:11:in `<class:Uglifier>'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/uglifier-3.2.0/lib/uglifier.rb:9:in `<top (required)>'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:91:in `require'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:91:in `block (2 levels) in require'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:86:in `each'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:86:in `block in require'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:75:in `each'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:75:in `require'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler.rb:107:in `require'
/home/f/app_foo/config/application.rb:7:in `<top (required)>'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:88:in `require'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:88:in `block in server'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:85:in `tap'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:85:in `server'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
/home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands.rb:18:in `<top (required)>'
bin/rails:4:in `require'
bin/rails:4:in `<main>'
Bundler Error Backtrace:
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:90:in `block (2 levels) in require'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:86:in `each'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:86:in `block in require'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:75:in `each'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:75:in `require'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.14.6/lib/bundler.rb:107:in `require'
	from /home/f/app_foo/config/application.rb:7:in `<top (required)>'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:88:in `require'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:88:in `block in server'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:85:in `tap'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:85:in `server'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
	from /home/f/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.0.2/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

system: debian 8.7
ruby -v: 2.4.1
rails -v: 5.0.2
bundle -v: 1.14.6
source 'https://ruby.taobao.org'

git_source(:github) do |repo_name|
  repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
  "https://github.com/#{repo_name}.git"
end

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 5.0.2'
# Use sqlite3 as the database for Active Record
gem 'sqlite3'
# Use Puma as the app server
gem 'puma', '~> 3.0'
# Use SCSS for stylesheets
gem 'sass-rails', '~> 5.0'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# Use CoffeeScript for .coffee assets and views
gem 'coffee-rails', '~> 4.2'
# See https://github.com/rails/execjs#readme for more supported runtimes
 gem 'therubyracer', platforms: :ruby

# Use jquery as the JavaScript library
gem 'jquery-rails'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.5'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 3.0'
# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'

# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

group :development, :test do
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug', platform: :mri
end

group :development do
  # Access an IRB console on exception pages or by using <%= console %> anywhere in the code.
  gem 'web-console', '>= 3.3.0'
  gem 'listen', '~> 3.0.5'
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
  gem 'spring-watcher-listen', '~> 2.0.0'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

That's weird. ExecJS is required by Uglifier and also coffee-rails in your Gemfile. I'm not sure if cancelling the bundle install could've ended in a weird state. I'd try reinstalling execjs: gem uninstall execjs and gem install execjs.

oh, that's strange, i'm gem uninstall execjs and gem install execjs
it's success run rails s
Gemfile
source 'https://rubygems.org'

git_source(:github) do |repo_name|
  repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
  "https://github.com/#{repo_name}.git"
end

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 5.0.2'
# Use sqlite3 as the database for Active Record
gem 'sqlite3'
# Use Puma as the app server
gem 'puma', '~> 3.0'
# Use SCSS for stylesheets
gem 'sass-rails', '~> 5.0'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# Use CoffeeScript for .coffee assets and views
gem 'coffee-rails', '~> 4.2'
# See https://github.com/rails/execjs#readme for more supported runtimes
# gem 'therubyracer', platforms: :ruby

# Use jquery as the JavaScript library
gem 'jquery-rails'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.5'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 3.0'
# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'

# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

group :development, :test do
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug', platform: :mri
end

group :development do
  # Access an IRB console on exception pages or by using <%= console %> anywhere in the code.
  gem 'web-console', '>= 3.3.0'
  gem 'listen', '~> 3.0.5'
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
  gem 'spring-watcher-listen', '~> 2.0.0'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

thanks


Dogfooding autosharding internally

This is a bug report/feature or change request/support request
Problem

Proposed Solutions

Notes

Blocked by https://github.com/waku-org/js-rln/issues/91

Should be addressed by following issues:

https://github.com/waku-org/js-waku/issues/1967
https://github.com/waku-org/js-waku/issues/1961


Add implementation types to examples table

Description
This PR is blocked by #2255 #2335
Also added missing examples from other PR's to not block any of them so we could merge them soon
Checklist

[ ] I have run cargo make pr-flow
[ ] I have reviewed my own code
[ ] I have added tests

@hamza1311 This Pr is no longer blocked. we can merge it in.


Allow using script preloaded with UID as a type

Addresses second item from https://github.com/godotengine/godot/issues/87418
Again, I'd like to add a test but since the tests currently only load one .gd file at a time, the test setup would need some extensive work

Preloading with only a UID is less readable and inconsistent with other resources. I would expect an optional uid argument for load/preload (preload("res://script.gd", "uid://uid")). See https://github.com/godotengine/godot/issues/87418#issuecomment-1907516677 for details.


How to use the same ViewController and not a new one every click

I'm using a tabbar as the main view and I have others VC in the menu.
When i leave the TabBarController, go to another VC and come back, this is a new tabbar not the old one.
How to reuse the same?
Thank!

Yes that is something interested


Wrong runtime selected when pom file has Liberty/OpenLiberty elements

Describe the bug
When running move2kube against a directory that contains a Maven pom.xml and the pom.xml contains elements for OpenLiberty/Liberty the tool creates a Dockerfile  that includes JBoss, not Liberty.
To Reproduce
Steps to reproduce the behavior:

create a pom.xml file with Liberty/OL plugin

    <parent>
        <groupId>io.openliberty.tools</groupId>
        <artifactId>liberty-maven-app-parent</artifactId>
        <version>3.2.1</version>
    </parent>

            <plugin>
                <groupId>io.openliberty.tools</groupId>
                <artifactId>liberty-maven-plugin</artifactId>
                <version>3.2.1</version>
                <configuration>
                    <assemblyArtifact>
                        <groupId>io.openliberty</groupId>
                        <artifactId>openliberty-runtime</artifactId>
                        <version>[<IP_ADDRESS>,)</version>
                        <type>zip</type>
                    </assemblyArtifact>
                    <serverName>${project.artifactId}Server</serverName>
                    <include>${packaging.type}</include>
                </configuration>
            </plugin>

Runt he move2kube command
move2kube plan -s directory

run the translate step
move2kube translate m2k.plan

The Dockerfile created is for JBoss:

FROM registry.access.redhat.com/ubi8/ubi:latest AS build_base
RUN yum install -y java-1.8.0-openjdk-devel
RUN yum install -y maven
COPY . /app
WORKDIR /app
RUN mvn package

# Run App
FROM registry.access.redhat.com/jboss-eap-6/eap64-openshift:latest
EXPOSE 8080
COPY --from=build_base /app/target/* /opt/eap/standalone/deployments/

Expected behavior
Either recognise that the image is for Liberty/OL and use Liberty/OL in the Dockerfile or ask which one should be chosen.
Desktop (please complete the following information):

OS: Linux (Unbuntu 18.04), MacOS 11.x
Version (from homebrew)

move2kube version
v0.1.0

Regards, Brian

Thanks @bpaskin-ibm for logging the issue. We will evaluate it and get back. The pom.xml sample is good. But was wondering whether it would be possible to share sample working code?

In v3 we all the user which runtime to use.


Create days.txt

password permutations of days of the week

Thanks for the suggestion, are you able to give the source to this.

Just a list of days in lowercase:
monday
tuesday
wednesday
thursday
friday
saturday
sunday

Thank you again!


[CI] Fix ios/android e2e tests

Summary
Fixes broken ios/android e2e tests for CI.
Changes:

Use npm pack instead of yarn pack during e2e test
Update test string assertions in ios/android e2e tests
Use Step One as candidate for source code replacement to test for page changes (since the "Welcome to React Native" string exists in the Header component instead of being on App.js)

Changelog

[Internal] [Fixed] - Fix ios/android e2e tests
Test Plan

ios/android e2e tests in CI stage should pass

I also verified that android e2e passes locally on my box. Let's see if we can re-enable android e2e on the CI :)

@hramos i know you're trying to land #24972, should be safe to reinstate the ios e2e test on the CI 👍


Bump controller-runtime to 0.15

The controller-runtime dependency needs to be updated to 0.15 to fix issues introduce by k8s.io/* v0.27.x This change bumps the controller-runtime dependency which will unblock to k8s.io upgrades.

We can't merge this until all of our dependent packages implement the required interface changes:
❯ go vet ./...
# github.com/openstack-k8s-operators/openstack-ansibleee-operator/api/v1alpha1
../../../go/pkg/mod/github.com/openstack-k8s-operators/openstack-ansibleee-operator/api@v0.0.0-20230523171900-dd4d0945d2d1/v1alpha1/openstackansibleee_webhook.go:76:27: cannot use &OpenStackAnsibleEE{} (value of type *OpenStackAnsibleEE) as admission.Validator value in variable declaration: *OpenStackAnsibleEE does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
# github.com/openstack-k8s-operators/nova-operator/api/v1beta1
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/nova_webhook.go:105:27: cannot use &Nova{} (value of type *Nova) as admission.Validator value in variable declaration: *Nova does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/novaapi_webhook.go:76:27: cannot use &NovaAPI{} (value of type *NovaAPI) as admission.Validator value in variable declaration: *NovaAPI does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/novacell_webhook.go:86:27: cannot use &NovaCell{} (value of type *NovaCell) as admission.Validator value in variable declaration: *NovaCell does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/novaconductor_webhook.go:76:27: cannot use &NovaConductor{} (value of type *NovaConductor) as admission.Validator value in variable declaration: *NovaConductor does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/novaexternalcompute_webhook.go:86:27: cannot use &NovaExternalCompute{} (value of type *NovaExternalCompute) as admission.Validator value in variable declaration: *NovaExternalCompute does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/novametadata_webhook.go:76:27: cannot use &NovaMetadata{} (value of type *NovaMetadata) as admission.Validator value in variable declaration: *NovaMetadata does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/novanovncproxy_webhook.go:76:27: cannot use &NovaNoVNCProxy{} (value of type *NovaNoVNCProxy) as admission.Validator value in variable declaration: *NovaNoVNCProxy does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/openstack-k8s-operators/nova-operator/api@v0.0.0-20230523154806-b03835b62112/v1beta1/novascheduler_webhook.go:76:27: cannot use &NovaScheduler{} (value of type *NovaScheduler) as admission.Validator value in variable declaration: *NovaScheduler does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
# github.com/metal3-io/baremetal-operator/apis/metal3.io/v1alpha1
../../../go/pkg/mod/github.com/metal3-io/baremetal-operator/apis@v0.2.0/metal3.io/v1alpha1/baremetalhost_webhook.go:37:27: cannot use &BareMetalHost{} (value of type *BareMetalHost) as admission.Validator value in variable declaration: *BareMetalHost does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)
../../../go/pkg/mod/github.com/metal3-io/baremetal-operator/apis@v0.2.0/metal3.io/v1alpha1/bmceventsubscription_webhook.go:37:27: cannot use &BMCEventSubscription{} (value of type *BMCEventSubscription) as admission.Validator value in variable declaration: *BMCEventSubscription does not implement admission.Validator (wrong type for method ValidateCreate)
		have ValidateCreate() error
		want ValidateCreate() (admission.Warnings, error)

It seems we will need to do MariaDB and Keystone first. Then we can look at tackling some of the above issues.

@bshephar does renovate bot automatically open PRs for this kind of version bump?

I don't see this in other operators (I just spot checked a few). Is it still needed?

I don't see this in other operators (I just spot checked a few). Is it still needed?

@bshephar Is this still needed?

All our other operators are still at v0.14.6
Closing out this PR due to age. Please re-open it when you (or someone) is able
to start working on it again.


Japes DK Cage remaining open if a DK portal spawns in the back

Potential solution: https://github.com/2dos/DK64-Randomizer/issues/2234
This issue might be a duplicate


Positioning tweaks

A picture tells a thousand words
Before this PR
After this PR

Coverage increased (+0.02%) to 40.42% when pulling b98d0d9418d4cf3d6858cb129a2054545ddcce43 on bjornreppen:align into 7d81e008b8ddf54c7b0f479f6cee71dd32136cef on Artsdatabanken:master.


Why the context and code text count is very small for copilot chat?

Please tell us why it's very low also when will we see improvements and have claude level context?

I would very much like to know this as well. What is the point of having a chat history and being able to add multiple files to the context if it is all forgotten almost immediately on the next chat because it can't store the context from the previous messages in the history. It makes copilot chat almost unusable.

Please provide concrete examples we can operate on where the context is not allowing you to complete the work you need. The context window is actually quite large, we just do not advertise it at this time.


5.4 Pattern filtering (equals/contains/starts with/wildcard)

API and in memory filtering support

Specs for strings:


Improve lode UI

Add some basic styles.
Add triangle count
Switch to one-pager
Add LOD Viewer

Looks promising 👍
I've added some comments.

Exceptions are fine, but please provide a comment on the reasoning. This helps - especially as we have no tests so far.


Namespace UI

Show namespace capabilities
Show interaction between namespace and server capabilities

Closes https://github.com/PelicanPlatform/pelican/issues/1724

New namespace card with the nested capabilities. Should give easy insight to what caches and servers are open to this namespace.

New namespace card with the nested capabilities. Should give easy insight to what caches and origins provide what capabilities for this namespace.

There are multiple "issuer" showing up here. I think it'd be better to differentiate them here in case causing further confusion. Here are my two cents to avoid semantic overload:

Change "issuer" under "token generation" to another word, like "generator". Also change its occurrences everywhere else
Add tool-tip to "issuer" and "generator", explaining their difference in Pelican project. For example:

Token Generator: a server that performs the technical task of creating the token
Token Issuer: a server as a trusted authority that vouches for the authenticity of the token

Could you take a quick look and share your thoughts on the necessity of this change @jhiemstrawisc


Upgrade from Python 3.9.x to 3.10.x

Generally speaking it would be beneficial to make the program somewhat backward compatible. It is fairly easy to install latest Python in Windows, on a Ubuntu Desktop (e.g. 20.04) only 3.8 is available.

Out of date issue - I have just upgraded to 3.11 as part of massive rework for v4 of Guru
Will raise new issue though to revisit Lars' concern about backwards compatibility


Add a way for the Quirrel API to communicate with the Quirrel Dashboard for DX-style communication (e.g. email "a job has failed")

Closed by #29


Give Dragons other uses besides flight.

Title. Like the normal dragon, they could shoot dragon fireballs and create dragon breath clouds. Other elemental dragons could also do this, but have other effects besides your usual high-damage AoE cloud.

There is an unused breath weapon mechanic, but this codebase is nightmarish to work with. I will probably need to commission someone to clean it up lmao.


Does not display the element while dragging in Safari

Describe the bug
When I drag an item, the selected item is not displayed. When dropped - it's fine, the item is in the needed place.
Or I have an SVG file inside a div and when I drag it, it shows the Chrome logo.
Screenshots

Desktop:

OS: macOS Monterey 12.6
Browser: Safari 16.0
Version: react-dnd 15.1.2 and react-dnd-html-5-backend 15.1.3. Also tried on 16.0.1

Additional context
I also added CSS styles:
   -webkit-touch-callout: none;
    -webkit-user-select: none;
    -khtml-user-select: none;
    -moz-user-select: none;
    -ms-user-select: none;
    user-select: none;

I had 2 bugs:

When I drag an item, the selected item is not displayed. Solution:
Adding transform: 'unset' to overlayClass. I use 'react-modal'. But the modal would be crushed. In this case, you need to change the styles to center the modal.
I have an SVG file inside a div and when I drag it, it shows the Chrome logo. Solution:
Adding attribute draggable="false" to tag img .

我使用了一个hacking方案，在这里
https://github.com/react-dnd/react-dnd/issues/2708


refactor: Reduce imperative shell logic in flake.nix

Regarding HOME specifically, I would've liked to eliminate that too as it's nearly always a pungent smell in Nix builds, but Burrito.Steps.Build.PackAndBuild seems to
want a writable path to work in. AFAICT it's purely for scratch space, so we can just directly reuse the Nix build's temporary directory.

Works on my x86 Linux and M1 Mac computers, will merge once you revert the bin/7z deletion


[Module request] Please support native drawer | side menu

Folks, most of mobile apps need drawer, please help integrate it into the starter kit
https://raw.githubusercontent.com/LukeBrandonFarrell/open-source-images/master/react-native-navigation-drawer-extension/left-drawer.gif
https://raw.githubusercontent.com/LukeBrandonFarrell/open-source-images/master/react-native-navigation-drawer-extension/bottom-drawer.gif
Ref: https://github.com/lukebrandonfarrell/react-native-navigation-drawer-extension
Cheers,

I would not agree that most of mobile apps need drawer. It's better to use tabs, more UX friendly. Plus, the drawer is not officially supported by RNN, it's just a workaround. However, if you'd like to use that library, I believe that there is easy installation instructions.


[barray] adopt the barray module

This is WIP, but I think we can do a quick review for it.
A few things in this change:

I added the iterator module to replace the sequence module. The OCaml
community seems have not reached an aggrement on how functional
iterator would look like, we've got gen, seq, Core.Sequence etc.  Dave suggested we use
this as well.
Adopted the barray module, added a Collection interface

Things to do:

add unit tests for barray

I also pretty much concur with everything @bordoley said

A couple of general comment (around the same theme)

"fromList is inefficient": this is all relative and really up to the user to decide whether it meets its performance requirement of not. The documentation should specify the complexity of the operation but I believe we should still have such function even though there exist more efficient (and more complex) solution to the problem. What if I am writing a quick and simple program where performance does not matter (unit test, little script ...)? (It's perfectly fine btw to advertise better solutions in the documentation but the decision is eventually left to the user).
usage of unsafeGet/Set, I very much like those options to be available. There are many use cases that are perfectly fine using those unsafe API (generated code, algorithm where the boundary are check ahead of accessing the elements, etc). [@@undefined_to_opt] is still not as efficient as an unsafe access.
Can we rename collection to bcollection.

Hey sorry for the delay, was on holiday for a while, I think this is ready for another review, I cleaned up the interface and added more unit tests. @glennsl @bordoley

@mransan:

"fromList is inefficient": this is all relative and really up to the user to decide whether it meets its performance requirement of not. The documentation should specify the complexity of the operation but I believe we should still have such function even though there exist more efficient (and more complex) solution to the problem. What if I am writing a quick and simple program where performance does not matter (unit test, little script ...)? (It's perfectly fine btw to advertise better solutions in the documentation but the decision is eventually left to the user).

I think it's good design to make performance costs explicit, since it makes the implications apparent when reading the code. It's usually considered bad practice to optimize prematurely, and most people don't read the documentation for basic stuff like this anyway, so it's important to have this information at read-time, not just write-time.
There might be better solutions than inconvenience of course, but I don't think documentation is a good substitute for poor design, and I don't think the design should optimize for scripting either.

@glennsl "fromList" is not inefficient, it's just a convertor between 2 containers. If we're discussing 2 implementations differences of that function and one is more performant than another I would agree, but here we're talking whether converting a list to an array should be exposed at all.
A simple case could be that you use an API (3rd party) that returns a list but then you need to perform a one-time algorithm which require random access. Why not simply converting one to the other using the fromList function ? Are there better alternatives ? If so, why the fromList is not implemented in terms of them ?

@mransan Conversion between any two containers is supported via sequence (xs |> Array.toSequence |> List.fromSequence). What you're requesting is that array and list should be special cased, which strongly implies that there is a more efficient conversion between them (in the absence of some other obvious reason for special treatment).

I lean towards not having fromList now since we have an iterator class Sequence which should play this role.

@glennsl @dorafmon, so fromSequence is:
let fromSequence i =
  of_list @@ Sequence.foldLeft (fun acc item -> acc @ [item]) [] i

So now I need to convert my list to a sequence which gets converted back to a list in order to
call the ofList function that has been removed from the interface.
a) it's slower
b) it's more complex to understand (ie new comers need to also understand sequence concept even for the 2 most popular containers in OCaml!)
I agree the unification of all containers to a general concept as a sequence is elegant and theoretically pleasing, but sometimes I just think it defeats the purpose of simplicity.

@mransan there is no reason that implementation can't be changed not to implement a list. Its basically impossible to implement Array.fromList/Seq, efficiently because it's always going to involved numerous o(n) copies. Another option is to require the caller you to pass the size as an argument to the Array.from* function, and raise an exception if you can't pull enough items from the list/seq to fill the array.

@mransan @bordoley  If we are only considering the BS case, then we can implement fromSequence like this:
let fromSequence i =
    let open Bsequence in
    let arr = ref [||] in
    let rec aux i =
        match i () with
        | Nil -> !arr
        | Cons(x, r) -> ignore @@ Js.Array.push x !arr; aux r
    in aux i

JS offers variable length array and thus it made it this possible. We only need 1 copy of the array in total.

A summary of outstanding issues and what I am thinking:

@glennsl  mentioned the formatting issue, I am using ocp-indent and I have checked in the configuration file. I think we should leave the formatting up to the autoformatter. @glennsl could you check and see if you see any particular issue we should address?
@mransan mentioned we should prefix modules with b. There are a few modules that do not follow this guideline. I will change it in a seperate PR.

Apart from these two can we get this PR merged if no other objection? I think the interface is nice now and I the tests should provide at least basic coverage.
Thanks a lot!

@dorafmon: Under the covers JSArray is effectively a C++ vector. There are optimizations for small arrays (and you can implement these yourself as well) but pushing into large arrays can also cause significant amounts of copying. You also sacrifice some array optimizations. This article has more details (old but still mostly accurate): https://gamealchemist.wordpress.com/2013/05/01/lets-get-those-javascript-arrays-to-work-fast/ See item 4.
For large datasets, it's likely that pushing into an immutable vector such as those implemented by Imm.re/Immutable.js will be faster than pushing into a native js array.

@bordoley out of curiosity I assume the immutable vector has a O(1) conversion to arrays?

@dorafmon I'll eat my words a little. I ran some perf tests last night, and indeed Array.push is fast (in absolute time) for pushing large number of items (faster than I thought it would be), beating even push into a Transient Vector by 30 to 50%. I tested with Arrays where n = 7M on node. This surprised me. I looked into the BS stdlib, and I suspect that I could optimize some array CopyOnWriteArray functions to use native implementations that could achieve equal or better performance.
That said push is still not "efficient" in absolute terms as pre-allocating a single array and then inserting elements into it should be faster since it avoids generating garbage.
Imm.re vectors do not have an O(1) conversion to array. However, it's possible to implement an conversion via one allocation, and O(N) copies. This is due to the fact that Imm.re vectors have an O(1) count operation.

@bordoley efficiency is relative and depends on the context.
Take this example: you call a third party API (no control over it) which returns a list, you then want to transform it into an array to make an expensive computation which required random access of the elements. So you must convert a 'a list to 'a array. What are the choices:

use fromList: does the minimum amount of allocations and has linear complexity. (yes 2 iterations one to get length and one to copy).
use to/fromSequence: duplicate the list into another list (linear number of allocation)
use Immuntable List/Array/Vector as intermediate data structure to get to an array: no matter what you will need to allocate more stuff.

So now in this example (which by the way is very common), the most efficient solution is a plain old fromList. Maybe I miss something :) let me know!

@bordoley @mransan Did some benchmarking, the code is here. So tired fromList and fromSequence on a large array and not surprisingly fromSequence is slower (around 5 times usually on my Mac). I used node's profiler and think the reason is that we grow the array too often.
So I modified the code a bit so it allocate a large array beforehand and then assign values to the array. But this version turns out to be even slower. I think we should include fromList, toList as the conversion happens a lot and in this case fromSeq/toSeq seems still suffers from performance.

@glennsl  I reformated the code a bit, especially the test. Is it better now?

@dorafmon The last half or so barray_test is better. Not as good as before you started messing with it, but good enough. bsequence_test, int_test and the first half of barray_test is still pretty bad.
There's also lots of remaining issues with 4-space indentation, even if you've fixed the config the code itself hasn't been reformatted.

@glennsl  I overhauled the whole formatting a bit. Should be a bit better.


Issue with run.sh script location

I have found an issue when trying to create a Dataproc custom image when following the Google Cloud Platform documentation[1].
Error Logs
[Daisy] Errors in one or more workflows:
  debian-error: FileIOError: stat /home/$USER/dataproc-custom-image/dataproc-custom-images/custom_image_utils/startup_script/run.sh: no such file or directory
Traceback (most recent call last):
  File "generate_custom_image.py", line 110, in <module>
    main()
  File "generate_custom_image.py", line 101, in main
    daisy_image_creator.create(args)
  File "/home/borjah/dataproc-custom-image/dataproc-custom-images/custom_image_utils/daisy_image_creator.py", line 43, in create
    os.path.abspath(args.daisy_path), workflow_script)
  File "/home/$USER/dataproc-custom-image/dataproc-custom-images/custom_image_utils/daisy_workflow_executor.py", line 42, in run
    raise RuntimeError("Error building custom image.")
RuntimeError: Error building custom image.

Steps to reproduce error
Executing the commands from the Google Clound Console ,these are the steps I followed in order to reproduce the error:

Cloned the repo: git clone https://github.com/GoogleCloudPlatform/dataproc-custom-images.git
Moved to the custom images folder: cd dataproc-custom-images
Enabled the Cloud Dataproc API, Compute Engine API, and Cloud Storage APIs.
Download daisy bin: wget https://storage.googleapis.com/compute-image-tools/release/linux/daisy
Give daisy binary file execution permission: chmod +x daisy
Created a Cloud Storage Bucket
Created initialisation script with the following commands inside:

  #! /usr/bin/bash
  apt-get -y update
  apt-get install python-dev -y
  apt-get install python-pip -y
  pip install numpy

Execute the image creation command:
python generate_custom_image.py --image-name custom-image-error --dataproc-version 1.4.7-debian9 --customization-script initialisation.sh --zone europe-west1-c --gcs-bucket gs://mybucket --daisy-path daisy

Workaround
As we can see in the error logs when executing the command, the create image script is looking for a script named “run.sh” but it does in the path custom_image_utils/startup_script/run.sh.
There is indeed a script named “run.sh” but not in this path, so the workaround is to make a folder called “startup_script” inside the folder “custom_image_utils” and copy the script run.sh that is located inside dataproc-custom-images/startup_script/run.sh
In order to solve this error follow next steps from the dataproc-custom-images base folder:
Create startup_script folder inside custom_image_utils:
mkdir custom_image_utils/startup_script
Copy run.sh from the startup_script folder into the newly created folder custom_image_utils_startup_script
cp startup_script/run.sh custom_image_utils/startup_script/

[1] https://cloud.google.com/dataproc/docs/guides/dataproc-images

@bhidalto thanks for the feedback, we'll fix it. We are deprecating Daisy, a workaround now is run the command without the --daisy-path flag.


Documentation of ActionTypes

The ActionTypes are currently not too well documented (there is no intellisense documentation or API documentation for it, the documentation is scattered over multiple pages); the sample on this page just uses ActionTypes.ImBack without explaining why and with no link as to where to find more explanation, even though it looks relevant for this page.
The types are briefly explained at https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-add-rich-cards?view=azure-bot-service-4.0. However there are not many details added, for example for the signin type and a discussion about whether to use an OAuthPrompt or this action?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 53446ad1-6b8c-1ccb-6e78-2cb14358b1c3
Version Independent ID: 8b0fcc34-e624-64a9-adc8-6c8fa3031f54
Content: Use button for input - Bot Service
Content Source: articles/v4sdk/bot-builder-howto-add-suggested-actions.md
Service: bot-service
Product: bot-framework
GitHub Login: @Kaiqb
Microsoft Alias: kamrani

+1.
No documentation about ActionTypes.

@Kaiqb - This issue shouldn't have been closed. That authentication document doesn't cover the card action types, and even if it did it would be off topic for that document.


[BUG] Loading checkpoint in the inference script fails when not using kernel injection

Describe the bug
I was running the DeepSpeed inference example with kernel injection set to False, and the script has trouble loading checkpoints. (If kernel injection set to True it works).
I'm using OPT model this time, and have reported the same issue here a few days for bloom but got no response. I believe this is model-agnostic script issue.
To Reproduce
deepspeed --num_gpus=2 inference/huggingface/text-generation/inference-test.py --name "facebook/opt-13b" --dtype float16 --use_meta_tensor --replace_method "auto" --ds_inference
Error:
[2023-04-18 19:18:25,836] [INFO] [launch.py:229:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2023-04-18 19:18:25,837] [INFO] [launch.py:235:main] nnodes=1, num_local_procs=2, node_rank=0
[2023-04-18 19:18:25,837] [INFO] [launch.py:246:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2023-04-18 19:18:25,837] [INFO] [launch.py:247:main] dist_world_size=2
[2023-04-18 19:18:25,837] [INFO] [launch.py:249:main] Setting CUDA_VISIBLE_DEVICES=0,1
[2023-04-18 19:18:27,949] [INFO] [utils.py:785:see_memory_usage] before init
[2023-04-18 19:18:27,949] [INFO] [utils.py:786:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-04-18 19:18:27,950] [INFO] [utils.py:793:see_memory_usage] CPU Virtual Memory:  used = 5.36 GB, percent = 2.9%
Fetching 23 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 68906.42it/s]
[2023-04-18 19:18:28,418] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.9.0, git-hash=unknown, git-branch=unknown
[2023-04-18 19:18:28,419] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter replace_method is deprecated. This parameter is no longer needed, please remove from your call to DeepSpeed-inference
[2023-04-18 19:18:28,419] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-04-18 19:18:28,419] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Traceback (most recent call last):
  File "/home/xx/DeepSpeedExamples/inference/huggingface/text-generation/inference-test.py", line 74, in <module>
    pipe.model = deepspeed.init_inference(pipe.model,
  File "/home/xx/venv/lib/python3.9/site-packages/deepspeed/__init__.py", line 324, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/xx/venv/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 155, in __init__
    self._load_checkpoint(config.checkpoint)
  File "/home/xx/venv/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 450, in _load_checkpoint
    load_path, checkpoint, quantize_config = sd_loader.load(self._config.tensor_parallel.tp_size,
AttributeError: 'dict' object has no attribute 'load'

Expected behavior
We should be able to load checkpoints with meta tensor when not using kernel injection.
ds_report output
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/xx/venv/lib/python3.9/site-packages/torch']
torch version .................... 2.0.0+cu117
deepspeed install path ........... ['/home/xx/venv/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.9.0, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7

Also running into this issue.

Hi @brevity2021,
Meta tensor checkpoint loading is only supported when kernel injection is enabled. Please provide the --use_kernel argument to the inference-test.py script when running this example and using meta tensors.
Additional assertions have also been added on the DeepSpeed InferenceEngine side to check that kernel injection is enabled when meta tensors are used https://github.com/microsoft/DeepSpeed/pull/2940.
Thanks,
Lev

@lekurile Is that still true after https://github.com/microsoft/DeepSpeed/pull/3102 was merged?


Pass logger from options to TransportFactory

I have noticed that a transport error was being logged in https://github.com/getsentry/sentry-php/blob/bfab90c7ef921221b52b75b5108201e87f0f4661/src/Transport/HttpTransport.php#L112-L115
but there always was default NullLogger registered. So I've found that even though I set logger via options it was not being passed to the Transport.
This fixes it.

ah cahngelog


Fix legacy typo

it used to be :.r, but not anymore (good)

@juliakm
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Thanks for the fix!
#sign-off


Multiple error messages

Microsoft PowerToys version
<IP_ADDRESS>
Running as admin

[X] Yes

Area(s) with issue?
PowerToys Run
Steps to reproduce
PowerToysReport_2022-05-26-17-19-32.zip
2022-05-26.txt
After waking my computer's monitors back up, I now see 7 different error messages titled "PowerToys Run ran into an issue" saying "Something went wrong." The error messages appeared as follows with the log file and report a bug zip folder attached above.

✔ Expected Behavior
No response
❌ Actual Behavior
No response
Other Software
No response

/dup #16975 , could you please add some details there about touch devices you might have that are causing this?


Plex sensor unknown since 0.101.x (remote plex server)

Home Assistant release with the issue:

0.101.0, 0.101.1, 0.101.2, 0.101.3
Last working Home Assistant release (if known):
0.100.x
Operating environment (Hass.io/Docker/Windows/etc.):

Hassio on RP3

arch
armv7l

dev
false

docker
true

hassio
true

os_name
Linux

python_version
3.7.4

timezone
Europe/London

version
0.101.3

virtualenv
false

Integration:

https://www.home-assistant.io/integrations/plex/
Description of problem:
I have a remote plex server.  Made necessary changes for 0.100.x and it worked.  Since upgrading to 0.100.0,1,2,3 my generated sensor.plex remains unknown.  Nothing in logs.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
plex:
  host: my.remote.plex.tv
  port: 32400
  token: mytokenhere
  server: plex
  ssl: true
  verify_ssl: false

Traceback (if applicable):

Additional information:
I've manually configured using configuration.yaml as when I try and configure using the frontend, I can successfully sign into plex, but when I get redirected, it always timesout, and the plex integration never appears.

Releated to https://github.com/home-assistant/home-assistant/issues/28564 I think.
I did search honestly!

An unknown sensor could have several causes, let's make sure it's the same before merging issues. Can you share any logs or tracebacks?

I thought I'd fixed it as I noticed that my token had changed, but have changed configuration.yaml to the new token, but still not working.  In 0.100.x versions, it successfully created a media_player.plex_tv_ue40h6400 player and the sensor did update.
Here's the plex sensor and tautulli sensor.

And here's plex snippets from my log
2019-11-11 10:43:43 DEBUG (MainThread) [homeassistant.bootstrap] Setting up {'config', 'media_player', 'scene', 'group', 'discovery', 'sensor', 'person', 'device_tracker', 'shopping_list', 'joaoapps_join', 'input_select', 'telegram_bot', 'hue', 'onboarding', 'webhook', 'tts', 'http', 'weather', 'timer', 'openuv', 'device_automation', 'updater', 'auth', 'hassio', 'plex', 'frontend', 'zone', 'cast', 'automation', 'notify', 'logbook', 'google', 'speedtestdotnet', 'cloud', 'sun', 'websocket_api', 'script', 'mqtt', 'mqtt_statestream', 'tplink', 'input_boolean', 'nest', 'api', 'lovelace', 'history', 'system_health', 'panel_custom', 'map'}

2019-11-11 10:43:51 INFO (MainThread) [homeassistant.setup] Setting up plex

2019-11-11 10:43:53 DEBUG (MainThread) [homeassistant.components.plex] Imported Plex configuration

2019-11-11 10:43:53 INFO (MainThread) [homeassistant.setup] Setup of domain plex took 2.3 seconds.

2019-11-11 10:43:56 DEBUG (MainThread) [homeassistant.components.plex] Connected to: plexy (https://REMOVED.eu:32400)
2019-11-11 10:43:56 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event component_loaded[L]: component=plex>

2019-11-11 10:43:57 INFO (MainThread) [homeassistant.components.media_player] Setting up media_player.plex

2019-11-11 10:44:38 DEBUG (MainThread) [plexwebsocket] Websocket connection refused

2019-11-11 10:44:43 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.plex

2019-11-11 10:44:43 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=sensor.plex, old_state=None, new_state=<state sensor.plex=unknown; unit_of_measurement=Watching, friendly_name=Plex (plexy) @ 2019-11-11T10:44:43.219134+00:00>>

2019-11-11 10:44:38 DEBUG (MainThread) [plexwebsocket] Websocket connection refused

Can you think of anything in your setup that could cause websocket connections to the Plex server to fail? A reverse proxy, firewall, etc?

Nov 11, 2019 10:43:53.927 [0x7f6036ffd700] DEBUG - (Capabilities) Platform 'Home Assistant' not matched by plugin platform requirements

Hmm, when you authenticate to Plex during the HA integration setup, are you using a managed/home user login or the main/admin account?

I think the websocket is my problem but I don't think there's anything on the server to cause issues. It's a dedicated server in a data centre and Plex isn't behind a reverse proxy and there's no firewall installed (will probably live to regret this one day). Do websockets require two way communication (I. E. Need to port forward on my home router?)
If I try and configure through the integration frontend page, I get taken to the plex login page. I successfully sign in with the admin account, then I get redirected to an invalid certificate warning. Clicking allow/continue I then get taken to something like https://mydomain.duckdns.org:8123/api/callback?=1234 which is 404

No, websockets communicate over the same port as the plain API calls and shouldn't need any special ports opened. A reverse proxy can need special setup, but that's not the case here. However something is causing the websocket connection to fail and I'm not sure what that is yet.
Is your baseurl set properly in HA? The redirect after logging into Plex sends you back to your HA instance. An invalid cert or 404 means that step is not working.

Thanks for your patience and help, and I'm pleased to say I'm sorted now and it's working perfectly. I re-read up on the baseurl docs and realised I'd messed up even though my set up had been working for ages.
For any future googlers,
I was accessing HA via https://mydomain.duckdns.org:8123 and I had baseurl: mydomain.duckdns.org in my config, with 8123 <> 8123 in the router.
So I changed baseurl to baseurl: https://mydomain.duckdns.org:8123, removed plex: from my config, and restarted. I then re-added plex via the integrations page, and the redirect back worked, and sensor.plex_myserver works perfectly and fast.

I'm not quite sure how that would affect the websocket connections, but I'm glad it's working now. I've created #28703 to actually log the error when connecting to the websocket so it can make future troubleshooting easier. Thanks for reporting and troubleshooting with me.


app-designer pushing to incorrect folder

Software and hardware versions
Android 7.1.1, Nexus 6p
Mac OS High Sierra 10.13.2
happens both with app-designer downloaded from website and cloned from github
Problem description
grunt adb-push-default-app creates a new config folder inside the config folder it should be pushing to, and pushes all data to the new folder.
Steps to reproduce the problem

delete opendatakit folder on phone
grunt adb-push-default-app
grunt adb-push-default-app again

the second push reliably creates the second folder, Although it seems that it sometimes happens on the first push.
Console snippit
[ 53%] /sdcard/opendatakit/default/config/config/assets/csv/large_dataset.15000.csv: 27%
[ 54%] /sdcard/opendatakit/default/config/config/assets/csv/large_dataset.15000.csv: 28%
[ 54%] /sdcard/opendatakit/default/config/config/assets/csv/large_dataset.15000.csv: 29%
[ 54%] /sdcard/opendatakit/default/config/config/assets/csv/large_dataset.15000.csv: 30%
[ 54%] /sdcard/opendatakit/default/config/config/assets/csv/large_dataset.15000.csv: 32%
(note the config/config)

confirmed on windows as well


Updated v0.6.4

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

license_file entry is missing, but is required.

For recipe:

noarch: python recipes are recommended to have a lower bound on the python version. This recommendation will become a requirement in the future.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

license_file entry is missing, but is required.

For recipe:

noarch: python recipes are recommended to have a lower bound on the python version. This recommendation will become a requirement in the future.

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

noarch: python recipes are recommended to have a lower bound on the python version. This recommendation will become a requirement in the future.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

noarch: python recipes are recommended to have a lower bound on the python version. This recommendation will become a requirement in the future.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

ValueError: License file given in about/license_file (/home/conda/recipe_root/LICENSE.md) does not exist in source root dir or in recipe root dir (with meta.yaml)

Looks like its' not in the sdist:

ValueError: License file given in about/license_file (/home/conda/recipe_root/LICENSE.md) does not exist in source root dir or in recipe root dir (with meta.yaml)

Looks like its' not in the sdist:


cannot download dataset

I found that the dataset like Efficient-Large-Model/sherlock_317K could not be downloaded now, and I got 404 when I enter it in huggingface's datasets.

Sorry, the README was accidentally overwritten by an old version when we release LongVILA. Please take a look at README again.


Major performance difference between Windows and Linux

There is a major performance difference using the same binaries between Windows and Mono. I have bundled my beta3-based application and run the application with the *.cmd and *.sh respectively.
On Windows, the first request for Home/Index takes approximately 1000ms, every subsequent one is in the 10-20ms range. On Linux, the first request takes about 25000ms (really, that's 25s) and every subsequent request still 3500ms. According to F12 tools the time is spent waiting for the response.
I can see this difference with both Kestrel and Nowin.vNext, so I would rule out the HTTP server component. Also, I don't see any notable difference in serving static content files which both systems handle in the two-digit-millisecond range.
The Windows system is a Core i7 Surface Pro 3.
The Linux is an embedded ARMv7 with 792 MHz.
I would expect a difference, but it's several orders of magnitude too large to result from the slower system speed.
Any advice where and how to start looking?

A simple content action (return "Hello";) show 4 ms / 28 ms on Windows / Linux.
If I remove _ViewStart.cshtml, I get 600ms / 15,000ms for the first page view and 7ms / 43ms for subsequent requests.

Without precompilation, without _ViewStart.cshtml I get 6000/15000 for the first request and fast as above for subsequent requests.
So I can say:

The precompilation done on Windows during bundle is not picked up on Linux.
An existing _ViewStart.cshtml file seems to invalidate the Razor compilation result on Linux for every request.

The numbers are the same for a blank ASP.NET 5 Preview Starter Web with added Kestrel

I've found two issues and from what I can see, both are already fixed on dev (albeit accidentally :-) ):
First, _ViewStart.cshtml files in beta3 are associated with a page's cache entry using the normalized path instead of the native one, so enumerating the possible locations fails and the cache property is null everywhere resulting in compilation for each page request. There might be a similar issue on current dev, but I haven't tried any of this.
Second, during pre-compilation in beta3, the each file's hash is not stored in the generated cache entries and doesn't land in the generated code. During the introduction of IMemoryCache, this has been fixed. The reason I could see this was because the Windows machine was running on CET and the Linux machine on UTC, so the timestamps didn't match when evaluating the cache entries.

The fixes where not accidental :)

For quite some time now I'm running a SPA. I won't be able to look into this anymore. Feel free to reopen if you think it's necessary


DCOS-10470: [8/10] Introducing utilities to operate on DSL expressions

This PR introduces the DSLUtil for processing prop types and high-level DSL AST operations and the DSLUpdateUtil that provides the utilities for dynamically updating DSL expressions.

Shall we start? Retest this please.

Slight pause until I fix some issues arised form a discussion with @orlandohohmeier yesterday.

Addressed the #nits, but there is a small bug on the delete function, fixing...

🚢 🚢 🚢