Implementation Issue for Super Mixins: DDC

What: Issue for implementation for DDC
ETA: 10/15 in Dart Repo.
For: SuperMixins Implementation and strawman documentation
Meta Issue: This is subsumed under the Super Mixins Meta Issue in the Language Repo
Comments: Feel free to break this further into further tasks and issues, or use as a meta issue for all the tasks in the implementation plan
Note: This is a P1 for Super Mixins

testing CL now: https://dart-review.googlesource.com/c/sdk/+/74965
Kernel support is not implemented yet because we can't tell which classes are these new mixins.

I landed inference tests, and I'm seeing some tests that are expected to produce a compile time error and which are passing on the analyzer, failing on ddc.  This seems unexpected so re-opening this for investigation.
mixin_declaration/mixin_declaration_inference_invalid_03_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167
mixin_declaration/mixin_declaration_inference_invalid_04_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167
mixin_declaration/mixin_declaration_inference_invalid_05_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167
mixin_declaration/mixin_declaration_inference_invalid_06_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167
mixin_declaration/mixin_declaration_inference_invalid_07_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167
mixin_declaration/mixin_declaration_inference_invalid_08_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167
mixin_declaration/mixin_declaration_inference_invalid_09_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167
mixin_declaration/mixin_declaration_inference_invalid_10_test: MissingCompileTimeError   # https://github.com/dart-lang/sdk/issues/34167

I'm going to open a new issue, since this one is about implementing the feature in DDC, which has been landed.
I don't see how inference/MissingCompileTimeError can be a DDC problem (we don't do inference, or produce compile errors, except in rare circumstances), but I'll take a look. It could be something related to Analyzer task model or summaries, both of which DDC uses, and Analyzer tests don't.

new bug is https://github.com/dart-lang/sdk/issues/34516


C++: use shortestDistances in PrimitiveBasicBlocks

The use of transitive closure for BB index calculation has been the cause of an out-of-memory error. This PR switches the calculation to use the shortestDistances HOP, which still has the problem that the result needs to fit in RAM, but at least the RAM requirements are sure to be linear in the size of the result. The shortestDistances HOP is already used for BB index calculation for the C++ IR and for C#.
We could guard even better against OOM by switching the calculation to use manual recursion, but that would undo the much-needed performance improvements we got from #123.
This change improves performance on Wireshark, which is notorious for having long basic blocks. When I benchmarked shortestDistances for #123, it was slower than TC. With the current evaluator, it looks like shortestDistances is faster. Performance before was:
PrimitiveBasicBlocks::Cached::getMemberIndex#ff ................... 9.7s (executed 8027 times)
#PrimitiveBasicBlocks::Cached::member_step#ffPlus ................. 6.6s
PrimitiveBasicBlocks::Cached::primitive_basic_block_entry_node#f .. 3.5s
PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff .... 2.3s

Performance with this commit is:
PrimitiveBasicBlocks::Cached::primitive_basic_block_entry_node#f ................................................................... 3.5s
shortestDistances@PrimitiveBasicBlocks::Cached::primitive_basic_block_entry_node#1@PrimitiveBasicBlocks::Cached::member_step#2#fff . 3s
PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ..................................................................... 963ms

What goes wrong if you implement primitive_basic_block_member using the obvious:
predicate primitive_basic_block_member(Node node, PrimitiveBasicBlock bb, int pos) {
    primitive_basic_block_entry_node(bb) and node =  bb and pos = 0
    or
    exists(Node prev |
      member_step(prev, node) and
      primitive_basic_block_member(prev, bb, pos-1)
    )
}

@markshannon That's what we had prior to #123, and it was a serious performance problem on large databases. That's also why C# uses shortestDistances.

Just to satisfy my curiosity, what was the problem? It should be perfectly linear in size.

I believe it was perfectly linear. It just had a high constant factor. I never figured out why #123 + #285 made it so much faster.

I did some more tests, and I think I can very easily be persuaded to switch to @markshannon's implementation, which appears to perform well enough with the current QL engine. See this Slack discussion.

I've added a commit to change the implementation back to plain recursion, like it was before #123. This performs better than the current implementation, and it doesn't have the cache flushing risks of shortestDistances. Benchmarks:
Chromium, recursion:
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ........................ 59.3s (executed 23636 times) (wallclock: 1m14s)
Chromium, TC:
    quickQuery.ql-2:#PrimitiveBasicBlocks::Cached::member_step#ffPlus ..................................... 1m5s
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::getMemberIndex#ff ....................................... 57.3s (executed 23636 times) (wallclock: 69s)
Chromium, shortestDistances:
    quickQuery.ql-2:shortestDistances@PrimitiveBasicBlocks::Cached::primitive_basic_block_entry_node#1@PrimitiveBasicBlocks::Cached::member_step#2#fff . 1m22s
                                                                                (+ 5s cache flush)
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ..................................................................... 10.8s

---

WebKit, recursion:
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ........................ 12.5s (executed 10003 times) (wallclock: 16s)
WebKit, TC:
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::getMemberIndex#ff ....................................... 12s (executed 10003 times) (wallclock: 16s)
    quickQuery.ql-2:#PrimitiveBasicBlocks::Cached::member_step#ffPlus ..................................... 10.3s
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ........................ 3.9s
WebKit, shortestDistances:
    quickQuery.ql-2:shortestDistances@PrimitiveBasicBlocks::Cached::primitive_basic_block_entry_node#1@PrimitiveBasicBlocks::Cached::member_step#2#fff . 6.6s
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ..................................................................... 1.6s

---

Firefox, recursion:
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ........................ 21.6s (executed 16568 times) (wallclock: 28s)
Firefox, TC:
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::getMemberIndex#ff ....................................... 20.5s (executed 16568 times) (wallclock: 26s)
    quickQuery.ql-2:#PrimitiveBasicBlocks::Cached::member_step#ffPlus ..................................... 19.1s
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ........................ 7.3s
Firefox, shortestDistances:
    quickQuery.ql-2:shortestDistances@PrimitiveBasicBlocks::Cached::primitive_basic_block_entry_node#1@PrimitiveBasicBlocks::Cached::member_step#2#fff . 12s
    quickQuery.ql-2:PrimitiveBasicBlocks::Cached::primitive_basic_block_member#fff ..................................................................... 3s


VM deployment form group template/iso

Open for https://github.com/apache/cloudstack-primate/issues/468

@blueorangutan package

@rhytd I think if the group template/iso by the featured, community, ... the old filter in the search bar will be removed, right?

@rhtyd a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Yes @utchoang there is no need of that filtering now, just a search is OK

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/470 (JID-2160)

@rhtyd cc @svenvogel I've completed this task. Please review it. Thanks.

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/470 (JID-2165)

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/470 (JID-2178)


Make Pin function Public

Simple PR to allow for access to the pin(to:) func swift helper in UIViewExtensions.
bumped podspec
pulled into the example
@vokal/ios-developers please review

@vokal-isaac any chance you can take a peak at this when you have time? You are probably the only other person with context of the repo while luby is out. Not super high priority but I did want to pull this small change in to use in AMA


Fix percent size calculation

Apply it for outer widget size instead of inner

TestHmGui.txt

And this is BFett's design I'm trying to build:


SwaggerParameterAttribute doesn't support classes

We have an endpoint with many query params. We decided to move them to a class and just refer to that for the params, kinda like this:
[HttpGet]
public IActionResult Get([FromQuery] QueryViewModel query)
{

Putting the SwaggerParameterAttribute on the whole class does nothing. The attribute is marked to only be usable on params:
[AttributeUsage(AttributeTargets.Parameter, AllowMultiple = false)]
public class SwaggerParameterAttribute : Attribute
{

That means we have no way to use that attribute anymore to add shema comments. I'm not sure if the xml route would work here. If you know of a workaround in the mean that, that would be nice.
Fixing this is probably not trivial. I imagine the crawler code for the schema would have to be reworked.  I realize that putting query params all into a class is not very common in MVC, so this may not be a priority, unfortunately, but I'd appreciate a fix ;)

+1 for this
We have a class:
public sealed class MyModel
{
  [FromQuery(Name = "firstName")]
  [Required, MinLength(2)]
  public string FirstName { get; set; }

  [FromQuery(Name = "lastName")]
  [Required, MinLength(2)]
  public string LastName { get; set; }
}

And controller's action:
[HttpGet]
public async Task<ActionResult> GetSomething(MyModel model)
{
  // here some code
}

But I cannot use [SwaggerParameter] there.
I think this issue has a major priority, because only classes (and only their properties, not fields) support built-in validation via ModelState, so it's a must-have.

I have the same problem and also prefer to put my parameters into separate classes. It should be possible to fix this issue by adding a new attribute type that can be used on properties in the class that is passed to the action method. I believe a new AnnotationParameterFilter needs to be created which uses PropertyInfo instead of ParameterInfo when fetching the attributes from the context.

I'm not sure exactly when this was resolved, but I tested this functionality on v5.6.0 and it works as desired.

I'm not sure exactly when this was resolved, but I tested this functionality on v5.6.0 and it works as desired.


Address Lustre file system limitations with phoSim workflow

The Lustre file system as implemented at SLAC has a 100 GB metadata store which is used to store, among other things, the full file path of each and every file/directory/symbolic link.  Running phoSim for Twinkles Run 3 typically requires ~12,000 production SEDs + another ~4,000 sprinkled SED files.  The file paths are necessarily long (but descriptive) which compounds the problem.  During the initial running of this workflow, all instanceCatalogs and SED files are preserved for downstream validation/debugging.  Finally, the goal of running 2500 concurrent instances of phoSim means many millions of link/files.  What was observed is that once this workflow began, the metadata usage within Lustre rose from ~38 GB to >100 GB at which point, no new files could be created.

This problem has been addressed by making more use of local /scratch space available on all batch nodes.  The amount of space needed is ~300 MB per phoSim instance and ~16,000 inodes.  SLAC batch farm currently has six generations of batch machines and all of them should be able to handle the space/inode demands, even if all available cores are dedicated to this workflow task.  One possible complication is that local /scratch space is voluntarily managed by the user who places data into that space; if that user fails to clean up after a job terminates, then the scratch space slowly fills up -- much like a memory leak in C++.
The implemented solution involves the following new steps in the workflow:

sprinkled SED files are generated in local /scratch, then a .tar.gz of the spectra_files directory is copied to Lustre to be used later.  (Instance catalog continues to be written directly to Lustre)

In preparation to running phoSim, the production SEDs are sym-linked to a location in local /scratch, wherein the Lustre copy of spectra_files.tar.gz (sprinkled SEDs) are unpacked.

The phoSim /work directory is placed in local /scratch

upon completion, the local /scratch is cleaned up (removed), leaving reference copies of the instanceCatalog, sprinkled SEDs, and production SEDs in Lustre.  The Lustre space can be cleaned up (manually) once the Twinkles group is satisfied they are no longer needed.  Again, my plan is to retain these data for the first NNNNN visits of Run3, where NNNNN is yet to be determined, but at least 1000.

Tests of this refactoring of the workflow appear successful.


Log Spam when process exits/PID changes even when not set to quit.

Same issue here when config is set to not close on app quit. Controllers seem to work fine but I seem to lose the ability to see half my screen

https://github.com/Alia5/GlosSI/commit/f5b71bf9c9a91adcbff088ed7eacb79992911e87


physic: Rename Frequency.Duration() to Frequency.Period()

It is more correct.
In v3.5.0, add the new method Period(). In v4.0.0, remove Duration().

Ported to https://github.com/periph/conn/issues/27.


[type: performance] optimize UpstreamCheckService

Make sure that:

[x] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[x] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #4386 (ff0eb2b) into master (ff7e980) will decrease coverage by 0.08%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #4386      +/-   ##
============================================
- Coverage     68.38%   68.30%   -0.08%
+ Complexity     7516     7511       -5
============================================
  Files          1020     1020
  Lines         28813    28813
  Branches       2568     2565       -3
============================================
- Hits          19704    19681      -23
- Misses         7575     7592      +17
- Partials       1534     1540       +6

Impacted Files
Coverage Δ

...henyu/admin/service/impl/UpstreamCheckService.java
66.09% <100.00%> (-0.58%)
:arrow_down:

...henyu/common/cache/MemorySafeWindowTinyLFUMap.java
73.33% <0.00%> (-8.34%)
:arrow_down:

...a/org/apache/shenyu/common/utils/VersionUtils.java
21.27% <0.00%> (-6.39%)
:arrow_down:

...rg/apache/shenyu/plugin/rewrite/RewritePlugin.java
66.66% <0.00%> (-5.56%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
58.51% <0.00%> (-5.32%)
:arrow_down:

...che/shenyu/sync/data/http/HttpSyncDataService.java
85.71% <0.00%> (-4.09%)
:arrow_down:

...in/response/strategy/NettyClientMessageWriter.java
71.42% <0.00%> (-3.58%)
:arrow_down:

...a/org/apache/shenyu/plugin/mock/util/MockUtil.java
77.55% <0.00%> (-2.05%)
:arrow_down:

...yu/admin/service/impl/PluginHandleServiceImpl.java
74.07% <0.00%> (-1.86%)
:arrow_down:

...client/brpc/BrpcContextRefreshedEventListener.java
68.57% <0.00%> (-1.43%)
:arrow_down:

... and 1 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Open Telemetry example

Sets up application with exporting metrics and traces to OpenTelemetry
Includes docker-compose that setups otel-collector, Prometheus, Jaeger and Grafana

@slashmo


Run Travis tests on an in-memory file system

/cc @bleything

@indirect is this actually something we have a prayer's chance of doing? Given RSpec.current_example is thread-safe, we could use it to allow parallel example execution, I supposer?

There are gems for in-memory file systems intended for testing purposes (ex. https://github.com/simonc/memfs). Haven't used any personally so not sure what the canonical or best solution is.
Having parallel tests that are each safely isolated would be amazing though. Though I will note that CodeClimate and similar analysis engines often don't do too well with test coverage and parallel rspecs or similar mechanisms.

@segiddins theoretically Travis provides a ramdisk on every machine specifically in case you would like to do this. I remember trying it once, seeing that not every single test passed, and then not having time to investigate further. If we can make it work, it would theoretically increase the speed of the tests by... a lot.

@RochesterinNYC we already don't get good coverage reports on our integration tests, and those are the ones that would be sped up by parallel execution and/or ramdisks… I'd be happy with either one, and ecstatic if we can pull off both.

@indirect I couldn't find any docs for Travis having a ramdisk

moves to backlog

I think we can close this as infeasible as long as we have mostly integration specs

Every Travis VM ships with a ramdisk mounted at (iirc) /var/ramfs. It will be at least 512MB, which I'm not sure about being large enough for our tests or not. Worth trying, because it seems like it would speed up our tests by a huge amount. Integration tests need fast IO.


Line breaks are not working! It doesn't print in the given pre tag format

Thank you for submitting an issue to jsPDF. Please read carefully.
Are you using the latest version of jsPDF?
Yes
Have you tried using jspdf.debug.js?
Yes
Steps to reproduce
Ideally a link too. Try fork this http://jsbin.com/rilace/edit?html,js,output
What I saw
public class Answer implements QuestionInterface { private String reverse(String
str){ int length = str.length(); String reverse=""; for(int i=length-1;i>=0;i--){ reverse
= reverse + str.charAt(i); } return reverse; }
What I expected
C CODE
public class Answer implements QuestionInterface {
private String reverse(String str){
int length = str.length();
String reverse="";
for(int i=length-1;i>=0;i--){
reverse = reverse + str.charAt(i);
}
return reverse; }

Are you using fromHTML?

Yes!

FromHTML doesn't parse pre-tags. You have to modify fromHTML sourcecode.

Can you explain me precisely where and what should i change in order to parse the pre-tags?

Precisely? Where and What?
How about you put yourself some effort and solve this issue by yourself? Dude, Have some Coder-honor!
I was checking the code and then I read your other comment in the other issue. Man, are you kidding?
Hire a professional Softwaredeveoper with analytic skills than you to fix that problem.
A hint:
You have to modify the drillForContent-Function
In this function a handler for the pre-Tag has to to be implemented. It has to be checked if the text node is in a pre tag and if so you should split the content by the newLine-Chars and then call for each line the addText-Method and incrementing the renderer.y.
If this is to complicated hire someone.

Thank you for your help! I've also found another answer for my table issue with your help! Thank you :)

Well done! Coder-Honor proven!


Fix selection applying to hidden Nodes when filtering Scene Tree Editor

Fixes https://github.com/godotengine/godot/issues/64399.
In this PR, Node TreeItems that are hidden in the Tree are ignored outright, when multi-selecting.

Thanks! :wink:


[WIP] chore(rls): move to feature flag and disable related view

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

Merging #11575 into master will decrease coverage by 4.59%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #11575      +/-   ##
==========================================
- Coverage   59.59%   55.00%   -4.60%
==========================================
  Files         832      412     -420
  Lines       40482    14288   -26194
  Branches     3666     3666
==========================================
- Hits        24126     7859   -16267
+ Misses      16187     6260    -9927
  Partials      169      169

Flag
Coverage Δ

cypress
55.00% <ø> (-0.01%)
:arrow_down:

python
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...set-frontend/src/dashboard/util/getDropPosition.js
77.77% <0.00%> (-1.59%)
:arrow_down:

superset/queries/saved_queries/commands/export.py

superset/annotation_layers/annotations/schemas.py

superset/databases/commands/update.py

superset/migrations/versions/7fcdcde0761c_.py

superset/examples/misc_dashboard.py

...t/migrations/versions/e866bd2d4976_smaller_grid.py

superset/utils/pandas_postprocessing.py

superset/charts/commands/bulk_delete.py

...tions/versions/4fa88fe24e94_owners_many_to_many.py

... and 405 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52145f8...d908993. Read the comment docs.


ARROW-9698: [C++] Remove -DNDEBUG flag leak in .pc file

This change eliminates the need to include the -DNDEBUG flag in arrow's .pc file, which leaks into downstream projects' builds unexpectedly. The CheckSize function definition is now included in both debug and release builds, though it will no-op in release builds (as expected). The flag is no longer required as a means to make the ABI consistent across build types for downstream projects, and so it is now removed.

I disagree with these changes. There no is reason to rename NDEBUG to another symbol. We should simply ensure when the ABI is the same with and without NDEBUG.

It looks the only place that would need fixing is in arrow/util/trie.h.

(also, you forgot to change the .pc file itself...)

(also, you forgot to change the .pc file itself...)

Whoops, must have gotten lost when I revised my commit. Had it in the original.

Actually, given the approach here, it wouldn't need to change -- my change would sets a different flag rather than NDEBUG.
In any case, this change was one approach suggested in the jira issue, but I am happy to change it.
From the code you pointed out, the NDEBUG check could simply be removed (not sure why it is necessary, if NDEBUG would also disable the assert anyway?). Or, it could be moved inside of the function definition and break/do nothing.
Thoughts or preferences?

I don't know why @kou suggested the flag change. If I look at headers in /usr/include on Ubuntu 18.04, several of them use NDEBUG in places: it doesn't seem to be causing any issues in practice.
As for arrow/util/trie.h, the method should simply always be defined even if not used (though I'm not even sure that matters, since it's an inline method...).

@pitrou Why are the asserts wrapped in NDEBUG? Isn't that redundant?

@bdunlay Where exactly? Do you have an example?

For example the lines you pointed out to me: src/arrow/util/trie.h
If this is a debug build, the function is included in the header, and the assert is evaluated. If it's a release build, the function is omitted, and the assert is obviously never evaluated. But if you removed the macro completely, in a release build, the function exist but would no-op (assert disabled).

Ah, yes, you are right, in those cases the #ifdef is pointless.

Think the linter is mad at me for an extra newline in the cmake file. Will remove that w/ an update once CI completes.

I will push a new change with that newline removed, but the linter is going to complain anyway because there are lots of changes that are not compliant with the linter in that cmake file that existed prior to my change.

If you installed archery, you can run archery lint --cmake-format. Otherwise we can do it for you.

I'll submit this for review as is if that's okay with you.

Hey look at that, it's happy now 😄

I suggested the renaming approach because using release build libarrow.so without NDEBUG increases needless (empty) CheckXXX() function calls.
If it's acceptable, just removing NDEBUG is OK to me.

@github-actions crossbow submit -g linux


Refactor/logging middleware

Logging has been reworked to be driven entirely by middleware in the amqp package. This resulted in a number of API changes to the general middleware.

Closes #19


Autocomplete: doesn't allow for '0' values

Dunno if its a bug, but autocomplete currenty verifies if the source prop has length: https://github.com/react-toolbox/react-toolbox/blob/master/components/autocomplete/index.jsx#L115
if (source.hasOwnProperty(length)) {
        return new Map(source.map(function (item) {

Note that length is not a string, and currently has value '0' (maybe it's an artifact from babel? I couldn't find any var declaration.
Maybe it should check for 'length' instead?
Anyway I did a PR: https://github.com/react-toolbox/react-toolbox/pull/163

Fixed by #163 (#12e9e4bf4f11a9ca9b6290063f1fc5ab182a88b7)


Refresh Cache

Description
Implements a cache that supports refresh
Contribution checklist:

[X] The pull request does not introduce [breaking changes]
[X] CHANGELOG is updated for new features, bug fixes or other significant changes.
[X] I have read the contribution guidelines.
[X] Pull request includes test coverage for the included changes.

General Guidelines and Best Practices

[X] Title of the pull request is clear and informative.
[X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR.

This looks acceptable to me, but I want to ensure @jimmyca15 and @zhenlan are okay with it as well. There's no cache expiring and it's entirely dependent on the ref changing. If someone is using a local file, they can't simply update the dict unless they're also updating the ref of "feature_flags".


this.props.data without query root field

I would like to ask if do something wrong. I have the following React Component:
import React from 'react';
import { graphql } from 'react-apollo';
import gql from 'graphql-tag';
import { connect } from 'react-redux';

import s from './menu.scss';

class Menu extends React.Component {

  render() {
    console.log(this.props);

    const { wp_query } = this.props.data;

    return (...);
  }
}

const getMenu = gql`
  query getMenu {
    wp_query {
      menu (name: "pl") {
        id
        title
      }
    }
  }
`;

const MenuWithData = graphql(getMenu)(Menu);

export default MenuWithData;

console.log returns:
{ data:
   { loading: true,
     errors: null,
     variables: undefined,
     refetch: [Function],
     fetchMore: [Function],
     updateQuery: [Function] } }

Unfortunately, there is no query root field: wp_query.
What might be wrong?
In advance, thank you for resolving this issue. I used the official documentation.

It is currently loading, since loading: true is set. Have you tried waiting for loading to complete?

Thank you for the fast reaction. I got the point.
I use menu.map in return (...), so the app was just crashing and the component never rendered correctly.
Now, I do something like this:
    const { loading } = this.props.data;

    if (loading) {
      return (
loading...
);
    }

    const { wp_query } = this.props.data;
    const { menu } = wp_query;

and it works. However, I have a question: do I need to always check loading flag or is there any option to render only when loading equals false?

@tommyJS there currently isn't although I've thought about that as an option

Thank you for the answers. Feel free to reopen.

@ghost Seems any failure inside Component will cause graphql to stop loading, and 'loading' kept in true. Do your agree?


Is it possible to consider adding log callBack on Android?

Please make sure that this is a feature request.
System information (Please provide as much relevant information as possible)

MediaPipe Solution (you are using): framework
Programming language : C++ / Android Java
Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state:
We have used mediapipe in the production environment, it is really great, but there will be some crashes in the production environment, because it is a production environment, so there is no symbol table of the native layer, because the crash is in C++, there are no some log, it is difficult to determine the call stack information. and context
Will this change the current api? How?
Will change because the log callback interface needs to be added to the JNI layer. But it only changes the JNI layer.
Who will benefit with this feature?
All developers using mediapipe on the Android platform
Please specify the use cases for this feature:
    AndroidLog.nativeInjectLoggable(new JNILogging((message, severity, tag) -> {
      if (severity == Loggable.Severity.LS_DEBUG) {
          // do something eg: Use other log libraries in the current application save debug level log
      } else if (severity == Loggable.Severity.LS_INFO) {
          // do something eg:  Use other log libraries in the current application save info level log
      } else if (severity == Loggable.Severity.LS_WARNING) {
         // do something eg:  Use other log libraries in the current application save warning level log
      } else if (severity == Loggable.Severity.LS_ERROR) {
         // do something eg:  Use other log libraries in the current application save error level log
      }
    }));

Any Other info:

Hi @lilinxiong, Thanks for raising this feature request. We will share this request with the internal team and will update if any specific update on the above request. Thank you!

@kuaashish Thanks for your reply, In fact, the day before yesterday, I had simply implemented it myself, but the writing was really ugly. It was just enough to satisfy my own use. Tomorrow I plan to tidy it up and rewrite it beautifully. But I'm still looking forward to your conclusions.

Hello @lilinxiong,
We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.
You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you


MacOS: Results Grid View - Smooth (Pixel-by-Pixel) Scrolling?

Hello! Wonderful project, thank you in advance for all of the great work that's been put in to make DBeaver so awesome!
As a MacBook Pro user, scrolling through the results grid with the trackpad accurately is EXTREMELY difficult to do. Frequently do I find myself 20 columns to the right/left of where my scroll position originally began (when only wanting to scroll vertically), and my trackpad sensitivities are not set above default levels for the MacOS operating system. Microscopic gestures in the wrong direction of your desired scroll will send you wildly off track in large data sets.
If the option to disable scroll stops on column and row endings existed, it would be much easier to control my scrolling.
Describe the solution you'd like
I think it would be a great idea to have the option to enable pixel-by-pixel (precise) scrolling in the results grid view to both the vertical (row) and horizontal (column) axis. This feature should have the ability to be toggled on/off, since use cases exist for scroll stops being locked to row & column endings.
Describe alternatives you've considered
There are not any alternatives to this issue, as the ability to control scroll behavior exists only within DBeaver's backend.
Additional context
Trackpads that support advanced gestures are extremely handy, and as time goes on they are becoming more and more precise with the input they provide to the operating system. Utilizing these features would go a long way to the user experience within DBeaver.
If i'm alone in feeling this way, or a solution already exists that I missed in my original research before posting this, please let me know and I'll do my best with your advice :smile:
Thanks again!

This option did not perform as expected:

Related to bug report #6536 that confirmed newer versions of DBeaver on MacOS do not take advantage of the "Database Editors" > "Data Editor" > "Appearance" > "Grid" > "Smooth Scrolling" property.

Thank you for the idea. Smooth scrolling works when you move table view by scroll bar now.

Шmproved smooth scrolling will be implemented in future releases

I'm using v7.3.0 and this is a brutally frustrating bug. The best way to describe it is "janky". It makes it incredibly difficult to maintain perspective as to what you're actually looking at.

@jamesdh please see answers in #7777 for hints how to make UI more responsive on MacOS.
In fact performance very depends on your MacOS and on UI settings.

@jamesdh please see answers in #7777 for hints how to make UI more responsive on MacOS.
In fact performance very depends on your MacOS and on UI settings.

Thank you for the idea. Smooth scrolling works when you move table view by scroll bar now.

I know it was SO long ago, but I did confirm that this worked in that version, as well as the version currently released for MacOS. Smooth scrolling works when using the scrollbar manually, but is still not applied when scrolling is initiated with a trackpad gesture.
Looking forward to the feature in the new milestone.

Thank you for the idea. Smooth scrolling works when you move table view by scroll bar now.

I know it was SO long ago, but I did confirm that this worked in that version, as well as the version currently released for MacOS. Smooth scrolling works when using the scrollbar manually, but is still not applied when scrolling is initiated with a trackpad gesture.
Looking forward to the feature in the new milestone.

@jamesdh please see answers in #7777 for hints how to make UI more responsive on MacOS.
In fact performance very depends on your MacOS and on UI settings.

@tati-kru his complaints are not due to a slow UI; they are due to how DBeaver handles scrolling from trackpad gestures within the results grid on MacOS. Scrolling in one direction alone without wildly losing your place is near impossible - as this issue describes.

@jamesdh please see answers in #7777 for hints how to make UI more responsive on MacOS.
In fact performance very depends on your MacOS and on UI settings.

@tati-kru his complaints are not due to a slow UI; they are due to how DBeaver handles scrolling from trackpad gestures within the results grid on MacOS. Scrolling in one direction alone without wildly losing your place is near impossible - as this issue describes.

@ooobii thank you for details, it's really helpful. We'll try to improve it after 8.0.

@ooobii thank you for details, it's really helpful. We'll try to improve it after 8.0.

+1 For this issue. It is very difficult to use the data view without smooth scrolling with the trackpad. Thank you!

This is a major usability issue, as viewing result data is one of the main use cases.

I am pinging back here again that this is indeed a major usability issue. Smooth scrolling is often not possible AT ALL on macOS because system scrollbars appear on scroll and then disappear and are very small. Even if you get the mouse over them fast enough they are very small and disappear very quickly so it is hard to grab them at all. You have to scroll with the trackpad to make them appear in order to grab one of them with the mouse so that you can smooth scroll and have accurate eyeball tracking in the results.

system scrollbars appear on scroll and then disappear

I recall Mac has a setting which enables the scrollbars to be shown at all times. Might help a little!

system scrollbars appear on scroll and then disappear

I recall Mac has a setting which enables the scrollbars to be shown at all times. Might help a little!

That's correct, it could. Though it would disrupt desired behavior for all the other apps on the machine as well.

Hey all, just would like to let you know that we haven't forgotten about this issue. But it's quite complicated to fix.
Couple recommendations to improve grid performance on Mac:

Set monospaced font for data grid: Window -> Preferences -> User Interface -> Appearance -> Colors and Fonts -> Database Query results -> Resultset Data types -> Results grid font.
Remove right-justify: Window -> Preferences -> Editors -> Data Editor -> Appearance -> Uncheck Right-justify numbers and Right-justify date/time values.

I'd like to add: This is an issue on windows as well. Exact same conditions. Scroll via trackpad only jumps by column. The scrollbar is smooth as expected.

system scrollbars appear on scroll and then disappear

I recall Mac has a setting which enables the scrollbars to be shown at all times. Might help a little!

On Mac, DBeaver always shows scroll bars since the Eclipse UI always shows them.
@tati-kru : I would like to note that using the scrollbars mimics the exact behavior that is desired with the trackpad horizontal scrolling. I know this probably doesn't present a programatic solution to the problem, but it will certainly help visualize the desired behavior for whoever ends up working on this 😄

This was a part of the milestone 21.0.1, but it was never achieved / issue never closed.
Not trying to sound impatient... Any updates on this @tati-kru?

We are waiting for some improvements from the Eclipse platform. As far as I know, there are some movements of this ticket on the Eclipse board. As soon as we get the update we will add this to DBeaver.

Just want to add that I am also waiting for this fix as well.

+1, this should be embarrassing

Coming back to confirm that this issue has been fixed, and it's a genuine pleasure to use DBeaver to scroll through large data sets!!
Thank you to everyone involved - from the community providing work arounds to developers (both the Eclipse platform & from DBeaver contributors) who implemented the fix! 👍🏼 😄


[9.102.x-prod] Update postgres to 15.9 version

Update postgres to version 15 for validation

This one will be on hold and it can be merged if we decide to go to a CR2.
@gmunozfe btw, was this PR opened upstream too?

@rgdoliveira if there's CR2, this could be included

We will need a CR2 because of SRVLOGIC-472, said that I'm merging this PR so it can be included.


Cron autoupdate cancellation

I checked the documentation and there's no way to cancel a update that's been initiated? I have that command set to check hourly due to all the updates of late and a player wanted to delay while they were doing a dino train as a result of the biome changes coming. Sadly, I don't think that's possible, according to what I've read. Maybe I missed it? It is late at night and I'm tired so it's entirely possible.

The cancelshutdown should cancel an update that is in the warning stage.


Added underline to navbar items on hover

Hey @purviporwal0 , this underline effect was merged recently from one of the PR.
Your previous commit to change the color of OS-CODE at footer had some conflicts, pls resolve that :)


TimeoutError

allianceauth-discordbot==3.8.1 is throwing the following error with both redis==4.6.0 and redis==5.0.8.
==> myauth/log/discordbot.log <==
ERROR:root:
Traceback (most recent call last):
  File "/home/ppfeufer/Development/Python/AllianceAuth/venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 997, in drain_events
    get(self._deliver, timeout=timeout)
  File "/home/ppfeufer/Development/Python/AllianceAuth/venv/lib/python3.12/site-packages/kombu/transport/redis.py", line 597, in get
    raise Empty()
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/sda1/Development/Python/AllianceAuth/allianceauth-discordbot/aadiscordbot/bot.py", line 317, in poll_queue
    self.message_connection.drain_events(timeout=0.01)
  File "/home/ppfeufer/Development/Python/AllianceAuth/venv/lib/python3.12/site-packages/kombu/connection.py", line 341, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ppfeufer/Development/Python/AllianceAuth/venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 1000, in drain_events
    raise socket.timeout()
TimeoutError

Happens with Python 3.11 and 3.12.

Ah. This is not actually an error. It's invalid log spam. I'll push a fix this morning.

fixed in 3.8.2


added function to speed up ray casting

Added a function to find the range of indices in the scan array that needs to be modified for opponent casting instead of looping over the full array.

I was getting some slight differences with the old code. I think there's an off by one error on this line in ray_cast:
for i in range(min_ind, max_ind):
should be
for i in range(min_ind, max_ind + 1):

Leaving this as-is now by fixing the off by one. Will add a unit test later to make sure new and old methods get the same scan.


Missing required module 'CDBus' in XCode

Thanks to this fantastic project.
I tried building DBus.framework and add it as a dependency into my own Swift project.
When I import DBus, it gives me an error like this

Anything helpful ? Thanks

I copied CDBus folder into root dir of project, and added it into Swift Compiler - Search Path


Images unable to build with docker "BuildKit"

Describe the bug
A comedy of errors:

Some images (so far we know harborregistry and harborregistryctl) cannot be built with the BuildKit docker build engine.
BuildKit is enabled by default now in Docker for Mac <IP_ADDRESS>.
Installing Docker for Mac fresh or resetting to factory defaults cause make build to fail.

To Reproduce
Steps to reproduce the behavior:

Enable BuildKit
Try to build lagoon images

Expected behavior
Users can build lagoon images no matter what build engine is enabled
Screenshots
n/a
Additional context
The error with harborregistry:
docker build --no-cache --build-arg LAGOON_VERSION=v1.10.0 --build-arg IMAGE_REPO=lagoon --build-arg UPSTREAM_REPO=uselagoon --build-arg UPSTREAM_TAG=latest --build-arg ALPINE_VERSION=3.11 -t lagoon/harborregistry -f services/harborregistry/Dockerfile services/harborregistry
[+] Building 0.7s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 37B                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => [internal] load metadata for docker.io/goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.10.0                                                        0.4s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 70B                                                                                0.0s
 => CACHED [1/4] FROM docker.io/goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.10.0@sha256:873c73954aaae94267cecdeee83676c04a03dbb3230473ba478ec94fa  0.0s
 => [2/4] COPY entrypoint.sh /home/harbor/                                                                                0.0s
 => [3/4] COPY --chown=harbor:root install_cert.sh /home/harbor/                                                                                0.0s
 => ERROR [4/4] RUN chmod -R g+w /etc/pki/tls/certs && chgrp -R root /etc/pki/tls/certs && chmod -R g+w /var/lib/registry && chmod -R g+w /etc/registry   0.2s
------
 > [4/4] RUN chmod -R g+w /etc/pki/tls/certs && chgrp -R root /etc/pki/tls/certs && chmod -R g+w /var/lib/registry && chmod -R g+w /etc/registry && chgrp -R root /home/harbor && chmod -R g+w /home/harbor && if [ -f /storage ]; then chmod -R g+w /storage; fi:
#8 0.171 chmod: /var/lib/registry: No such file or directory
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c chmod -R g+w /etc/pki/tls/certs && chgrp -R root /etc/pki/tls/certs && chmod -R g+w /var/lib/registry && chmod -R g+w /etc/registry && chgrp -R root /home/harbor && chmod -R g+w /home/harbor && if [ -f /storage ]; then chmod -R g+w /storage; fi]: runc did not terminate sucessfully
make: *** [build/harborregistry] Error 1

Not sure why the build fails, but the docs to state there is a known limitation that it's Only supported for building Linux containers.
Easiest fix now is to just update our documentation to warn people to disable BuildKit, but on the off chance that a user depends on that, it's not ideal that we require them to disable it just to run lagoon.
Should we open/search for upstream issues so that the broken images can work with buildkit?
We might be able to disable BuildKit by setting the DOCKER_BUILDKIT environment variable in our Makefile: https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds.

fixed in uselagoon/build-deploy-tool#143


Unable to run image built from this repository

Hey, I was looking into building my own image of this to start modifying it to my own needs, but upon building using the resources provided in this repository, I am unable to run the docker image.
This is the command I am building with:
docker build samba -t [my registry ip]/samba:latest --no-cache
This is the error I get upon running the image I built:
2024-11-12 23:10:59 env: can't execute 'bash
': No such file or directory

This is happening with no modifications made to any file, directly after cloning this repository.

Yeah, it's puzzling because pulling the image directly from Docker Hub works fine, but building it doesn't.
Here is what my build logs output, maybe this could help identify the problem if you don't mind helping me look through it...
[internal] load build definition from Dockerfile
	transferring 671/0 0.01
[internal] load metadata for docker.io/library/alpine:edge
[internal] load .dockerignore
[internal] load build context
	transferring 58/0 0.002
[1/4] FROM docker.io/library/alpine:edge@sha256:8431297eedca8df8f1e6144803c6d7e057ecff2408aa6861213cb9e507acadf8
	resolve docker.io/library/alpine:edge@sha256:8431297eedca8df8f1e6144803c6d7e057ecff2408aa6861213cb9e507acadf8 0/0 0.014
[2/4] RUN set -eu &&     apk --no-cache add     tini     bash     samba     tzdata     shadow &&     addgroup -S smb &&     rm -f /etc/samba/smb.conf &&     rm -rf /tmp/* /var/cache/apk/*
	fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
	fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
	(1/62) Installing ncurses-terminfo-base (6.5_p20241006-r3)
	(2/62) Installing libncursesw (6.5_p20241006-r3)
	(3/62) Installing readline (8.2.13-r0)
	(4/62) Installing bash (5.2.37-r0)
	Executing bash-5.2.37-r0.post-install
	(5/62) Installing gmp (6.3.0-r2)
	(6/62) Installing nettle (3.10-r1)
	(7/62) Installing libunistring (1.2-r0)
	(8/62) Installing libidn2 (2.3.7-r0)
	(9/62) Installing libffi (3.4.6-r0)
	(10/62) Installing libtasn1 (4.19.0-r2)
	(11/62) Installing p11-kit (0.25.5-r2)
	(12/62) Installing gnutls (3.8.8-r0)
	(13/62) Installing icu-data-en (74.2-r0)
	Executing icu-data-en-74.2-r0.post-install
	*
	* If you need ICU with non-English locales and legacy charset support, install
	* package icu-data-full.
	*
	(14/62) Installing libgcc (14.2.0-r4)
	(15/62) Installing libstdc++ (14.2.0-r4)
	(16/62) Installing icu-libs (74.2-r0)
	(17/62) Installing talloc (2.4.2-r1)
	(18/62) Installing tevent (0.16.1-r0)
	(19/62) Installing samba-util-libs (4.20.5-r0)
	(20/62) Installing libwbclient (4.20.5-r0)
	(21/62) Installing libcap2 (2.70-r0)
	(22/62) Installing jansson (2.14-r4)
	(23/62) Installing gdbm (1.24-r0)
	(24/62) Installing libsasl (2.1.28-r8)
	(25/62) Installing libldap (2.6.8-r0)
	(26/62) Installing lmdb (0.9.33-r0)
	(27/62) Installing tdb-libs (1.4.10-r0)
	(28/62) Installing ldb (2.9.1-r0)
	(29/62) Installing popt (1.19-r4)
	(30/62) Installing samba-libs (4.20.5-r0)
	(31/62) Installing samba-common (4.20.5-r0)
	(32/62) Installing acl-libs (2.3.2-r1)
	(33/62) Installing acl (2.3.2-r1)
	(34/62) Installing skalibs-libs (<IP_ADDRESS>-r0)
	(35/62) Installing utmps-libs (<IP_ADDRESS>-r2)
	(36/62) Installing linux-pam (1.6.1-r1)
	(37/62) Installing libauth-samba (4.20.5-r0)
	(38/62) Installing dbus-libs (1.14.10-r4)
	(39/62) Installing libintl (0.22.5-r0)
	(40/62) Installing avahi-libs (0.8-r19)
	(41/62) Installing samba-common-server-libs (4.20.5-r0)
	(42/62) Installing samba-client-libs (4.20.5-r0)
	(43/62) Installing cups-libs (2.4.11-r0)
	(44/62) Installing liburing (2.8-r0)
	(45/62) Installing samba-server (4.20.5-r0)
	(46/62) Installing libbz2 (1.0.8-r6)
	(47/62) Installing libexpat (2.6.4-r0)
	(48/62) Installing lz4-libs (1.10.0-r0)
	(49/62) Installing xz-libs (5.6.3-r0)
	(50/62) Installing zstd-libs (1.5.6-r1)
	(51/62) Installing libarchive (3.7.7-r0)
	(52/62) Installing libformw (6.5_p20241006-r3)
	(53/62) Installing libpanelw (6.5_p20241006-r3)
	(54/62) Installing libsmbclient (4.20.5-r0)
	(55/62) Installing samba-client (4.20.5-r0)
	(56/62) Installing samba-common-tools (4.20.5-r0)
	(57/62) Installing samba (4.20.5-r0)
	(58/62) Installing libmd (1.1.0-r0)
	(59/62) Installing libbsd (0.12.2-r0)
	(60/62) Installing shadow (4.16.0-r1)
	(61/62) Installing tini (0.19.0-r3)
	(62/62) Installing tzdata (2024b-r1)
	Executing busybox-1.36.1-r31.trigger
	OK: 67 MiB in 76 packages
[3/4] COPY --chmod=755 samba.sh /usr/bin/samba.sh
[4/4] COPY --chmod=664 smb.conf /etc/samba/smb.default
exporting to image
	exporting layers 0/0 2.137
	exporting manifest sha256:2f39cc33d61d6116aa9edf709190a647db7d0c2184eb2dbca07b07198def2675 0/0 0.008
	exporting config sha256:71c7b78cda3d872af5c54c194d06889ee38dcaa60e4a757f74322de7db980ef5 0/0 0.007
	exporting attestation manifest sha256:ca509c161e004f093891b7bb4d8ac3c77279dac6e112832c52814d5de4166a79 0/0 0.015
	exporting manifest list sha256:eecbafab219a83df262e3c26b5ea329352f27ac09e6ae86153a5c3865edca8c9 0/0 0.01
	naming to [registry]/samba:latest 0/0 0.001
	unpacking to [registry]/samba:latest 0/0 0.381

Nevermind, fixed it by building with buildkit. Sorry about that.

Nevermind, fixed by building with buildkit. Sorry about that.


got error

When I press a button
RELIGHT
I get an "ERROR".  error in the image window
While in the terminal the steps are still running and have not been completed

i fix it
installed gradio-3.48.0 gradio-client-0.6.1


Please update ReadMe with where to get the service account json

https://console.cloud.google.com/iam-admin/serviceaccounts

@davvit  Maybe submit it as a pull request:)


Updated subprocess docs with clarification for command name

What does this PR do?
Clarifies subprocess documentation about command name in the get_subprocess_output command
Motivation
Documenting potentially surprising behavior.
Additional Notes
N/A
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached
[ ] If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

Please add changelog/no-changelog and qa/skip-qa labels


Error finding AWS credentials

Trying to run terragrunt apply on a code i have.
AWS CLI is configured and profile is defined.
To Reproduce
Steps to reproduce the behavior, code snippets and examples which can be used to reproduce the issue.
  terraform_code:
    needs: pre-requisites
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-central-1
      - name: Configure AWS CLI
        run: |
          aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws configure set default.region eu-central-1
          aws configure set default.output json
      - name: Install Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.5.7
          terraform_wrapper: true
      - name: Setup Terraform version
        run: terraform --version
      - name: Setup Terraform wrapper path
        run: |
          which terraform
          pwd
      - name: Setting AWS env. vars.
        run: |
          printenv
      - name: Deploy
        uses: gruntwork-io/terragrunt-action@v2
        with:
          tf_version: 1.5.7
          tg_version: 0.54.21
          tg_dir: /github/workspace/Terraform/
#          tg_dir: $GITHUB_WORKSPACE/Terraform/
          tg_command: 'apply'

Expected behavior
A clear and concise description of what you expected to happen.
Nice to have

[ ] Terminal output
[ ] Screenshots

Versions

Terragrunt Action version: v2
Environment details (Terragrunt version, Terraform version, etc.):
terragrunt version: 0.54.21
terraform version: 1.5.7

Additional context
the error as provided in the action:
Run gruntwork-io/terragrunt-action@v2
/usr/bin/docker run --name ee65ec3775[19](https://github.com/Giladt2789/p81/actions/runs/7681275632/job/20934296506#step:10:20)63575d48d293f14f0922bd57c7_266c52 --label ee65ec --workdir /github/workspace --rm -e "REGION" -e "STATE_FILES_BUCKET" -e "PROFILE_NAME" -e "STATEFILE_LOCK_TABLE" -e "ACCOUNT_NUMBER" -e "IAM_USER_NAME" -e "TASK_BUCKET_NAME" -e "ORIGIN_ID" -e "TERRAFORM_CLI_PATH" -e "INPUT_TF_VERSION" -e "INPUT_TG_VERSION" -e "INPUT_TG_DIR" -e "INPUT_TG_COMMAND" -e "INPUT_TG_COMMENT" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/p81/p81":"/github/workspace" ee65ec:37751963575d48d293f14f0922bd57c7
[20](https://github.com/Giladt2789/p81/actions/runs/7681275632/job/20934296506#step:10:21)24-01-27 21:30:54 Starting Terragrunt Action
Installing Terraform v1.5.7
Downloading release tarball from https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_linux_amd64.zip

#####                                                                      7.6%
##############                                                            20.0%
############################                                              39.9%
#############################################                             63.6%
############################################################              83.6%
######################################################################## 100.0%
Downloading SHA hash file from https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_SHA256SUMS
Not instructed to use Local PGP (/home/runner/.tfenv/use-{gpgv,gnupg}) & No keybase install found, skipping OpenPGP signature verification
Archive:  /tmp/tfenv_download.4hc54V/terraform_1.5.7_linux_amd64.zip
  inflating: /home/runner/.tfenv/versions/1.5.7/terraform
Installation of terraform v1.5.7 successful. To make this your default version, run 'tfenv use 1.5.7'
Switching default version to v1.5.7
Default version (when not overridden by .terraform-version or TFENV_TERRAFORM_VERSION) is now: 1.5.7
Terragrunt version environment variable: 0.54.[21](https://github.com/Giladt2789/p81/actions/runs/7681275632/job/20934296506#step:10:22)
20[24](https://github.com/Giladt2789/p81/actions/runs/7681275632/job/20934296506#step:10:25)/01/[27](https://github.com/Giladt2789/p81/actions/runs/7681275632/job/20934296506#step:10:28) 21:30:55 Creating directory for terragrunt: /home/runner/.terragrunt.versions/
2024/01/27 21:30:55 Creating directory for terragrunt: /home/runner/bin
Unable to write to: /usr/local/bin/terragrunt
Creating bin directory at: /home/runner/bin
RUN `export PATH=$PATH:/home/runner/bin` to append bin to $PATH
Downloading https://github.com/gruntwork-io/terragrunt/releases/download/v0.54.21/terragrunt_linux_amd64 to terragrunt_linux_amd64
Downloading ...
688[29](https://github.com/Giladt2789/p81/actions/runs/7681275632/job/20934296506#step:10:30)375 bytes downloaded.
Switched terragrunt to version "0.54.21"
time=2024-01-27T21:[30](https://github.com/Giladt2789/p81/actions/runs/7681275632/job/20934296506#step:10:31):56Z level=error msg=Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
time=2024-01-27T21:30:56Z level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1
time=2024-01-27T21:30:56Z level=error msg=Suggested fixes:
Missing AWS credentials. Provide credentials to proceed.

Hi,
try to pass credentials through env:
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: 'us-west-2'

I guess, its because this action running in a container and don't have access to ~/.aws/credentials
According to output of step:
Run gruntwork-io/terragrunt-action@v2
/usr/bin/docker run --name asdasdas123--label abbf75 --workdir /github/workspace --rm -e "tf_version" -e "tg_version" -e "AWS_DEFAULT_REGION" -e "AWS_REGION" -e "AWS_ACCESS_KEY_ID" -e "AWS_SECRET_ACCESS_KEY" -e "AWS_SESSION_TOKEN" -e "GITHUB_TOKEN" -e "INPUT_TF_VERSION" -e "INPUT_TG_VERSION" -e "INPUT_TG_COMMAND" -e "INPUT_TG_DIR" -e "INPUT_TG_COMMENT" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_ID_TOKEN_REQUEST_URL" -e "ACTIONS_ID_TOKEN_REQUEST_TOKEN" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/REPONAME":"/github/workspace" abbf75:3c680dc150b440e3ba5b1e5dda8883b2

I fixed the issue by providing path to the AWS credentials file
      - name: Setup SSH
        shell: bash
        run: |
          # This will copy our creds to the workspace, which, GitHub will mount to the action's container
          cp -r ~/.aws .

      - name: Terragrunt Apply
        uses: gruntwork-io/terragrunt-action@v2
        env:
          # /github/workspace is path in container, aka /home/runner/workspace.. in GHA Runner
          AWS_SHARED_CREDENTIALS_FILE: /github/workspace/.aws/credentials
        with:
          tf_version: ${{ env.tf_version }}
          tg_version: ${{ env.tg_version }}
          tg_command: run-all apply
          tg_dir: ${{ matrix.folder }}

To Reproduce
Steps to reproduce the behavior, code snippets and examples which can be used to reproduce the issue

Hey just a tip, instead of using fixed access and secret keys you can use dynamically generated keys that only last for the run of the action, this reduces the risk of leaking keys and manage of them. This is achieved by using OIDC, see https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services

I fixed the issue by providing path to the AWS credentials file
      - name: Setup SSH
        shell: bash
        run: |
          # This will copy our creds to the workspace, which, GitHub will mount to the action's container
          cp -r ~/.aws .

      - name: Terragrunt Apply
        uses: gruntwork-io/terragrunt-action@v2
        env:
          # /github/workspace is path in container, aka /home/runner/workspace.. in GHA Runner
          AWS_SHARED_CREDENTIALS_FILE: /github/workspace/.aws/credentials
        with:
          tf_version: ${{ env.tf_version }}
          tg_version: ${{ env.tg_version }}
          tg_command: run-all apply
          tg_dir: ${{ matrix.folder }}

Hey just a tip, instead of using fixed access and secret keys you can use dynamically generated keys that only last for the run of the action, this reduces the risk of leaking keys and manage of them. This is achieved by using OIDC, see https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services

Tried passing env variable to container for using OIDC which didn't worked.
  - name: Plan
        uses: gruntwork-io/terragrunt-action@v2
        env:
          AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
          AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
          AWS_SESSION_TOKEN: $AWS_SESSION_TOKEN
        with:
          tf_version: ${{ env.tf_version }}
          tg_version: ${{ env.tg_version }}
          tg_dir: ${{ env.working_dir }}
          tg_command: "plan"

So, did a workaround as of @yevhen-kalyna

      - name: Setup AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          audience: sts.amazonaws.com
          aws-region: ${{ env.AWS_REGION }}
          role-session-name: GH-Actions-${{ github.run_id }}-${{ github.run_attempt }}
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}

      - name: Setup
        shell: bash
        run: |
          mkdir -p .aws
          cat >> .aws/credentials << EOF
          [default]
          aws_access_key_id = $AWS_ACCESS_KEY_ID
          aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
          aws_session_token = $AWS_SESSION_TOKEN
          EOF

      - name: Plan
        uses: gruntwork-io/terragrunt-action@v2
        env:
          AWS_SHARED_CREDENTIALS_FILE: /github/workspace/.aws/credentials
        with:
          tf_version: ${{ env.tf_version }}
          tg_version: ${{ env.tg_version }}
          tg_dir: ${{ env.working_dir }}
          tg_command: "plan"


RequireHttpsAttribute should implement IFilter

RequireHttpsAttribute does not implement IFilter so it can't be used as a global filter like:
// Startup.cs
services.Configure<MvcOptions>(opt => opt.Filters.Add(typeof(RequireHttpsAttribute)));

Sorry, it implements IAuthorizationFilter  which contains IFilter.


IOS 10.2 I6 reboots/ Crashes

I Don't know if this is an issue or a bug, Just wanted to let you know even on beta 7 I am getting this random crash.
I read the issue posting guidelines.

At random times in the jailbroken state it just reboots.

I'm having this problem as well except my Cydia crashes when a tweak is finishing downloading.

@jitterbvg Should be a new repo and update for Cydia. (Round 3 of the Cydia betas) that fixes that problem. Go to /r/Jailbreak's front page somewhere to find it.

Here's the repo: http://apt.saurik.com/beta/cydia-arm64/
This is just odd, considering my phone is doing just fine on beta 7 (not even joking). And it's an iPhone 6.

does that repo fix the issue? or ? and the beta 3 link download isnt working anymore

The repo does not necessary "fix" the whole restart thing, but it will make Cydia less buggy, despite it being in beta right now (1.1.28~b9 at the time of this comment).
Just keep using beta 7 of the Yalu jailbreak. Honestly, that's all we have.

thanks! it actually worked, no more crashing

I'm having similar problems on the iPhone 6, iOS 10.2:

The jailbreak process sometimes fails on the first attempt after a reboot and restarts the device in a non-jailbroken mode (JB apps do not start). Sometimes the second attempt works, sometimes another manual reboot is required before it works again.
Sometimes I have random freezes of the device (e.g. while navigting on homescreen). A hard reboot is required afterwards.
May not be JB related: Substrate Safe Mode automatically restarts springboard after unlock. Afterwards the device is in a state where mobile substrate, PreferenceLoader etc. is not loaded at all. However, jailbreak apps can be started and complain about missing MobileSubstrate/RocketBootstrap. Only a respring (e.g. initiated over the Activator app) returns the device in the proper jailbroken state.
Yesterday it showed me a message stating that my SIM Card was not found. However, the phone unlocked without problems... Could be unrelated.

Cydia64_1.1.28~b9 = less problems

the latest cydia beta fixed the crashes for me so far. I just installed the latest maybe 8 hours ago. haven't had a crash

Alrwady have the latest Cydia version. Will keep note if it fixed the issues.

Springboard just randomly crashed while using the Youtube app (no associated tweaks installed) and afterwards was again in this jailbroken mode without mobile substrate and without safe mode. A respring got my device to stuck at a black screen without spinning wheele. I had to do a hard reset to restart the phone.

Just had this again: "The jailbreak process sometimes fails on the first attempt after a reboot and restarts the device in a non-jailbroken mode (JB apps do not start). Sometimes the second attempt works, sometimes another manual reboot is required before it works again."
So: No, the new Cydia did not fix the problem.

Ok even a restart did nit directly fix this and the jailbreak did only work on the second attempt. There are indeed some hick-ups in the JB process.

Il check if there is an update, Do I have to update the Yalu app?

I'm on beta 9... still crashes occur

Wtf beta 9? That's not possible. Are you talking about Cydia?

send logs next time

Here you go. Random panic while doing nothing special...

Have you updated safe mode yet?

Just now. From the changelog it seems that the safe mode problem was fixed:
"(In specific, the expected dialog now appears, but if you are using the older Yalu for 10.0-10.1.1 jailbreaks, including the final "b3", clicking Restart will cause the device to lock up so you will need to reboot. This continues to be a known issue with this jailbreak.)"
As i'm not on b3 this should not explain the lock ups / freezes.
Also the panic should not be related to this. Here another fresh ome that happened while looking at pictures in the photo app and one from yesterday:

Just now. From the changelog it seems that the safe mode problem was fixed:
"(In specific, the expected dialog now appears, but if you are using the older Yalu for 10.0-10.1.1 jailbreaks, including the final "b3", clicking Restart will cause the device to lock up so you will need to reboot. This continues to be a known issue with this jailbreak.)"
As i'm not on b3 this should not explain the lock ups / freezes.
Also the panic should not be related to this. Here another fresh ome that happened while looking at pictures in the photo app and one from yesterday:

Updated cydia to beta 9 and it still is happening, not as much but still happening, yes I also updated substrate safe mode

I looked at some logs and it says a lot of springboard loops with UIKit

Hi i was using 10.2 yalu jailbreak and succesfully jailbroked
but today after reboot on winterboard installation
am unable to open yaly102
it keep crashing, please help

@sanjubaby re install Yalu beta 7 0n your pad a Weill have to every 7 days

Yalu beta 7 random reboots here. After 2 days of use

What device crash log ? And would say tweeks  will be problem remove resent tweeks as lot are not updated to ios10


[Suggestion] Include timestamp / DatetimeIndex for plotting

After some more exploring, one thing I realized is, that plotting (with pyfolio / quantstats) to evaluate the results is harder than it needs to be. Currently we have no timestamps easily available which are needed for direct plotting or using the above libraries.
The Demo_FinRL_Meta_Integrate_Trends_data_to_DOW_Jones.ipynb is a good example.
It is done there, but with a (hacky) workaround using a Custom_DataProcessor / the yahoofinance dataframe timestamps.
A solution I could think of would be adding 3 functions to the env that return equal weight portfolio returns, buy-and-hold returns and the agent-returns including the timestamps. A pd.Series with DateTimeindex could work.
This would also be a good preparation for more advanced reward functions like (differential / deflated) sharpe, comparing to buy-and-hold etc. that rely on a return series.
What do you think?

Great suggestions! Thanks a lot. We will consider adding them in the future. And if you have ideas on the specific implementation, you may share them with us through the Pull Request. We will much appreciate it.

I think the issue/solution lies in return price_array, tech_array, turbulence_array in the data_processor / run function. If you would pass the whole dataframe all necessary data would be available. Not sure what best practice is, but I would suggest always using the main dataframe with all data. That way also would make it most flexible. Meaning the agent should also accept just the df instead of individual arrays.
For example this part of Demo_MultiCrypto_Trading.ipynb raised those questions:

Why is it necessary to pass the array data twice? Via env(config=data_config) and directly to the DRLAgent_erl?

    #process data using unified data processor
    DP = DataProcessor(data_source,  **kwargs)
    price_array, tech_array, turbulence_array = DP.run(ticker_list, start_date
                                                       , end_date, time_interval,
                                                       technical_indicator_list,
                                                       if_vix)
    data_config = {'price_array':price_array,
                   'tech_array':tech_array,
                   'turbulence_array':turbulence_array}
    #build environment using processed data
    env_instance = env(config=data_config)

    #read parameters and load agents
    current_working_dir = kwargs.get('current_working_dir','./'+str(model_name))

    if drl_lib == 'elegantrl':
        break_step = kwargs.get('break_step', 1e6)
        erl_params = kwargs.get('erl_params')

        agent = DRLAgent_erl(env = env,
                             price_array = price_array,
                             tech_array=tech_array,
                             turbulence_array=turbulence_array)

More flexible:
    DP = DataProcessor(data_source,  **kwargs)
    df = DP.run(ticker_list, start_date , end_date, time_interval,  technical_indicator_list,   if_vix)

    agent = DRLAgent_erl(env = env,  df = df)

Then inside the Agent get the necessary columns from the df (and convert them to an array where necessary).
For that to work the columns would have to be named accordingly of course like:

open, high, low, close
indicator_rsi, indicator_macd, ....
feature_weather, feature_customvolatility

With those naming conventions, you know what to get from the df inside the agent. For example with https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.filter.html you can get all columns containing "feature_" and so on.

After looking at other examples and (gym)environments I think best practice would be doing it inside the env:
https://github.com/AminHP/gym-anytrading/blob/master/gym_anytrading/envs/forex_env.py#L19
Init the env with the df and then processing the data there. The agent should just get the env as input.

Example:
# get the dataframe from the dataprocessor
DP = DataProcessor(data_source, **kwargs)
df = DP.run(ticker_list, start_date, end_date, time_interval, technical_indicator_list, if_vix)

# build environment using dataframe
env_instance = env(df=df)

# agent gets only the enviroment
agent = DRLAgent_erl(env=env)

class DRLAgent:
    """Provides implementations for DRL algorithms
    Attributes
    ----------
        env: gym environment class
            user-defined class
    Methods
    -------
        get_model()
            setup DRL algorithms
        train_model()
            train DRL algorithms in a train dataset
            and output the trained model
        DRL_prediction()
            make a prediction in a test dataset and get results
    """

    def __init__(self, env, price_array, tech_array, turbulence_array):
        self.env = env
        self.price_array = env._get_price_array
        self.tech_array = env._get_tech_array
        self.turbulence_array = env._get_turbulence_array

class Enviroment:
    def __init__(self, df):
        self.df = df
        self.price_array = self._get_price_array

    @property
    def _get_price_array(self):
        unique_ticker = self.df.tic.unique()
        price_array = np.column_stack([self.df[df.tic == tic].close for tic in unique_ticker])
        return price_array

    @property
    def _get_tech_array(self):
        ...

    @property
    def _get_turbulence_array(self):
        ...

    # This is very easy now using the dataframe:
    @property
    def _get_buy_and_hold(self):
        return pd.Series(self.df['close'].pct_change(), index=self.df['time'])

Example:
# get the dataframe from the dataprocessor
DP = DataProcessor(data_source, **kwargs)
df = DP.run(ticker_list, start_date, end_date, time_interval, technical_indicator_list, if_vix)

# build environment using dataframe
env_instance = env(df=df)

# agent gets only the enviroment
agent = DRLAgent_erl(env=env)

class DRLAgent:
    """Provides implementations for DRL algorithms
    Attributes
    ----------
        env: gym environment class
            user-defined class
    Methods
    -------
        get_model()
            setup DRL algorithms
        train_model()
            train DRL algorithms in a train dataset
            and output the trained model
        DRL_prediction()
            make a prediction in a test dataset and get results
    """

    def __init__(self, env, price_array, tech_array, turbulence_array):
        self.env = env
        self.price_array = env._get_price_array
        self.tech_array = env._get_tech_array
        self.turbulence_array = env._get_turbulence_array

class Enviroment:
    def __init__(self, df):
        self.df = df
        self.price_array = self._get_price_array

    @property
    def _get_price_array(self):
        unique_ticker = self.df.tic.unique()
        price_array = np.column_stack([self.df[df.tic == tic].close for tic in unique_ticker])
        return price_array

    @property
    def _get_tech_array(self):
        ...

    @property
    def _get_turbulence_array(self):
        ...

    # This is very easy now using the dataframe:
    @property
    def _get_buy_and_hold(self):
        return pd.Series(self.df['close'].pct_change(), index=self.df['time'])

Your suggestion is good. We consider to use the df['key'] to pass the data. We also consider using dict, d['key']: np.array to pass data, which is more efficient. In #46, we described this idea.
With respect to data processors. Now, we use several functions. In fact, some data may be missing, so we use bfill or ffill to fill the missing values. If users want to process it in their own way, they should define their own functions. We will consider to use one function and return the data.

@cryptocoinserver great idea

Your suggestion is good. We consider to use the df['key'] to pass the data. We also consider using dict, d['key']: np.array to pass data, which is more efficient. In #46, we described this idea.

I still think the dataframe approach inside the dataprocessor would be better. It has a way better usability. The conversion to numpy for performance makes sense, but isn't it enough to do it in the enviroment at _get_price_array or maybe even later in the agent's __init__.

With respect to data processors. Now, we use several functions. In fact, some data may be missing, so we use bfill or ffill to fill the missing values. If users want to process it in their own way, they should define their own functions. We will consider to use one function and return the data.

Yes, but couldn't you handle that in the individual processor either directly in download_data or adding DP.clean_data() like this:
DP = DataProcessor(data_source, **kwargs)
# Get candle data (added to the dataframe)
DP.download_data(ticker_list, start_date, end_date, time_interval)
DP.clean_data()
# Add indicators
DP.add_technical_indicator(technical_indicator_list)
# Very flexibel as I construct the df step by step. For example now we skip turbulance. Just using candle data and indicators. Then in the end we get the df.
df = DP.get_df()

I think the example I posted would be the most "pythonic" way to do it, and deep learning best practice too. Making everything more flexible and structured. Currently all the different data processor use different function names, varying column names (for example 'date' vs 'time'.
I understand though that this requires many changes and you aren't excited about that. I believe though it would be well invested, as it also would make many future implementations much easier. Also it would make a lit more consistent and easier to maintain. I could help with it if you are interested.

The dataframe or the future numpy inside the class is good. We will update it in the next days.
Currently all the different data processors have differences regarding function names, varying column names (for example 'date' vs 'time') and usage. For example some use .run() to return the final data and alpaca fetch_latest_data().
response: In the new version, we have revised it to ensure all the column names are the same. The function fetch_latest_data should be deleted since the download_data can replace it.

Awesome. BTW regarding the decision between the dataframe vs numpy. Using a dataframe as long as possible ensures compatiblity to other libraries.
An example would be tsfresh which is an incredible library to generate more features and help with feature selection / filtering.
Using the dataframe approach like this and converting to numpy later in env._get_price_array or the agent, would make using it very easy:
DP = DataProcessor(data_source, **kwargs)
# Get candle data (added to the dataframe)
DP.download_data(ticker_list, start_date, end_date, time_interval)
# Add indicators
DP.add_technical_indicator(technical_indicator_list)
# Add features
DP.add_upper_shadow_feature()
DP.add_lower_shadow_feature()
DP.add_parkinson_volatility_feature()
# Very flexibel as I construct the df step by step. For example now we skip turbulance. Just using candle data and indicators. Then in the end we get the df.
df = DP.get_df()

from tsfresh.utilities.dataframe_functions import roll_time_series
df_rolled = roll_time_series(df, column_id="tic", column_sort="time")
from tsfresh import extract_features
df_features = extract_features(df_rolled, column_id="tic", column_sort="time")

Thank you for your great work.

It's a pleasure to get your suggestions.
We will use  several incredible  libraries such as tsfresh to calculate  more features.
Converting to numpy can be put to the final step based on the generated dataframe, which is easy. This option can be set by users in config
We will update the code in several days.


test-framework2.4.0 is missing

Describe the bug
The POM for org.unidal.framework:test-framework:jar:2.4.0 is missing, no dependency information available
and in mvn-repo branch , there is no 2.4.0

2
is there not exists
import com.dianping.cat.configuration.client.entity
import com.dianping.cat.configuration.client.transform
two package?

To Reproduce
Steps to reproduce the behavior:
build, then error
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

test-framework可以直接修改版本，不影响构建。
找不到entity相关类的说明这个也可以在WIKI中找到。为了方便阅读，欢迎阅读这部分Cat-Cookbook，


Code Review: Basis for preferences window

Just wanting feedback on what I've written so far. This isn't ready for a merge.
Any review/suggestions would be great. Currently it doesn't seem to update the downloadPath, gotta look into that. Secondly, it doesn't currently display the existing downloadPath because I'm not sure what the best way to handle that is. I've tried to mimic the pattern used in the main app, which involved putting the dialogue box call in menu.js, not sure how I feel about that, but it's a central point that has access to windows.js. I figured there's a reason not to include windows.js within the preferences.js file itself.

Looks nice so far! Left some comments. Thanks for doing this.
When you think the prefs dialog looks good, I recommend pasting a screenshot into the PR. Github makes this v smooth. On Ubuntu it's Alt + PrintScr, Copy to Clipboard, then Ctrl V directly into a Github comment.

Thanks for the feedback @dcposch, @ungoldman! I've made several changes, fixing the issues mentioned and taking suggestions from both of you :).
The only issue I'm having now is actually retrieving the state.downloadPath. Currently, the main window creates a new App() instance with state passed in. I could take a similar approach and make a Preferences() class, but then we'd be maintaining two state objects, which seems like a bad idea.
I'm not sure what the best way to tackle this is. Any ideas? I feel like this is a nontrivial problem, solutions may require a large code refactor.

@ChrisMorrisOrg i think there's two things we could do:

Move state to the main process. This is a big refactor and might suck in other ways, but it would ensure that there's exactly one state object.
Make preferences their own section of the state, state.saved.prefs. (The rest of state.saved contains things like the torrent list, which is persisted but is not part of "Preferences".) Then, every time we open the prefs modal, we pass in state.saved.prefs. The modal has its own copy of that object, which it modifies as the user interacts with the form inputs. Finally, if the user clicks OK, the we send an IPC to the main process, setPreferences, which sets the new value of state.saved.prefs .

I like option 2. It's easier to implement, and it means that all prefs changes take effect simultaneously when the user clicks OK, rather than piece by piece as they interact with the prefs modal.

@dcposch's option 2 sounds pretty good to me as well.

:+1: to option 2. Also, it's worth noting that on OS X, Preferences windows don't have "Save" buttons. When you change a setting, it's immediately saved. We can add the OK, Cancel buttons for other OSs, since users will expect them,

Superseded by #426, right?


Adding black and white or greyscale color setting for the apps

Adding black and white or greyscale color setting for the apps used within yam will be a useful feature for avoiding distraction within those apps

The launcher itself can be configured to only use black and white colours, but I can't do anything for apps. The function to turn your screen grayscale is built into Android (you can find it in accessibility or display settings depending on your device and there might even be a quick settings tile for it), so you can use that.
While it is possible to enable grayscale through code, it requires the app user to enable a permission through adb that they shouldn't be enabling for an app like this in the first place. Considering the redundancy and security concerns, this won't be implemented.


Add benchmarks

Would it even make sense if it's a localhost benchmark...?

https://github.com/bheisler/criterion.rs
That would probably be a good place to start


Ability to process Flat to OPC and OPC to Flat in memory.

Added a couple of overloads to make it possible to transfer a flat opc structure to a docx package and vice versa without writing to disk.

@lpperras Hi, thank you!
I am in the final stages of putting together a cool new module for Open-Xml-PowerTools - watch for this near the beginning of next week.  :-)
Following that, I will be attending to some other Open-Xml-PowerTools issues, and will be looking at this and other PRs in detail.  If OK, please be patient - I will attend to this as soon as I release this new module.
Cheers, Eric

@lpperras and @EricWhiteDev The functionality to convert between OPC and Flat OPC in memory exists in the Open XML SDK. I had a PR for that and @EricWhiteDev merged that PR at the time.

hi, what is the status of this functionality? thanks, Ran

@lpperras and @rashlab Are you still interested in this PR? If so, have you checked out the functionality that is already included in the Open XML SDK? What would this PR add to that?


How to get IP address from a VM

Assuming a VM --name 'foo' in --resource-group 'bar', how to get the public IP address assigned to that machine using bash to pass it into ssh?
IP=$(az ??????)
ssh ${IP}

VM show with -d.


exchange frequency not passed to run_replica_exchange in example

Looks like the replica exchange example uses the default 'exchange attempts', which is the total number of exchange attempts throughout the whole simulation. The default is total time steps / 1000, or 10 if < 10000 steps. The pickle data storage command uses 'exchange frequency' which we took as the number of time steps between exchange attempts. Seems like we can simplify things by passing only frequency into run_replica_exchange? And should we make this a mandatory argument for run_replica_exchange?

Definitely should be easier to set this, especially since flexibility in this it is likely to be key in improving scaling to more nodes.


fail with simple sum gradient on CUDA

The following script gives a segmentation fault (after hanging for a few minutes)
using Enzyme, CUDA
x = CUDA.ones(2)
dx = CUDA.zeros(2)
autodiff(Reverse, sum, Active, Duplicated(x,dx))

[1332974] signal (11.1): Segmentation fault
in expression starting at /home/lucibello/.julia/dev/Flux/t.jl:4
typekeyvalue_hash at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jltypes.c:1622 [inlined]
lookup_typevalue at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jltypes.c:1059
jl_inst_arg_tuple_type at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jltypes.c:2157
jl_f_tuple at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:868 [inlined]
jl_f_tuple at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:863
absint at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/absint.jl:116
abs_typeof at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/absint.jl:302
unknown function (ip: 0x7f98b76ac15e)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
check_ir! at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler/validation.jl:500
check_ir! at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler/validation.jl:208
check_ir! at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler/validation.jl:178
check_ir at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler/validation.jl:157 [inlined]
#codegen#488 at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler.jl:4376
codegen at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler.jl:4340 [inlined]
_thunk at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler.jl:5430
_thunk at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler.jl:5430 [inlined]
cached_compilation at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler.jl:5464 [inlined]
#532 at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler.jl:5530
JuliaContext at /home/lucibello/.julia/packages/GPUCompiler/U36Ed/src/driver.jl:47
unknown function (ip: 0x7f99113c1265)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
#s1883#531 at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/compiler.jl:5482 [inlined]
#s1883#531 at ./none:0
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
GeneratedFunctionStub at ./boot.jl:602
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_call_staged at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/method.c:540
ijl_code_for_staged at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/method.c:593
get_staged at ./compiler/utilities.jl:123
retrieve_code_info at ./compiler/utilities.jl:135 [inlined]
InferenceState at ./compiler/inferencestate.jl:430
typeinf_edge at ./compiler/typeinfer.jl:920
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2889
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_45276.1 at /home/lucibello/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_type_infer at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2480 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2368
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2886 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
autodiff at /home/lucibello/.julia/packages/Enzyme/wR2t7/src/Enzyme.jl:224
unknown function (ip: 0x7f9911360dca)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
include_string at ./loading.jl:2076
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
_include at ./loading.jl:2136
include at ./Base.jl:495
jfptr_include_46403.1 at /home/lucibello/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
exec_options at ./client.jl:318
_start at ./client.jl:552
jfptr__start_82738.1 at /home/lucibello/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/cli/loader_exe.c:58
unknown function (ip: 0x7f9928680d8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 159273970 (Pool: 159095605; Big: 178365); GC: 99

@CarloLucibello I don't see any function f defined here?

sorry, I edited the post

While Enzyme shouldn't fail in the way you are seeing right now. Currently we do not support CUDA on the host side, that is ongoing work,

Like mentioned here https://github.com/EnzymeAD/Enzyme.jl/issues/1454#issuecomment-2260921347 reverse mode kerenel support for CUDA.jl has now been added. This doesn't include reductions, however. As the relevant code is in CUDA.jl itself -- please open an issue there and cc me.

Issue opened at https://github.com/JuliaGPU/CUDA.jl/issues/2455.


How to run a script headless, which contain some steps via skuli and some via normal element identifier.

How to run a script headless, which contain some steps via skuli and some via normal element identifier.
Try to create a new script for skuli  steps and call the same in another tag
tagui skuil.tag -n
and run the same script using -h but it is not allowing

Hi @Anujpalu24
For calling subflows, you are unable to add run options. However, you can chain the different tag files like this in a .cmd file. One with headless mode and one with no browser mode, like this:
@echo off
call tagui a.tag -h
call tagui b.tag -n