Arrow Row Format

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I think this crate has pretty good stories for operating on individual columns, either by downcasting to a concrete type, or invoking a dyn kernel.
The stories for multi-column operations are substantially weaker, with patchy support for common multi-column operations such as sorts, groupings, aggregations, reassembly, etc... We have some pieces such as MutableArrayData, DynComparator, but they're not especially performant, making extensive use of dynamic dispatch at the row-level, nor easy to use.
Describe the solution you'd like

Having a first-class row representation will not only allow us to implement more performant versions of existing kernels such as lexsort, but also provide a pretty compelling primitive to downstreams with which to implement more advanced operations such as streaming merges, joins, aggregates, etc... There is also precedent, with the C++ arrow library providing its own row format.
Goals

Each row should be encoded as a single sequence of bytes
Comparison of the byte arrays should be sufficient to establish ordering of the rows
It should be possible to convert a selection of rows back to arrays

Non-Goals

Support introspection or mutation of the row values
Provide a stable encoding for FFI, IO, etc...
Provide "optimal" encoding, rather a reasonable out-of-the-box baseline for common use-cases

Describe alternatives you've considered

We could extend the row format in DataFusion, however, this would limit its benefits to DataFusion. I think a row-oriented representation is such a fundamental primitive that it makes sense for inclusion in arrow-rs, so that it can be both used in its kernels and by downstreams that don't make use of DataFusion.
Additional context

Hello,
Would this help with processing row/record-based data, where the number of columns (that is, the structure of the row/record) vary depending on the type of row/record in the file. Example: An ndjson file where rows can be of n different variants.

This will not allow operations on data with multiple schema, same as with RecordBatch.
That being said, in the case of rows with different variants, nulls will be inserted by the JSON reader for the columns found in other variants but not present in the current record. The data is effectively unified to a single schema.
This trades off memory efficiency for the abilitybto efficiently process data in a columnar fashion, without per-value dynamic dispatch.
This is likely acceptable, however, partitioning the data so that different schema aren't interleaved will lead to better performance and memory efficiency. The row format could help with implementing this, but does not alter the nature of arrow schema

Thanks for clarifying. As you say, I'll have to make a separate stream for each row-type.


Output of large_residuals no longer seems to work with bad_toa

The last couple times I've gotten a run lite_utils.large_residual and then plugged the output into the bad-toa block, the bad-toa block seems to now give me errors.
I ran
lu.large_residuals(fo, threshold) for some data and got back

[guppi_56614_J1713+0747_0004.15y.x.ff, 20, 1]

Per my understanding/what the doc string says, I popped that directly back into the bad-toa block in my config and got
traceback.txt
Have I messed up large_residuals somewhere along the way or has something changed in the excision that makes the output of large_residuals no longer just work?

It looks like the error message is complaining about the lack of a subint flag in at least some of the TOAs you're using. The NANOGrav 15-year TOAs all have this flag, and bad-toa tries to match against it -- the two numbers at the end of the large_residuals() output are meant to be the values of the chan and subint flags.

Using bad-toa-averaged could be a workaround here, but I'm not entirely clear on how that would work in a case like this. Presumably it would remove other TOAs from this epoch too, which we don't want.
The right way to deal with this is probably to check for the existence of the chan and subint flags on a per-TOA basis before trying to match against them. It would require some fiddling with the apply_ignore() function. If done properly, though, it could make bad-toa flexible enough that we wouldn't need a separate bad-toa-averaged field.

I merged this into the main branch. This should solve the problems you were having @gooddc .
~ Joe G.


x/net/http2: generate static table in hpack module before runtime

As the static version of headerFieldTable in hpack module is generated in runtime, we may use the go:generate to prepare the struct before the initialization phase.
This is supposed to save init time and allocations for many binaries, as net/http imports hpack.
Before:
init golang.org/x/net/http2/hpack @1.1 ms, 0.097 ms clock, 21240 bytes, 29 allocs
After:
init golang.org/x/net/http2/hpack @0.67 ms, 0.015 ms clock, 8120 bytes, 9 allocs
We also could add isStatic field, so we will be able to remove less readable pointer comparison in the idToIndex method and again reduce allocations. BenchmarkEncoderSearchTable is okay with this:
EncoderSearchTable-8  6.75µs ± 7%  6.57µs ± 6%   ~     (p=0.099 n=10+10)

CC @neild, @tombergan.


[Map] Read pins from database locations collection

So far, pin locations were served from a static file. Now the request is redirected to a backend function that queries the pins from the locations collection in the database. For now, pin details are still loaded from static content.

Now pin details for the plasma type are also loaded from the database.


topics with no same-name command give odd help info

Describe the bug
When I have a topic that does not have a same-name command, the help output produced for the topic describes a command in the topic rather than being generic to the topic.
To Reproduce
Steps to reproduce the behavior:

Clone the topic-test repository.
Build it by running npm install && npm build.
Run ./bin/run topic-only --help to see help for the topic that has no same-name command associated with it.
See error

Expected behavior
The description at the top should mention this is a topic only and there should be no USAGE stanza. Ideally the description would be configurable somehow.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):

OS & version: MacOS and Linux for sure but I don't think this is an OS-specific thing.

Additional context
Perhaps there is already a way to fix this? We use oclif in the SmartThings CLI and have this problem there. We have an edge topic but no edge command. For now, I've created an edge command that has a description of edge topic and just displays a message asking the user run --help.
https://github.com/SmartThingsCommunity/smartthings-cli/pull/444

@rossiam
The usage section in this case indicates that you can use the topic by specifying a sub command,
USAGE
  $ topic-test topic-only COMMAND

I believe this is a valid usage string so I'm not convinced that we should get rid of it but I'm open to hearing arguments about why it should.
Also, the description for the topic can be configured by adding it to the oclif section of your package.json. For instance,
"oclif": {
"topics": {
      "topic-only": {
        "description": "This topic has no top-level command"
      }
    }
}

Thanks for the help. I wasn't aware of the configuration in package.json. I'm closing this although I still think there is room for improvement. A couple of errors I made in the original description of this:

I shouldn't have said there "there should be no USAGE stanza" stanza under expected behavior. The default USAGE stanza is describing the sub-commands perfectly. It's only the description that is off.
I should have defined "same-name command". What I mean by this is a topic with no index.ts and no <topic-name>.ts like topic-only in the example.
I could have been more clear that I think the current behavior is correct when there is a same-name command. For example, the topic topic-and-command produces a good description. I just think the default description for a topic with no same-name command should be different or even left out entirely.


Style improvements after review

8: plat? Why not use an existing word?)
10,11: Lines can be merged in one import.
29: Why do you comment this code in comments?
41: Can be replaced with Promise.all(...).catch(...).
43: const process = files.map(...).filter(...).
49: try...catch inside of then can be replaced with then(...).catch(...).
92: Sometimes C:/Windows/ isn't a path to the Windows directory. Using $PATH can solve this problem.
93: Hacky trim, split, slice, and crowded map can be replaced with split, chain of maps, and filter.
113: Wrapping switch in the try...catch feels grumpy. I will show an alternative later.
TBH on the first sign this file feels like a bunch of copy-pasted code fragments with chaotic structure and tricky data-flow.
I think each function (both of them, yes) in this file should only choose platform specific methods from separate files and handle some errors in unified way.
Smth like this:

export async function killProcByPID(pidString) {
  const pid = parseInt(pidSting, 10);

  if (!Number.isInteger(pid)) {
    return {
      result: null, // []?
      error: 'PID is not a number' };
  }

  const killByPIDFn;

  switch (platform) {
    case 'win32':
      killByPIDFn = require('./platform/win32').killByPID;
    case 'linux':
    case 'darwin':
      killByPIDFn = require('./platform/unix').killByPID;
    default:
      killByPIDFn = () => throw new Error(OSErrMsg);
  }

  return await killByPIDFn(pid)
    .then(({stdout, stderr}) => {
      result: stdout,
      error: stderr
    })
    .catch(e => {
      result: null, // []?
      error: e.stderr || e.message
    })
}

Switch...case in this example can be replaced with a decision map ofc.
This is an draft of directory structure:
└── package-dir
    ├── index.js
    └── platform
        ├── darwin.js
        ├── win32.js
        └── unix.js

I like your attempt of extracting getLinuxProc. All other functions should be extracted from switch...case too. (edited)


Fix updates to globalSettings from Chrome Policy

Yeah, I'm sorry.

This change is

OK, I think I see what you're trying to do - although I'm lacking context on why this relates to Chrome Policy.

Eek, I forgot to put in a real PR description, sorry about that.

Review status: 0 of 17 files reviewed at latest revision, 5 unresolved discussions.

src/generic_core/uproxy_core.ts, line 791 [r1] (raw file):
Previously, trevj wrote…
> why don't you use `updateGlobalSetting`?

It doesn't seem to make much sense to call the method that's going to end up taking these settings individually and then just set them with double calls to `storage.save` since we're already in the core.  I think you do make a very good point though that we should probably wait for `globals.loadSettings` in the core before setting anything.

src/generic_core/uproxy_core.ts, line 792 [r1] (raw file):
Previously, trevj wrote…
> weird formatting

Eek.  Okay, I have no idea whatsoever how I even managed to type this.  Fixed.

src/interfaces/uproxy_core_api.ts, line 33 [r1] (raw file):
Previously, trevj wrote…
> This is extremely abstract. Many other types here make it explicit for which operation they are, e.g. `CloudOperationArgs`. How about `UpdateGlobalSettingArgs`?

Okay, fixed.

src/interfaces/uproxy_core_api.ts, line 147 [r1] (raw file):
Previously, trevj wrote…
> A comment here describing the payload would be great (though I see now literally no other value here specifies it, either).

Done.

src/interfaces/uproxy_core_api.ts, line 184 [r1] (raw file):
Previously, trevj wrote…
> Again, a comment describing the payload.

Done.

Comments from Reviewable

:thumbsup:

Review status: 0 of 17 files reviewed at latest revision, 1 unresolved discussion.

src/generic_core/uproxy_core.ts, line 791 [r1] (raw file):
Previously, jpevarnek (Jonathan Pevarnek) wrote…
> It doesn't seem to make much sense to call the method that's going to end up taking these settings individually and then just set them with double calls to `storage.save` since we're already in the core.  I think you do make a very good point though that we should probably wait for `globals.loadSettings` in the core before setting anything.

Ah! Got it :-)

Comments from Reviewable


mongodb configuration file.

Hi,
Please can you told mongodb.conf file not found, mongodb.conf.origin file found. please help me. it's urgently.

Can you give all the commands you ran and any relevant files or logs for reproducing the issue
If you wanted to use a custom file it would look like this https://github.com/docker-library/docs/tree/master/mongo#using-a-custom-mongodb-configuration-file

The image doesn't come with a configuration file because without one, MongoDB already has pretty sensible defaults and providing/using a configuration file is optional (you're welcome to provide your own via bind mount, creating a new image that's FROM this one, Docker Config/Secret objects, etc).


setting tracked timestamp to x1 rather than x2 in H-test

Addresses #21. Creates a member variable called x1_timestamp to store input timestamp from RadClass associated with the first measurement in a hypothesis test: x1. This is preferable since, in the event of a rapid decrease that causes the null hypothesis to be rejected (which is anecdotally more common), the first timestamp is more relevantly associated with the event spectrum.

Thanks @stompsjo


SliM seems not to be maintained and therefore isn't recommended by archlinux

Quotation from archwiki:

Warning: The SliM project has been abandoned (the project homepage is down, leaving a github mirror), and is not fully compatible with systemd, including logind sessions. Consider using a different Display manager or Xinitrc.

What do toy think about switching to lightdm for example?

Thanks for the reminder. At the end of 2015 I looked at most of the alternatives and failed to find anything I liked. Slim has not actually broken yet, so I was happy to stay with it.
I'll try the alternatives again.

LightDM seems fine.


Override default read-replica names.

TL;DR
Make it possible to override the default name that this module generates for read replicas on a per-replica basis. This would be useful when importing existing resources into Terraform, or other considerations require a custom naming scheme.
Terraform Resources
No response
Detailed design
E.g. something like:
In variables.tf:
// Read Replicas
variable "read_replicas" {
  description = "List of read replicas to create. Encryption key is required for replica in different region. For replica in same region as master set encryption_key_name = null"
  type = list(object({
    name            = string
    name_override   = optional(string)
...

In read_replica.tf:
resource "google_sql_database_instance" "replicas" {
  provider             = google-beta
  for_each             = local.replicas
  project              = var.project_id
  name                 = each.value.name_override == null ? "${local.master_instance_name}-replica${var.read_replica_name_suffix}${each.value.name}" : each.value.name_override

Unfortunately this approach requires opting into the module_variable_optional_attrs experiment, but perhaps there are other approaches?
Additional information
No response

Sorry for the delay @mans0954
I think we can add an override attrib. Regarding optional agreed, it would be cleaner if we had that and this would not be a breaking change which is coming in 1.3 https://github.com/hashicorp/terraform/releases/tag/v1.3.0-beta1


dbt seed raise error mismatched input 's' when there is 's in csv file

Hi team,
I initially raised this issue to dbt-core and it seems fine with dbt-snowflake.
Would you please take a look? the dbt seed will fail if there is a ' character in the seed file
details:
https://github.com/dbt-labs/dbt-core/issues/6306
Thanks

Decided to close off this issue as this repo is not longer maintained, re-submit at https://github.com/dbt-athena/dbt-athena/issues


[$250] Multi tags - Unable to save GL code for dependant tags

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.9-1
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Issue reported by: Applause - Internal Team
Action Performed:
Precondition:

Workspace is under Control plan
Workspace is imported with dependant tags on Old Dot

Go to staging.new.expensify.com
Go to workspace settings > Tags
Click on any tag
Click on any subtag
Click GL code
Enter the GL code and save it
Click on the same subtag again

Expected Result:
GL code will be saved
Actual Result:
User is unable to save GL code for dependant tags
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/user-attachments/assets/18503140-f29f-4c99-82af-74d998c346a9
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01d849cbdc023dce84
Upwork Job ID:<PHONE_NUMBER>845810989
Last Price Increase: 2024-07-30

Issue OwnerCurrent Issue Owner: @brunovjk

@anmurali  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors

We think that this bug might be related to #wave-control

cc @yuwenmemon you might want to take a look at this one. We don't yet support dependent multi-tags on NewDot yet, so that's might what is going on?

I was unable to reproduce the issue.

Awesome! Thank you @gijoe0295, I will try to reproduce it once more after lunch. I will review your proposal after that. Thank you very much.

@gijoe0295 your proposal looks good to me, I tried your suggestion and it works fine. I have a question, do you believe it is necessary update the tagListName here as well?
https://github.com/Expensify/App/blob/d4104a73c1a6159e61c170bbbaf3949865991852/src/libs/actions/Policy/Tag.ts#L351-L352

Great, thank you so much @gijoe0295 for your hard work.
@gijoe0295's proposal looks good to me
🎀👀🎀 C+ reviewed

[x]  [@brunovjk] The PR that introduced the bug has been identified. Link to the PR: N/A
[x]  [@brunovjk] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment: N/A
[x]  [@brunovjk] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion: N/A
[x]  [@brunovjk] Determine if we should create a regression test for this bug. Yes
[x]  [@brunovjk] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.

Regression Test Proposal
Precondition:

Workspace is under Control plan.
Workspace is imported with dependant tags on Old Dot or using a .csv file.

Steps:

Go to URL https://staging.new.expensify.com/.
Log in with an account that has access to the imported workspace.
Navigate to Workspace settings > Tags.
Click on any tag list.
Click on any tag.
Click on GL code.
Enter the GL code and save it.
Verify that the GL code is saved correctly.
Click on State > California.
Run the following snippet in the Console to mock tag errors:

Onyx.merge('policyTags_POLICY_ID_HERE', {State: {tags: {California: {errors: {'test': 'Random error'}}}}});

Dismiss the "Random error" error.
Verify that the error is dismissed.

Do we agree 👍 or 👎

Update: Still waiting for payment. @rlinoz do you know if @anmurali is OOO for a few days? Thanks.

Oh you got it right, until Aug 23rd, let's see if we can get someone else to proceed with payment here

Thank you :D

Contributor: @gijoe0295 paid $250 via Upwork
Contributor+: @brunovjk paid $250 via Upwork
Test case created

https://github.com/Expensify/Expensify/issues/422010

Thanks y'al!


added a line to make it return null if nothing is found

the get_by function returns an empty array if no record is found even when fetching a single record .. added a line to make it return null if nothing is found .. makes it more logical

well i was following fuelphp's way of doing things ...i.e,
->a function which returns true or false for a case should return true or false only
->a function which returns an array of objects when records are found should return an empty array if nothing is found
->a function which returns a single object when a record is found should return null if nothing is found
however, its your library, just post one more time if you want me to change the return value to false when no record is found


Update header css

Updates on-demand header css to include mega menu heading indentation updates
Changes

Updates header.css

Testing

Verify that new header.css contains .o-mega-menu_group-heading rules

Checklist

[ ] Changes are limited to a single goal (no scope creep)
[ ] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the front end playbook
[ ] Passes all existing automated tests
[ ] New functions include new tests
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[ ] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)

It contains .o-mega-menu_group-heading 👍

Thanks, @anselmbradford!


Run-time errors when using cypress-image-diff with Cypress/Typescript and cypress-cucumber-preprocessor

Hello,
I've tried using your tool in Mocha Cypress and it did work and I liked the results and simplicity of use.
We use Cypress with Typescript and BDD/Cucumber.  The I see run-time errors which I don't know how to fix.
my project structure looks like this

Keep in mind that because of Cucumber screenshot were generated but with specific name.  Then I saw an error like this

My cypress.json looks like this
{
"env": {},
"numTestsKeptInMemory": 50,
"watchForFileChanges": true,
"includeShadowDom": false,
"defaultCommandTimeout": 8000,
"execTimeout": 60000,
"taskTimeout": 60000,
"pageLoadTimeout": 60000,
"requestTimeout": 5000,
"responseTimeout": 30000,
"fileServerFolder": "root project folder",
"fixturesFolder": "fixtures",
"integrationFolder": "features",
"testFiles": "**/*.feature",
"pluginsFile": "plugins/index.ts",
"supportFile": "support/index.ts",
"componentFolder": "",
"videosFolder": "videos",
"htmlReportDir": "./test-results/html",
"reporterOptions": {
"reportDir": "",
"overwrite": true,
"html": false,
"json": true
},
"screenshotsFolder": "screenshots",
"screenshotOnRunFailure": true,
"trashAssetsBeforeRuns": true,
"videoCompression": 32,
"video": true,
"videoUploadOnPasses": true,
"viewportWidth": 1440,
"viewportHeight": 1080,
"nodeVersion": "bundled"
}
commands.ts looks like this
// ***********************************************
// This example commands.js shows you how to
// create various custom commands and overwrite
// existing commands.
//
// For more comprehensive examples of custom
// commands please read more here:
// https://on.cypress.io/custom-commands
// ***********************************************
//
//
// -- This is a parent command --
// Cypress.Commands.add("login", (email, password) => { ... })
//
//
// -- This is a child command --
// Cypress.Commands.add("drag", { prevSubject: 'element'}, (subject, options) => { ... })
//
//
// -- This is a dual command --
// Cypress.Commands.add("dismiss", { prevSubject: 'optional'}, (subject, options) => { ... })
//
//
// -- This will overwrite an existing command --
// Cypress.Commands.overwrite("visit", (originalFn, url, options) => { ... })
// cypress/support/commands.js
// @ts-ignore
declare global {
// eslint-disable-next-line @typescript-eslint/no-namespace
namespace Cypress {
interface Chainable {
compareSnapshot(a: string, b:number): void;
}
}
}
const compareSnapshotCommand = require('cypress-image-diff-js/dist/command')
compareSnapshotCommand()
plugins/intex.ts is below

My feature is

And teps under the feature look like
import { Given, When, Then, Before } from "cypress-cucumber-preprocessor/steps";
beforeEach(() => {
cy.viewport(1000, 600 );
cy.exec('ls');
})
Given(/^I am finally on CAIR login page$/, function () {
///  cy.visit("http://qe.ist.bina.com/login");
cy.visit("https://www.ups.com");
cy.wait(5000);
///  cy.url().should ('eq', 'yahoo.com')
});
Given(/^I make my own screenshot$/, function () {
//  cy.screenshot('');
// @ts-ignore
cy.compareSnapshot('wholePage', 0.1)
});

When I tried to remove report related code from Support/index.ts
I saw this error now

Think that actual snapshots for comparison using your tool had benn generated ina screenshot folder
Please help if you can.  I really want to move forwar and use your tool in my upcomimg project
Thanks in advance
Jeff

So, is it a hope that the above enhancement will be done in a feasible future. Please, let me know
Thanks
Jeff

End of year is a bit tough for me.. if no one in the community picks it up soon I will work on it in the first weeks of the new year.

I’ll patiently wait because out of few tools yours is one of the best I’ve tried. I mean the way it works a straight forward implementation and use
Thanks in advance
Jeff
Sent from my iPhone

On Dec 24, 2020, at 4:52 AM, Pippo Raimondi<EMAIL_ADDRESS>wrote:

End of year is a bit tough for me.. if no one in the community picks it up soon I will work on it in the first weeks of the new year.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Please, let me know if you'll be able to get to my issue in a feasible future one way or another?  Otherwise I'll stop waiting and will think to use some other approach to Visual testing
Thanks
Jeff

Please, let me know if you'll be able to get to my issue in a feasible future one way or another?  Otherwise I'll stop waiting and will think to use some other approach to Visual testing
Thanks
Jeff

Please, let me know if you'll be able to get to my issue in a feasible future one way or another? Otherwise I'll stop waiting and will think to use some other approach to Visual testing
Thanks
Jeff

@jradom the issue seems specific to generating the report. You might not be able to use after blocks on cucumber setups since it's a mocha thing... can you remove the below from cypress/support/index.js:
after(() => {
  cy.task('generateReport')
})

I can research a way for you to generate the report using cucumber framework and will update the readme. Please let me know if this works for you for now after removing the above.

Please, let me know if you'll be able to get to my issue in a feasible future one way or another? Otherwise I'll stop waiting and will think to use some other approach to Visual testing
Thanks
Jeff

@jradom the issue seems specific to generating the report. You might not be able to use after blocks on cucumber setups since it's a mocha thing... can you remove the below from cypress/support/index.js:
after(() => {
  cy.task('generateReport')
})

I can research a way for you to generate the report using cucumber framework and will update the readme. Please let me know if this works for you for now after removing the above.

Sounds like a great plan. So, I'll try removing report related part but it'll be nice to have the ability to generate a report.
Thanks
Jeff

Sounds like a great plan. So, I'll try removing report related part but it'll be nice to have the ability to generate a report.
Thanks
Jeff

Unfortunately, your suggestion didn't help.  I'm getting now this

Please, advise
Thanks a lot in advance
Jeff

Unfortunately, your suggestion didn't help.  I'm getting now this

Please, advise
Thanks a lot in advance
Jeff

@jradom have you tried merging plugins/intex.ts is below?
so have the below written at the end of your second module that handles cucumber preprocessor:
  const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
  getCompareSnapshotsPlugin(on, config)

This is not a problem with this plugin and more how you are setting it up... can you try that and share your new plugin/index file content so we can take it from there?

@jradom have you tried merging plugins/intex.ts is below?
so have the below written at the end of your second module that handles cucumber preprocessor:
  const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
  getCompareSnapshotsPlugin(on, config)

This is not a problem with this plugin and more how you are setting it up... can you try that and share your new plugin/index file content so we can take it from there?

my index.ts looks like this
module.exports = (on:any, config:any) => {
const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
getCompareSnapshotsPlugin(on, config)
}
const browserify = require('@cypress/browserify-preprocessor');
const cucumber = require('cypress-cucumber-preprocessor').default;
const resolve = require('resolve');
module.exports = (on:any, config:any) => {
const options = {
...browserify.defaultOptions,
typescript: resolve.sync('typescript', { baseDir: config.projectRoot }),
};
on('file:preprocessor', cucumber(options));
};

my index.ts looks like this
module.exports = (on:any, config:any) => {
const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
getCompareSnapshotsPlugin(on, config)
}
const browserify = require('@cypress/browserify-preprocessor');
const cucumber = require('cypress-cucumber-preprocessor').default;
const resolve = require('resolve');
module.exports = (on:any, config:any) => {
const options = {
...browserify.defaultOptions,
typescript: resolve.sync('typescript', { baseDir: config.projectRoot }),
};
on('file:preprocessor', cucumber(options));
};

module.exports = (on:any, config:any) => {
const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
getCompareSnapshotsPlugin(on, config)
}
const browserify = require('@cypress/browserify-preprocessor');
const cucumber = require('cypress-cucumber-preprocessor').default;
const resolve = require('resolve');
module.exports = (on:any, config:any) => {
const options = {
...browserify.defaultOptions,
typescript: resolve.sync('typescript', { baseDir: config.projectRoot }),
};
on('file:preprocessor', cucumber(options));
};

Can you change to:
const browserify = require('@cypress/browserify-preprocessor');
const cucumber = require('cypress-cucumber-preprocessor').default;
const resolve = require('resolve');
module.exports = (on:any, config:any) => {
const options = {
...browserify.defaultOptions,
typescript: resolve.sync('typescript', { baseDir: config.projectRoot }),
};
on('file:preprocessor', cucumber(options));

const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
getCompareSnapshotsPlugin(on, config)
};

module.exports = (on:any, config:any) => {
const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
getCompareSnapshotsPlugin(on, config)
}
const browserify = require('@cypress/browserify-preprocessor');
const cucumber = require('cypress-cucumber-preprocessor').default;
const resolve = require('resolve');
module.exports = (on:any, config:any) => {
const options = {
...browserify.defaultOptions,
typescript: resolve.sync('typescript', { baseDir: config.projectRoot }),
};
on('file:preprocessor', cucumber(options));
};

Can you change to:
const browserify = require('@cypress/browserify-preprocessor');
const cucumber = require('cypress-cucumber-preprocessor').default;
const resolve = require('resolve');
module.exports = (on:any, config:any) => {
const options = {
...browserify.defaultOptions,
typescript: resolve.sync('typescript', { baseDir: config.projectRoot }),
};
on('file:preprocessor', cucumber(options));

const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
getCompareSnapshotsPlugin(on, config)
};

I'll try and let you know

I'll try and let you know

Good news.  It looks like it went to end for positive case with no diff.  I'll try later a negative case.  Also, I'd be very interested to generate a report.  Can you please try making it work for Cypress/Cucumber configuration?  Is is a particular low Cpress version I need to use?
Thanks a  lot
Jeff

Good news.  It looks like it went to end for positive case with no diff.  I'll try later a negative case.  Also, I'd be very interested to generate a report.  Can you please try making it work for Cypress/Cucumber configuration?  Is is a particular low Cpress version I need to use?
Thanks a  lot
Jeff

now that we fixed the plugin, try following the report documentation... it most likely will work as well...

now that we fixed the plugin, try following the report documentation... it most likely will work as well...

Hi Pippo,
Reporting startedto work but in a strange way.  When I run several features only one report was created.  Any next one overwrites a previous.  I'd prefer consolidated report or separate repots based on feature files.  Also, it's not clear to me how to assert in order to verify a failed or passed scenario
Please, advise
Thanks
Jeff

Hi Pippo,
Reporting startedto work but in a strange way.  When I run several features only one report was created.  Any next one overwrites a previous.  I'd prefer consolidated report or separate repots based on feature files.  Also, it's not clear to me how to assert in order to verify a failed or passed scenario
Please, advise
Thanks
Jeff

below are my step samples
Given(/^I am finally on NIGHTLY login page$/, function () {
cy.visit('http://nightly.ist.bina.com/');
});
Then(/^I take my own screenshot NIGHTLY$/, function () {
let uploadToS3Local: string = 'aws s3 sync ' + BASELINE_LOCAL_PATH + ' ' + BASELINE_LOCAL_AWS_PATH;
let uploadToS3Jenkins: string = 'aws s3 sync ' + BASELINE_LOCAL_PATH + ' ' + BASELINE_JENKINS_AWS_PATH;
cy.compareSnapshot('wholePageYahoo', 0.1)

if (getOS() === 'Mac OS') {
    cy.exec(uploadToS3Local);
} else {
    cy.exec(uploadToS3Jenkins);
}

});
Then(/^I take my own screenshot and compare with actual YAHOO$/, function () {
let downloadToS3Local: string = 'aws s3 sync ' + BASELINE_LOCAL_AWS_PATH + " " + BASELINE_LOCAL_PATH;
let downloadToS3Jenkins: string = 'aws s3 sync ' + BASELINE_JENKINS_AWS_PATH + ' ' + BASELINE_LOCAL_PATH;
if (getOS() === 'Mac OS') {
    cy.exec(downloadToS3Local);
} else {
    cy.exec(downloadToS3Jenkins);
}

cy.compareSnapshot('wholePageYahoo', 0.1)

});

below are my step samples
Given(/^I am finally on NIGHTLY login page$/, function () {
cy.visit('http://nightly.ist.bina.com/');
});
Then(/^I take my own screenshot NIGHTLY$/, function () {
let uploadToS3Local: string = 'aws s3 sync ' + BASELINE_LOCAL_PATH + ' ' + BASELINE_LOCAL_AWS_PATH;
let uploadToS3Jenkins: string = 'aws s3 sync ' + BASELINE_LOCAL_PATH + ' ' + BASELINE_JENKINS_AWS_PATH;
cy.compareSnapshot('wholePageYahoo', 0.1)

if (getOS() === 'Mac OS') {
    cy.exec(uploadToS3Local);
} else {
    cy.exec(uploadToS3Jenkins);
}

});
Then(/^I take my own screenshot and compare with actual YAHOO$/, function () {
let downloadToS3Local: string = 'aws s3 sync ' + BASELINE_LOCAL_AWS_PATH + " " + BASELINE_LOCAL_PATH;
let downloadToS3Jenkins: string = 'aws s3 sync ' + BASELINE_JENKINS_AWS_PATH + ' ' + BASELINE_LOCAL_PATH;
if (getOS() === 'Mac OS') {
    cy.exec(downloadToS3Local);
} else {
    cy.exec(downloadToS3Jenkins);
}

cy.compareSnapshot('wholePageYahoo', 0.1)

});

Hi Pippo,
Reporting startedto work but in a strange way. When I run several features only one report was created. Any next one overwrites a previous. I'd prefer consolidated report or separate repots based on feature files. Also, it's not clear to me how to assert in order to verify a failed or passed scenario
Please, advise
Thanks
Jeff

Please open a new issue for the report with instructions and screenshots of the issue.
Assert is part of the compareScreenshot command. If it has a baseline and the new comparison image fails the diff it will fail the test.

Hi Pippo,
Reporting startedto work but in a strange way. When I run several features only one report was created. Any next one overwrites a previous. I'd prefer consolidated report or separate repots based on feature files. Also, it's not clear to me how to assert in order to verify a failed or passed scenario
Please, advise
Thanks
Jeff

Please open a new issue for the report with instructions and screenshots of the issue.
Assert is part of the compareScreenshot command. If it has a baseline and the new comparison image fails the diff it will fail the test.

I see.
Thanks for clarifying
Jeff

I see.
Thanks for clarifying
Jeff


Static Code Analysis

The changes I proposed are light but now you have far bigger issues with your branch. I tried running only your analysis, without crawling or cleaning, and after maybe 30 minutes I some thread starvation warnings, which culminated in an OutOfMemoryError (log).
I figured that this might be an issue with the number of analysis threads, so I reduced them to 3 while enabling the crawling and cleaning jobs. This time however, your analysis inexplicably stopped at one point... (log)
The last cloning thread was made at 2023-04-02 18:12:04.442. Checking with VisualVM, your cloning threads were reported as inactive.

Its also worth noting that while that run did not end in an OutOfMemoryError, it did consume more than ten times the resources the app normally would. Here's the heap space graph running only crawling and cleaning:

As you can see the heap space is consumed and freed at regular intervals.  I suspect that this might be an issue with the entities being preserved in a cache by the EntityManager while your analysis is running. In any case you need to figure out the source of the excessive heap consumption, otherwise the app will be completely unusable.

@dabico
Garbage collector now seems to properly clean up memory.

https://github.com/seart-group/ghs/blob/4cb452217fcb59351c5140fbbacc7cc4465abb15/src/main/java/usi/si/seart/service/MetricLanguageServiceImpl.java#L30-L47
Here I'm clearing the cache as I want to make sure other subsequent threads have visibility of the newly created MetricLanguage entity in the database. I wasn't able to do so without invoking method entityManager.clear().
Now, I have noticed no side effects until now. Changes made by Cleanup and Crawl job are persisted regularly, but at the moment I'm not 100% sure if there will be side effects.
Documentation for EntityManager.clear():

Clear the persistence context, causing all managed entities to become detached. Changes made to entities that have not been flushed to the database will not be persisted.

I wasn't able to do so without invoking method entityManager.clear().

I find this to be a bit odd. Have you also considered using saveAndFlush? Look at the documentation of clear:

Changes made to entities that have not been flushed to the database will not be persisted.

Regular saves are flushed at the discretion of the EntityManager. Also, look into the detach method from the EntityManager.

@dabico I refactored the Criteria classes in order to implement the code metrics filters later on. [24beec4]
I tested the search results and they seem to coincide with the results from the previous version. Can you please check it yourself and confirm it ? Thanks

There is a bug Im working now right now:

As you can see only the first entry of the metrics has the data stored as attributes, the rest is stored as tags


Preserve original kernel file for isolate spawn requests.

This change gives incremental kernel files ".incremental.dill" extension thereby preserving original kernel file. It also reverts workaround in asset_bundle introduced for https://github.com/flutter/flutter/issues/17012.
This fixes https://github.com/flutter/flutter/issues/16084, https://github.com/flutter/flutter/issues/17012,  https://github.com/flutter/flutter/issues/17983.
Note that while this is big improvement over current state of things, it is not quite ultimate solution as this means that isolates will be spawned from the original, rather than edited and hot-reload code.

@Hixie wrote:

@aam What is your plan with this PR? (No particular rush, I'm just cleaning out the review queue.)

This is still relevant, I'm hoping to get this reviewed and landed.

Spawning an isolate from the original instead of the edited but hot reloaded code could lead to weird behavior, isolate messaging may encounter incompatible types.

Spawning an isolate from the original instead of the edited but hot reloaded code could lead to weird behavior, isolate messaging may encounter incompatible types.

That is true and hopefully can be addressed next, but what we currently have is worse - after hot reload no isolate can be spawned because full kernel file is overwritten with incremental kernel file.


Close Menu, "Close other tabs" and "Close tabs to right" always operate on tab index 0, are always enabled

I'm not sure if this is an issue in main or an issue in feature/settings-ui.
/cc @mpela81

Looks fine on master, maybe it wasn't fully merged onto the feature branch?

Excellent. In that case, it's on me :smile:

Uh oh, I'm seeing this on the  1.5.2941.0 selfhost the team is working with as well.

Fortunately, that is built off feature/settings-ui!

Looks like this was @carlos-zamora's merge commit
commit 0ce05b5405df96a829d4756f44e4b1daffe28083
Merge: e00275232 743283e43
Author: Carlos Zamora<EMAIL_ADDRESS>Date:   Fri Oct 16 12:48:21 2020 -0700

    Merge branch 'master' into feature/settings-ui

Carlos, this commit lost the call to _EnableCloseMenuItems in the various tab implementations that controlled whether the menu items were enabled/disabled.


chore: use base images from mcr

use base images from mcr

Codecov Report

Merging #1396 (96e573a) into master (514318b) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1396   +/-   ##
=======================================
  Coverage   61.78%   61.78%
=======================================
  Files          11       11
  Lines        1078     1078
=======================================
  Hits          666      666
  Misses        386      386
  Partials       26       26


Update docs to include current way to show img with alt text

I thought that the img tag was broken.  The documented method has brackets for the title and alt text.  It currently appears to work only if you use double and single quotes.
Docs:
{% img [class names] /path/to/image [width] [height] [title text [alt text]] %}
Working:
{% img [class names] /path/to/image [width] [height] "title text 'alt text'" %}
https://stackoverflow.com/questions/47665964/hexo-alt-text-doesnt-work
https://github.com/hexojs/hexo/issues/2887

Thanks for collaboration. This will save many people a day. :)


Update dependencies due to button release

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Resolves: # (issue, if applicable)
Summary:
Please summarize the scope of the changes you have submitted, what the intent of the work is and anything that describes the before/after state of the project.
Update dependency versions for auro-button and all other dependencies
Type of change:
Please delete options that are not relevant.

[ ] New capability
[ ] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[x] Other (please elaborate)

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

:tada: This PR is included in version 1.2.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Merge main into release-1.1 for v1.1.2 RC image

Issue #, if available:
N/A
Description of changes:
Merge main into release-1.1 for v1.1.2 RC image
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I see this PR - https://github.com/aws/aws-network-policy-agent/pull/268/commits/93be822fe5503ee65394819d8521efcb5d1c3888 is missing..


How does AG-assistant convert a task description to a prompt?

I am curious about the underlying process. Which LLM papers / approaches does it use to do this?

The core of AutoGluon Assistant (AG-A) is built around the concept of a Task, which encapsulates all necessary information for a tabular prediction problem. Task Inference prompts are processed by instruction-tuned LLM APIs via LangChain. The responses from these LLMs are ultimately parsed and used to populate the Task metadata. To ensure, robustness and fault tolerance, AG-A implements validation checks and appropriate fallbacks for metadata in case of an LLM based failure during the Task Inference step.


make the hasher class pluggable in HashClient

I was doing some benchmarking and I needed this patch to be able to plug in ewdurbin/clandestined-python as the hash implementation.
In passing, there's a separate issue where pymemcache calls the relevant method get_node, but clandestined-python calls it find_node.
Thanks for your time.

@slingamn What is this change meant to fix? It doesn't appear to change the underlying functionality, just the way you pass information to the function. In addition, this change is not backwards compatible with existing clients, so we'd have to release it as a new major version.

Without this patch, the code only works when the default None value of hasher is not modified. When it is modified, you get a crash because self.hasher is never initialized:
Traceback (most recent call last):
  File "/nail/home/shivaram/benchmarks/memcache_bench.py", line 94, in <module>
    main()
  File "/nail/home/shivaram/benchmarks/memcache_bench.py", line 59, in main
    with client_manager() as client:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/nail/home/shivaram/benchmarks/memcache_bench.py", line 41, in make_pymemcache_client
    yield pymemcache.client.hash.HashClient(servers, hasher=RendezvousHash)
  File "/nail/home/shivaram/benchmarks/pymemcache/client/hash.py", line 93, in __init__
    self.add_server(server, port)
  File "/nail/home/shivaram/benchmarks/pymemcache/client/hash.py", line 107, in add_server
    self.hasher.add_node(key)
AttributeError: 'HashClient' object has no attribute 'hasher'

This change is not backwards compatible in the sense that it is a bug fix.

Got it, thanks for the fix!

@slingamn Are you publishing your benchmarks anywhere?

I'm not planning on a formal writeup, but I could share the numbers here.


bullet ends

Signed-off-by: Ludan Stoeckle<EMAIL_ADDRESS>Thanks for contributing!
Please check if the PR fulfills these requirements 👌:

[x] Confirm the PR related to a dedicated issue
[x] Confirm your PR corrects a single bug or implements a single feature
[x] Your code is linted locally prior to submission
[x] Your code is fully tested: 99% to 100% coverage
[x] Everything is correct on Sonar dashboard
[x] Confirm your code is under Apache 2.0 license
[x] Confirm your documentation is under CC-BY-4.0 license
[x] Confirm license and copyright content is created/updated in each file (see CONTRIBUTING.md)
[x] Confirm changelog.adoc is updated
[ ] If corrects vulnerabilities, confirm changelog.adoc contains CVE IDs

Also remember: each commit MUST contain a sign off message (see CONTRIBUTING.md) 🙄
Indicate related issue:
issue #148
Explain what is done, scope, limitations etc.
does what is asked in the feature request
Does this PR introduce a breaking change? 😁
should not

It's ok for me regarding the issue I opened.
Thanks.


Remove function-process links to 'modification' terms in the process ontology

This would be helpful in the short term.
For example:
GO:0006508 proteolysis AnnotationPropagation P GO:0004298 threonine-type endopeptidase activity IBA
GO:0006468 protein phosphorylation AnnotationPropagation P GO:0004674 protein serine/threonine kinase activity IBA
GO:0035335 peptidyl-tyrosine dephosphorylation AnnotationPropagation P GO:0004725 protein tyrosine phosphatase activity IBA
GO:0006470 protein dephosphorylation AnnotationPropagation P GO:0004721 phosphoprotein phosphatase activity IBA
GO:0006508 proteolysis AnnotationPropagation P GO:0004222 metalloendopeptidase activity IBA
GO:0016311 dephosphorylation AnnotationPropagation P GO:0016791 phosphatase activity IBA
there are lots.
It is difficult to weedout these annotations, because for the ones with an experimental source on the MF, they look like annotations we have made.

There are still 14,000 EXP, but many are from MF connections
remove links between
(645) | ubiquitin-protein transferase activity
(375) | protein tyrosine kinase activity
(568) | ubiquitin protein ligase activity
(117) | histone acetyltransferase activity
(104) | SUMO transferase activity
(98) | ubiquitin conjugating enzyme activity
(92) | histone H3K4 methyltransferase activity

I think these are now done


onLoad on the Chart component doesn't work

Hi!
First of all, thank you for your nice library!
I have found an issue which is that the load event doesn't seem to work, here is a minimal example: https://codesandbox.io/s/qxn6z5r589
I have copied an example from the documentation and used the onLoad and onRender callback on the Chart component. If you open the DevTools you'll notice that the render even is triggered but the load event is not.
Maybe there is something I didn't understand but this example should work (here is a pure highcharts example).
Can you have a look?
Thank you!

Ah damn, that's annoying.
As a workaround (depending on your use case) you might be able to use the callback option added in 2.0.3 - see here?

Hi @whawker, thanks for the quick reply! I tried the callback option, it seems to work although at the time it is executed, the series are not on the chart, here is the updated example: https://codesandbox.io/s/qxn6z5r589. When I log the series from the onRender, I get 5 series but 0 when I log them from callback.

I found out that for my use case, the load event wouldn't work either as the series are added after the initial load in this library.

What exactly were you trying to achieve with the load event?

What I need to do is to execute a drilldown when opening the graph (in my use case the graph should be shown "drilled down" to a specific point from the beginning). I rolled back to react-highcharts and it works great for my use case, here is an example of what I'm trying to achieve: https://codesandbox.io/s/rr82porw4q
As you can see I want to use doDrilldown on a specific point of a series on load but in react-jsx-highcharts the series are added after the initial load so this won't work as evt.target.series (from my example) is an empty array. I need a callback that is executed after everything is loaded and added to the graph.

Ah, ok. I do need to add proper support for drilldown (see #29).

Hi @whawker for now is there any workaround ?. so far I haven't started my project but very soon i will start but i am sure I need this functionality in my project.  Is there any work around to get this feature ?

@janakp2411 here is a workaround, it's pretty nasty and I wouldn't advise doing this way usually, but it works.
See here

@ricopella as all the axes and series are added to the chart dynamically, you would be best hooking into a Series render event and updating your component state.
Example here: https://stackblitz.com/edit/rjh-on-load-workaround-wvnbfx?file=MyChart.js


is there a reason DISCO fails to resolve a tree if the tree has more than four species?

Hello James,
I've been using DISCO to resolve a set of multicopy gene trees. I noticed that a big portion of the trees are not resolved because they have less than four species (they don't meet the minimal species criterion). But, I also noticed that a small portion (~1%) of the trees have between four and six species and DISCO failed to produce a resolved tree. Here's an example...
Tree 81: Tree has 6 species.
Best root had score 792 with 778 duplications; there were 3 ties.
Outgroup: {Me_px_Sstr_Streblomastix_strix}
Decomposition strategy outputted 0 tree(s) with minimum size 4 .
Do you know a reason why this is happening?
Thank you very much!

Hi,
Sorry for the late response. My guess it that, even though there are more then 4 species in the entire tree, there is no set of more then 4 leaves in the tree with the same label that are considered orthologous by DISCO.
A simple example might be (((a,b),c),((a,b),d)); this tree would have 4 species; however, c and d are considered paralogous, since their LCA is a duplication event. In this case, the largest subtree of orthologous leaves would be of size 3.
I hope this helps! I can also take a look at your input to confirm this is what is happening if you think that would help.
Best,
James


onHover event in ComboBox changes selected_item

I am trying to extract the selected value from a ComboBox with combo_box.selected_text() through the win32 backend, with no success as each time my mouse is hovering on another not selected item, this function returns the hovered item's text and not my selected item's text.
Am I missing something or is this intentional behavior on onHover event?
I've tried numerous ways to extract the selected value + tried to understand if the combo box list is visible (which means the user might hover over some items) but with no success at all.
I noticed (through Accessibility Insights) that there is a certain property in ComboBox item indicating the selectedValue, how do I extract it?

You can use backend="uia" and method get_value() which should return "Anesthesiology" as seen on the screenshot. I'm not sure about standard behavior of "win32" events... "win32" combo box method selected_index() returns value from window message CB_GETCURSEL. Microsoft docs has this comment:

An application sends a CB_GETCURSEL message to retrieve the index of the currently selected item, if any, in the list box of a combo box.

I guess "win32" method window_text() should return actual edit box text of the combo box.


DOM is not rendered directly inside shadow root

Code to reproduce the issue:
As a base use the code from here.
Then replace in src/index.ts the following code block:
var span = document.createElement("span");
root.appendChild(span)
createUserGreeter(span);

with
createUserGreeter(root);

Expected behavior:
The website shows My Awesome Cycle.js app!.
I've a similar react example which is working when using the shadow root.
Actual behavior:
The website is empty and does not throw any exception or show an error.
Versions of packages used:
"@cycle/dom": "^13.0.0",
"@cycle/xstream-run": "^3.1.0",
"xstream": "^7.0.0"

I believe this was fixed in Cycle DOM v14.2. Please put a comment if you see this issue still happening.


parser: disallow into-outfile clause in some place

What problem does this PR solve?
Part of pingcap/tidb#19097.
What is changed and how it works?
This PR validate the 'into outfile' option. This clause is only allowed in the last select statement.
Check List
Tests

Unit test
Integration test

Code changes

Has exported function/method change

Side effects

Increased code complexity

Related changes

Need to cherry-pick to the release branch

I'd prefer we do this kind of check outside of parser.y as a separation of concern. Hmm perhaps the semantic checker can really be moved from pingcap/tidb into pingcap/parser.
MySQL 8 and MariaDB 10.4 have different error messages for these errors. And I think the MySQL 8 is clearer.
In MySQL 8, using SELECT INTO in the wrong place generates the 3954 error instead of the standard 1064 "syntax error" (this PR):
mysql> (select 1 into outfile '1.txt') union select 1;
ERROR 3954 (HY000): Misplaced INTO clause, INTO is not allowed inside subqueries, and must be placed at end of UNION clauses.

also CREATE VIEW AS SELECT INTO generates the 1350 error:
mysql> create view c as select 1 into outfile '1.txt';
ERROR 1350 (HY000): View's SELECT contains a 'INTO' clause

Thus I suspect MySQL is checking after the AST is built, not during.
Perhaps as a side effect, MySQL allows the following code which has been parse errors in MariaDB:
-- NO ERROR, the 'into outfile' part is just ignored (looks wrong to me)
insert into a select 1, 2 into outfile '1.txt';
create table c as select 1 into outfile '1.txt';

-- same as removing the parenthesis
(((((select 1 into outfile '1.txt')))));

BTW the following should work:
create binding for
    select 1 into outfile '1.txt'
using
    select 1 into outfile '1.txt';

I agree with kenny,   semantic check should be done in the TiDB part.

Will be fixed by https://github.com/pingcap/tidb/pull/21956.


Some fields in APBILL and APBILLITEM cannot be queried

Hello,
I'm seeing an error when querying some fields in these two objects using IA.Functions.Common.NewQuery.Query.
ResultException [Error]: Result status: failure for Control ID: 65d824c7-68c1-4a9c-9cea-cc4007c76ac6 - XL03000006 The following fields cannot be queried: [         apbill: [PAYTOCONTACTNAME, RETURNTOCONTACTNAME] ] [Support ID: LYF3ZEB034%7EYeogyP0F2YJ-hwZWD-iCgQAAABU]
These are the fields that I've run into issues with:
APBILL

PAYTOCONTACTNAME
RETURNTOCONTACTNAME

APBILLITEM

GLACCOUNTNO

I'm using the version 2.1.1 of the @intacct/intacct-sdk

Hi @justin-ch, please be sure to build/validate your fields for the new query function by using the lookup function:
Developers Site for query
Lookup function

@justin-ch, did the above comment resolve your issue?

Closing this issue.... if it is persisting, please reopen and we will revisit.


Does/will lit-parrot support qlora?

Qlora requires much less GPU, currently lit-parrot only supports lora. Is there any plan for supporting qlora as well?
It would be a huge advantage for users like us.

Tracking in https://github.com/Lightning-AI/lit-gpt/issues/176! Thanks for the suggestion

@thanhnew2001  should be supported now via #275 :)


Developer UI: Use Gnosis' infra contracts on networks where they're deployed

For known networks such as rinkeby, mainnet etc, does it make sense to reuse the existing canonical versions? i.e. no need to re-reploy?

Should it always use gnosis' contracts or suggest doing so?

Would there even be any benefit if not using the canonical version?

@rmeissner Are there any differences between Gnosis's deployed infra contracts and your own ones? For example, there are methods that can be executed only by owner, etc
From usage POV there shouldn't be any differences


[Explorer]: update playpause animation

Description

Add animation for playpause

Test Plan
How did you test the new or updated feature?

If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.
Type of Change (Check all that apply)

[ ] protocol change
[ ] user-visible impact
[ ] breaking change for a client SDKs
[ ] breaking change for FNs (FN binary must upgrade)
[ ] breaking change for validators or node operators (must upgrade binaries)
[ ] breaking change for on-chain data layout
[ ] necessitate either a data wipe or data migration

Release notes

I think once this is in place with the refetching logic, it will be expected that the table refetching occurs at the same time as the animation completing.
Also it looks like the filled animation is a little larger than the unfilled path - is that intended?

Thanks for the comments, I am updating so we can sync the animation to the refetch


PPP and collated code

Coverage remained the same at 92.063% when pulling 4f48fc44449e1af7217fcbbcc2103fb45743ba7f on HanYaodong:doc-profile-collate-code into 0427d51c520799e179b08781c1abffee2bf9b025 on CS2103AUG2017-F11-B4:master.

Coverage remained the same at 92.063% when pulling 6dc1a8e0bf268f0be7933513a8ae7bba25200bd9 on HanYaodong:doc-profile-collate-code into 0427d51c520799e179b08781c1abffee2bf9b025 on CS2103AUG2017-F11-B4:master.

Coverage remained the same at 92.063% when pulling d2068d8df6edb03ce382c36f70565d4b4db7097f on HanYaodong:doc-profile-collate-code into 0427d51c520799e179b08781c1abffee2bf9b025 on CS2103AUG2017-F11-B4:master.

Coverage remained the same at 92.063% when pulling d2068d8df6edb03ce382c36f70565d4b4db7097f on HanYaodong:doc-profile-collate-code into 0427d51c520799e179b08781c1abffee2bf9b025 on CS2103AUG2017-F11-B4:master.

Coverage remained the same at 92.063% when pulling 29103ceb0fbd96e01b57d812defa213e5ef2ece7 on HanYaodong:doc-profile-collate-code into d38e172a5145761de7fbeb6f35dc38936752634b on CS2103AUG2017-F11-B4:master.


HADOOP-19048. ITestCustomSigner failures

MultiObject delete fails with x-amz-checksum* not found. This fix adds the correct execution attribute to use the correct interceptor.
Contributed by Harshit Gupta
How was this patch tested?
New test. It fails, even though this (existing) patch was meant to fix it.
For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 49s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
47m 13s

trunk passed

+1 :green_heart:
compile
0m 43s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 33s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
0m 31s

trunk passed

+1 :green_heart:
mvnsite
0m 40s

trunk passed

+1 :green_heart:
javadoc
0m 26s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 32s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
1m  8s

trunk passed

+1 :green_heart:
shadedclient
37m 51s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 29s

the patch passed

+1 :green_heart:
compile
0m 35s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
0m 35s

the patch passed

+1 :green_heart:
compile
0m 26s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
0m 26s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 20s

the patch passed

+1 :green_heart:
mvnsite
0m 31s

the patch passed

+1 :green_heart:
javadoc
0m 14s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 24s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
1m  6s

the patch passed

+1 :green_heart:
shadedclient
38m  9s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
3m 10s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

140m 30s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6471/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6471

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 629b81065159 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 0075ac8f9488db9c748c7d1e8ccd8920a138a62e

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6471/1/testReport/

Max. process+thread count
533 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6471/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


userguide: update what is Kill Bill doc

First pass.

Merging, so I can submit a Second Pass but a couple of points above are still open.


deletetweets command not found

i think there's something to do with this

Run it like ~/.local/bin/delete-tweets ... or adjust your PATH.


Constraint on number of assets?

What are you trying to do?
Setting a max number of assets to allocate weights to. For example observe a universe of 500 stocks, I only want the weights for max 10 stocks and the sum should ofcourse be 100%.
What data are you using?
SP500 data universe.

Hi @sword134,
Unfortunately, that's not possible because the constraint is not convex. Please see #45  and #101 for suggestions. As I have mentioned in other threads, my perspective is that portfolio optimisation != asset selection: you should choose which assets you are interested in before using PyPortfolioOpt to optimise weights.
Best,
Robert

Hi @sword134,
Unfortunately, that's not possible because the constraint is not convex. Please see #45  and #101 for suggestions. As I have mentioned in other threads, my perspective is that portfolio optimisation != asset selection: you should choose which assets you are interested in before using PyPortfolioOpt to optimise weights.
Best,
Robert

@sword134: as mentioned by @robertmartin8, the cardinality constraint you ask for is not convex.
However, CXPY also supports MILP problems, if you have the required solver installed.
For small problems with less than 1000 variables and constraints, you can use the community version of CPLEX:
pip install cplex
Consider this simple example for minimizing the portfolio volatility given a maximum of 10 securities to invest in, with weights between 0 and 1 for each asset. Slight modifications are required if you want to choose different bounds.
import cvxpy as cp
from tests.utilities_for_tests import setup_efficient_frontier
ef = setup_efficient_frontier(solver=cp.CPLEX)
booleans = cp.Variable(len(ef.tickers), boolean=True)
ef.add_constraint(lambda x: x <= booleans)
ef.add_constraint(lambda x: cp.sum(booleans) <= 10)
ef.min_volatility()
ef.weights
array([0.        , 0.01503686, 0.09891734, 0.12603716, 0.        ,
       0.        , 0.        , 0.17711377, 0.        , 0.11697861,
       0.17732243, 0.        , 0.        , 0.18942864, 0.        ,
       0.00072097, 0.        , 0.08792907, 0.        , 0.01051514])

Some remarks:

These types of problems are generally much harder to solve
You should consider Robert's comment about optimization vs. asset selection.

@sword134: as mentioned by @robertmartin8, the cardinality constraint you ask for is not convex.
However, CXPY also supports MILP problems, if you have the required solver installed.
For small problems with less than 1000 variables and constraints, you can use the community version of CPLEX:
pip install cplex
Consider this simple example for minimizing the portfolio volatility given a maximum of 10 securities to invest in, with weights between 0 and 1 for each asset. Slight modifications are required if you want to choose different bounds.
import cvxpy as cp
from tests.utilities_for_tests import setup_efficient_frontier
ef = setup_efficient_frontier(solver=cp.CPLEX)
booleans = cp.Variable(len(ef.tickers), boolean=True)
ef.add_constraint(lambda x: x <= booleans)
ef.add_constraint(lambda x: cp.sum(booleans) <= 10)
ef.min_volatility()
ef.weights
array([0.        , 0.01503686, 0.09891734, 0.12603716, 0.        ,
       0.        , 0.        , 0.17711377, 0.        , 0.11697861,
       0.17732243, 0.        , 0.        , 0.18942864, 0.        ,
       0.00072097, 0.        , 0.08792907, 0.        , 0.01051514])

Some remarks:

These types of problems are generally much harder to solve
You should consider Robert's comment about optimization vs. asset selection.

@phschiele that throws me the following error:
Traceback (most recent call last):
  File "d:\Finansiering. Modern Portfolio Theory Projekt\MPT. Finansierings projekt.py", line 41, in <module>
    weights = ef.min_volatility()
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\efficient_frontier.py", line 166, in min_volatility
    return self._solve_cvxpy_opt_problem()
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\base_optimizer.py", line 222, in _solve_cvxpy_opt_problem
    opt.solve(solver=self._solver, verbose=self._verbose)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\cvxpy\problems\problem.py", line 396, in solve
    return solve_func(self, *args, **kwargs)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\cvxpy\problems\problem.py", line 754, in _solve
    self.unpack_results(solution, solving_chain, inverse_data)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\cvxpy\problems\problem.py", line 1068, in unpack_results
    "Try another solver, or solve with verbose=True for more "
cvxpy.error.SolverError: Solver 'CPLEX' failed. Try another solver, or solve with verbose=True for more information.

I have no idea where to add the verbose=True.

@phschiele that throws me the following error:
Traceback (most recent call last):
  File "d:\Finansiering. Modern Portfolio Theory Projekt\MPT. Finansierings projekt.py", line 41, in <module>
    weights = ef.min_volatility()
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\efficient_frontier.py", line 166, in min_volatility
    return self._solve_cvxpy_opt_problem()
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\base_optimizer.py", line 222, in _solve_cvxpy_opt_problem
    opt.solve(solver=self._solver, verbose=self._verbose)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\cvxpy\problems\problem.py", line 396, in solve
    return solve_func(self, *args, **kwargs)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\cvxpy\problems\problem.py", line 754, in _solve
    self.unpack_results(solution, solving_chain, inverse_data)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\cvxpy\problems\problem.py", line 1068, in unpack_results
    "Try another solver, or solve with verbose=True for more "
cvxpy.error.SolverError: Solver 'CPLEX' failed. Try another solver, or solve with verbose=True for more information.

I have no idea where to add the verbose=True.

@sword134 Did you try to reproduce my example or did you add the constraints to your code? As mentioned above, the community version of CPlEX only supports 1000 variables and constraints. Note that this includes the boolean variables as well. It also does not only count the variables defined in CVXPY, but the variables in the canonical form produced by CVXPY. So the S&P 500 optimisation problem might already exceed these limits.

@sword134 Did you try to reproduce my example or did you add the constraints to your code? As mentioned above, the community version of CPlEX only supports 1000 variables and constraints. Note that this includes the boolean variables as well. It also does not only count the variables defined in CVXPY, but the variables in the canonical form produced by CVXPY. So the S&P 500 optimisation problem might already exceed these limits.

@phschiele so you say 1000 variables, what does this entail? How big can the stock universe be then? I added the code to my own data etc. to see if it worked. I didn't try it out of the box as you posted, because i don't have the "test" folder from which it is pulling the "setup_efficient_frontier" function, so I just added the solver parameter manually.

@phschiele so you say 1000 variables, what does this entail? How big can the stock universe be then? I added the code to my own data etc. to see if it worked. I didn't try it out of the box as you posted, because i don't have the "test" folder from which it is pulling the "setup_efficient_frontier" function, so I just added the solver parameter manually.

@sword134 This can easily be verified for your particular problem by excluding n assets from the universe and let n go to 0.
Check out the CVXPY Docs for further information on the problem size.

@sword134 This can easily be verified for your particular problem by excluding n assets from the universe and let n go to 0.
Check out the CVXPY Docs for further information on the problem size.

@phschiele ok I brought down the number assets so it works. However I cant use max_shape()
Traceback (most recent call last):
  File "d:\Finansiering. Modern Portfolio Theory Projekt\MPT. Finansierings projekt.py", line 41, in <module>
    weights = ef.max_sharpe(risk_free_rate=riskfreerate)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\efficient_frontier.py", line 226, in max_sharpe
    self._solve_cvxpy_opt_problem()
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\base_optimizer.py", line 226, in _solve_cvxpy_opt_problem
    raise exceptions.OptimizationError from e
pypfopt.exceptions.OptimizationError: Please check your objectives/constraints or use a different solver.

I want the optimal tangency portfolio (not min variance) so what solver would I use for sharpe ratio?

@phschiele ok I brought down the number assets so it works. However I cant use max_shape()
Traceback (most recent call last):
  File "d:\Finansiering. Modern Portfolio Theory Projekt\MPT. Finansierings projekt.py", line 41, in <module>
    weights = ef.max_sharpe(risk_free_rate=riskfreerate)
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\efficient_frontier.py", line 226, in max_sharpe
    self._solve_cvxpy_opt_problem()
  File "D:\Anaconda\envs\Finansiering\lib\site-packages\pypfopt\base_optimizer.py", line 226, in _solve_cvxpy_opt_problem
    raise exceptions.OptimizationError from e
pypfopt.exceptions.OptimizationError: Please check your objectives/constraints or use a different solver.

I want the optimal tangency portfolio (not min variance) so what solver would I use for sharpe ratio?

Max-Sharpe does not play well with complex constraints, because to find the max-sharpe portfolio, PyPortfolioOpt has to make a change of variables (the maths is a little tricky).
A while ago I was contracted to build max-sharpe portfolios and I was in a similar position to you (had a constraint on the number of assets).
My solution was to vary the L2 reg parameter, which produces different amounts of nonzero portfolios. This short script constructs several max sharpe portfolios and prints the number of assets with more than 0.1% weight:
for i in range(1, 20, 2):
    ef = EfficientFrontier(mu, sigma)
    ef.add_objective(objective_functions.L2_reg, gamma=0.05 * i)
    ef.max_sharpe()
    print(f"gamma = {0.05*i}, num nonzero = {sum(ef.weights > 0.001)}")

Then, you can pick the value of gamma that corresponds to your desired number of assets.
I know it's not ideal, but we are somewhat constrained (no pun intended) by the mathematics here :/

Max-Sharpe does not play well with complex constraints, because to find the max-sharpe portfolio, PyPortfolioOpt has to make a change of variables (the maths is a little tricky).
A while ago I was contracted to build max-sharpe portfolios and I was in a similar position to you (had a constraint on the number of assets).
My solution was to vary the L2 reg parameter, which produces different amounts of nonzero portfolios. This short script constructs several max sharpe portfolios and prints the number of assets with more than 0.1% weight:
for i in range(1, 20, 2):
    ef = EfficientFrontier(mu, sigma)
    ef.add_objective(objective_functions.L2_reg, gamma=0.05 * i)
    ef.max_sharpe()
    print(f"gamma = {0.05*i}, num nonzero = {sum(ef.weights > 0.001)}")

Then, you can pick the value of gamma that corresponds to your desired number of assets.
I know it's not ideal, but we are somewhat constrained (no pun intended) by the mathematics here :/

Hi @sword134,
gamma does not directly control weights. It is an indirect way of penalising portfolios with many nonzero weights.
Try and run the script I provided, you'll get some output like:
gamma = 0.05, num_nonzero = 500
gamma = 0.1, num_nonzero = 300
gamma = 0.15, num_nonzero = 200
...

Let's say you wanted 250 nonzero assets, this means you should choose gamma between 0.1 and 0.15. You could do another for loop with a more granular range of gammas if you wanted.
This approach does not work well if you absolutely must have a fixed number of assets. It is more of a heuristic for getting approximately the right number of nonzero-weight assets in your portfolio.
Best,
Robert

Hi @sword134,
gamma does not directly control weights. It is an indirect way of penalising portfolios with many nonzero weights.
Try and run the script I provided, you'll get some output like:
gamma = 0.05, num_nonzero = 500
gamma = 0.1, num_nonzero = 300
gamma = 0.15, num_nonzero = 200
...

Let's say you wanted 250 nonzero assets, this means you should choose gamma between 0.1 and 0.15. You could do another for loop with a more granular range of gammas if you wanted.
This approach does not work well if you absolutely must have a fixed number of assets. It is more of a heuristic for getting approximately the right number of nonzero-weight assets in your portfolio.
Best,
Robert

@robertmartin8 it seems like you have a table over the approximate gamma to assets list. How big would gamma be for say 10 assets?

@robertmartin8 it seems like you have a table over the approximate gamma to assets list. How big would gamma be for say 10 assets?

@sword134
There is no a priori answer to this, you have to run the snippet and play around with different gamma ranges to find the value that works for you

@sword134
There is no a priori answer to this, you have to run the snippet and play around with different gamma ranges to find the value that works for you

@robertmartin8 ok thank you. I am unsure how the final code should look like. How do I select that from the loop which gives me the number of assets I need to do clean_weights on?

@robertmartin8 ok thank you. I am unsure how the final code should look like. How do I select that from the loop which gives me the number of assets I need to do clean_weights on?

@robertmartin8 ok thank you. I am unsure how the final code should look like. How do I select that from the loop which gives me the number of assets I need to do clean_weights on?

To reword it. I want to run the loop and find gamma = 0.XX, num nonzero = 10 and then export that gamma and use in my clean_weights

@robertmartin8 ok thank you. I am unsure how the final code should look like. How do I select that from the loop which gives me the number of assets I need to do clean_weights on?

To reword it. I want to run the loop and find gamma = 0.XX, num nonzero = 10 and then export that gamma and use in my clean_weights

Actually scratch that. I am fighting a headache and doing complete brainfarts. Here is the final code for anybody that wants.
Num_assets_wanted = 12 #number of assets to want in the portfolio
for i in range(1, 100, 1):
    ef = EfficientFrontier(mu, sigma)
    ef.add_objective(objective_functions.L2_reg, gamma=0.00001 * i)
    ef.max_sharpe(risk_free_rate=riskfreerate)
    if sum(ef.weights > 0.001)==Num_assets_wanted:
        print(f"gamma = {0.00001*i}, num nonzero = {sum(ef.weights > 0.001)}")
        break

cleaned_weights = ef.clean_weights()
ef.save_weights_to_file("weights.csv")  # saves to file
ef.portfolio_performance(verbose=True, risk_free_rate=riskfreerate)
print(sum(ef.weights > 0.001))

Actually scratch that. I am fighting a headache and doing complete brainfarts. Here is the final code for anybody that wants.
Num_assets_wanted = 12 #number of assets to want in the portfolio
for i in range(1, 100, 1):
    ef = EfficientFrontier(mu, sigma)
    ef.add_objective(objective_functions.L2_reg, gamma=0.00001 * i)
    ef.max_sharpe(risk_free_rate=riskfreerate)
    if sum(ef.weights > 0.001)==Num_assets_wanted:
        print(f"gamma = {0.00001*i}, num nonzero = {sum(ef.weights > 0.001)}")
        break

cleaned_weights = ef.clean_weights()
ef.save_weights_to_file("weights.csv")  # saves to file
ef.portfolio_performance(verbose=True, risk_free_rate=riskfreerate)
print(sum(ef.weights > 0.001))

@robertmartin8 have you tried making a backtest on these weights, rebalancing every month/quarter/year? I've been looking and trying for 4 hours now with zipline and backtrader and I simply cant get it to work.

@robertmartin8 have you tried making a backtest on these weights, rebalancing every month/quarter/year? I've been looking and trying for 4 hours now with zipline and backtrader and I simply cant get it to work.

@sword134 it's on the roadmap, but not currently a priority.
Because portfolio optimisation tends to be used on longer time horizons, "for loop backtests" work pretty well in my opinion.
However, this might be something to ask on the Backtrader forums.

@sword134 it's on the roadmap, but not currently a priority.
Because portfolio optimisation tends to be used on longer time horizons, "for loop backtests" work pretty well in my opinion.
However, this might be something to ask on the Backtrader forums.

Alternatively, trace out the efficient frontier using multiple efficient_risk() / efficient_risk() calls (+ cardinality constraint) and select the sharpe ratio maximizing portfolio from the resulting portfolios.
I do not argue against the L2 regularization method, but it does more than just changing the cardinality, which is something you might or might not want.

Alternatively, trace out the efficient frontier using multiple efficient_risk() / efficient_risk() calls (+ cardinality constraint) and select the sharpe ratio maximizing portfolio from the resulting portfolios.
I do not argue against the L2 regularization method, but it does more than just changing the cardinality, which is something you might or might not want.

My optimizer quickly optimizes actively rebalanced portfolios of thousand assets on my laptop. You can't unfortunately do that with available optimizers without getting stuck into deceptive local optima(s). (In my method, the asset pre-selection is not necessary but you can still do that or give return estimates to introduce investor bias.) Here is an example QuantConnect back-test since 2009 for such a rebalanced portfolio that is found by my method. I actually want to integrate a reliable pre-selection method so that I can optimize larger universes of e.g. ten-thousand assets. What would be the best method for that? Hierarchical Risk Parity?

My optimizer quickly optimizes actively rebalanced portfolios of thousand assets on my laptop. You can't unfortunately do that with available optimizers without getting stuck into deceptive local optima(s). (In my method, the asset pre-selection is not necessary but you can still do that or give return estimates to introduce investor bias.) Here is an example QuantConnect back-test since 2009 for such a rebalanced portfolio that is found by my method. I actually want to integrate a reliable pre-selection method so that I can optimize larger universes of e.g. ten-thousand assets. What would be the best method for that? Hierarchical Risk Parity?

My optimizer quickly optimizes actively rebalanced portfolios of thousand assets on my laptop. You can't unfortunately do that with available optimizers without getting stuck into deceptive local optima(s). (In my method, the asset pre-selection is not necessary but you can still do that or give return estimates to introduce investor bias.) Here is an example QuantConnect back-test since 2009 for such a rebalanced portfolio that is found by my method. I actually want to integrate a reliable pre-selection method so that I can optimize larger universes of e.g. ten-thousand assets. What would be the best method for that? Hierarchical Risk Parity?

My optimizer quickly optimizes actively rebalanced portfolios of thousand assets on my laptop. You can't unfortunately do that with available optimizers without getting stuck into deceptive local optima(s). (In my method, the asset pre-selection is not necessary but you can still do that or give return estimates to introduce investor bias.) Here is an example QuantConnect back-test since 2009 for such a rebalanced portfolio that is found by my method. I actually want to integrate a reliable pre-selection method so that I can optimize larger universes of e.g. ten-thousand assets. What would be the best method for that? Hierarchical Risk Parity?

My optimizer quickly optimizes actively rebalanced portfolios of thousand assets on my laptop. You can't unfortunately do that with available optimizers without getting stuck into deceptive local optima(s). (In my method, the asset pre-selection is not necessary but you can still do that or give return estimates to introduce investor bias.) Here is an example QuantConnect back-test since 2009 for such a rebalanced portfolio that is found by my method without a pre-selection. I actually want to integrate a reliable pre-selection method so that I can optimize an universe of ten-thousand assets. What would be the best for that? Hierarchical Risk Parity?

I cant answer your question on pre-selection. However I want to ask you this, is this portfolio made with Markowitz Portfolio optimization or something else?

My optimizer quickly optimizes actively rebalanced portfolios of thousand assets on my laptop. You can't unfortunately do that with available optimizers without getting stuck into deceptive local optima(s). (In my method, the asset pre-selection is not necessary but you can still do that or give return estimates to introduce investor bias.) Here is an example QuantConnect back-test since 2009 for such a rebalanced portfolio that is found by my method without a pre-selection. I actually want to integrate a reliable pre-selection method so that I can optimize an universe of ten-thousand assets. What would be the best for that? Hierarchical Risk Parity?

I cant answer your question on pre-selection. However I want to ask you this, is this portfolio made with Markowitz Portfolio optimization or something else?

@sword134 I don't use methods from 1950. I develop ones for 2050.
I have done the stock pre-selection. Here is long-short 3000 stocks:

@sword134 I don't use methods from 1950. I develop ones for 2050.
I have done the stock pre-selection. Here is long-short 3000 stocks:

@kayuksel then why post on a library that focuses on 1950's methods?

@kayuksel then why post on a library that focuses on 1950's methods?

@robertmartin8 I think the original question has been answered. Do you think the example should be preserved somewhere (I think it has come up multiple times before).
@kayuksel has raised another question about pre-selection and HRP. If that question is still relevant, perhaps a new issue should be opened to keep things organized. My initial view would be that perhaps (methods of) HRP could indeed also be used for pre-selection, e.g. by only choosing only one asset from a very similar cluster, although I have never tried that empirically.

Hi. I just wanted to add that there is a very real use-case for this. Using gamma to set the number of assets is a painful/inefficient way of doing it.
For example- I'm interested in a married put option strategy for my portfolio. Because options contracts have a fixed quantity of 100 underlying securities, I would like to allocate the securities in my portfolio based on multiples of 100s.
It would be great if we could please include the asset number constraint as an enhancement.
Thank you so very much.

Hi @linda390, I am absolutely aware that the constraint on the number of assets is an important feature. Unfortuantely, I am constrained (no pun intended) by the fact that it is a difficult problem mathematically.
It's possible that in a future version, some variant of @phschiele's solution may be built in (e.g ef.add_num_assets_constraint), but a lot of thought needs to go into it to make sure it plays nice with other constraints/bounds.
@phschiele I agree this should be preserved somewhere. Perhaps an FAQ page on the documentation site?

Hi, Have anyone tried SCIP solver for cardinality constraint?
@phschiele @robertmartin8 @sword134

@sword134: as mentioned by @robertmartin8, the cardinality constraint you ask for is not convex. However, CXPY also supports MILP problems, if you have the required solver installed. For small problems with less than 1000 variables and constraints, you can use the community version of CPLEX: pip install cplex Consider this simple example for minimizing the portfolio volatility given a maximum of 10 securities to invest in, with weights between 0 and 1 for each asset. Slight modifications are required if you want to choose different bounds.
import cvxpy as cp
from tests.utilities_for_tests import setup_efficient_frontier
ef = setup_efficient_frontier(solver=cp.CPLEX)
booleans = cp.Variable(len(ef.tickers), boolean=True)
ef.add_constraint(lambda x: x <= booleans)
ef.add_constraint(lambda x: cp.sum(booleans) <= 10)
ef.min_volatility()
ef.weights
array([0.        , 0.01503686, 0.09891734, 0.12603716, 0.        ,
       0.        , 0.        , 0.17711377, 0.        , 0.11697861,
       0.17732243, 0.        , 0.        , 0.18942864, 0.        ,
       0.00072097, 0.        , 0.08792907, 0.        , 0.01051514])

Some remarks:

These types of problems are generally much harder to solve
You should consider Robert's comment about optimization vs. asset selection.

@phschiele  Is there a non-community version available for purchase? I can't find anything here: https://www.cvxpy.org/install/

@sword134: as mentioned by @robertmartin8, the cardinality constraint you ask for is not convex. However, CXPY also supports MILP problems, if you have the required solver installed. For small problems with less than 1000 variables and constraints, you can use the community version of CPLEX: pip install cplex Consider this simple example for minimizing the portfolio volatility given a maximum of 10 securities to invest in, with weights between 0 and 1 for each asset. Slight modifications are required if you want to choose different bounds.
import cvxpy as cp
from tests.utilities_for_tests import setup_efficient_frontier
ef = setup_efficient_frontier(solver=cp.CPLEX)
booleans = cp.Variable(len(ef.tickers), boolean=True)
ef.add_constraint(lambda x: x <= booleans)
ef.add_constraint(lambda x: cp.sum(booleans) <= 10)
ef.min_volatility()
ef.weights
array([0.        , 0.01503686, 0.09891734, 0.12603716, 0.        ,
       0.        , 0.        , 0.17711377, 0.        , 0.11697861,
       0.17732243, 0.        , 0.        , 0.18942864, 0.        ,
       0.00072097, 0.        , 0.08792907, 0.        , 0.01051514])

Some remarks:

These types of problems are generally much harder to solve
You should consider Robert's comment about optimization vs. asset selection.

@phschiele Is there a non-community version available for purchase? I can't find anything here: https://www.cvxpy.org/install/

@gowombats CVXPY is the modeling language, which is completely open-source. It supports multiple solvers, one of them is the above-mentioned CPLEX solver. There are also other solvers which would work. What's the issue that you are facing? Ideally, open a new issue for that, as it seems not directly related to this thread.


Delete Assignment

can now delete assignment
@Sumukh ready.

Please add permissions + permissions tests to (tests/unittests/test_assignment.py).
The user need here is delete. You'll have to add the check to the API and the actual permissions needed to delete in models.py

@Sumukh It looks like delete already has tests and permissions checks, so I went back - looks like the BaseAPI has a delete method. However, this is only accessible to admin. I overrode it and allowed staff to delete as well.

Did you make changes? I don't see any new commits.
Also the model will still have to be changed. The base API works with the HTTP delete method - not with a POST.

I just pushed, but wait, I'm calling a new delete method in the Assignment API. Am I supposed to call the default? The default only allows admin, and I don't want to change that.

I think we should relax the permissions then. The "owner" of an assignment
is the instructor for the course not an admin.
On Sunday, June 21, 2015, Alvin Wan<EMAIL_ADDRESS>wrote:

I just pushed, but wait, I'm calling a new delete method in the Assignment
API. Am I supposed to call the default? The default only allows admin, and
I don't want to change that.
—
Reply to this email directly or view it on GitHub
https://github.com/Cal-CS-61A-Staff/ok/pull/472#issuecomment-113959955.

--
Sent from Phone

Looking good.
You can actually get rid of the delete method (it's almost exactly the same as the BaseAPI method). The base method just defers to the models permissions. And now that the model accounts for delete permissions, we can just do that.
Then you'd just make the angular service submit a DELETE request instead of POST.
There should be a test to ensure an anon user cannot delete an assignment.

Ohh. That's convenient. I'll call the default then. And, I'll add in the user test as well.

@Sumukh I'm having some trouble. I'm using the Angular service's default delete method. However, server is responding "Method not allowed". Should I just leave the newly-added delete method?

Can you push up the code? I'll take a look
Do you have
        'delete': {
            'methods': set(['DELETE'])
        }

as a method in the methods list?

haha Yeah, I eventually figured out I needed that. x) Thanks

@Sumukh Ready to go. Added tests for anon users and all assignment views reoriented to course.

Looks good. Left a minor comment - feel free to fix that if you want and I'll merge once travis comes in.

Fixed. Why don't don't we use set literals though?

I'm not sure why it was done that way. That being said, I prefer using set([...]) rather than { }


Refactorizacion de clase Posicion, MapaLineal y Tablero para desarrol…

…lar Test

Codecov Report
Attention: 26 lines in your changes are missing coverage. Please review.

Comparison is base (cc20c0b) 5.61% compared to head (006c157) 5.05%.
Report is 8 commits behind head on main.

Files
Patch %
Lines

src/main/java/Entidades/Tablero/MapaLineal.java
0.00%
14 Missing :warning:

...c/main/java/Entidades/Obstaculos/FieraSalvaje.java
0.00%
3 Missing :warning:

src/main/java/Entidades/Jugadores/Gladiador.java
0.00%
2 Missing :warning:

src/main/java/Entidades/Obstaculos/Bacanal.java
0.00%
2 Missing :warning:

src/main/java/Entidades/Obstaculos/Piedra.java
0.00%
2 Missing :warning:

src/main/java/Entidades/Tablero/Casillero.java
0.00%
2 Missing :warning:

src/main/java/Entidades/Tablero/Posicion.java
0.00%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##              main      #6      +/-   ##
==========================================
- Coverage     5.61%   5.05%   -0.57%
  Complexity       5       5
==========================================
  Files           25      25
  Lines          178     198      +20
  Branches         1       3       +2
==========================================
  Hits            10      10
- Misses         168     188      +20

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Fixed Issue #6105

Fixed docstring on scipy/misc/common.py, solves ticket #6105.
Thank you.

Thanks @Newman101!

You're welcome, happy to help. =)


Fix Publisher JUnit output to Slave workspace

Fix:
When running on a Slave node, the Performance Publisher's JUnit output file would not be written to the (remote) workspace correctly.

Codecov Report

Merging #150 into master will increase coverage by 0.03%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #150      +/-   ##
==========================================
+ Coverage    68.3%   68.33%   +0.03%
==========================================
  Files          44       44
  Lines        3896     3894       -2
  Branches      678      677       -1
==========================================
  Hits         2661     2661
+ Misses        989      987       -2
  Partials      246      246

Impacted Files
Coverage Δ

...dson/plugins/performance/PerformancePublisher.java
83.82% <0%> (+0.25%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad06070...a6c8e20. Read the comment docs.

@undera Can you guys please merge this fix before the 3.4 release? Thanks


how do you use tensorflow gpu with this ?

Is it possible to utilise a gpu when processing text with this script ?
I have it working fine but it uses my CPU and I have very large chunks of text I want to process and it take to long with just the cpu.

Beofre anyone asks .. yes my other scripts use the gpu I have k80

Hi @skintflickz,
the code checks if a gpu is available and uses it automatically. Last time I checked this it worked fine.

I thought it uses tensorflow
Doh