build: Cross-compile to Scala 2.13

Related Issues/PRs

Builds on https://github.com/microsoft/SynapseML/pull/1771
What changes are proposed in this pull request?
Make cross-compilation to Scala 2.12 and 2.13 work.
How is this patch tested?
Using sbt test:compile && sbt ++2.13.10 test:compile without issues.
Does this PR change any dependencies?

[x] Yes. Make sure the dependencies are resolved correctly, and list changes here.
Shapeless was a transitive dependency before and resolved to 2.3.2.
This version does not have a Scala 2.13 build, so I added 2.3.4 explicitly which is the first version that does.

Does this PR add a new feature? If so, have you added samples on website?

[x] No. You can skip this section.

/azp run

Codecov Report

Merging #1772 (4b034b8) into master (7ab63a1) will decrease coverage by 63.80%.
The diff coverage is 17.50%.

@@             Coverage Diff             @@
##           master    #1772       +/-   ##
===========================================
- Coverage   86.55%   22.74%   -63.81%
===========================================
  Files         275      275
  Lines       14571    14573        +2
  Branches      782      756       -26
===========================================
- Hits        12612     3315     -9297
- Misses       1959    11258     +9299

Impacted Files
Coverage Δ

...e/synapse/ml/cognitive/vision/ComputerVision.scala
29.70% <0.00%> (-60.67%)
:arrow_down:

.../azure/synapse/ml/geospatial/AzureMapsTraits.scala
14.54% <ø> (-56.37%)
:arrow_down:

...oft/azure/synapse/ml/nn/BoundedPriorityQueue.scala
0.00% <ø> (ø)

...soft/azure/synapse/ml/automl/EvaluationUtils.scala
4.00% <0.00%> (-44.00%)
:arrow_down:

...ft/azure/synapse/ml/automl/HyperparamBuilder.scala
0.00% <0.00%> (-45.46%)
:arrow_down:

...ft/azure/synapse/ml/core/env/StreamUtilities.scala
11.11% <ø> (-74.08%)
:arrow_down:

...oft/azure/synapse/ml/explainers/ICEExplainer.scala
20.14% <0.00%> (-67.92%)
:arrow_down:

...t/azure/synapse/ml/explainers/KernelSHAPBase.scala
15.68% <0.00%> (-84.32%)
:arrow_down:

...crosoft/azure/synapse/ml/explainers/LIMEBase.scala
26.15% <0.00%> (-66.16%)
:arrow_down:

...soft/azure/synapse/ml/explainers/TabularLIME.scala
10.29% <0.00%> (-85.30%)
:arrow_down:

... and 264 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@nightscape thanks for this PR, synapse-internal tests are down right now (not your fault) but Style looks like something that can be fixed

@nightscape can we also meet at some point to discuss what this change means for the jar publishing foot print. If you reach out to<EMAIL_ADDRESS>we can set up a slot :)

Hi @mhamilton723, thanks for getting back so quickly!
I just sent a mail to the address you posted above.
Regarding your comment about improving the Style:
Are you talking about some code quality check in the PR validation that I can take as a starting point, or just to improve the style generally?
Some things in the PR read not so nicely because I needed to write them in a way that is compatible with 2.12 and 2.13, but maybe there's a way to make them nicer.

Hi @mhamilton723, thanks for getting back so quickly! I just sent a mail to the address you posted above. Regarding your comment about improving the Style: Are you talking about some code quality check in the PR validation that I can take as a starting point, or just to improve the style generally? Some things in the PR read not so nicely because I needed to write them in a way that is compatible with 2.12 and 2.13, but maybe there's a way to make them nicer.

I just meant the buid gate, you can run this locally with sbt scalastyle test:scalastyle

I looked at the explainers, exploratory and onnx packages, and the changes looked good to me.

@mhamilton723 I fixed the Scalastyle issue and rebased on the latest master.

I almost forgot this: We're still blocked by isolation-forest not being released for Scala 2.13:
https://github.com/linkedin/isolation-forest/pull/35

/azp run

I almost forgot this: We're still blocked by isolation-forest not being released for Scala 2.13: linkedin/isolation-forest#35

Just messaged the maintainer to ask him to take a look at this!

A Scala 2.13 version of isolation-forest has been released.
We're ready to proceed!

The error in the build is because of isolation forest, makign the changes i suggested should remove that blocker

@mhamilton723 I created a separate PR for the dependency update https://github.com/microsoft/SynapseML/pull/1776

Hey, folks. It seems that isolation-forest dependency has been upgraded in master separately. Can anyone please give this issue another try?


runtime: misleading error when reflect type names clash

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version devel +456a01a Wed Sep 28 16:20:41 2016 +0000 linux/amd64

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/mvdan/go"
GORACE=""
GOROOT="/home/mvdan/tip"
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build526272435=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?
Run https://github.com/mvdan/reflect-names (play.golang.org does not do since I need multiple packages to trigger this).
What did you expect to see?
An error message that made me understand what was wrong.
What did you see instead?
panic: interface conversion: interface is foo.Bar, not foo.Bar

The first solution that comes to mind is that if the two names clash, the full import path should be included. For example:
panic: interface conversion: interface is github.com/mvdan/reflect-names/p1/foo.Bar, not github.com/mvdan/reflect-names/p2/foo.Bar

Alternatively, to avoid unnecessarily long lines a common prefix could be stripped:
panic: interface conversion: interface is p1/foo.Bar, not p2/foo.Bar

I could come up with a patch for this, but first I want input on whether this makes sense at all. @quentinmit I assume that's what NeedsFix means?

It's fine to send a patch for this (see https://golang.org/doc/contribute.html).  Thanks.
Note that the compiler is supposed to already do this.  See the code in Sym.symfmt in cmd/compile/internal/gc/fmt.go.  I don't know why it is not working for your test case.

@ianlancetaylor thanks for the pointer. I've been playing around with debug prints. I assume you were referring to:
// If the name was used by multiple packages, display the full path,
if s.Pkg.Name != "" && numImport[s.Pkg.Name] > 1 {
        return fmt.Sprintf("%q.%s", s.Pkg.Path, s.Name)
}

But that code will not run when my error is being prepared - when the two _type.string() calls are made in the runtime - as Sym.symfmt sees fmtmode == FTypeId. To my understanding, it should be FErr, as the switch case clearly says case FErr: // This is for the user and the code that we want is in its body.
The way in which fmtmode is set is very confusing, and I haven't been able to track this down. This is also a dance between runtime and gc, two packages I've never touched before. Trying to tinker with this function at all mostly just results in cmd/compile breaking and me having to redo a full make.bash.
Do you have any pointers? Any idea why fmtmode could be wrong? Any way to debug this better than searching via prints?
Thanks!

@mvdan I'm sorry, I don't know the answers to your questions.  You could try asking on golang-dev.

@rsc you seem to have touched this code more than anyone else, any idea? I'd rather keep golang-dev as a last resource.

@mvdan I'm happy to help you look into fixing this.

Thanks @mdempsky! See https://github.com/golang/go/issues/17283#issuecomment-256935475. If that's going in the wrong direction, how would you go about figuring this out without breaking the compiler at every step?

Okay, familiarized myself a little with the code.
So to summarize the problem, we're storing short package-name-qualified names for types like "foo.Bar", which we use in runtime.TypeAssertionError panics (and a couple other places).  The problem is if we have two different package paths "x/foo" and "y/foo" that both use the package name "foo", the resulting error message is ambiguous.
I don't think simply using package path for ambiguous package names will work for a couple reasons:

We currently emit the "foo.Bar" strings when compiling the "x/foo" and "y/foo" packages, at which time we don't know whether "foo" will be ambiguous in the resulting program.
Because package names and package paths are separate namespaces (though in practice related), mixing the two can introduces their own ambiguities.

I suspect to really fix this, we would need to change TypeAssertionError to always use package path qualification, rather than package name qualification.

Sounds fair. If the path up to the package name ends up being the same, perhaps TypeAssertionError.Error() could omit that part from the output string, to keep the current behaviour when the types are not ambiguous.

That may work, but to reiterate: package name is not guaranteed to be a suffix of package path (or necessarily related at all).  For example https://godoc.org/google.golang.org/api/plus/v1 has package path "google.golang.org/api/plus/v1" but package name "plus".

Ah, right. Feel free to take over on the patch, as this seems non-trivial to fix.

I'm having a second crack at this, and finally made a bit of progress. Will send a work-in-progress CL soon, and will ask for pointers from there.

The Go compiler/runtime team discussed this and decided that we should change
interface is foo.Bar, not foo.Baz

to instead say
interface is foo.Bar (x/y/z), not foo.Baz (a/b/c)

I.e., if the package names are the same (even if the type name is different), then we include the package path afterwards.
Also that this isn't a blocking issue for 1.10.

@mdempsky thank you - that seems like a good decision. I'll update my CL shortly before the 1.11 tree opens to implement this.


TypeError: (0 , _reactStatic.cleanPath) is not a function

After upgrading to 5.3.4 I get this error when running yarn start or yarn build:
=> Exporting HTML...
TypeError: (0 , _reactStatic.cleanPath) is not a function
    at render (/Users/Trent/Projects/airship/airship/teamairship.com/dist/static.3a5b76ee.js:921:42)
    at Route.render (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/react-router/Route.js:124:13)
    at resolve (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/react-dom/cjs/react-dom-server.node.development.js:2149:18)
    at ReactDOMServerRenderer.render (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/react-dom/cjs/react-dom-server.node.development.js:2260:22)
    at ReactDOMServerRenderer.read (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/react-dom/cjs/react-dom-server.node.development.js:2234:19)
    at renderToString (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/react-dom/cjs/react-dom-server.node.development.js:2501:25)
    at renderToStringAndExtract (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/gitdocs/node_modules/react-static/lib/static.js:453:68)
    at _callee2$ (/Users/Trent/Projects/airship/airship/teamairship.com/static.config.js:46:18)
    at tryCatch (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/regenerator-runtime/runtime.js:62:40)
    at Generator.invoke [as _invoke] (/Users/Trent/Projects/airship/airship/teamairship.com/node_modules/regenerator-runtime/runtime.js:296:22)

Fixed in beta


[DEP0040] DeprecationWarning - punycode module is deprecated

See also: https://github.com/igorshubovych/markdownlint-cli/issues/437
I used NODE_OPTIONS='--trace-deprecation' to get this stack trace:
markdownlint-cli2 v0.10.0 (markdownlint v0.31.1)
(node:10616) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
    at node:punycode:3:9
    at BuiltinModule.compileForInternalLoader (node:internal/bootstrap/realm:392:7)
    at BuiltinModule.compileForPublicLoader (node:internal/bootstrap/realm:328:10)
    at loadBuiltinModule (node:internal/modules/helpers:101:7)
    at Module._load (node:internal/modules/cjs/loader:1001:17)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.<anonymous> (/opt/homebrew/Cellar/markdownlint-cli2/0.10.0/libexec/lib/node_modules/markdownlint-cli2/node_modules/markdown-it/lib/index.js:14:20)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)

It appears to be coming from markdown-it node_modules/markdown-it/lib/index.js via markdownlint.

Per my comment in the linked issue, please open an issue against that project.

Hi! Now that markdownlint v0.33.0 is released that includes a fix for this, could we have a new release of the cli that incorporates that version too? I can make a PR if that's useful?

Dependencies have already been updated, thanks.
https://github.com/DavidAnson/markdownlint/blob/main/doc/ReleaseProcess.md


Significant MessageBox and Dialog Improvements

This PR requires #586
It includes many fixes, mostly focused on improving the look, feel, and usability of Dialog and MessageBox
The following bugs should be closed by this PR:

Closes #299 - MessageBox now auto sizes
Closes #557 - MessageBoxes on small screens
Closes #432 - MessageBox does not deal with long text; width/height params are goofy
Closes #521 - MessageBox should take ustrings (BREAKING CHANGE)
Closes #35 - Dialog should have 1 char padding around edges
Closes #570 - Dialog should use computed layout for buttons
Closes #470 - UI Catalog: Add Dialogs Scenario

In addition we get these new features:

MessageBox.Query called with width and height == 0 get auto-size behavior. A new constructor is added making this easy to use.
Multi-line MessageBoxes are now supported. Just use \n to add lines. The height of the MessageBox will adjust automatically.
The MessageBoxes Scenario in UI Catalog provides a full demo/test-case.
Dialog called with width and height == 0 are sized to 85% container. A new constructor is added making this easy to use.
Dialog (and MessageBox Buttons are now dynamically laid out using Computed layout.
A Dialogs Scenario has been added to UI Catalog making it easy to test the API.
Button now supports BOTH specifying a hotkey with '' and the old behavior of using the first uppercase char (if '' is not found).
All UI Catalog scenarios that use Dialog or MessageBox now use the simplified API.
Terminal.Gui.dll now has version metadata and UI Catalog's about box displays it as a test case.
Button, Dialog, and MessageBox API documentation has been updated/revised.
View, Window, FrameView, and Dialog have been upgraded to use the new ConsoleDriver.DrawFrameWindow API directly.

This has to go in as part of my other big PR. @migueldeicaza : Please comment.

I am in awe at this work Charlie - thank you so much for this contribution.
I looked at the samples, and it looks like the only breaking change was the incoming parameters for the message box, we could add an API that ignores those parameters?
Did I miss something else?
I think this can go in, given that you have a mondo pull request with too many good enhancements, and we can later add a compatbility layer if someone complains.

I am in awe at this work Charlie - thank you so much for this contribution.
I looked at the samples, and it looks like the only breaking change was the incoming parameters for the message box, we could add an API that ignores those parameters?
Did I miss something else?
I think this can go in, given that you have a mondo pull request with too many good enhancements, and we can later add a compatbility layer if someone complains.

Miguel, I'm afraid we are going to need to do a 'breaking changes' release prior to 1.0 if we really want this project to "sing" long-term. There are enough deep bugs due to code churn/atrophy and inconsistencies (such as Frame v Bounds, weirdness around embedded ContentViews, and inconsistencies in how events are defined) that I really think we should bite the bullet and clean some stuff up. This will require breaking existing apps.
IMO, we should do this ONCE.
I think the build we are converging on (which I am calling 0.90) should be that ONCE.
The latest release notes show the breaking changes I've already identified (there may be more, but I've not done any testing to see).

That said, I'll ensure the old constructors still work on this one.

Now that Miguel's approved, I'm closing this PR as it will be incorporated in mondo #600


error nprinting-adsync

I have this error when i run the script
ADSyncDemoApp.exe -u ....... -p ....
Performing a full sync.
System.DirectoryServices.DirectoryServicesCOMException (0x80072030): There is no
such object on the server.
at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
at System.DirectoryServices.DirectoryEntry.Bind()
at System.DirectoryServices.DirectoryEntry.get_AdsObject()
at System.DirectoryServices.DirectorySearcher.FindAll(Boolean findMoreThanOne
)
at System.DirectoryServices.DirectorySearcher.FindAll()
at ADSyncCSharp.Service.FullSyncOperation.execute(String querySearchFilter, I
EntryProcessor`1 entryProcessor)
at ADSyncCSharp.Service.ActiveDirectorySyncService.FullSync()
at ADSyncDemoApp.Component.AdSync.Sync()
at ADSyncDemoApp.Program.Main(String[] args)

I check my Active Directory and Nprinting configuration in adsyncnprinting.config file , the syntax and the configuration is correct
Thank you for your help

What search filter have you configured? Have you checked if it is a valid Active Directory Query?

yes i checked, and i created group in my AD (the name is NPrinting)
Below my adsyncnprinting.config file :

I change the group name in AD to "NPrinting_NewsStand User" and i have the same problem !!!

Looking at the error it  seems the problem lays in the path to the domain controller
In adsyncnprinting.config these path are configured with the following properties:
rootDN = <root of the directory tree, e.g. DC=yourdomain,DC=com>
syncBaseDN = <pointer to the subtree containing all groups: e.g. OU=NPrinting,DC=yourdomain,DC=com>

yes i checked several times, but still the same problem, i run all the steps like the video in youtube
(Synchronize Active Directory to Qlik NPrinting - Qlik Tuesday Tips and Tricks)
And below my adsyncnprinting.config file :
add key="url" value="xxx.xx.xx.xxx:389" />
add key="domainName" value="mydomainname" />
add key="rootDN" value="OU=NPrinting,DC=mydomainname,DC=com" />
add key="syncBaseDN" value="DC=mydomainname,DC=com" />
add key="searchFilter" value="(&(objectCategory=group)(name=NP*))"/>
add key="searchDeletedObjectsFilter" value="(&(&(objectClass=group)(lastKnownParent={0}))(isDeleted=TRUE))"/>
add key="npUrl" value="https://mynprintingserver:4993" />

I deleted all the code in "adsyncnprinting.config" and I still have the same problem !!!
I think the problem do not come from this file ....
HELP ME...

now i have this error
Performing a full sync.
Processing Group: NPrinting_NewsStand User
API Error: Qlik.NPrinting.Rest.Client.Client.ApiException: Error calling RolesGe
t: {"message":"No HTTP resource was found that matches the request URI 'http://mynprintingserver:4996/api/v1/roles?rolename=NewsStand User'.","messageDetail":"No
route data was found for this request."}

Do you have NPrinting version 17.4 or later?

no, my version is <IP_ADDRESS>.
why, is working only with the 17.4 or later ?

these are new APIs added in version 17.4

I upgrade NPrinting and is working
Thank you

ok cool! Closing the issue then

@vus-qlik I suggest you add the requirement of supported NP versions in the README.md.

How to import users without group ?

Users can only be imported through Groups. You would have to define your logic to handle Users and  NP Roles.


[Feature] - Custom 404 Not Found page added

Fixes Issue #69

Changes proposed
tailwind.coffig.js - Added the background image of 404 page
Custom404.jsx - the file containing the 404 page

Screenshots

Note to reviewers
I have tested this on multiple places -- do perform ur check, if there is a problem, I am open to soleving it.

@Nishitbaria Kidnly review the PR

Thank you for your contribution @sambitsingha

@Nishitbaria Please add the gssoc and level label as it is important

@Nishitbaria Please label it


cbuild: invalid layer detection variables when pack is on a different drive

Describe the bug
cbuild setup detects valid layers from packs and writes the configuration variables to *.cbuild-idx.yml. However the variables are invalid when the pack is on a different drive than the project. Also an absolute path should be used when the relative path is not within the solution directory.
To Reproduce
Use an example which requires layer detection and put it on a different drive than the pack with the layer. Run cbuild setup *.csolution.yml and examine the configuration variables in *cbuild-idx.yml.
You can also use the following project and pack to reproduce this:
Project:
FileSystem.csolution.yml:

add under target-types:

type: H743
board: STMicroelectronics::STM32H743I-EVAL

File_Demo.cproject.yml:

remove provides: - CMSIS-RTOS2 (not needed with the upcoming CMSIS-Toolbox 2.5.0)

Pack:
STM32H743I-EVAL_BSP: install as local pack
Run cbuild setup FileSystem.csolution.yml
FileSystem.cbuild-idx.yml contains the following when the pack is on a different drive than the project:
  variables:
    - Board-Layer: $SolutionDir()$/
      description: STM32H743I-EVAL board setup for IoT
      file: Board.clayer.yml
      copy-to: Board/STM32H743I-EVAL

Board-Layer: is invalid and path is missing.
FileSystem.cbuild-idx.yml contains the following when the pack is on the same drive as the project:
  variables:
    - Board-Layer: $SolutionDir()$/../../../../Open-CMSIS-Pack/pack-examples/STM32H743I-EVAL_BSP/Layers/IoT/Board.clayer.yml
      description: STM32H743I-EVAL board setup for IoT
      path: ../../../../Open-CMSIS-Pack/pack-examples/STM32H743I-EVAL_BSP/Layers/IoT
      file: Board.clayer.yml
      copy-to: Board/STM32H743I-EVAL

Board-Layer: and path are correct however they use a relative path. An absolute path should be used when the relative path is not within the solution directory.
Expected behavior
Variables should be correct regardless if the pack is installed on a different drive than the project. Also an absolute path should be used when the relative path is not within the solution directory.
Environment:

[cbuild]
Version 2.4.0
OS: Windows

The issue I see here is that the output in the *.cbuild-idx.yml is intended to be copied into the CMSIS Solution project files.
The problem is that the CMSIS Solution project files must not use absolute paths (unless dynamically constructed by access sequences) for portability. Therefore it is "impossible" to create a reference to another drive.

Specification is here: https://github.com/Open-CMSIS-Pack/cmsis-toolbox/blob/main/docs/YML-CBuild-Format.md#configurations
@jkrech it should be possible to locate the pack content on a different (central) drive then the actual application. The path: must then include a drive specification, otherwise the IDE cannot copy the layer.
It is however acceptable to have no value for Board-Layer: in this situation.


swift version it 's not finish

ModernLookOSXSampleSwift.zip

ModernLookOSXSampleSwift.zip

It looks good, but what you expect from me?  I suggest to create a new repo for this, then I can extend the description adding a link and stating that there is a Swift version maintained by you.

ok I agree

here is the link
under construction
https://github.com/thierryH91200/ModernLookOSXSampleSwift

Thanks, README updated.


Docs - Adding a MIT Licenses badge and some text to Licenses section in readme

Is your feature request related to a problem? Please describe.
The readme file of the repo should have a Licenses badge and some text regarding licenses in the  Licenses section to make it more professional.

Describe the solution you'd like
adding a licenses section
Describe alternatives you've considered
No response
Additional context
Please assign me this issue under GSSoC'23
Code of Conduct

[X] I agree to follow this project's Code of Conduct

This issue is not considerable.
Closing it.


Create bubble_sort implementation in JavaScript

I can do this

Algorithms ->sort -> bubble_sort.
attach language name and time complexity to filename acc. to Contributing Guidelines


[3.5][Feature][Ready] Add orderLogic option to columns

Fixed merge conflicts from https://github.com/Laravel-Backpack/CRUD/pull/1178 .
It does apply the closure, this worked well for me for Articles:
            [  // Select
               'label' => "Category",
               'type' => 'select',
               'name' => 'category_id', // the db column for the foreign key
               'entity' => 'category', // the method that defines the relationship in your Model
               'attribute' => 'name', // foreign key attribute that is shown to user
               'orderable' => true,
               'orderLogic' => function ($query, $column, $columnDirection) {
                    return $query->leftJoin('categories', 'categories.id', '=', 'articles.select')
                        ->orderBy('categories.name', $columnDirection)->select('articles.*');
                }
            ],

For monsters:
            [  // Select
               'label' => "Category",
               'type' => 'select',
               'name' => 'select', // the db column for the foreign key
               'entity' => 'category', // the method that defines the relationship in your Model
               'attribute' => 'name', // foreign key attribute that is shown to user
               'orderable' => true,
               'orderLogic' => function ($query, $column, $columnDirection) {
                    return $query->leftJoin('categories', 'categories.id', '=', 'monsters.select')
                        ->orderBy('categories.name', $columnDirection)->select('monsters.*');
                }
            ],

Please notice:

select('original_entity_table_name.*') is also required in that custom query, so that the end result has a single ID (otherwise the buttons and details_row will stop working, because there are two IDs)
leftJoin() instead of join(), so that results without a category will show up;

Todo:

[x] add this feature to the columns docs

Hi, when using joins in orderLogic option, how to avoid ambiguousness for columns names?
For example if I have 'name' column in current table and also 'name' column in related table, after joining two tables, current's table value is replaced  by value of the column with the same name, from related table


Feature second refactoring

This pr (second refactoring) is going to include:

code cleanups
code improvements
documentation
code formatting

This version now works, thank you


inconsistent url when calling fetch mock with proxy setting

Hi there, I was using fetch-mock 7.2.2, typescript 3.1.3 and I have the following code
fetchMock.config = Object.assign(fetchMock.config, {
        Request: Request,
        Response: Response,
        fetch: fetch //fetch in typescript library
    });

fetchMock.get("/default", data.list_data);

fetchMock.get("/default/provinces", data.get_data);
fetchMock.get("/default/NBA", 404);

fetchMock.delete("https://mydictionary/default/provinces", 200);
fetchMock.delete("https://mydictionary/default/NBA", 404);

...// some other code here

it("", () => {
    await foo(); // foo calls fetch
});

I have a proxy set in package.json:
proxy: "https://mydictionary"
The issue I'm having right now is the following:

When doing a GET, it sends the request to "/default/..."
When doing a DELETE, it sends the request to "https://mydictionary/default/..."

Is this the expected behavior? I can provide more detail if needed.

fetch-mock doesn't send any request - it's your application that does that. It sounds to me that it's probably an issue in your code somewhere, but if you can provide a repo I can clone that reproduces the issue I'm happy to take a look

I think I figured this out. It was a dumb mistake I made. In foo(), I was fetching from "https://mydictionary/default/..." instead of just "/default/...", causing the issue above. Please close this issue, thank you very much.


variable-maxlength type

To have the benefits of both fixed & variable string:

A type is defined with a max length
The code generator generates similar to fixed-length type but also encodes the size as prefix to the data (similar to variable type).   Exception is raised if data being copied exceeds the specified length.

This doesn't force the client to perform any padding, the field can be random-accessed, the flyweight can query its size.    (The cost is a few unused bytes depending on variance of field data length)

A length accessor has been added to variable length strings/vardata. I've also provided convenience methods to get and set strings on vardata.
This additional support does not make sense for array data especially in the context of variable byte character encoding.


Update sbt-scalafix to 0.13.0

About this PR
📦 Updates ch.epfl.scala:sbt-scalafix from 0.12.1 to 0.13.0
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "ch.epfl.scala", artifactId = "sbt-scalafix" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "ch.epfl.scala", artifactId = "sbt-scalafix" }
}]

labels: sbt-plugin-update, early-semver-major, semver-spec-minor, commit-count:1

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 48.19%. Comparing base (263d117) to head (8d7ca1c).
Report is 2 commits behind head on develop.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #592   +/-   ##
========================================
  Coverage    48.19%   48.19%
========================================
  Files            5        5
  Lines           83       83
  Branches         6        6
========================================
  Hits            40       40
  Misses          43       43

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


getCurrentBSSID() returns incorrect macaddress

Hi,
The getBSSID() function returns incorrect bssids for some macaddresses.
Here's an example: the bssid of my network is:
Cirrent-personal a8:a6:68:d9:71:0b -31  11      Y  -- WPA(PSK/AES,TKIP/TKIP) WPA2(PSK/AES,TKIP/TKIP)
But when i connect to it and call getCurrentBSSID() i get back a8:a6:68:d9:71:b
Notice the last bit - the actual bssid has a 0b, and this just has b. When i convert the hex to decimal the values are quite different. Do you think this is a quick fix?
Thanks a lot!

This actually correct per se since it is a zero. Just different kinds of notations.

Facing same issue. In iOS pre zeros are deleted. i forked an android version which does bssid for android, in case of android it works fine


SBOM db image requires authentication for pulling

What happened:
The SBOM db image seems to require authentication to pull.
$ docker pull ghcr.io/cisco-open/kubeclarity-sbom-db:v2.0.0
Error response from daemon: Head https://ghcr.io/v2/cisco-open/kubeclarity-sbom-db/manifests/v2.0.0: unauthorized

What you expected to happen:
It should not require authentication.
How to reproduce it (as minimally and precisely as possible):
$ docker pull ghcr.io/cisco-open/kubeclarity-sbom-db:v2.0.0

Hi @phantasia15, thanks for reaching out, the images are now public. You are welcome to continue testing and feedback on the new release.


docs: nav section isn't expanded when using search

When searching the docs, when you navigate to a search result, the section of the navigation containing that page isn't expanded.
Also, when hitting a component page url directly, the section containing that page is expanded, but the page isn't highlighted.

Hey @sweeting , I can't seem to repro the error. I tried in FF, Chrome and Safari (see below). I used Enter key and mouse click to select the search result. Can you still make this happen?

@jaketrent It looks like this happens if any of the side nav sections have been manually expanded or collapsed.
Steps to reproduce:

Open docs site
Expand "Text" section
Search for "Button" and navigate to it from search results
Notice that "Text" section remains expanded and "Actions" is remains collapsed

I think the behavior should be when a page is navigated to via search results or by hitting a direct url (which may be the same thing), all other side nav sections are collapsed except the section that page is in and the nav item should be indicated as being active.
I noticed the item in the side nav isn't in the active state (blue line, higher contrast text) when you navigate to it via search results or by hitting the url directly. This happens regardless of having manually expanded/collapsed any sections.

Hey Brian, thanks for the sleuthing and extra detail.
Agreed:

Nav via search or direct url, should open drawer with relevant link
Nav link of page should have active styling

Less sure, because it's different than what we've code to date:

all other side nav sections are collapsed except the section that page is in

Today, instead, we track all open drawers and reopen them. We store this per session. We only do this for user-clicked drawers. Again (see "Agreed" above), it seems good to open the drawer on these other events, but I'm not sure about closing the rest. It seems like a non-intuitive case, where a user's drawers stay open through clicks but then get closed on search. I'd prefer we open any new drawer on search and left the user-clicked drawers open too. Seem more intuitive.
What are you thinking regarding this feedback?

@jaketrent I agree it would be better when searching that any sections expanded remained that way as long as the section that contains the page you choose from search results is also expanded. It looked as though navigating via search was causing a page refresh, so I wasn't sure if that had something to do with Algolia.
I'd love it if the side nav didn't repaint each time you navigated to a different page, that way you wouldn't have to scroll the side nav to browse between items in a section.

Compare that to the behavior of the side nav when navigating between pages in a section on material-ui.com:

@sweeting Check, we will keep user-clicked drawers open regardless.
Re the refresh. Yes, that's algolia. If you click from page to page, like the gif in material above, you will avoid the refresh in a similar way to material's site.

@jaketrent It may avoid the refresh, but it still scrolls back to the top each time you click on an item.

@sweeting Nice catch on the scroll. I'll catch that in this issue or capture in another.

Verified scroll restoration works in prod. Latest hypothesis on why would be that gatsby scroll restore was active in prod only and not local, so its values were overwriting ours. After I more completely diverged in our implementation from what was copied from gatsby, keys for sessionStorage weren't colliding.


Exception displaying an image from an asset

Steps to Reproduce

I am trying to display an image in a flutter app.
I have added an images directory and have a file in that directory called iu.png.
In pubspec.yaml I have:
assets:
- images/iu.png
in main.dart I have:

child: new Column(
    children: <Widget>[
        new Text('Image Demo'),
        new Image.asset('images/iu.png'),
    ],
),

When I run the code in either an Android or iOS simulator I get the following exception:
I/flutter ( 9252): ══╡ EXCEPTION CAUGHT BY IMAGE RESOURCE SERVICE ╞════════════════════════════════════════════════════ I/flutter ( 9252): The following assertion was thrown resolving an image codec: I/flutter ( 9252): Unable to load asset: images/iu.png I/flutter ( 9252): I/flutter ( 9252): When the exception was thrown, this was the stack: I/flutter ( 9252): #0 PlatformAssetBundle.load (package:flutter/src/services/asset_bundle.dart:221:7) I/flutter ( 9252): I/flutter ( 9252): #1
AssetBundleImageProvider._loadAsync (package:flutter/src/painting/image_provider.dart:427:44) I/flutter ( 9252): I/flutter ( 9252): #2
AssetBundleImageProvider.load (package:flutter/src/painting/image_provider.dart:412:14) I/flutter ( 9252): #3 ImageProvider.resolve.. (package:flutter/src/painting/image_provider.dart:266:86) I/flutter ( 9252): #4 ImageCache.putIfAbsent (package:flutter/src/painting/image_cache.dart:143:20) I/flutter ( 9252): #5 ImageProvider.resolve. (package:flutter/src/painting/image_provider.dart:266:63) I/flutter ( 9252): (elided 8 frames from package dart:async) I/flutter ( 9252): I/flutter ( 9252): Image provider: AssetImage(bundle: null, name: "images/iu.png") I/flutter ( 9252): Image key: AssetBundleImageKey(bundle: PlatformAssetBundle#54de0(), name: "images/iu.png", scale: I/flutter ( 9252): 1.0) I/flutter ( 9252): ════════════════════════════════════════════════════════════════════════════════════════════════════
I am using the latest flutter master and flutter doctor output is:
Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel master, v0.10.2-pre.21, on Mac OS X 10.13.6 17G65, locale en-GB) [✓] Android toolchain - develop for Android devices (Android SDK 28.0.3) [✓] iOS toolchain - develop for iOS devices (Xcode 10.0) [✓] Android Studio (version 3.2) [✓] IntelliJ IDEA Community Edition (version 2018.2.3) [✓] Connected device (2 available)
Anyone have any clues?
Logs

MBP:layout3 lol$ flutter analyze
Analyzing layout3...
No issues found! (ran in 2.1s)
MBP:layout3 lol$

MBP:layout3 lol$ flutter doctor -v
[✓] Flutter (Channel master, v0.10.2-pre.47, on Mac OS X 10.13.6 17G65, locale
    en-GB)
    • Flutter version 0.10.2-pre.47 at /Volumes/Data/dev/flutter
    • Framework revision 9dfc0f3aaa (11 hours ago), 2018-10-24 07:22:08 +0100
    • Engine revision 2586e94122
    • Dart version 2.1.0-dev.7.1.flutter-45f9462398

[✓] Android toolchain - develop for Android devices (Android SDK 28.0.3)
    • Android SDK at /Users/lol/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling
      support)
    • Platform android-28, build-tools 28.0.3
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_152-release-1136-b06)
    • All Android licenses accepted.

[✓] iOS toolchain - develop for iOS devices (Xcode 10.0)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.0, Build version 10A255
    • ios-deploy 2.0.0
    • CocoaPods version 1.5.3

[✓] Android Studio (version 3.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 29.1.1
    • Dart plugin version 181.5656
    • Java version OpenJDK Runtime Environment (build
      1.8.0_152-release-1136-b06)

[✓] IntelliJ IDEA Community Edition (version 2018.2.3)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 28.0.4
    • Dart plugin version 182.4323.44

[!] Connected device
    ! No devices available

! Doctor found issues in 1 category.

Please make sure your pubspec.yaml file looks like this. if you made a folder called assets and then under it a folder called images then your pubspec.yaml file should look like this.
assets:
    - assets/images/

Do care about the indentation here. That matters. For loading the images do something here like this
Image(image: AssetImage('assets/images/googlehome.jpg'),)

my folder structure is like below
assets
-images
--googlehome.jpg

Please consider asking support questions in one of the other channels listed at http://flutter.io/support .


Support multi-targeted libraries by decoupling .Net and EF Core versions

Summary of the issue
I'm having trouble integrating this library with a multi-targeted EF Core project.
My EF Core 3.1 code is in a data access class library that is configured with multiple target frameworks:
<TargetFrameworks>net5.0;netstandard2.0;net471;</TargetFrameworks>
That library is used by 3 different projects: a .Net 4.7 web application, a .Net Core 2.1 WebAPI (compatible with .Net Standard 2.0), and a .Net 5 console application.
This has been working fine and now I want to add caching, but it seems that the .Net version and EF Core version are tightly coupled in EFCoreSecondLevelCacheInterceptor.  I have read the explanations in the previous issues here and here and here, but I still believe that is not 100% correct.
EFCoreSecondLevelCacheInterceptor sees the net5.0 target and is forcing my class library to use EF Core 5.  While it's true that EF Core 5 requires .Net 5, the reverse is not true.  It says here:

.NET 5 and all future versions will continue to support .NET Standard 2.1 and earlier.  (emphasis mine)

This means that .Net 5 applications are still compatible with .Net Standard 2.0, and therefore work with EF Core 3.1 as well.  .Net 5 (and future) versions should not be coupled to specific versions of EF Core libraries.
(Side note: if you're wondering why this is needed, it's part of the migration path to .Net 5.  Eventually everything will be on the same version, but things get messy along the way.  This is supported by the EF Core team.)
Environment
.NET (Core) SDK version:  net5.0;netstandard2.0;net471;
Microsoft.EntityFrameworkCore version: 3.1.11
EFCoreSecondLevelCacheInterceptor version: 3.0.0
Database: MySQL
Cache provider: Redis

Example code/Steps to reproduce:
Create a multi-target class library:
Data.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net5.0;netstandard2.0</TargetFrameworks>
    [...]
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="EasyCaching.Redis" Version="1.1.0" />
    <PackageReference Include="EFCoreSecondLevelCacheInterceptor" Version="3.0.0" />
    <PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="3.2.4" />
    [...]
  </ItemGroup>
</Project>

Create two applications that reference that class library:
Net5App.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <ProjectReference Include="..\Data\Data.csproj" />
  </ItemGroup>
</Project>

NetCoreApp.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <ProjectReference Include="..\Data\Data.csproj" />
  </ItemGroup>
</Project>

Output:
Build error:
Error	NU1107	Version conflict detected for Microsoft.EntityFrameworkCore.Relational. Install/reference Microsoft.EntityFrameworkCore.Relational 5.0.0 directly to project Data to resolve this issue.
 Data -> EFCoreSecondLevelCacheInterceptor 3.0.0 -> Microsoft.EntityFrameworkCore.Relational (>= 5.0.0)
 Data -> Pomelo.EntityFrameworkCore.MySql 3.2.4 -> Microsoft.EntityFrameworkCore.Relational (>= 3.1.8 && < 5.0.0).

This library does support multiple target frameworks.

.Net 5 applications are still compatible with .Net Standard 2.0

No. .NET 5, is .NET 5. The last version of .Net Standard was 2.1 and now It's just 5 and nothing else. .Net Standard is dead for good.

.NET 5 and all future versions will continue to support .NET Standard 2.1 and earlier

This means you can use the older libraries with .NET 5 apps and not the opposite!
EF-Core is bound to specific .NET and .NET core versions and it's respected in this library. Open its NuGet page, select an specific version of it and then look at its base framework (dependencies). You should use an specific version of it for different frameworks.
Pomelo.EntityFrameworkCore.MySql 3.2.4 is compiled for EF-Core 3x only. You can't use it freely. It muse be bound to an specific version this way.

Thank you for the reply, apologies if my wording was confusing.   I am not trying to use EF Core 5 with an older framework, I know that will not work.  I am using an older library (EF Core 3.1) with a .Net 5 app as you say, and indeed my applications have been working fine for months until I tried to add EFCoreSecondLevelCacheInterceptor.
I suspect that this is the problem, it seems that it should read like this instead:
  <ItemGroup Condition="('$(TargetFramework)' == 'net5.0')">
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="[3.1,)" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="[3.1,)" />

It's true that EF Core 3.1 has a framework dependency on .Net Standard 2.0, but EF Core 3.1 is still compatible with .Net 5 applications, and should be allowed.  The way the condition is currently written prevents the transitive dependency from working correctly.

This library is not limiting you, Pomelo.EntityFrameworkCore.MySql 3.2.4 is restricting the usage of EF-Core:
 Microsoft.EntityFrameworkCore.Relational (>= 3.1.8 && < 5.0.0)

I'm using EF Core 3.1.11, which falls within that range.

Your posted error message means you are not limiting the boundaries of EF-Core correctly as I mentioned. If you don't bound it, the last version will be used.

Ok, I have tried your suggestion.  I added the boundaries to try to force it to use EF Core 3.1.11:
  <ItemGroup Condition=" ('$(TargetFramework)' == 'net5.0')">
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="[3.1.11,4)" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="[3.1.11,4)" />
  </ItemGroup>

Now I'm getting a different error:
Error	NU1605	Detected package downgrade: Microsoft.EntityFrameworkCore from 5.0.0 to 3.1.11. Reference the package directly from the project to select a different version.
 Net5App -> Data -> EFCoreSecondLevelCacheInterceptor 3.0.0 -> Microsoft.EntityFrameworkCore (>= 5.0.0)
 Net5App -> Microsoft.EntityFrameworkCore (>= 3.1.11 && < 4.0.0)

It looks like EFCoreSecondLevelCacheInterceptor is trying to force me to use EF Core 5 even when I limit the boundaries.
I have tried adding EF Core 3.1.11 directly to the Net5App project as suggested in the error (instead of pulling it in as a transitive dependency via the Data project) but it has the same error.
I've created a minimal sample project to reproduce the issue (using SqlServer instead of Pomelo.)  If you remove the cache interceptor package on this line then it compiles.

I can't change the lower bound of EF-Core 5x here (I can't use Version="[3.1,)" for it), because they have changed the signature of DbCommandInterceptor in EF-Core 5x and it's not compatible with EF-Core 3x anymore. They used Task<int> in v3x and now they are using ValueTasks.

Suppose you want to create a new .NET 5x based library and you want to use the EF-Core in it and also you want to support EF-Core 3x and EF-Core 5x at the same time in your library. Can you do that?
The answer is "no"! Because it's not possible to differentiate between different piece of your code based on the different versions of EF-Core (a given dependency). Compiler directives (#if/#else) can only be created based on the specific target frameworks (.NET5 here) and not based on the different versions of the specific dependency and … EF-Core 3x and 5x are not compatible at all. The old one uses Task based methods and the second one loves ValueTasks and uses it everywhere. Now our interceptor should support both Task based methods for EF-Core 3x and at the same time ValueTasks based methods for the EF-Core 5x code. Is it possible? No. Because compiler directives don't support such a scenario. I can't say #If EF-Core5x Then use ValueTasks. I have to say #If NET5 Then use only this specific dependency and I can't define a range of versions (select EF-Core>=3.1 for .NET5) here because again this range doesn't mean having a consistent and completely compatible libraries in it.
But … I can create lots of different EFCoreSecondLevelCacheInterceptor binaries with names such as EFCoreSecondLevelCacheInterceptor-for-NET5-and-EFCore5x or EFCoreSecondLevelCacheInterceptor-for-NET5-and-EFCore3x and so on for every and each available target framework and … this is not logical.

Thank you for the detailed explanation, I think I understand the problem much better now.  My only other thought was to use reflection to determine the EF Core package version, but it sounds like maybe that can't be done at runtime because that interceptor signature has changed.
I will try to restructure my projects to work around these issues, thanks for taking the time to explain the problems.  This package looks awesome and I'm excited to get it working in my project!


i18n: Update translation data from Transifex.

Generated by tools/i18n/sync-translations -b 6.x.
The changes in Arabic seem bugged; there's also a lot of line-wrapping changes that I think must be caused by some Transifex side server change?

Used the new scripts made available in #23988 to update this with the latest stuff from Transifex as a unit that should be independently reviewable and mergeable from that PR.

Added 7ef3c9bc0a, a backport of the unescaping script from #23988

Merged, thanks @klardotsh!


Transition testing framework to pytest

Pull Request Checklist
Thank you for taking the time to improve Arrow! Before submitting your pull request, please check all appropriate boxes:

[x] 🧪 Added tests for changed code.
[x] 🛠 All tests pass when run locally (run tox or make test to find out!).
[x] 📚 Updated documentation for changed code.
[x] ⏩ Code is up-to-date with the master branch.

If you have any questions about your code changes or any of the points above, please submit your questions along with the pull request and we will try our best to help!
Description of Changes
Closes: https://github.com/crsmithdev/arrow/issues/594
This is the first iteration of the transition to Pytest. We should ultimately move away from the x-unit style of setUp() functions to native Pytest fixtures, but this will do for now.

Codecov Report

Merging #751 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master   #751   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           9      9
  Lines        1669   1669
  Branches      282    282
=====================================
  Hits         1669   1669

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c856ab9...3d667ed. Read the comment docs.

Hey @jadchaar I was able to successfully clone your branch and run the new tests, everything passed fine.

Fantastic. I have locked the dependency versions like you requested.

Looks like the last version of pytest to support 2.7 is 4.6 (https://docs.pytest.org/en/latest/py27-py34-deprecation.html). 😞

@systemcatch I pushed a fix for this in the latest commit.

I don't have anything to add :). I'll go ahead and merge this.


FEATURE REQUEST: Option to limit number of entries in the different card types.

I love these cards. However they're very large. It would be nice to be able to limit the number of result entries in the different cards. For example on the results card I'd like to be able to only show the top 3 finishers. The same goes for the driver and constructor standings.

Hmm I think you are one of few that would like this feature. The whole point of the cards is to display the stats, and not part of the stats.
You could make the card smaller with card mod. Just hide the rows you don't want to display.
I will close this request, but if a few others request it I will realize this feature. So don't consider this as a 'won't do' but as a archive action.


Reversed camera tilt, part 2

The "reverse tilt" was first reported in #1970, and fixed in #2006, with one caveat:

There is still an issue in certain situations with tilting after using a shift-left drag to look. That will have to be fixed post-1.0.

I couldn't find an open issue for this, but shift-left-drag certainly causes problems.  In the animation below, I shift-drag the globe to the left, and then hold the middle-mouse and drag the mouse up and down.  The globe tilts left-and-right while the mouse goes up-and-down.
Then, I shift-drag the globe such that the center of the globe is above the center of the screen, the middle tilt-drag will be completely upside-down.

@emackey we have issue #2877 open, I added a comment to also see the details in this issue


Contrib guidelines

Additional instructions for beginner contributors.

This looks great! @Anselmoo also added a PR template, do you think anything else should be added here?

@davidbuniat That template looks good to me. Depending on the quality incoming PRs, I can also write a "Unit Testing Guidelines" too.


Installing in a server production

Hi,
When I try to run it Is necessary to use the php internal server? Because I get this error when log in, actually I'm using apache.

That error looks like the api directory is not writable. See #7 for some troubleshooting options.

All files and folders have 777 permissions

Is Apache setup to allow that directory to override settings? Something like:
<Directory /var/www/html/TaskBoard>
    AllowOverride All
    Options All
    Require all granted
</Directory>

Thanks that was the problem, I haven't added it :)


Integrity checks in snapshot load routines

Hello.
We need to add integrity checks to snapshot loader to ensure that an incorrectly composed snapshot wouldn't lead to memory corruption or other issues during the load procedure.

It should verify checksum and engine's version at least.

There is snapshot version field in current version of snapshot header that is checked upon load.

So, what about adding checksum?)

I think that this is good idea, but the checksum comparison wouldn't catch errors in case snapshot is intentionally malformed.

Such issues can be checked only with appropriate testing, i.e snapshot can successfully pass internal test-suite and test262.

@egavrin, could you, please, clarify what kind of issues do you mean?

There was an offline discussion on the issue and we've concluded that this is issue is about integrity checks that should guarantee that only snapshot images that could be successfully read in conformity to the current snapshot format, should be considered correct by snapshot loader, and all other snapshot images should lead to correct indication of inconformity by the loader, without any memory corruptions or produce of incorrect run-time structures etc.

@egavrin, @ruben-ayrapetyan, @zherczeg, can we close this issue?

I think we could think about this later for untrusted snapshots, but currently all snapshots are trusted. I close this for now, but it can be reopened later.


dartdev: Add empty project template

As I am currently working on an OpenAPI code generator I was looking for a way to create a barebones dart project without any boilerplate. Therefore I propose to add an empty template only adding a minimal configuration.

[X] I’ve reviewed the contributor guide and applied the relevant portions to this PR.

cc @kevmoo @mit-mit @bkonyi
It looks like we currently have:
          [cli]                  A command-line application with basic argument parsing.
          [console] (default)    A command-line application.
          [package]              A package containing shared Dart libraries.
          [server-shelf]         A server app using package:shelf.
          [web]                  A web app that uses only core Dart libraries.

When thinking about the project templates we offer, one of the main things that I would consider would be reducing new user confusion; reducing the questions we ask them that they don't yet have good context to answer. I think having fewer template options here is strictly better (even if each additional template would add some value in its own right). Above, I would actually strip out one (possibly two) options, and wouldn't add any new ones without an overwhelming value add.
(For the existing options above, we definitely don't need two command line starter templates, and its not clear that we need both a command-line and a package template).

Very Good CLI  already provides eight templates for users (those in Very Good Templates).
For what you're asking you can do:

Just wanted to bring up that there are community solutions for templates (might help support @devoncarew point). For example, Very Good CLI  already provides eight templates for users (those in Very Good Templates). It also has initiatives for creating a minimal templates.
In addition, there is also a whole ecosystem for templates available in BrickHub.
Overall, I'm not sure how many templates would the Dart Dev tool aim to support and maintain, and where to trace the line between what templates get in or not.

What is the status of this PR?

I think adding this particular template is reasonable, but moving forward we should avoid adding more templates without investigating our options in this area for the Dart and Flutter CLI tooling.

Hi @felixwoestmann there is one tiny change requested here; are you able to get that done?

Hello @mit-mit I fixed the copyright notice. Thanks for the reminder.


chore: surface errors, requeue, refactor

The initial idea in this PR was to surface the underlying errors that happen when r.Client.Get()ing an object from the API which is not found. So i.e. if we try to get a non-existing Secret we'll now get the actual error instead of a misleading api key is blank error.
Additionally, all Reconcile() functions now return ctrl.Request{}, err when a r.Client.Get() error occurs, thus making use of the retry mechanism which is built-in into the controller.
However, some of the strict pre-commit hooks started not passing because of a cyclomatic complexity of 21 in some Reconcile() functions, so I ultimately refactored a bunch of them. I basically moved the resource creation/update part out of Reconcile() and into a new writeNewRelicAlertCondition() function.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Looks good, confirmed is working as expected
level":"error","ts":1602783834.0332706,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"alertspolicy","request":"default/my-policy","error":"Secret \"nr-api-key\" not found","stacktrace":"github.com/go-logr/zapr...


Added Wordpress theme support

Added badges for Wordpress theme downloads (total) and theme rating.

Fixes #869.

No problem! Added validation using the 'querystring' library in accordance with other services. Also added tests for all 6 Wordpress integrations.

Also, test would not run without installing joi and minimist - not sure if that was intentional, but added them to dev-dependencies.

Thanks!
minimist should definitely be added to the dev dependencies. It's used directly by our test runner.
joi gets installed by icedfrisby and we should make sure only one version of it gets installed. If possible I'd rather avoid directly adding it until MarkHerhold/IcedFrisby#52 is resolved.
Which node / npm version are you using?

Removed joi. I'm using Node 6.9.2, npm 3.10.9

So strange! I was on Node 6 / npm 3 and didn't have the problem then, nor after I upgrade.
Thanks. I'll try to tackle MarkHerhold/IcedFrisby#52 so this can be resolved more satisfactorily.


Getting canvas usage data from Canvas analytics API

Fixes #25
Pulling Canvas usage data from the Canvas Analytics API

I am currently making a parallel call for pulling the data takes about ~5minutes(sequential calling take about 30 minutes) and generates about 500k records. I am deleting all the records and updating them with new data. Analytics API doesn't have a filter parameter by date for just query incremental approach. The deleting and insert 500k records went fine locally.
I am retrying for a 500 status code. I see a lot of 404 from the canvas when ran the process, what that tells me is the data I am requesting for a course doesn't exist in the data source canvas is pulling from. I am not going to retry for those. I am having my workers defaults to 8 since I don't feel I need to increase more than that. I didn't see any rate-limiting during the most used canvas time.

just as an fyi - here's the Canvas doc on what page views and participations mean in the context of the analytics: https://community.canvaslms.com/docs/DOC-16681-analytics-page-views-and-participations#jive_content_id_Overview
We'll pull the text from that for explaining what the data means.

_ means private access i.e in-class access only. You don't want the
instance of the class calling it outside of the class. It doesn't need to
be like that, I just chose it
On Fri, Apr 3, 2020 at 3:27 PM Sam Sciolla<EMAIL_ADDRESS>wrote:

@ssciolla commented on this pull request.
In canvas/canvas_course_usage.py
https://github.com/tl-its-umich-edu/course-inventory/pull/51#discussion_r403262424
:

   try:

       analytics_data = json.loads(response.result().text)

   except JSONDecodeError as e:

       logger.error(f"Error in parsing the response due to {e.msg}")

       return

   if not analytics_data:

       logger.info(f"Response for fetching canvas course usage is empty {analytics_data}")

       return

   new_dic['course_id'] = course_id

   new_dic['analytics'] = analytics_data

   self.canvas_usage_courses.append(new_dic)

def _get_canvas_course_views_participation_data(self, retry_courses=None):

Can you educate me on what the leading _ means? I've seen people do that
before, never understood it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tl-its-umich-edu/course-inventory/pull/51#pullrequestreview-387524065,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACBOVPZTWGU25IHNKWKWTB3RKY2AZANCNFSM4L2RVQNQ
.

I felt I really don't want to retry there if there is some data specific
error. Retying may not resolve it. I felt that is overusing of retry
mechanism
On Fri, Apr 3, 2020 at 3:34 PM Sam Sciolla<EMAIL_ADDRESS>wrote:

@ssciolla commented on this pull request.
In canvas/canvas_course_usage.py
https://github.com/tl-its-umich-edu/course-inventory/pull/51#discussion_r403267721
:

   try:

       analytics_data = json.loads(response.result().text)

   except JSONDecodeError as e:

       logger.error(f"Error in parsing the response due to {e.msg}")

       return

I think you could also put this in the re-try mechanism, right? Maybe use
a flag, something like what I did in my async PR with problem_encountered?
I think at some point we may want to refactor so this mechanism is the same
in all the classes we've written.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tl-its-umich-edu/course-inventory/pull/51#pullrequestreview-387528215,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACBOVP4EBP45AIG66KBJYL3RKY23NANCNFSM4L2RVQNQ
.

It is sending a tuple like<PHONE_NUMBER>,) so I just pulled the first one
On Fri, Apr 3, 2020 at 3:36 PM Sam Sciolla<EMAIL_ADDRESS>wrote:

@ssciolla commented on this pull request.
In canvas/canvas_course_usage.py
https://github.com/tl-its-umich-edu/course-inventory/pull/51#discussion_r403269222
:

   except JSONDecodeError as e:

       logger.error(f"Error in parsing the response due to {e.msg}")

       return

   if not analytics_data:

       logger.info(f"Response for fetching canvas course usage is empty {analytics_data}")

       return

   new_dic['course_id'] = course_id

   new_dic['analytics'] = analytics_data

   self.canvas_usage_courses.append(new_dic)

def _get_canvas_course_views_participation_data(self, retry_courses=None):

   logger.info("Starting of _get_canvas_course_views_participation_data call")

   with FuturesSession() as session:

       headers = {'Content-type': 'application/json', 'Authorization': 'Bearer ' + self.canvas_token[0]}

Can you explain what the indexing of canvas_token is doing here? It
worked in your other PR, but don't you want the whole token? I must be
missing something.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tl-its-umich-edu/course-inventory/pull/51#pullrequestreview-387529333,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACBOVP7BFHXEXAKFNAQU4WLRKY3C7ANCNFSM4L2RVQNQ
.

Done
On Fri, Apr 3, 2020 at 3:30 PM Sam Sciolla<EMAIL_ADDRESS>wrote:

@ssciolla commented on this pull request.
In canvas/canvas_course_usage.py
https://github.com/tl-its-umich-edu/course-inventory/pull/51#discussion_r403264839
:

   self.canvas_url = canvas_url,

   self.canvas_token = canvas_token,

   self.course_ids = course_ids

   self.retry_attempts = retry_attempts

   self.canvas_usage_courses = []

   self.course_retry_list = []

   self.retry_count = 0

def parsing_canvas_course_usage_data(self, response):

   logger.info("parsing_canvas_course_usage_data Call")

   new_dic = {}

   if response is None:

       logger.info(f"For Canvas Course usage response is None ")

       return

   logger.info(f"CanvasCourseUsage date collected so far : {len(self.canvas_usage_courses)}")

Do you mean data here?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tl-its-umich-edu/course-inventory/pull/51#pullrequestreview-387526001,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACBOVPZKMZ3IO3K5KT5EOTDRKY2MZANCNFSM4L2RVQNQ
.

Oops, that was not needed. Removed it
On Fri, Apr 3, 2020 at 3:35 PM Sam Sciolla<EMAIL_ADDRESS>wrote:

@ssciolla commented on this pull request.
In canvas/canvas_course_usage.py
https://github.com/tl-its-umich-edu/course-inventory/pull/51#discussion_r403268379
:

   if status != 200:

       logger.info(f'retry_count: {self.retry_count}')

       if self.retry_count < self.retry_attempts and status in status_codes_to_retry:

           logger.info("Append to retry list")

           self.course_retry_list.append(course_id)

       logger.info(f"Response not successful with status code {status} due to {response.result().text}")

       return

   try:

       analytics_data = json.loads(response.result().text)

   except JSONDecodeError as e:

       logger.error(f"Error in parsing the response due to {e.msg}")

       return

   if not analytics_data:

       logger.info(f"Response for fetching canvas course usage is empty {analytics_data}")

Do we need to log analytics_data here if it's empty?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tl-its-umich-edu/course-inventory/pull/51#pullrequestreview-387528677,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACBOVP65L77QPDLOB2FCFJTRKY26TANCNFSM4L2RVQNQ
.

I felt I really don't want to retry there if there is some data specific error. Retying may not resolve it. I felt that is overusing of retry mechanism

Okay, I'm asking because this is specifically something I DID want to re-try for in my other Canvas calls. Sometimes the response is just invalid JSON, but the next attempt is fine. See make_request_using_api_utils in inventory.py

Each time I retry I one try failed cases from last time
Round 1: Course1, Course2, Course3
Round 2: Course 2 ( try since it failed again), Course1, course3 (was able to get data from this try)
Round 3: Course 2 still has a problem, I exit at that time with this check if len(self.course_retry_list) != 0 and self.retry_count < self.retry_attempts:
make sense?

I have a commit for the canvas token indexing, which was annoying I did not even realize that I was declaring the canvas token/url as tuple
https://stackoverflow.com/questions/27654671/member-variable-string-gets-treated-as-tuple-in-python

For the Return None , I am type hinting for now. Thanks for more insights on that

@ssciolla , I think I have resolved your recent review suggestion as well

@ssciolla , I think I have resolved your recent review suggestion as well

Thanks!

I want to run this Branch on test instance before I merge it master. I just want to see how the MYSQL test instance handles 500K row of date delete and insert on the AWS Mysql instance


GitHubComment task doesn't accept Token connection type

Question, Bug, or Feature?
Type: Bug
Enter Task Name: GitHubCommentV0
list here (V# not needed):
https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/GitHubCommentV0
Environment

Server - Azure Pipelines
N/A
Agent - Private:
N/A

Issue Description
The GitHubComment task doesn't accept Token type of connection, although it reads the Token scheme in its source code. I believe it's a matter of updating input parameters restriction.
[Include task name(s), screenshots and any other relevant details]

Experiencing this as well

@ammohant @ds-ms can this be prioritised? This seems like an extremely simple fix. I'd suggest a PR myself, but I have no way of testing it

@ds-ms Any idea when this fix is going to be available to general public?

@ds-ms It's been three weeks now since the change was merged, but I still can't see it being deployed and available.


Miscellaneous base changes

⚠ PLEASE DO NOT DELETE THE TEXT BELOW ⚠
Pull request's description
Miscellaneous changes in the structure of the repo
Checklist before merging (please do not edit)
You must review you work before to submit the review to others.
Self-review of your content
Remember the content must be readable and understandable by someone else than yourself.

[x] From a technical point of view.
[x] From a grammatical point of view (spelling mistakes, typos, clarity, etc.), and following the guidelines at the bottom.

External reviews of your content

[x] You requested a technical review.
[ ] You requested a grammatical review.
[x] You validated with @K0rdan or @alexgeron-Luos that it is safe to merge.

Some guidelines to keep in mind

Colons (:), semi-colons (;), exclamation (!), and interrogation points (?) are not preceded by a space (like full stops and commas). E.g.: Colons!

File names and/or address are put in italic. E.g.: The file main.c.

Functions, variables, or more generally short codes are put between grave accents. E.g.: To obtain code(), type `code()`.

Long codes are put into blocks of code with three grave accent on each side, and the language's name:
```c
// Some C language
```

"Luos engine" has a upper case on the L for "Luos" and lower case on the e for "engine".

We call it "Luos engine" as a proper name, and not "the Luos engine".

Following that fashion, anything that's owned by "Luos engine" implies that we must use 's as the standard English rule to show the possessive case, e.g. "Luos engine's API".

The names pipe, gate, inspector, sniffer, app or application, driver, etc. have a lower case and can be refereed with the determiner the. E.g.: The inspector.

Following the latest changelog of Docusaurus : https://github.com/facebook/docusaurus/releases/tag/v2.0.0-beta.21

Vercel configuration had to be changed to run on Node 16.x as described by https://github.com/facebook/docusaurus/pull/7501
@lorenzi-nicolas you should have a look at : https://github.com/facebook/docusaurus/pull/7469 it could introduce usefull features for SEO


[release] 0.25.3

Reviewer instructions

Check the changelog to ensure the version increment is consistent with https://semver.org/.

If a different version should be released, follow the instructions.

Review the changeset as a sanity check.
Approve the PR, if it pases muster.
Merge this branch (do not squash or rebase), but DO NOT DELETE THE BRANCH.
After the release has been published to github, delete this branch.

Release notes
0.25.3 - 2022-05-19
Breaking changes

Introduce dedicated exit codes for type-checking errors, parser errors, and evaluation errors (e.g., due to unsupported language constructs or operators), see #1749

Features

Support sound records (over rows) in the model checker, see #1717

Bug fixes

Fix potential non-determinism when picking from [S -> T], see #1753

Codecov Report

Merging #1790 (e9a983e) into unstable (4f7cb4c) will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           unstable    #1790      +/-   ##
============================================
+ Coverage     76.83%   76.85%   +0.01%
============================================
  Files           383      383
  Lines         11826    11826
  Branches        561      567       +6
============================================
+ Hits           9087     9089       +2
+ Misses         2739     2737       -2

Impacted Files
Coverage Δ

...he/io/annotations/parser/CommentPreprocessor.scala
96.70% <0.00%> (+1.09%)
:arrow_up:

...a/at/forsyte/apalache/tla/lir/TlaLevelFinder.scala
100.00% <0.00%> (+2.63%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f7cb4c...e9a983e. Read the comment docs.


Deprecating CreatedByLabel in favor of OwnerReferences

What this PR does / why we need it:
We wish to replace CreatedByLabel, that virt-launcher is set with, with the a controller reference.
Unfortunately, we cannot do it right now due to backward compatibility issue (we might find running VMIs that are not set with a controller in their OwnerReferences).
This PR does the following:

Replace our implementations of GetControllerOf and IsControlledBy with those in Kubernetes.
Encapsulates the retrieval of the controller of virt-launcher in a way that it would be clear that it overrides that in Kubernetes. We first try to use the Kubernetes way and fall back to the existing logic that relies  on CreatedByLabel.
Deprecates CreatedByLabel (using a comment).

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1369
Special notes for your reviewer:
Release note:

None

/cc @rmohr

@ahadas on a first glance, I don't see issues in the code but the tests show that something is not right. Could you have a look? I would suggest to pick one or two of the tests which fail in all lanes, focus on them locally and check what's wrong ...
For instance this test:
[Fail] Storage Starting a VirtualMachineInstance With a HostDisk defined With 'DiskExists' type [It] Should use existing disk image and start
/root/go/src/kubevirt.io/kubevirt/tests/utils.go:1496

fails across all lanes.

@rmohr ack, thanks, looking

ci test please

ci test please

@rmohr good catch, you were right - the controller crashed due to improper handling in case of controllerRef==nil so I modified resolveControllerRef in virt-controller/watch/vmi.go to handle that correctly. the funtional tests look ok now when I run them locally, but seems there is some problem when they ran in jenkins. I'll trigger them again.

ci test please

@ahadas looks great!


Test flakes

Please provide a description for what this PR is for.
And to help us figure out who should review this PR, please
put an X in all the areas that this PR affects.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Pull Request Attributes
Please check any characteristics that apply to this pull request.
[ ] Does not have any changes that may affect Istio users.

/test unit-tests_istio

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

/test unit-tests_istio

/test unit-tests_istio

/test unit-tests_istio

/test unit-tests_istio


Apache blocks DP request with dirty headers

The first DP request contains a header with an empty value. This request will be blocked by Apache

see #366

Merged


How to inferrence with more than 1 arguments

Question
I have trainned and export a model from the pytorch tutorial(https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html), and the model definition is:
import torch.nn as nn
import torch.nn.functional as F
class TextSentiment(nn.Module):
    def __init__(self, vocab_size, embed_dim, num_class):
        super().__init__()
        self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)
        self.fc = nn.Linear(embed_dim, num_class)
        self.init_weights()

    def init_weights(self):
        initrange = 0.5
        self.embedding.weight.data.uniform_(-initrange, initrange)
        self.fc.weight.data.uniform_(-initrange, initrange)
        self.fc.bias.data.zero_()

    def forward(self, text, offsets):
        embedded = self.embedding(text, offsets)
        return self.fc(embedded)

but there are 2 paremeter in forward method, but the Predictor.predict in djl  only have one parameter.
how can i pass both input and offsets to the predict method?

When you build the NDList input to the model, you can have it with two NDArrays: one for the text and one for the offsets. The Predictor exists to convert between the numeric NDArrays and convenient Java classes. So, you could create a simple POJO or record to pass both the text and offset into the predictor and then the predictor preprocessing would convert them into the two NDArrays for your model.

When you build the NDList input to the model, you can have it with two NDArrays: one for the text and one for the offsets. The Predictor exists to convert between the numeric NDArrays and convenient Java classes. So, you could create a simple POJO or record to pass both the text and offset into the predictor and then the predictor preprocessing would convert them into the two NDArrays for your model.

@zachgk If I create a pojo like below, How  can I implement the Translator?
public class MyPOJO{
    private NDArray text;
    private NDArray offsets;

   // getter and setters
   // ....
}

@zachgk If I create a pojo like below, How  can I implement the Translator?
public class MyPOJO{
    private NDArray text;
    private NDArray offsets;

   // getter and setters
   // ....
}

our input is a NDList which can takes multiple NDArray
// assume you have created text and offsets NDArray
NDList inputs = new NDList(text, offsets);
predict.predict(inputs);

our input is a NDList which can takes multiple NDArray
// assume you have created text and offsets NDArray
NDList inputs = new NDList(text, offsets);
predict.predict(inputs);

It would be:
class MyTranslator {
  NDList processInput(TranslatorContext ctx, MyPOJO input) {
    return new NDList(input.getText(), input.getOffsets());
  }
}

Usually, you want to create the text and offsets NDArrays inside of the processInput method though. Then, you can use the ctx's NDManager which will automatically take care of the memory for you.

It would be:
class MyTranslator {
  NDList processInput(TranslatorContext ctx, MyPOJO input) {
    return new NDList(input.getText(), input.getOffsets());
  }
}

Usually, you want to create the text and offsets NDArrays inside of the processInput method though. Then, you can use the ctx's NDManager which will automatically take care of the memory for you.

@zachgk @stu1130 Thanks very much!  the inference works all right!

@zachgk @stu1130 Thanks very much!  the inference works all right!

@zachgk @stu1130 Thanks very much!  the inference works all right!

@zachgk @stu1130 Thanks very much!  the inference works all right!


playback takes a couple of seconds to start when the plugin is on, but processing hasn't happened.

update: cant repro anymore?


Sauce and fixes

@cscott It looks like this integration has stopped working: https://travis-ci.org/paulmillr/es6-shim/jobs/58485773 I'm not sure why. Can you look into it?


Some count values result on no color shown

Have an example where setting data to the below results in NO color showing up for the day with count of 5:

Results in this:

Interestingly, if the count value is > 7:

It seems to work correctly:

Also - without the first 2 days and just having the one with the 5 count, the coloring DOES appear to work properly.

Thanks for reporting the issue!
I will take a look, asap. Currently I'm just a bit swamped.

I found some time to work on the plugin and the issue should be resolved with the latest release now (v.1.2.0).


Can i finetune falcon-7b with 8gb Vram?

Hi, i want to ask if its possible to finetunefalcon-7b with 8gb Vram.

No, you need at least 48G RAM but it also depends on the size of your training dataset.

@thanhnew2001 so 48GB of RAM and 16GB of GPU do you think is it enough?

I was able to fine tune falcon-7b and stablelm-7b with Alpaca 50K on a GPU 24GB of RAM, but I had to make the micro_batch_size 1 and set bf16-true.  I could fine tune with both lora and adapter, but not adapter_v2, even when I set override_max_seq_length to 500.  I doubt you'll be able to fine tune either with only 16GB; the 7B parameters take ~14GB of GPU RAM as bfloat16, and there's plenty of other structures in addition to the fine tuning training dataset that takes up GPU memory.
There are some pretty smart folks who have managed to fine tune models with 4-bit floats (https://github.com/artidoro/qlora).  You may want to look into their work.

@thanhnew2001 Thanks!

@patrickhwood Thank you so much! I am also creating a google colab (in its free version) to finetuning falcon7b and other models

No, it didn’t work with that small memory. I tried many times  but yes if
some one else could provide their own experience, it would be great
Thanh
On Wed, 12 Jul 2023 at 05:41 Patrick Wood @.***> wrote:

I was able to fine tune falcon-7b and stablelm-7b with Alpaca 50K on a GPU
24GB of RAM, but I had to make the micro_batch_size 1 and set bf16-true. I
could fine tune with both lora and adapter, but not adapter_v2, even when I
set override_max_seq_length to 500. I doubt you'll be able to fine tune
either with only 16GB; the 7B parameters take ~14GB of GPU RAM as bfloat16,
and there's plenty of other structures in addition to the fine tuning
training dataset that takes up GPU memory.
There are some pretty smart folks who have managed to fine tune models
with 4-bit floats (https://github.com/artidoro/qlora). You may want to
look into their work.
—
Reply to this email directly, view it on GitHub
https://github.com/Lightning-AI/lit-gpt/issues/242#issuecomment-1631599936,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAY4GWFC5FFX3AWKYMSY5BLXPXJDLANCNFSM6AAAAAA2CFO4ME
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Best regards,
Thanh

qlora finetuning support is tracked in #176. I have a PR open that adds qlora inference support in #253

Well done for the inference support for qlora @carmocca ! :tada: . Do you know how hard is it to also add qlora for finetuning? Just initializing the model with quantization is not enough?
cf:
with fabric.init_module(empty_init=True), quantization(quantize):
        model = GPT(config)

Or do we need more change in the finetuning process itself?
I'm not super familiar with quantization things yet, else I would help.

That might be enough. I don't advertise that we support fine-tuning because I haven't played extensively with it to be confident. Feel free to give it a shot by adding the quantization argument to the script and instantiating the context manager as you said.
The QLoRA paper also uses "Paged Optimizers", which seem like CPU offloading. The paper says

While paged optimizers are critical to do 33B/65B
QLORA tuning on a single 24/48GB GPU, we do
not provide hard measurements for Paged Optimizers since the paging only occurs when processing
mini-batches with long sequence lengths, which is
rare.

So overall the pieces are there, somebody just needs to put them together

Thanks! Will try during the week. I saw somewhere you talked about a parameter called 'offload_cpu', but didn't saw it.

Well done for the inference support for qlora @carmocca ! tada . Do you know how hard is it to also add qlora for finetuning? Just initializing the model with quantization is not enough? cf:
with fabric.init_module(empty_init=True), quantization(quantize):
        model = GPT(config)

Or do we need more change in the finetuning process itself? I'm not super familiar with quantization things yet, else I would help.

I tried
with fabric.init_module(empty_init=True), quantization("bnb.int8"):
which worked fine after changing  "bf16-mixed" to "bf16-true" and micro_batch_size = 1.  The training and validation losses were just barely better than without quantization (still have to set "bf16-true"), although they were so close it could simply be statistical noise.  With quantization("bnb.nf4"), the training loss was worse, but the validation loss is very close:
bf16 training loss:
Min     1st Qu.   Median   Mean    3rd Qu.  Max     Std
0.40    2.15      2.50     2.54    2.91     4.95   0.526
int8 training loss:
Min     1st Qu.   Median   Mean    3rd Qu.  Max     Std
0.39    2.15      2.49     2.52    2.89     4.92   0.520
nf4 training loss:
Min     1st Qu.   Median   Mean    3rd Qu.  Max     Std
0.50    2.35      2.72     2.76    3.15     5.38   0.559
bf16 validation loss (sampled every 10 iterations):
Min     1st Qu.   Median   Mean    3rd Qu.  Max     Std
10.62   10.69     10.73    10.72   10.76    10.83   0.045
int8 validation loss (sampled every 10 iterations):
Min     1st Qu.   Median   Mean    3rd Qu.  Max     Std
10.59   10.66     10.70    10.69   10.73    10.80   0.046
nf4 validation loss (sampled every 10 iterations):
Min     1st Qu.   Median   Mean    3rd Qu.  Max     Std
10.68   10.74     10.78    10.77   10.81    10.87   0.043
This seems useful, so I'll submit a pull request with the quantization as a CLI option.


Tooltip sticks when exiting edit mode

Date is invalid tooltip sticks until you resize the browser. Tooltip should disappear when edit mode is exited.
Steps to reproduce:
-Select custom columns
-Turn on Editable
-Click pencil to edit row
-Click x to close edit mode

Working with this...


Make port queue an ordered set

Our port queue wound up having duplicates in it. We never want
duplicates in our port queue as this results in multiple processes
trying to use the same port leading to EADDRINUSE errors. Since unique
port entries is a requirement, this changes the port queue to be an
ordered set queue which, by design, will only contain unique port
values.

[approve ci autest]

[approve ci autest]

[approve ci autest]

Testing Notes:
I initially created this PR which changes the port queue to be a unique set as a draft PR. I included with that draft a reversion of the expansion of the port queue size (see #8292). Thus the initial port queue size was 1000 instead of 2000, as it was when we were seeing test failures. As a draft PR, I ran the autests for this PR four times and the tests passed all four times. I conclude that making the port container a unique set of ports fixes the issue with the recycled ports.
I'll now update this PR from a draft to a regular PR. In doing so, I'll remove the commit that reverted #8292 (the expansion of the initial port queue size) and make reverting the initial port queue size a separate PR.

[approve ci autest]

[approve ci autest]

This should port everywhere cleanly and will alleviate issues with port recycling for autest. Therefore I'm marking it for our supported releases.


Rethink build system

Today, we have buildAndTest script duplicated almost everywhere. We need to find a build method that would be more efficient to read/write

My suggest would be to use Fake, I could make a code proposition if that's right for you

@klettier Thanks for the idea. To be fairly honest, I've never seen it in action and I have no idea how it would fit in the actual layout. How long would it be for you provide a sample for the main library (Src/zipkin4net) for example ?

It depends of what you expect, do you want CSharp script files as described here or classic FSharp script files ?
In either cases we should implement those tasks

Clean
Build
Test
Pack

I will work on it today's evening and come back to you then.

Is there a big difference between both ? If not, I would rather stick with C# in order for everyone to understand it easily.
Otherwise, +1 on the tasks. Thanks a lot for your help and insight on this @klettier !

indeed great and thanks for trying impactful ideas @klettier

Here is my proposition for the main library and here is the travis related build.
---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target               Duration
------               --------
Clean                00:00:00.0119151
DotnetRestoreTools   00:00:10.7937422
DotnetBuild          00:00:12.0265212
DotnetTest           00:00:02.6369149
DotnetPackage        Failure
Total:               00:00:27.0637661
---------------------------------------------------------------------
Status:              Failure
---------------------------------------------------------------------

The build is in failing state because the pack for .NETFramework 4.5 could not be done on Linux but it will work well on Windows.
I tried to use CSharp script files but the Csi tool isn't available on Linux, FSharp is a concise language and it's pretty fun to use it :)
Please let me know if you have any question and if you want me to add the other projects as well.

@klettier Thanks a lot for this work! I think it's pretty easy to understand (even in F#).
Regarding the package target, we can make it optional in the compile run for the moment as it was not packaging with the current "tools". Can we still keep it to be able to run it manually ?
cc @crntn @gboucher90

Yes you can run it manually as well
You can pass any know target name to the script file and it will stop the pipeline to the one you provided.
if you don't provide any it will execute the default one that is specified at the end of the script.
You could even execute the script in fsharp interactive by selecting just the lines you want to execute.
Beauty of FSharp :)

Is the proposition suit you ?

@klettier Sorry the misunderstanding. Yes, the proposition suits me perfectly

Nice, I'll finish the set-up ASAP


Razor Strings from database or File structure

I was trying to implement pulling razor string from database or file structure. When i am running the template facing an error.
An unhandled exception of type 'RazorEngine.Templating.TemplateCompilationException' occurred in RazorEngine.dll
Can some one help me with this ?? . I went through all documentation even thought i am unable to get where i am missing.

[RazorTemplats.zip](https://github.com/Antaris/RazorEngine/files/619957/RazorTemplats.zip)

 var layoutText = File.ReadAllText(@"C:\Users\yelcdi01\Desktop\RazorTemplats\LayOut.cshtml");
var templateText = File.ReadAllText(@"C:\Users\yelcdi01\Desktop\RazorTemplats\BusinessReport.cshtml");
var service = RazorEngineService.Create();
service.AddTemplate("layout", layoutText);
service.AddTemplate("template", templateText);
service.Compile("template");
var result = service.Run("template");

Please try to enable http://antaris.github.io/RazorEngine/#Debugging and carefully read the message of the thrown exception.
If you still cannot figure out what's wrong please post the message of the exception.

I figured out what is the issue is. It is on my razor template.
Please close this issue.

Hi @vyelchuri,
I know is an issue from 2017 but i've the same problem...
Can you describe how to solve this?
Thanks in advanced!


NOAA C-CAP Example Notebook

added noaa-c-cap example  notebook
added dataset example notebook link to readme file

Thanks!

@TomAugspurger Is it helpful for me to update all the links in the README.md (quickstarts, datasets, tutorials)?

Yeah, I've let that README fall out of date. If you have a chance to link them that'd be great!


[vm/ffi] Test FFI with high heap

Currently native heap addresses are freely cast between void*, intptr_t and size_t. We need to make sure there are no surprises when the heap lives in the upper half of the address space.

There's also some interesting questions on 64-bit runtimes:
final p = Pointer<Int8>.fromAddress(0);
p[-1]; // Should be disallowed.
p[0x8000000000000000]; // Same here, since index is -9223372036854775808

final p2 = Pointer<Int8>.fromAddress(0x8000000000000000); // Should be allowed.

I don't see why p[-1] should be disallowed. I think the C interpretation should be (p.address + -1 * elementSize).load().
And the same for the rest. But, I intended this issue to be specifically about our handing of malloc returning large pointers. So it's obsolete after https://github.com/dart-lang/sdk/issues/38520


x/website: test page returns 500

What version of Go are you using (go version)?

$ go version
go version go1.17 darwin/amd64

(not applicable, this is a website issue)
Does this issue reproduce with the latest release?
Yes, this is a website issue
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/fgm/Library/Caches/go-build"
GOENV="/Users/fgm/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/fgm/src/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/fgm/src/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org|direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/fgm/src/go/src/github.com/golang/website/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/0g/p780bc554njc4qj110_8rmbr0000gn/T/go-build4101997795=/tmp/go-build -gno-record-gcc-switches -fno-common"

(irrelevant, this is a website issue)
What did you do?
Navigate to https://golang.org/test/
What did you expect to see?
A normal page
What did you see instead?
A 500 page with that content:
Error
template execution: template: site.tmpl:81:18: executing "site.tmpl" at <breadcrumb .URL>: error calling breadcrumb: template: site.tmpl:68:16: executing "breadcrumb" at <slice $elems 0 (sub (len $elems) 2)>: error calling slice: index out of range: -1

Bug apparently introduced when https://go-review.googlesource.com/c/website/+/339403 added use of {{breadcrum .URL}} in site.tmpl while moving it around.

CC @dmitshur

Thanks for reporting and bisecting @fgm.
CC @rsc as the author of CL 339403.


Feature/rr-2-jwt_tokens

Type of Change

[x] ✨ New feature (non-breaking change which adds functionality)
[ ] 🛠 Bug fix (non-breaking change which fixes an issue)
[ ] ❌ Breaking change (fix or feature that would cause existing functionality to change)
[ ] 🧹 Code refactor
[ ] ✅ Build configuration change
[ ] 📝 Documentation
[ ] 🇺🇸 Translation
[ ] 🗑 Chore

@MateuszBednarczyk The PR doesn't pass the lint check. Could you commit changes regarding golangci-lint?


updates documentation

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (234aaa6) 76.54% compared to head (7d64ba2) 76.54%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #21   +/-   ##
=======================================
  Coverage   76.54%   76.54%
=======================================
  Files           7        7
  Lines         584      584
=======================================
  Hits          447      447
  Misses        137      137

Flag
Coverage Δ

unittests
76.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


error: linking with link.exe failed: exit code: 1181

Hey, all
I'm following the getting started guide from diesel.
when i run the command
cargo install diesel_cli --no-default-features --features postgres
i get the following error code in the console
error: linking with link.exe failed: exit code: 1181
Does somebody has a solution for my problem?

That sounds like your linker is missing libpq as native library somewhere. See our CI setup for a working example.
I close this issue because this is nothing actionable for the diesel team.


Bug fixes

Added the verificationDelay parameter, which defines the time in milliseconds that must pass before the verification process starts.
Added health check for the explorer links before the verification process starts
Added BigInt.prototype.toJSON to the BigInt type to fix the issue with the BigInt type in the JSON.stringify function.

Readme for the new param?