Allow external override of variables without edit

I'm building this script into some automation, and forcing users to edit this script when these variables are different is super inconvenient.  It would be better to allow users to override them with environment variables.

I'm not willing to maintain my docker image, so feel free to either merge this or not.  I don't need these changes.


Add a validate-datasets command

After thinking about some ways to reduce cognitive load on users, I think we need an optional command that would allow users to verify that all of the fields in their databases are included in their dataset manifest files. This would then throw an automatic error when a user adds a dataset/collection/field but doesn't document it in their manifests
There are two possible options that I can think of for implementation here (please suggest other options as you think of them!):

user can pass a map of {fides_key: database_url} via the config/env vars
we could store these credentials (safely) within the API/db

I lean towards #1 from a design perspective, not to mention if we can avoid storing sensitive credentials in the db we should at all costs
Example 1 for embedding the validation values into the TOML file
[cli]
dataset_validation = {dataset_1 = "<database_url_1>", dataset_2 = "<database_url_2>"}

Example 2 for embedding the validation values into the TOML file
[datasets]
dataset_1 = "database_url_1"
dataset_2 = "database_url_2"

This is one of my personal favourites too - restated differently, I'd like a feature where fidesctl can tell me if my database schema is fully annotated by one (or more!) datasets!
My thoughts here in two parts: (1) the command itself, and (2) how to provide secrets.
CLI Command
Personally, I'd prefer to "flip" this around and instead of validate-dataset <dataset> <database> do something like:
fidesctl coverage-report <database_url> <dataset_key>

And have that return something like:
`postgresdb1` annotation coverage: 82%

The following fields do not have any data category annotations:

- postgresdb1.users.name
- postgresdb1.users.id
- postgresdb1.users.email
- ...(ie. <dataset_key>.<collection name>.<field name>)

...and have that return an error code if annotation coverage is not 100%.
Note that this makes it behave most similarly to generate-dataset in that the target of the command is the database, not the dataset. I also think it should be valid to have your database annotations split across multiple datasets (e.g. fidesctl coverage-report <database_url> <dataset_1> <dataset_2> <dataset_3> ...) but that feels like a v2 thing, because I'd want that to just implicitly work without specifying the dataset keys in the CLI, but we don't currently have a clear way to link databases <--> datasets today so we'd want to think about a couple options.
Secrets
In Fidesops, we use the connections API to configure persistent connections to a system (see https://ethyca.github.io/fidesops/guides/database_connectors/). Long-term I think a persistent server-side database connection makes the most sense, rather that injecting those secrets into the CLI at runtime. This is especially true in more realistic production environments where database secrets need to be tightly protected, so injecting them into CI jobs  may be infeasible.
However, I see no issue with having the CLI support the ability to read in a DB URL from the TOML/ENV too if that works for the user, and that also seems like a better place to start before we commit to a pattern for server-side secrets! So I'd happily see us try a simple version with the ENV variables like you mentioned, e.g.
[database_urls]
database_key_1 = "database_url_1"
database_key_2 = "database_url_2"

Note that, again, this is me wanting databases to be one-level above datasets. For a large production database you may have dozens of schemas and 100s of tables used for multiple applications, so you'd reasonably want to use datasets to declare subsets of that schema. A good example of this is DBT models - you have individual .yml files for each SQL model, all of which ladder up into a unified database schema in the end. So, architecturally, I think we have to be careful about assuming datasets will always map 1:1 to databases. I may need to explain this with a whiteboard at some point if I'm not making sense...

@NevilleS that's a great idea and i love the user experience of having a coverage check, the only thing i would do is add a flag that lets them choose a failure threshold instead of always failing below 100%
As for the dataset/database differentiation, I totally understand what you were describing and agree we should have the ability to slice it up

Cool - sounds good. Definitely agree that 100% coverage is strict so having a param for that makes sense 😁
So do you agree with:

The "flipping around" of the command to be "validate database is covered by dataset(s)" instead of "validate dataset matches database"?
The change in command name to "coverage-report" instead?
The need to collect coverage from multiple datasets? (Although we could make that a follow up enhancement, because I think it'd be confusing to have to explicitly tell the CLI command which datasets to look for...)

This is a great change, totally agree

Not self-descriptive enough, I'd go with something like database-coverage

For sure, multiple datasets per database is an inevitability, so we should plan for that. Even if it isn't implemented this time around. We just need to check that each field is mentioned at least, so here is my idea of the logic:

do a generated-dataset for the database in question. This gives us all of the fields in the database and shows what 100% coverage looks like
Count the number of fields (to use for our eventual percentage)
Compare the generated dataset to each user-defined dataset . If there is a matching field, remove it from the generated-dataset (again, we only care if it is mentioned at least once)
At the end, all of the leftover fields in the generated dataset are the ones that weren't described by any dataset.yml files

Does that make sense? Seem like a good implementation?


Doesn't work with recent MT

I know the main repo is now in minetest-mods, but for some reason I don't get to open issues there... so I'm dropping it here...
The problem is a change in the stairs.register_slab api. I'll attach a patch later.

Here is a trivial patch


Update index.md

Proposed changes

Unreleased project version (optional)

Related issues (optional)

I like it. :) Thanks, @scjane !


Thrift Service :  javadoc doesn't generate correct javadoc for methods returning void

    @SuppressWarnings("unused")
    protected static class Delete_response
            implements net.morimekta.providence.PUnion<Delete_response,Delete_response._Field>,
                       Comparable<Delete_response>,
                       java.io.Serializable,
                       net.morimekta.providence.serializer.rw.BinaryWriter {
        private final static long serialVersionUID = -5679378591674977061L;

        private final _Field tUnionField;

        private volatile int tHashCode;

        /**
         * @param value The union value
         * @return The created union.
         */
        public static Delete_response withSuccess() {
            return new _Builder().setSuccess().build();
        }

See above from example from a generate thrift service with a method having void as return type.

Fixed in 0.3.8.


Facebook Share count not working

Hey,
I noticed that my facebook wasn't working and I went to see what was happening. I've found out that the facebook API you are using was deprecated today, 18th August.
Can you take a look at this? Will it possible using the new API or not?
Regards!

Hello, @rrfranco92 ! Thank you for the report. Looks like facebook completely disabled their rest api. Now we can't get share counter from browser without access token. Currently i can't find the way how to get around this.
Also you can use graph api https://developers.facebook.com/docs/graph-api

Try this:
    function facebook(url, callback){

        $.ajax({
            type: 'GET',
            dataType: 'jsonp',
            url: 'http://graph.facebook.com?id='+url,
        })
        .done(function (data){callback(data.share.share_count)})
        .fail(function(){callback(0);})
    }

but apparently they squeeze a part of the result

Hello @ldo-tamedia ! Thanks for your comment. Yes your solution works, but it should not work :). Regarding documentation we should provide access_token, but we can see it works well without token.
And everything fine, we again found unofficial method to fetch share counters bypassing official graph API, but if you try to multiple refresh this url you can get error like this:
/**/ callback({
   "error": {
      "message": "(#4) Application request limit reached",
      "type": "OAuthException",
      "is_transient": true,
      "code": 4,
      "fbtrace_id": "DfFpebWbEsl"
   }
});

Because they have limits requests per user IP.
But anyway I decided to add this method, be careful with it and be ready that not every user can see this counter... Also facebook can disable this hack at any time.


switch to using eodash/eodash_catalog

all

This PR integrates the eodash/eodash_catalog tool to generate the catalog to avoid double development.
I've manually tested all examples of different endpoints, I could think of, but let's try to wait for the all pipeline to finish (at least the generation part) before merging.


Type of variable erroneously narrowed to 'never'

TypeScript Version:  ~3.5.3
Code
import { Component } from "@angular/core";

export interface Abc {
  name: string;
}

export interface Abcd {
  toto: string;
}

@Component({
  selector: "my-app",
  templateUrl: "./app.component.html",
  styleUrls: ["./app.component.css"]
})

export class AppComponent {
  name = "Angular 6";
  a: Abc | Abcd;

  constructor() {
    this.a = {
      name: "sss"
    };
    console.log(typeof this.a, "---");
    console.log(this.instanceOfA(this.a));
    if (this.instanceOfA(this.a)) {
      console.log(this.a.name);
    } else {
      console.log(this.a.toto); // Why is this.a type marked as 'never' here? Should be 'Abcd', right?
    }
  }

  instanceOfA(object: any): object is Abc {
    return "name" in object;
  }
}

Expected behavior:
Type of 'this.a' variable is narrowed to 'Abc ' inside the if statement and to 'Abcd' inside the else statement.
Actual behavior:
Type of 'this.a' variable is narrowed to 'Abc|Abcd ' inside the if statement and to 'never' inside the else statement, causing runtime error "TS2339: Property 'toto' does not exist on type 'never'."
Playground Link:
https://stackblitz.com/edit/angular-cluhyc?file=src%2Fapp%2Fapp.component.ts
Related Issues:
https://github.com/microsoft/TypeScript/issues/27103

The control flow analysis can see that this.a is never re-assigned again within the function, so the compiler ignores your type annotation of Abc | Abcd, assuming the type is always Abc (because that's what the compiler sees).
If you adjust your code you will see the behavior you expect:
this.a = Math.random() > 0.5 ? { name: "sss" } : { toto: 'foo' };

This is not something I can change in the actual code, here I used a simplified code to showcase my problem, but in the actual code "this.a" is affected somewhere else and it can actually contain both values, the behavior is still the wrong one though.

but in the actual code "this.a" is affected somewhere else

In that case you're affected by #9998.
Your workaround is to use a type assertion to specify the type:
this.a = { name: "sss" } as Abc | Abcd;

Or with slightly less redundancy:
this.a = { name: "sss" } as this['a']

Okay so I thought the point of "object is Abc" was precisely to avoid having to specify manually "as Abc" whenver I want to use my variable. (variable that may at some point be instanciated from JSON representation, which is why I cannot always explicitly specify the type of it using "as Abc" during instancitaion).

Okay so I thought the point of "object is Abc" was precisely to avoid having to specify manually "as Abc" whenver I want to use my variable.

You misunderstood me. I said as Abc | Abcd, not as Abc. Then still use your instanceOfA check.
You're running into an edge case where the control flow analysis narrows the type of your variable because it's unaware of the modification within other functions that you call. As a result the type of your this.a variable is always inferred to be Abc (ignoring your explicit type annotation), while it actually should be Abc | Abcd (due to the side-effects in your other functions).

Hmmm then I bet I will have to

Here's a simplified example without all that Angular crap (sorry, not sorry :-P):
type A = { a: string }
type B = { b: string }

function isA(value: A | B): value is A { return 'a' in value }
function modifyValue(): void { value = { b: 'b' } }

let value: A | B = { a: 'a' }
modifyValue()

// Despite the type annotation "A | B" on value the type is inferred to be "A",
// because the control-flow analysis does not see another assignment to the variable.
// It is unaware of the modification within the "modifyValue()" function (issue #9998).
if (isA(value)) { // value is of type A.
  value
} else {
  // Because according to the type checker the type of value can only ever be "A",
  // and we checked for "A" already in the if-branch, the type now is narrowed to "never".
  value
}

// To fix it you could use a type assertion to widen the type again:
let value2: A | B = { a: 'a' } as A | B
if (isA(value2)) { // value2 is now of type A | B.
  value
} else {
  value
}

Okay so thanks to your example I was able to summarize even better what problem I am actually facing, which is:
type A = { a: string }
type B = { b: string }

function isA(value: A | B): value is A { return 'a' in value }
function modifyValue(): void { value = { a: 'a' } }

let value: A | B
modifyValue()

if (isA(value)) {
  value
} else {
  value // here type is 'never'
}

The problem is, this only happen in the actual code, the example does not have this issue if I update the code on stackblitz... Could this have been fixed in some later angular version?

In your updated example you don't assign any value to value. So the compiler does not make the assumption of "it's always A" anymore.

Yes that's how it is in the actual code, yet I am still facing the erroneous 'never' type in the else in the actual code, while the updated example works fine...

Yes that's how it is in the actual code

This is not how it is, neither in the code of your first post or in your Stackblitz: https://stackblitz.com/edit/angular-cluhyc?file=src%2Fapp%2Fapp.component.ts
In line 20 you assign a value to the variable. So within that function the compiler assumes the value is always of the type of your assigned value.
My example adjusted would be:
let value: A | B

function doStuff() {
  value = { a: 'a' }
  modifyValue()

  if (isA(value)) {
    value
  } else {
    value // here type is 'never'
  }
}

This is not how it is, neither in the code of your first post or in your Stackblitz: https://stackblitz.com/edit/angular-cluhyc?file=src%2Fapp%2Fapp.component.ts

You are right, what I mean is: the original example was an inaccurate representation of what is actually happenning in my application. Which is why I posted an updated code sample here: https://github.com/microsoft/TypeScript/issues/41270#issuecomment-717211986

Should I update the main code with this?

:shrug: You should update the example. Because the code from https://github.com/microsoft/TypeScript/issues/41270#issuecomment-717211986 does not show the behavior of narrowing to never. Not in the TypeScript playground, nor in the Stackblitz of yours (copying over the code).
Without seeing the actual code or a matching example I can just guess that your problem is a combination of the two issues I mentioned earlier.


Bevy still crashes when mesh attributes are different lengths

Bevy version
main
Relevant system information
AdapterInfo { name: "Apple M1 Max", vendor: 0, device: 0, device_type: IntegratedGpu, driver: "", driver_info: "", backend: Metal }
SystemInfo { os: "MacOS 13.5.1 ", kernel: "22.6.0", cpu: "Apple M1 Max", core_count: "10", memory: "64.0 GiB" }

What you did
Add one more UV in the generate_custom_mesh example and cargo run --example generate_custom_mesh.
diff --git a/examples/3d/generate_custom_mesh.rs b/examples/3d/generate_custom_mesh.rs
index a4aa02502..51b0fae24 100644
--- a/examples/3d/generate_custom_mesh.rs
+++ b/examples/3d/generate_custom_mesh.rs
@@ -181,6 +181,8 @@ fn create_cube_mesh() -> Mesh {
             [0.0, 0.45], [0.0, 0.2], [1.0, 0.2], [1.0, 0.45],
             // Assigning the UV coords for the forward side.
             [0.0, 0.45], [0.0, 0.2], [1.0, 0.2], [1.0, 0.45],
+            // extra
+            [0.0, 0.0]
         ],
     );

What went wrong
Warning is emitted, but Bevy still crashes.
The warning seems to imply that Bevy should not be crashing.
2023-10-25T20:12:41.865953Z  WARN bevy_render::mesh::mesh: MeshVertexAttributeId(2) has a different vertex count (28) than other attributes (24) in this mesh, all attributes will be truncated to match the smallest.

thread 'Compute Task Pool (0)' panicked at crates/bevy_render/src/mesh/mesh/mod.rs:370:46:
range end index 800 out of range for slice of length 768
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_render::render_asset::prepare_assets<bevy_render::mesh::mesh::Mesh>`!
thread 'Compute Task Pool (3)' panicked at crates/bevy_render/src/pipelined_rendering.rs:145:45:
called `Result::unwrap()` on an `Err` value: RecvError

Additional information
I attempted to bisect this, but I see the same crash even with #9699 where the warning was introduced.
First noticed here: https://github.com/bevyengine/bevy/pull/10259#pullrequestreview-1698045156.

I think the issue is here in get_vertex_buffer_data():
for attribute_data in self.attributes.values().take(vertex_count) { // <<<<
 // <snip>

This limits the number of attribute buffers that will be iterated over, not the number of values in the attribute itself.
The inner loop over chunks of data from the attribute buffer should have the .take(vertex_count).
- for (vertex_index, attribute_bytes) in
-                attributes_bytes.chunks_exact(attribute_size).enumerate()
+ for (vertex_index, attribute_bytes) in
+                attributes_bytes.chunks_exact(attribute_size).take(vertex_count).enumerate()

Quick testing shows that moving the .take() into the inner loop prevents the crash.


Docs: fix index

This is generating from the README.md file: https://deploy-preview-2218--devel-docs-dagger-io.netlify.app/

So weird! I don't understand docusaurus at all 😅 I'm just trying to change the current behavior which causes docs.dagger.io to land on getting started instead of introduction.

Ah! I think I got it to work.


Bug: Permissions ignored in Promotions query

What are you trying to achieve?
Using a token from my admin account to query promotions just like i query other protected stuff yields the "To access this path, you need one of the following permissions: MANAGE_DISCOUNTS", error. I verified through the UI that I have the permissions on my account.
Steps to reproduce the problem
Create fresh saleor instance w version 3.20, only things i changed was i added new shipping zone and channel. Its supposed to work regardless
What did you expect to happen?
That i can manage the discounts when i have the proper permissions ....
Logs
No response
Environment
Saleor version: 3.20, same w dashboard
OS and version: …

Checked this with 3.20, and I am getting a correct results: user with manage_discounts is able to fetch promotions

User without manage_discounts receives permission error:

Make sure, that you're using a correct token, or provide more details like:
	me{
		userPermissions{
			code
		}
		permissionGroups{
			permissions{
				code
			}
		}
	}

Some screenshots, and details of what type of auth layer you're using - is it built-in one? OIDCPlugin with SSO/without SSO?

I will close the issue for now as it seems this is actually not a bug. Feel free to reopen if it keeps happening 🙌


Update b4 submission space

confirmed:B4
address:39WxKwMicf2tXFeXfGmQijomVdVa5VdJeJo4gCybMCaQUQ1Bye
baker:186

confirmed:B4
address:39WxKwMicf2tXFeXfGmQijomVdVa5VdJeJo4gCybMCaQUQ1Bye
baker:186

Form submitted at 17:50 23/04/2022 CEST


incomplete response from complete

I'm trying to diagnose a SH freeze that happens when invoking the complete command. However, the command line version is hosed:
% stack --stack-yaml=stack-ghc-8.2.2.yaml exec -- hsdev complete --file ~/play/emuwork/src-emubase/Machine/Types.hs
Missing:

Usage: hsdev complete [-w|--wide] (-f|--file path)
  show completions for input

Currently, complete will cause SH to freeze for 5 minutes (the default timeout for hsdev responses). Would like to figure out why. The JSON is similar to the command line:
HsDevClient.call[6] cmd 'complete' opts
{'command': 'complete',
 'file': '/Users/me/play/emuwork/src-emubase/Machine/Types.hs',
 'id': '6',
 'no-file': True,
 'prefix': '',
 'wide': False}

(BTW: I propagated the new and changed hsdev backend methods to the NoneBackend and the HaskellBackend classes. Really should consider running pylint after you hack on SH. You would be surprised at the number of people who actually use the "none" backend, which makes its functionality very important. As important as hsdev. :-)

The problem manifests itself as a short read on the SH client -- SH sees an incomplete (short) response. Somewhat wondering if the socket needs to be forcibly flushed. I'll experiment and report back.

However, the command line version is hosed
complete command expects prefix, but you can pass empty one:

hsdev complete "" --file <file>

Yes, it's slow. I'll try to figure out, what's wrong

@mvoidex: I've captured a packet trace between SH and hsdev and confirmed that complete does send an incomplete response. I don't think the problem is hsdev itself because wireshark does show that hsdev stops sending data, with subsequent "invalid file descriptor messages" in stderr. That would lead me to conclude that the network-<IP_ADDRESS> package is reaping socket/file descriptors before their time or that the sendAll function is misimplemented or something else funky is going on in encode.
Doesn't matter if hsdev is built with 8.0.2 or 8.2.2.

Is it fixed with #87?

Unable to stall hsdev... so far.

Try to install from master with stack.yaml
Or from ghc-8.4.2 with stack.yaml or nightly.yaml


Fix 12/24h format on event list due date-time by using an 5.1.1 Andro…

…id version of the DateUtils method. #396

Fixed it as discussed on chat.
I just added the old version of the method as a private method of DateFormatter as I assume it's best to keep every date-time formatting code there until Time is removed form the project. We can refactor after that.
Btw, found the bug on the official tracker: https://issuetracker.google.com/issues/37127319

How about using the workaround only if we have to. I.e. detect whether the device 24h/12h setting is different from the current locale 24h/12h setting and use the original method if it's the same?
It's probably best if we use workarounds only to the extend we have to and stick to the original methods otherwise.

I've updated it as suggested, so only use old version when locale has AM/PM by default. Ready for check again.


Add case macro for switch-case constructs

Pull Request Test Coverage Report for Build 757370861

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.237%

Totals

Change from base Build 757105199:
0.0%

Covered Lines:
33436

Relevant Lines:
48292

💛  - Coveralls

Nice!


Advertising not working?

Thank you very much for this great Project!
I've successfully compiled the Project on NET 3.5 Client Profile Settings and the Receiver works great. I can detect any beacons near my Bluetooth dongle. I've started the Advertising as a beacon as well (just like your sample code), but I can't receive anything on my other devices (like an android phone). The same dongle worked fine under Linux, I was able to use it as a Beacon together with my android phone successfully.
Do you have any hints that you can give me, why the transmit doesn't work?
using (var hub = new BeaconHub(0x0A12, 0x0001))
{
    hub.BeaconDetected += (sender, e) => Console.WriteLine(e.Beacon.ToString());
    hub.EnableAdvertising(new Beacon("B9407F30-F5F8-466E-AFF9-25556B57FE6D", 1000, 2000, -52));
    Console.ReadKey();
}

With further testing, I've hard-encoded the hcitool-command that I was using under Linux in the program:
public void EnableAdvertising(byte[] advertisementData)
        {
            byte[] DEBUGadvertisementData = { 02, 01, 26, 26, 255, 76, 00, 02, 21, 226, 10, 57, 244, 115, 245, 75, 196, 161, 47, 23, 209, 173, 07, 169, 97, 00, 00, 00, 00, 200, 00};
            SendCommand(new LeSetAdvertisingDataCommand(DEBUGadvertisementData));
            SendCommand(new LeSetAdvertisingEnableCommand(true));
        }

With this code, the advertisment WORKS. Up to "02, 01, 26, 26, 255, 76, 00, 02, 21," everything is identical. As I don't really know what I've changed by this (the hcitool-code comes from http://www.orangenarwhals.com/2014/06/bluetooth-low-energy-4-0-on-ubuntu-13-10-advertisements-sending-and-receiving/), can this Information maybe lead you to why the sample code does not work?

Is #23 a fix for this?

No, it's not a fix, sorry. I can only advertise, when I hard-code the advertismentData as shown above.

Apart from having a different UUID, Major, Minor and TX value (which should be the problem here), I see that you're sending 31 bytes, while Beacon.ToAdvertisingData() only generates 30 bytes. Can you add a zero to the result-array at the end in Beacon.cs ToAdvertisingData() method and see if that works?

The spec says SetAdvertisingDataCommand takes 31 octets, so this is probably a bug. Waiting for your test results :)

So quiet in here, I believe someone is enjoying holidays :)

Yes, I can confirm that adding a zero byte fixes the bug, the advertising is working. I figured the missing byte on my own, but I was forced to go on holiday before I could reply. :fearful: :wink:

Another issue closed :) Thank you!


fixes #590

Run the test
$ rake test_mssql TEST=test/db/mssql/unit_test.rb

I could not run this fix against MS SQL database yet as I have some trouble with running this from a Mac.

commits are picked on 1-3-stable and are expected to land in 1.3.18 ... thanks again for the PR


Replace in files doesn't work as expected with match containing newline

Type: Bug
Steps to reproduce

Create new file and fill the following.

child: asdf; asdf
}

Press Cmd+Shift+H, turn on regex option (Cmd+Option+R), fill find text with child: (\w+);(?=[\S\s\n]*?\}), replacement text with $1.

See preview or do replace.

Expected behavior
child: asdf should be replaced with asdf.
Actual behavior
child: asdf is replaced with $1.

Additional context
Normal single file replace works as expected.
If match contains no newline, it also works as expected.
VS Code version: Code - Insiders 1.75.0-insider (b229eb5fa2ecadb8a5238ccb9f6f49f7ccba9d2b, 2023-01-27T22:17:24.641Z)
OS version: Darwin x64 22.1.0
Modes:
Sandboxed: Yes

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz (6 x 3000)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
2, 2, 6

Memory (System)
24.00GB (0.02GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions: none
A/B Experiments
vsliv695:30137379
vsins829:30139715
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256197
vslsvsres303:30308271
pythonvspyl392:30422396
pythontb:30258533
pythonptprofiler:30281269
vshan820:30294714
pythondataviewer:30285072
vscod805:30301674
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30404738
cppdebug:30492333
vsclangdf:30492506
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30618038
pylantcb52:30590116
pyindex848:30611229
nodejswelcome1:30587009
pyind779:30611226
pythonsymbol12:30651887

I'll put a note here.

Problem
For me regex match group substitution was not working on Unicode characters.

Replace match groups with $X ($0 $1, $2, $3 ...) failed.

After testing, I removed "This is a test-", line, and now it's not even finding a match in the file even though "കണ്ടന്നുമുതൽ തൻ-", sill exists.
Solution
But then I learned from here even though \w matches Unicode characters, it is preferable to use \p{L}.

@jeiea, in your case, maybe you've to escape the \ before matching \n like so: \\n?
Given:
child: asdf; asdf
}

If you've to replace child: asdf with asdf why use regex at all? You could simply do:
F: child: asdf;
R: asdf;

That's simplified example for the reporting. I don't remember original use case but that's not important. Just because asdf can be anything, I had to use regex.

കണ്ടന്നുമുതൽ തൻ-

I see that \p{L} is syntax only used in PCRE, but open files should use the JS engine for processing regexes (https://github.com/microsoft/vscode/wiki/Search-Issues#notes-on-regular-expression-support). This being said, you likely need to look for something that works for both engines and is clean, unfortunately

I found a simpler repro for this:
In the single file find & replace, replace regex \s with   and nothing will happen to the newlines in the file


Underscore 1.2.3 Mixin not working

Use case:
var _ = require('underscore')._;
module.exports = function() {
_.mixin( require('underscore.deferred') );
var dfd = new _.Deferred();
}
RESULT: Error Message "undefined is not a function"
It still works independently of underscore, just won't mixin.

Underscore was recently changed, have you tried
var _ = require("underscore");

Just tried that and it is the same result.

thanks for opening an issue! this was happening before of how we were detecting the presence of module. This is fixed as of v0.1.2, which is on the npm registry now.


GUVNOR-2869 : Guided Decision Table V&V - Fix Verification reporting levels

https://issues.jboss.org/browse/GUVNOR-2869
Reporting was still a bit too eager, this silences it down. Quite a lot to do on reporting configuration, but given the time I think best just to harden the core.

@manstis @jomarko Pretty simple, just silences the verification when it is too noisy. Still I could use a review, please.

Hi @Rikkola. PR is completely OK. Maybe just small comment, I am not sure if is the best practice to name test methods like get....... However, I am also for merging.


Feature Request: Manage "Only allow Label" & "Exclude Label" in the user sharing restrictions tab

Is your feature request related to a problem? Please describe.
I have a few collections I assigned the label "Admin" that should really only be visible to me so I manually added "Admin" as a label to exclude for everyone of my users in their sharing restrictions.
For the future if a user shares their preferred method of a watchlist with me I want to create a collection for them and then share that collection only with them.
Describe the solution you'd like
While building the collection with PMM I can choose a user or users for which the collection shall be visible.
If someone wants an account for a kid that could be a way to assign the kid account labels they are allowed to watch as well.
Describe alternatives you've considered
Considered and practiced: adding those labels manually.

Feature Requests have been moved to https://features.metamanager.wiki.
This Request


Page footer overwritten when using auto size with 'canGrow'

Hi,
I'm using a PageFooterBand with Height = 0 and CanGrow = true, to autosize it. This works for the footer alone, but the data content is then written over the footer, when the content happens to fit on the page as if there was no footer.
Looking into the code and debugging a little, I found that ReportEngine.CalcHeight is only called at the point in time when the footer is rendered, although the footer height is already used earlier (esp. in ReportEngine.FreeSpace).
Is this a bug, or am I missing something?
I managed to find a dirty workaround, by handling the StartPage event of the page, and then calling page.PageFooter.CalcHeight there.

Hello!
Thanks for waiting. We have fixed this issue in FastReport .NET Core. A bugfix here will be available with next sync after few days and in next release too, of course.


i18n(de): Translate client-side-scripts.mdx

What kind of changes does this PR include?

Translated content

Description

Translate client-side-scripts.mdx to german

Nice work @sarasate! This page was recently changed, could you please add the changes made? I'll review & merge soon after!
https://github.com/withastro/docs/commits/main/src/content/docs/en/guides/client-side-scripts.mdx?since=2022-12-21T15:02:53.000Z

@Yan-Thomas Update done.

@Yan-Thomas Thanks for the review. Fixed your suggestions, seems to be fine now..

Ok, didn't know about that one, will keep that in mind. Updated.


[Bug]: Loading mistral-7B-instruct-v03 KeyError: 'layers.0.attention.wk.weight'

Your current environment
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Rocky Linux 8.8 (Green Obsidian) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)
Clang version: Could not collect
CMake version: version 3.29.2
Libc version: glibc-2.28
Python version: 3.9.13 (main, Oct 13 2022, 21:15:33)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.18.0-513.9.1.el8_9.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA H100 PCIe
Nvidia driver version: 535.129.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              256
On-line CPU(s) list: 0-255
Thread(s) per core:  2
Core(s) per socket:  64
Socket(s):           2
NUMA node(s):        2
Vendor ID:           AuthenticAMD
CPU family:          25
Model:               17
Model name:          AMD EPYC 9534 64-Core Processor
Stepping:            1
CPU MHz:             2450.000
CPU max MHz:         3718.0659
CPU min MHz:         1500.0000
BogoMIPS:            4900.22
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            32768K
NUMA node0 CPU(s):   0-63,128-191
NUMA node1 CPU(s):   64-127,192-255
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] triton==2.3.0
[pip3] vllm-nccl-cu12==<IP_ADDRESS>.4.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] torch                     2.3.0                    pypi_0    pypi
[conda] triton                    2.3.0                    pypi_0    pypi
[conda] vllm-nccl-cu12            <IP_ADDRESS>.4.0             pypi_0    pypiROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.4.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	NIC0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	SYS	5-6,133-134	0		N/A
NIC0	SYS	 X
Legend:
X    = Self
SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX  = Connection traversing at most a single PCIe bridge
NV#  = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_bond_0
🐛 Describe the bug
I am trying to load the new Mistral 7B instruct v03 model. However, it gives KeyError: 'layers.0.attention.wk.weight'. Curiously it seems to use the llama model loader (see stack trace). I am not sure if that is intended.
KeyError                                  Traceback (most recent call last)
Cell In[13], line 43
     40 else:
     41     raise ValueError(model)
---> 43 llm = LLM(
     44     model=model_path,
     45     dtype="float16",
     46     max_model_len=max_model_len,
     47     gpu_memory_utilization=gpu_memory_utilization,
     48     **kwargs
     49 )

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/entrypoints/llm.py:123, in LLM.__init__(self, model, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, enforce_eager, max_context_len_to_capture, max_seq_len_to_capture, disable_custom_all_reduce, **kwargs)
    102     kwargs["disable_log_stats"] = True
    103 engine_args = EngineArgs(
    104     model=model,
    105     tokenizer=tokenizer,
   (...)
    121     **kwargs,
    122 )
--> 123 self.llm_engine = LLMEngine.from_engine_args(
    124     engine_args, usage_context=UsageContext.LLM_CLASS)
    125 self.request_counter = Counter()

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/engine/llm_engine.py:292, in LLMEngine.from_engine_args(cls, engine_args, usage_context)
    289     executor_class = GPUExecutor
    291 # Create the LLM engine.
--> 292 engine = cls(
    293     **engine_config.to_dict(),
    294     executor_class=executor_class,
    295     log_stats=not engine_args.disable_log_stats,
    296     usage_context=usage_context,
    297 )
    298 return engine

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/engine/llm_engine.py:160, in LLMEngine.__init__(self, model_config, cache_config, parallel_config, scheduler_config, device_config, load_config, lora_config, vision_language_config, speculative_config, decoding_config, executor_class, log_stats, usage_context)
    156 self.seq_counter = Counter()
    157 self.generation_config_fields = _load_generation_config_dict(
    158     model_config)
--> 160 self.model_executor = executor_class(
    161     model_config=model_config,
    162     cache_config=cache_config,
    163     parallel_config=parallel_config,
    164     scheduler_config=scheduler_config,
    165     device_config=device_config,
    166     lora_config=lora_config,
    167     vision_language_config=vision_language_config,
    168     speculative_config=speculative_config,
    169     load_config=load_config,
    170 )
    172 self._initialize_kv_caches()
    174 # If usage stat is enabled, collect relevant info.

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/executor/executor_base.py:41, in ExecutorBase.__init__(self, model_config, cache_config, parallel_config, scheduler_config, device_config, load_config, lora_config, vision_language_config, speculative_config)
     38 self.vision_language_config = vision_language_config
     39 self.speculative_config = speculative_config
---> 41 self._init_executor()

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/executor/gpu_executor.py:23, in GPUExecutor._init_executor(self)
     17 """Initialize the worker and load the model.
     18
     19 If speculative decoding is enabled, we instead create the speculative
     20 worker.
     21 """
     22 if self.speculative_config is None:
---> 23     self._init_non_spec_worker()
     24 else:
     25     self._init_spec_worker()

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/executor/gpu_executor.py:69, in GPUExecutor._init_non_spec_worker(self)
     67 self.driver_worker = self._create_worker()
     68 self.driver_worker.init_device()
---> 69 self.driver_worker.load_model()

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/worker/worker.py:118, in Worker.load_model(self)
    117 def load_model(self):
--> 118     self.model_runner.load_model()

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/worker/model_runner.py:164, in ModelRunner.load_model(self)
    162 def load_model(self) -> None:
    163     with CudaMemoryProfiler() as m:
--> 164         self.model = get_model(
    165             model_config=self.model_config,
    166             device_config=self.device_config,
    167             load_config=self.load_config,
    168             lora_config=self.lora_config,
    169             vision_language_config=self.vision_language_config,
    170             parallel_config=self.parallel_config,
    171             scheduler_config=self.scheduler_config,
    172         )
    174     self.model_memory_usage = m.consumed_memory
    175     logger.info("Loading model weights took %.4f GB",
    176                 self.model_memory_usage / float(2**30))

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/model_executor/model_loader/__init__.py:19, in get_model(model_config, load_config, device_config, parallel_config, scheduler_config, lora_config, vision_language_config)
     13 def get_model(
     14         *, model_config: ModelConfig, load_config: LoadConfig,
     15         device_config: DeviceConfig, parallel_config: ParallelConfig,
     16         scheduler_config: SchedulerConfig, lora_config: Optional[LoRAConfig],
     17         vision_language_config: Optional[VisionLanguageConfig]) -> nn.Module:
     18     loader = get_model_loader(load_config)
---> 19     return loader.load_model(model_config=model_config,
     20                              device_config=device_config,
     21                              lora_config=lora_config,
     22                              vision_language_config=vision_language_config,
     23                              parallel_config=parallel_config,
     24                              scheduler_config=scheduler_config)

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/model_executor/model_loader/loader.py:224, in DefaultModelLoader.load_model(self, model_config, device_config, lora_config, vision_language_config, parallel_config, scheduler_config)
    221 with torch.device(device_config.device):
    222     model = _initialize_model(model_config, self.load_config,
    223                               lora_config, vision_language_config)
--> 224 model.load_weights(
    225     self._get_weights_iterator(model_config.model,
    226                                model_config.revision,
    227                                fall_back_to_pt=getattr(
    228                                    model,
    229                                    "fall_back_to_pt_during_load",
    230                                    True)), )
    231 for _, module in model.named_modules():
    232     quant_method = getattr(module, "quant_method", None)

File ~/miniconda/envs/project-experiments-py39-vllm3/lib/python3.9/site-packages/vllm/model_executor/models/llama.py:415, in LlamaForCausalLM.load_weights(self, weights)
    413 if name.endswith(".bias") and name not in params_dict:
    414     continue
--> 415 param = params_dict[name]
    416 weight_loader = getattr(param, "weight_loader",
    417                         default_weight_loader)
    418 weight_loader(param, loaded_weight)

KeyError: 'layers.0.attention.wk.weight'

I am also facing the same issue
NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.2
A100 - 80GB

Package
Version

torch
2.3.0

torchmetrics
1.4.0.post0

torchvision
0.18.0

nvidia-cublas-cu12
<IP_ADDRESS>

nvidia-cuda-cupti-cu12
12.1.105

nvidia-cuda-nvrtc-cu12
12.1.105

nvidia-cuda-runtime-cu12
12.1.105

nvidia-cudnn-cu12
<IP_ADDRESS>

nvidia-cufft-cu12
<IP_ADDRESS>

nvidia-curand-cu12
<IP_ADDRESS>

nvidia-cusolver-cu12
<IP_ADDRESS>

nvidia-cusparse-cu12
<IP_ADDRESS>

nvidia-ml-py
12.550.52

nvidia-nccl-cu12
2.20.5

nvidia-nvjitlink-cu12
12.5.40

nvidia-nvtx-cu12
12.1.105

numpy
1.26.4

vllm
0.4.2

vllm-nccl-cu12
<IP_ADDRESS>.4.0

sentence-transformers
2.7.0

transformers
4.41.0

+1

+1 same issue

+1 same issue

Fixed by https://github.com/vllm-project/vllm/pull/5005

@robertgshaw2-neuralmagic
still face the issue of vllm version 0.5.4
llm = LLM(model= "mistralai/Mistral-7B-Instruct-v0.3" , dtype='float16', max_model_len=4000, tensor_parallel_size=1, gpu_memory_utilization=1,
       quantization="bitsandbytes", load_format="bitsandbytes", enforce_eager=True
   )

WARNING 08-20 23:25:13 config.py:1454] Casting torch.bfloat16 to torch.float16.
WARNING 08-20 23:25:13 config.py:254] bitsandbytes quantization is not fully optimized yet. The speed can be slower than non-quantized models.
WARNING 08-20 23:25:13 config.py:1342] bitsandbytes quantization is not tested with LoRA yet.
INFO 08-20 23:25:13 llm_engine.py:174] Initializing an LLM engine (v0.5.4) with config: model='mistralai/Mistral-7B-Instruct-v0.3', speculative_config=None, tokenizer='mistralai/Mistral-7B-Instruct-v0.3', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4000, download_dir=None, load_format=LoadFormat.BITSANDBYTES, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=bitsandbytes, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=mistralai/Mistral-7B-Instruct-v0.3, use_v2_block_manager=False, enable_prefix_caching=False)
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████| 141k/141k [00:00<00:00, 12.8MB/s]
INFO 08-20 23:25:15 model_runner.py:720] Starting to load model mistralai/Mistral-7B-Instruct-v0.3...
INFO 08-20 23:25:15 loader.py:871] Loading weights with BitsAndBytes quantization.  May take a while ...
INFO 08-20 23:25:15 weight_utils.py:225] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:05,  1.76s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:03<00:03,  1.80s/it]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ubuntu/moa/test_add.py", line 16, in
[rank0]:     llm = LLM(model= args.llm_name, dtype='float16', max_model_len=4000,
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 158, in init
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 445, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 249, in init
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 47, in init
[rank0]:     self._init_executor()
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 36, in _init_executor
[rank0]:     self.driver_worker.load_model()
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 139, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 722, in load_model
[rank0]:     self.model = get_model(model_config=self.model_config,
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 21, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 946, in load_model
[rank0]:     self._load_weights(model_config, model)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 885, in _load_weights
[rank0]:     model.load_weights(qweight_iterator)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 513, in load_weights
[rank0]:     param = params_dict[name]
[rank0]: KeyError: 'layers.0.attention.wk.weight'
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:03<00:03,  1.92s/it]

Same here!

@torch.library.impl_abstract("xformers_flash::flash_bwd")
INFO 08-21 12:34:00 model_runner.py:720] Starting to load model /home/barbatus/finetuning/models...
INFO 08-21 12:34:00 selector.py:170] Cannot use FlashAttention-2 backend due to sliding window.
INFO 08-21 12:34:00 selector.py:54] Using XFormers backend.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
[rank0]: Traceback (most recent call last):
[rank0]:   File ".local/bin/vllm", line 8, in
[rank0]:     sys.exit(main())
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/scripts.py", line 149, in main
[rank0]:     args.dispatch_function(args)
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/scripts.py", line 30, in serve
[rank0]:     asyncio.run(run_server(args))
[rank0]:   File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
[rank0]:     return loop.run_until_complete(main)
[rank0]:   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
[rank0]:     return future.result()
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 342, in run_server
[rank0]:     async with build_async_engine_client(args) as async_engine_client:
[rank0]:   File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
[rank0]:     return await anext(self.gen)
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 102, in build_async_engine_client
[rank0]:     async_engine_client = AsyncLLMEngine.from_engine_args(
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 471, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 381, in init
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 552, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 249, in init
[rank0]:     self.model_executor = executor_class(
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 47, in init
[rank0]:     self._init_executor()
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 36, in _init_executor
[rank0]:     self.driver_worker.load_model()
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/worker/worker.py", line 139, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 722, in load_model
[rank0]:     self.model = get_model(model_config=self.model_config,
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 21, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 327, in load_model
[rank0]:     model.load_weights(
[rank0]:   File ".local/lib/python3.10/site-packages/vllm/model_executor/models/llama_embedding.py", line 84, in load_weights
[rank0]:     param = params_dict[name]
[rank0]: KeyError: 'layers.0.attention.wk.weight'
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]

The issues you are reporting are likely due to other arguments like the bitsandbytes quantization
I just ran the model on 0.5.4 and on main with default arguments and it loaded fine:
vllm serve mistralai/Mistral-7B-Instruct-v0.3

INFO 08-21 16:11:14 api_server.py:339] vLLM API server version 0.5.4
INFO 08-21 16:11:14 api_server.py:340] args: Namespace(model_tag='mistralai/Mistral-7B-Instruct-v0.3', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, model='mistralai/Mistral-7B-Instruct-v0.3', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None, dispatch_function=<function serve at 0x7fb01e0ec820>)
WARNING 08-21 16:11:15 config.py:1454] Casting torch.bfloat16 to torch.float16.
INFO 08-21 16:11:15 llm_engine.py:174] Initializing an LLM engine (v0.5.4) with config: model='mistralai/Mistral-7B-Instruct-v0.3', speculative_config=None, tokenizer='mistralai/Mistral-7B-Instruct-v0.3', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=mistralai/Mistral-7B-Instruct-v0.3, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 08-21 16:11:16 model_runner.py:720] Starting to load model mistralai/Mistral-7B-Instruct-v0.3...
INFO 08-21 16:11:16 weight_utils.py:225] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:01,  1.67it/s]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:01<00:00,  1.32it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.23it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.28it/s]

INFO 08-21 16:11:19 model_runner.py:732] Loading model weights took 13.5083 GB
INFO 08-21 16:11:23 gpu_executor.py:102] # GPU blocks: 27438, # CPU blocks: 2048
INFO 08-21 16:11:24 model_runner.py:1024] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 08-21 16:11:24 model_runner.py:1028] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 08-21 16:11:35 model_runner.py:1225] Graph capturing finished in 11 secs.
WARNING 08-21 16:11:35 serving_embedding.py:171] embedding_mode is False. Embedding API will not work.

INFO:     Application startup complete.
INFO:     Uvicorn running on http://<IP_ADDRESS>:8000 (Press CTRL+C to quit)

yes, without these arguments --quantization bitsandbytes --load_format bitsandbytes --enforce_eager,  mistralai/Mistral-7B-Instruct-v0.3 works fine.
this issue only applies to mistral, while llama-3.1 is not affected.

Could you investigate and submit a fix? This is likely due to this Mistral model having multiple copies of the checkpoint with slightly different state dicts

I am not sure to understand because I never set those argument anywhere. I am launching it the same way than you. May be my issue come from the config.json, I used the params.json delivered with the model and had to add that:
"model_type": "mistral",
"architectures": ["MistralModel"]

I am not sure to understand because I never set those argument anywhere. I am launching it the same way than you. May be my issue come from the config.json, I used the params.json delivered with the model and had to add that: "model_type": "mistral", "architectures": ["MistralModel"]

Post your config.json?

{
    "dim": 4096,
    "n_layers": 32,
    "head_dim": 128,
    "hidden_dim": 14336,
    "n_heads": 32,
    "n_kv_heads": 8,
    "norm_eps": 1e-05,
    "vocab_size": 32768,
    "rope_theta": 1000000.0,
    "model_type": "mistral",
    "architectures": ["MistralModel"]
}

This is not a valid HF transformers config.json
I am not sure what the params.json is in the checkpoint, but vLLM supports the official transformers config.json only

how can I find an exemple?

how can I find an exemple?

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/blob/main/config.json

I had to rename the tokenizer from tokenizer.model.v3 to tokenizer.model othewise it does not find it

Original checkpoint had two copies

one with merged qkv and merged gate_up
one with unmerged

vLLM supports loading the unmerged one. It seems like your checkpoint has the merged weights.
I’m not sure how you saved or made this checkpoint, it seems like you’re not saving it in the hugging face format via saved_pretrained()

I am using the tarball from mistral, I found here:
I used it to run some fine-tuning but it does not modify the base model (the lora checkpoint is in an other directory), so it might be an issue from mistral directly?

I am using the tarball from mistral, I found here: https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar
I used it to run some fine-tuning but it does not modify the base model (the lora checkpoint is in an other directory), so it might be an issue from mistral directly?

Not sure - don’t know anything about how you did the finetuning. Either way, you need to save the model in the hf format with unfused linear layers to use it with vLLM

Thank you for your help, so I downloaded all file from huggingface:
~/finetuning/models/7B-bis$ ll
total 28313268
drwxrwxr-x 2 barbatus barbatus        4096 Aug 22 15:11 ./
drwxrwxr-x 4 barbatus barbatus        4096 Aug 21 21:41 ../
-rw-rw-r-- 1 barbatus barbatus         601 Aug 22 09:12 config.json
-rw-rw-r-- 1 barbatus barbatus<PHONE_NUMBER>2 Aug 22 09:35 consolidated.safetensors
-rw-rw-r-- 1 barbatus barbatus<PHONE_NUMBER> Aug 22 09:38 model-00001-of-00003.safetensors
-rw-rw-r-- 1 barbatus barbatus<PHONE_NUMBER> Aug 22 09:41 model-00002-of-00003.safetensors
-rw-rw-r-- 1 barbatus barbatus<PHONE_NUMBER> Aug 22 09:43 model-00003-of-00003.safetensors
-rw-rw-r-- 1 barbatus barbatus      587404 Aug 22 09:16 tokenizer.model

$ cat config.json
{
  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.42.0.dev0",
  "use_cache": true,
  "vocab_size": 32768
}

yet:
$ vllm serve ~/finetuning/models/7B-bis/
INFO 08-22 15:15:31 api_server.py:339] vLLM API server version 0.5.4
INFO 08-22 15:15:31 api_server.py:340] args: Namespace(model_tag='/home/barbatus/finetuning/models/7B-bis/', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, model='/home/barbatus/finetuning/models/7B-bis/', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None, dispatch_function=<function serve at 0x7fedd58e5510>)
WARNING 08-22 15:15:31 config.py:1454] Casting torch.bfloat16 to torch.float16.
INFO 08-22 15:15:31 llm_engine.py:174] Initializing an LLM engine (v0.5.4) with config: model='/home/barbatus/finetuning/models/7B-bis/', speculative_config=None, tokenizer='/home/barbatus/finetuning/models/7B-bis/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/home/barbatus/finetuning/models/7B-bis/, use_v2_block_manager=False, enable_prefix_caching=False)
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
INFO 08-22 15:15:32 model_runner.py:720] Starting to load model /home/barbatus/finetuning/models/7B-bis/...
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py", line 217, in run_rpc_server
    server = AsyncEngineRPCServer(async_engine_args, usage_context, port)
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py", line 25, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(async_engine_args,
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 471, in from_engine_args
    engine = cls(
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 381, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 552, in _init_engine
    return engine_class(*args, **kwargs)
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 249, in __init__
    self.model_executor = executor_class(
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 47, in __init__
    self._init_executor()
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 36, in _init_executor
    self.driver_worker.load_model()
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/worker/worker.py", line 139, in load_model
    self.model_runner.load_model()
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 722, in load_model
    self.model = get_model(model_config=self.model_config,
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
    return loader.load_model(model_config=model_config,
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 327, in load_model
    model.load_weights(
  File "/home/barbatus/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 513, in load_weights
    param = params_dict[name]
KeyError: 'layers.0.attention.wk.weight'
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]

I also opened the consolidated file and I see the right key:
<A8>z^@^@^@^@^@^@{"layers.0.attention.wk.weight":{"dtype":"BF16","shape":[1024,4096],"data_offsets":[0,8388608]},"layers.0.attention.wo.weight":{"dtype":"BF16","shape":[4096,4096],"data_offsets":[8388608,41943040]},"layers.0.attention.wq.weight":{"dtype":"BF16","shape":[4096,4096

I am sure I missing something but I don't understand what

Thank you for your help, I finally downloaded all from hugging face and be able to make it works. I will try to finetune from here and see


Update network-watcher-nsg-flow-logging-powershell.md

There is no space on row 47 between nsgRG-Name The Name parameter is being treated as a string and breaking the cmdlet example. I added a space to fix the example

@jrudley Thanks for catching and fixing.  #sign-off


500KHz tone in output

I think this is coming from the power supply (the upstream switching
converter switches at 500KHz).
The first step is to measure the power supply noise and determine if
that is indeed the source of the noise. If so, the contents below
apply. If not, I don't yet have an alternate theory.
The tone comes from the TPS5420D switching converter. The problem
is some combination of the following:

TPS5420D produces too much ripple. Could use the old L7980 instead.
TPS7A7001 linear regulator (downstream) has inadequate PSRR. In
this case, it should be replaced with the old TPS7A91 (we'll need
2, 1 for the mixer and another for the power amp, in order to meet
max current draw).

Possible solutions (some combination of these):

Revert to L7980 and TPS7A91. Have mixer and power amp use separate
TPS7A91 to accommodate current draw.
Use a capacitance multiplier (see the art of electronics section
8.15.1) between the converter and regulator.

New design reverts to L7980.


Fixed an issue with an unrecognized FILL MISSING FIELD option for the COPY command

Problem description:
In cases where the COPY command is used with new syntax where options are listed in parentheses, "FILL MISSING FIELD" option is causing a syntax error.
Old syntax without parentheses works fine.
Root cause:
Parser's rule for every option in the list of options (nonterminal symbol copy_generic_opt_elem) currently expects that every option has a form of "string without spaces" plus optional "space" and "argument".
'FILL MISSING FIELD' doesn't match this pattern.
Fix:
Added handling of the "FILL MISSING FIELDS" option to the grammar file for the nonterminal symbol copy_generic_opt_elem.

Allure report https://allure-ee.adsw.io/launch/60231

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/877297

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/877298

Allure report https://allure-ee.adsw.io/launch/61997

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/975477

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/975478

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/975468


Allow deeper enforcement of types

(From a Terracotta internal review)
Both CacheManager.createCacheBuilder(…) and CacheManager.getCache(…) should include Class parameters matching the supplied generic types - this will allow implementations that so choose to validate the supplied generic types against any pre-existing cache (disk persistent or distributed) and also against any future configuration settings.  The method signatures would then become:
<K, V> CacheBuilder<K, V> createCacheBuilder(String cacheName, Class keyType, Class valueType)
<K, V> Cache<K, V> getCache(String cacheName, Class keyType, Class valueType)
Implementations that wish to ignore the classes could be free to do so, but those that wish to validate the generic type of what they are returning can do so.  For example an implementation that supports disk-persistent caches could save the class-name (or even a more full featured descriptor) to the disk that could be validated against the supplied key and value types when that cache is retrieved in a subsequent JVM execution.

I think this is a good idea.
To give full effect to it we would need to state in the Javadoc that implementations are required to check for valid types. Or that implementations may check for valid types.
Another way to go is to have two versions of each method; one like it is now and the new ones. If you use the non Class ones then types are not enforced. If you use the Class method then types are enforced.

I support the idea in general, but would restict this declaration to the configuration side of the story (e.g. to createCacheBuilder). I believe that in practice, most implementations will instantiate caches based on configuration file(s). There will be one source of truth (equivalent to a dtd/database schema) and that is where the enforcement will happen.
So, maybe
String CacheConfiguration.getKeyClassName() --- or Class CacheConfiguration.getKeyClass()
String CacheConfiguration.getValueClassName() --- or Class CacheConfiguration.getValueClass()
and corresponding support in builder. I have a preference to String rather than Class because of possible classloader issues.
I do not think getCache should specify the classes - this gives the impression that the consumer of the Cache can specify (alter) its type

I don't agree on getCache - the intention with the original proposal was that an implementation that supported type enforcement on caches could prevent users from avoiding that enforcement without them explicitly casting things.  The parameters passed to CacheManager.getCache(...) would not be used to control the generic type of the cache but rather they would be used to help enforce the type safety of the returned cache.
Does that make sense?

OK, I stand corrected on getCache. But let me just try this out in code. I think the below captures the proposal (CacheManager1 standing in for CacheManager). Though I agree that getCache() adds a bit earlier failure (on getCache rather than on get) I'm still not convinced that the added complexity of the API is worth that gain. I may be missing something else though. Ideas?
public interface CacheManager1 extends CacheManager {
    public <K, V> CacheBuilder<K, V> createCacheBuilder(String cacheName, Class<? extends K> keyType, Class<? extends V> valueType);
    public <K, V> Cache<K, V> getCache(String cacheName, Class<? extends K> keyType, Class<? extends V> valueType);
}

public void foo() {
    String cacheName = "myCache";
    Cache<Integer, String> cache =
            getCacheManager().createCacheBuilder(cacheName, Integer.class, String.class).build();

    Cache<Integer, String> cache1G = getCacheManager().getCache(cacheName);
    cache1G.put(1, "a");
    Cache<String, Integer> cache1B = getCacheManager().getCache(cacheName);
    cache1B.put("a", 1); // This should cause some exception because <Integer, String>

    Cache<Integer, String> cache2G = getCacheManager().getCache(cacheName, Integer.class, String.class);
    cache2G.put(1, "a");
    Cache<String, Integer> cache2B = getCacheManager().getCache(cacheName, String.class, Integer.class); // This should cause some exception because <Integer, String>
    cache2B.put("a", 1);
}

private CacheManager1 getCacheManager() {
    throw new UnsupportedOperationException();
}

I think not passing the key and value types in to getCache(...) might work although as you point out this is at the expense of getting late failures.  Personally I'm much more comfortable with the idea of an early failure.  Maybe the answer to this quandry is to go with:
public interface CacheManager {
  public <K, V> CacheBuilder<K, V> createCacheBuilder(String cacheName, Class<K> keyType, Class<V> valueType);

  public <K, V> Cache<K, V> getCache(String cacheName, Class<K> keyType, Class<V> valueType);
  public <K, V> Cache<K, V> getCache(String cacheName);
}

Also I'm confused as to why you declared the generic  types of the classes as Class<? extends K>.  It seems we want the exact class corresponding to the generic type of the returned cache passed in here, not an arbitrary subtype of that type.

A strong +1 on createCacheBuilder but I think the non-typed getCache shouldn't use generics at all:
  public <K, V> CacheBuilder<K, V> createCacheBuilder(String cacheName, Class<K> keyType, Class<V> valueType);

  public <K, V> Cache<K, V> getCache(String cacheName, Class<K> keyType, Class<V> valueType);
  public Cache<Object, Object> getCache(String cacheName);
}```

Really getCache("foo") would be a shortcut for getCache("foo", Object.class, Object.class).

With that sort of API I think the spec would be safe requiring fail-fast type checking on all typed Cache APIs. Then if a developer wants to have a type safe cache they use the 3arg getCache, if they don't care about types they use the 1arg getCache.

I also agree that it doesn't make sense to have Class<? extends K>. The place to use ? extends would be on specific Cache APIs that need it to allow subtypes of K and V to be used. Effective Java v2 has VERY good examples of how and where to use extends with generics.

Yep, I like that idea Eric - although I'm torn between that and a couple of other forms for the plain getCache:

public Cache getCache(String cacheName);
This kind of illustrates the badness of the method since it will trigger a nice raw-type compiler warning.
public Cache<?, ?> getCache(String cacheName);
This signature actually still enforces the type-safety of our caches (in the absence of casting) because it would require an explicit cast to pass anything in.
public Cache<Object, Object> getCache(String cacheName);
This signature is the most flexible but ruins the type-safety of our caches.

In the absence of any further input I'll go with Eric's original suggestion however (Option 3).

1 is a bit of a cop-out, it would be better to just not have getCache(String) than return something without type params (EJ Item23)
2 Makes it impossible to interact with the cache. A method parameter with a ? type can ONLY have null passed in as value, not even an Object can be passed.
3 Describes what is really happening, creating a cache that allows any Object for both the key and the value.

Agreed, it was an intentional cop-out basically a subtle way of saying "you really shouldn't be doing this - but okay".
Yep I'm aware that this cripples use of this method - it's just in my head a stronger version of 1.

I'm fine with adopting 3.

Well 2 isn't really a stronger version, it is a completely useless version. Any method on Cache that takes a generic parameter would be non-functional.

Yep, I freely admit that option 2 is pretty useless - it's intended to aggressively protect the type-safety of the cache at compile time.  Obviously it's a little too strong for what we want but I figured may as well have the full gamut of solutions represented.

Seems like a reasonable API for the createCacheBuilder.
For the previous comment, there is no (reasonable) way in java to support anything beyond the primary type. So we can capture the top most level Class types but not sub-types (like what you have for collections), thats just a limitation of what generics in Java can do.

That seems like a reasonable best-effort. When I get time I'll re-read the generics chapter in effective java 2 and see if there is any other insight.


Scrren Getting Auto Blurred

Type: Bug
If i use any other application in another display it is getting auto blurred it feels very irrtating if i want to cross check the code
VS Code version: Code 1.80.0 (660393deaaa6d1996740ff4880f1bad43768c814, 2023-07-04T15:06:02.407Z)
OS version: Windows_NT x64 10.0.22621
Modes:

System Info

Item
Value

CPUs
AMD Ryzen 9 5950X 16-Core Processor             (32 x 3400)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
undefined

Memory (System)
63.90GB (48.15GB free)

Process Argv
--crash-reporter-id 64db72d8-06e8-4321-8e24-84a66260ca17

Screen Reader
no

VM
0%

Extensions (34)

Extension
Author (truncated)
Version

vscode-django
bat
1.10.0

npm-intellisense
chr
1.4.4

path-intellisense
chr
2.8.4

vscode-mysql-client2
cwe
6.5.11

dart-code
Dar
3.68.1

flutter
Dar
3.68.0

vscode-eslint
dba
2.4.2

vscode-html-css
ecm
1.13.1

prettier-vscode
esb
9.19.0

auto-close-tag
for
0.5.14

auto-rename-tag
for
0.1.10

code-runner
for
0.12.0

vscode-mysql
for
0.4.1

go
gol
0.39.0

vscode-docker
ms-
1.25.2

vscode-edge-devtools
ms-
2.1.2

data-workspace-vscode
ms-
0.5.0

mssql
ms-
1.19.1

sql-bindings-vscode
ms-
0.4.0

sql-database-projects-vscode
ms-
1.1.1

python
ms-
2023.12.0

vscode-pylance
ms-
2023.7.10

remote-wsl
ms-
0.80.2

cpptools
ms-
1.16.3

color-highlight
nau
2.5.0

material-icon-theme
PKi
4.28.0

prisma
Pri
4.16.2

prisma-insider
Pri
31.0.2413

java
red
1.20.0

LiveServer
rit
5.7.9

intellicode-api-usage-examples
Vis
0.2.7

vscodeintellicode
Vis
1.2.30

vscode-icons
vsc
12.4.0

JavaScriptSnippets
xab
1.8.0

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263:30776702
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
282f8724:30602487
pyind779:30671433
f6dab269:30613381
pythonsymbol12:30671437
showlangstatbar:30737416
vsctsb:30748421
pythonms35:30701012
03d35959:30757346
7ij38806:30736111
pythonfmttext:30731395
pythoncmvfstrcf:30756944
fixshowwlkth:30771522
pythongtdpath:30769146
i26e3531:30780429
pythonnosmt12:30779714
pythonidxpt:30784022
pythonnocebcf:30776496
dsvsc013:30784100
cmakestatusbarv2:30779594

/gifPlease

Thanks for reporting this issue! Unfortunately, it's hard for us to understand what issue you're seeing. Please help us out by providing a screen recording showing exactly what isn't working as expected. While we can work with most standard formats, .gif files are preferred as they are displayed inline on GitHub. You may find https://gifcap.dev helpful as a browser-based gif recording tool.
If the issue depends on keyboard input, you can help us by enabling screencast mode for the recording (Developer: Toggle Screencast Mode in the command palette). Lastly, please attach this file via the GitHub web interface as emailed responses will strip files out from the issue.
Happy coding!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


Bower release version outdated

The latest version available for install via Bower is 3.0.0, but the latest release for install via npm is 3.5.0.

Hey, good point. Just tagged the latest commit version with 3.5.0. Shall I do something else or bower will take it from here.

That did it, thanks!
I was going to mention maybe adding version to the bower.json, but apparently that gets ignored now, according to the spec.


Getting media by folder not working

When I make the request to get a collection of media in a specific folder, it is returning all the most recent media for my current filter. Looking at the request, folder is not added to the query.
Is this functionality missing, or is there another way to get media folders returned?

Thanks for the find @brystearns I just pushed version 3.2.8 which now has the working folder param on getMedia.  Please install the latest and confirm that it's working now.


Fix pagination in Firefox.

Demo: https://kpt-dev-ee2a9--ff-pagination-a4fmlslg.web.app

LGTM

Verified it works.


Doc: Add a note on how to dive into the code with Pry [ci skip]

Follow up of https://github.com/bundler/bundler/pull/4919#issuecomment-242706463.
Preview DEVELOPMENT.md

@homu r+

:pushpin: Commit 6667a9d has been approved by indirect

:hourglass: Testing commit 6667a9d with merge 4c4fd8e...

:boom: Test timed out

@homu retry

:hourglass: Testing commit 6667a9d with merge 74c359d...

:boom: Test timed out

We can't merge stuff with [ci skip] in the title

@homu retry

:hourglass: Testing commit 6667a9d with merge fc35baa...

:broken_heart: Test failed - status

@homu retry

:hourglass: Testing commit 6667a9d with merge 2b4b826...

:sunny: Test successful - status


Upgrade salvo web server to latest 0.44 and add logging and add webserver configuration

rmqtt uses 0.37.9 salvo and the interface has changed in 0.44.
step one, would be to upgrade rmqtt to use salvo 0.44 api.
step two, is to add logging to http requests.
step three, add configuration for salvo web server in rmqtt config.
I am trying to add static webpages to http-api/api.rs:router function using StaticDir but I am getting 404, not sure where it is looking for the files.

The reason why salvo 0.40 and later versions were not used is because salvo 0.37 uses the latest official version of super ("0.14"), while salvo 0.4x uses super ("1.0.0 rc.3")

The function of adding HTTP request logs has been added to the development plan.

What salvo web server configurations need to be added? Can you provide examples?

Using StaticDir but I am getting 404. It is recommended to post your question in the salvo community.

Finally, thank you for your attention and use of rmqtt.

I would like rmqtt have a web interface with http-api so it would be able to function as a web-server also. MQTT is great for communication but for a user interface a web server is the best.

My understanding is that webserver is not a necessary feature of mqtt broker. Based on your requirements, it is recommended to deploy additional webserver services, such as nginx or haproxy.

It is not necessary but it is a nice feature;
Use Case
A SBC controls a network of IoT using mqtt protocol. It has a web application written to control the system. Space is limited. rmqtt server host the static app, another web server is not needed for one application.
Create a plugin; http-web-server

Based on your business situation, you can implement the ‘http-web-server’ plugin to solve your problem. But this is not a part of rmqtt, so you can implement this ‘http-web-server’ plugin and recompile rmqtt. The plugin implementation can refer to: https://github.com/rmqtt/rmqtt/tree/master/rmqtt-plugins/rmqtt-plugin-template


JUnit report treats all severities as failures

First of all, thanks for this great tool and the JUnit report format output option. Much appreciated!
We use the JUnit report format to create reports for Jenkins so that a build fails if an OpenAPI file fails to pass linting.
Not all issues that are found are of the same importance to us, so we use a custom ruleset to disable or change/downgrade the serverity of certain rules.
However it turns out that the JUnit report creates failure tags in the output XML for all severities.
For "error" that is clear, for "warn" it is debatable, but for "info" and "hint" it is outright wrong. Those should not be treated as failures.
The failure tags include the severity as "type" attrribute but that makes no difference to applications that interpret the result XML. All they see is a failure. AFAIK, applications commonly only discriminate between errors, failed and skipped tests.
As a workaround we needed to disable all rules that we actually only wanted to downgrade in severity.
Ideally I'd like to have only "error" result in a JUnit report failure, "warn" maybe depending on an extra flag, and "info" and "hint" getting ignored for this report format. The latter two severities would then only appear in "lint" output.
Regarding the "treat warnings as errors" flag, maybe let yourself inspire by this Spectral issue: https://github.com/stoplightio/spectral/issues/935

As said before this tool as great and junit IS very cool for CI.
Do you think it's possible to add pass test when check is OK. Cause it looks so red 😅


Differences between official and open-source version

Hi,
I have another question in addiction to my last opened issue here https://github.com/thunderkeys/pdfconvertme-public/issues/6 ....
I've noticed that if I send a mail with an inline image and two Word attachments, I get returned all of them converted (body and all attachments).
It's possible to obtain this result also in open source version or is a prerogative of the official version offered by pdfconvert.me website?
Thank you very much.
Marco

working on updating the opensource version with the redo that I did of the main service version.  nothing super secret, just been a time constraint.  Will update when I have a new release.

give this branch a try ... it has a lot of changes, so things may break :)
https://github.com/thunderkeys/pdfconvertme-public/tree/oct2022_update


Filename missing when SyntaxError thrown

I extract from .flow.js files, and babel parses these quite well. But if I make a syntax error, the output isnt very helpful (see below). Could we catch the SyntaxError here and perhaps re-throw it with a filename?
/node_modules/i18n-extract/node_modules/babylon/lib/index.js:788
    throw err;
    ^

SyntaxError: Unexpected token, expected "," (27:17)
    at _class.raise (.../node_modules/i18n-extract/node_modules/babylon/lib/index.js:776:15)
...
    at _class.parse (.../node_modules/i18n-extract/node_modules/babylon/lib/index.js:5304:17)
    at parse (.../node_modules/i18n-extract/node_modules/babylon/lib/index.js:10095:38)
    at extractFromCode (.../node_modules/i18n-extract/lib/extractFromCode.js:79:32)
    at .../node_modules/i18n-extract/lib/extractFromFiles.js:30:54
    at Array.forEach (<anonymous>)
    at extractFromFiles (.../node_modules/i18n-extract/lib/extractFromFiles.js:27:10)
    at bootstrap_node.js:609:3

What do you think of the suggested solution. Should I create a PR?

@zbycz You are right to report this problem. It's important to have meaningful error messages. It can save people a lot of time.
I'm happy with the solution that changes extractFromCode.js to catch the error, log in the console the source and rethrow the original error.


Latent Modulation Implementation

Hi,
I have a question about latent modulation. The paper says that the shift modulation at each layer comes from a linear map on the latent vector. However, the code seems to map latent vector through a ReLU MLP and extract the intermediate layer output as the shift modulation for the original network.
Is this a typo in the paper? Or is it a new type of modulation that works better?

Hi, note that for the line you linked:
self.forward = hk.nets.MLP(self.layer_sizes + (self.output_size,), activation=activation)
the self.layer_sizes is set to (), an empty tuple here in the experiment.
Hence this mapping is just a linear map in practice. We found that using an MLP with more layers (i.e. non-empty layer_sizes) actually led to worse performance, rather counter-intuitively. We suspect this is an optimization/initialization issue, but didn't get round to resolving it. If you can find a way to make it work better with more layers, that would be a nice contribution! Hope that helps :)

Thank you very much for the explanation! That makes sense!


Support conversion to videos

Support for converting to video will be very useful, as it is smaller than GIF files and can take advantage of GPU hardware acceleration when playing.
There is a SkottieTool in Skottie that generates mp4 files from FFmpeg, I don't know if this is something I could learn from.
If you plan to support video, I can port this library to Node.js via napi-rs.

I'm not going to add nor maintain any code other than what is necessary to convert lottie files to gif files. Feel free to use the rlottie crate and build your own conversion library with it.

Thanks!