AwsCognitoJwtAuthenticationFilter bean is registered twice when placed in Spring Security Filter Chain

Spring automatically registers all filter beans and puts it in the top level filter chain.
This means that when adding the AwsCognitoJwtAuthenticationFilter to the spring security filter chain (using  addFilterBefore or addFilterAfter) it will be registered twice. Once in the top-level chain and once in the spring security filter chain.
Adding this will avoid spring registering it in the top level filter.
    @Bean
    public FilterRegistrationBean registration(AwsCognitoJwtAuthenticationFilter awsCognitoJwtAuthenticationFilter) {
        FilterRegistrationBean registration = new FilterRegistrationBean(awsCognitoJwtAuthenticationFilter);
        registration.setEnabled(false);
        return registration;
    }

What you're seeing here is the top level filter chain that the spring context is building. (All filters that are either registered via a FilterRegistrationBean, or GenericFilterBeans).
If you want to make use of Spring Security features, I think it's best to make sure your filter is hooked into the springSecurityFilterChain. To do that you make sure Spring doesn't register it automatically, and you create a new security config (via a WebSecurityConfigurer) to hook your filter in the spring security filter before or after an existing filter.
It is also mentioned here : https://spring.io/guides/topicals/spring-security-architecture/
The fact that all filters internal to Spring Security are unknown to the container is important, especially in a Spring Boot application, where all @Beans of type Filter are registered automatically with the container by default. So if you want to add a custom filter to the security chain, you need to either not make it a @Bean or wrap it in a FilterRegistrationBean that explicitly disables the container registration.

with this configuration all works fine but i don't know if this is a correct way
@Configuration
public class HcaWebSecurityConfigurerAdapter extends WebSecurityConfigurerAdapter {
	@Autowired
	private AwsCognitoJwtAuthenticationFilter awsCognitoJwtAuthenticationFilter;

	@Autowired
	private HcaAuthenticationEntryPoint hcaAuthenticationEntryPoint;

	@Override
	protected void configure(HttpSecurity http) throws Exception {
		http.cors().and().authorizeRequests().anyRequest().authenticated().and()
				.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS).and()
				.exceptionHandling().authenticationEntryPoint(hcaAuthenticationEntryPoint).and()
				.addFilterBefore(awsCognitoJwtAuthenticationFilter, UsernamePasswordAuthenticationFilter.class)
				.addFilterBefore(new MyHlJwtAuthenticationFilter(), UsernamePasswordAuthenticationFilter.class)
		;
	}

	@Bean
	public FilterRegistrationBean unregister(AwsCognitoJwtAuthenticationFilter awsCognitoJwtAuthenticationFilter) {
		FilterRegistrationBean registration = new FilterRegistrationBean<>(awsCognitoJwtAuthenticationFilter);
		registration.setEnabled(false);
		return registration;
	}
}

Looks ok to me ..  I'm probably going to disable the filter registration by default in the auto config of this library so  you don't have to do it in code anymore. That will solve this ticket and if the config above works for you we can also close issue #12

"disable the filter registration by default" i think is the correct coice
i have a question about:
.addFilterBefore(awsCognitoJwtAuthenticationFilter, UsernamePasswordAuthenticationFilter.class)
.addFilterBefore(new MyHlJwtAuthenticationFilter(), UsernamePasswordAuthenticationFilter.class)

i put aws and custom filter beforce UsernamePasswordAuthenticationFilter, but is not clear the order, actually works correctly but i fear that is a coincidence.
i need my custom fitler chained after Cognito, this is the correct way? i don't want that in a future version of spring change somethings internal and my code stop works.
i tried somethings like that but spring not start
.addFilterBefore(awsCognitoJwtAuthenticationFilter, UsernamePasswordAuthenticationFilter.class)
.addFilterBefore(new MyHlJwtAuthenticationFilter(), AwsCognitoJwtAuthenticationFilter.class)

Yes, misread your original config ....  that indeed didn't make the order explicit.
Your last snippet should work ... what is the error you are getting ?

sorry seams something wrong before, now works
just i receive 2 invocation per fiter:
@Configuration
public class HcaWebSecurityConfigurerAdapter extends WebSecurityConfigurerAdapter {
	@Autowired
	private AwsCognitoJwtAuthenticationFilter awsCognitoJwtAuthenticationFilter;
	@Autowired
	private MyHlJwtAuthenticationFilter myHlJwtAuthenticationFilter;
	@Autowired
	private HcaAuthenticationEntryPoint hcaAuthenticationEntryPoint;

	@Override
	protected void configure(HttpSecurity http) throws Exception {
		http.cors();
		http.authorizeRequests().anyRequest().authenticated();
		http.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
		http.exceptionHandling().authenticationEntryPoint(hcaAuthenticationEntryPoint);
		http.addFilterBefore(myHlJwtAuthenticationFilter, UsernamePasswordAuthenticationFilter.class);
		http.addFilterBefore(awsCognitoJwtAuthenticationFilter, MyHlJwtAuthenticationFilter.class);
	}

	@Bean
	public FilterRegistrationBean awsBean(AwsCognitoJwtAuthenticationFilter awsCognitoJwtAuthenticationFilter) {
		FilterRegistrationBean registration = new FilterRegistrationBean<>(awsCognitoJwtAuthenticationFilter);
		registration.setEnabled(false);
		return registration;
	}

	@Bean
	public FilterRegistrationBean myhlBean(MyHlJwtAuthenticationFilter myHlJwtAuthenticationFilter) {
		FilterRegistrationBean registration = new FilterRegistrationBean<>(myHlJwtAuthenticationFilter);
		registration.setEnabled(false);
		return registration;
	}
}

@Component
public class MyHlJwtAuthenticationFilter extends GenericFilterBean {}

@ddewaele also with only your filter using FilterRegistrationBean (enabled false) and addFilterBefore i receive 2 invocation.
another question are you sure to extends GenericFilterBean and not AbstractAuthenticationProcessingFilter ?

@ddewaele can you make version without autoregistration and read my pull request about a little fix ?


Move to streaming output

As discussed in #32, we should have a streaming output. Since there is so much data throughput, it is ideal to progressively write to disk rather than all at once. I was hoping to rewrite this with #32 but it looks like it won't be the case.

From a few month old notes:

New export method export/exportStream for engines

This is supported as of 3.0.0


Unable to open linux kernel .o and .ko files

Attempting to open an intermediate .o file or .ko module from the Linux kernel fails with:
Function entry point x86_64:0x0 is invalid

Dupe of #63.


Fix filetype confusion and CSV generation bug

#124 pointed out that the CSV representation of bitstreams is output into a FASM file instead of a CSV file - this fixes this and adds a couple of comments to be a little clearer. Also noticed a bug in the generation of this CSV, which was just thanks to some incorrect indentation so that's fixed here too.

Thanks, this now works for me and also the bits are not all zeros and there are also all tiles included!


Much higher IOPS results in random read test than other tools

I ran diskspd with these commands, to test out the storage system performance(4KB random read, QD16x16T) of different hardware platforms.
diskspd.exe -b4K -t16 -r -o16 -d39 -Sh E:\iobw.tst
Ran on EPYC2-7702 and PM1725a, with a 20GB prebuilt file with pseudo random content created with iometer. All 3 tests are done with the same file. But the results is way higher than the number given by fio and iometer, which both offer the similiar results, about 400K iops. But on intel platform with xeon platinum 8275CL cpu the results are quite consistent(all tests are about 800K iops).
I tried to create a new file for testing, but all 3 tests produced the same results as before. What could go wrong? Thank you.

BTW there's a similiar result by one of my friends using a PM981 512GB on a EPYC2 7502 platform, which was rated about 440K iops, but the result is also much higher(583K).
`C:\Windows\system32>diskspd.exe -b4K -t8 -r -o32 -d16 -Sh #1
Command Line: diskspd.exe -b4K -t8 -r -o32 -d16 -Sh #1
Input parameters:
    timespan:   1
    -------------
    duration: 16s
    warm up time: 5s
    cool down time: 0s
    random seed: 0
    path: '#1'
            think time: 0ms
            burst size: 0
            software cache disabled
            hardware write cache disabled, writethrough on
            performing read test
            block size: 4096
            using random I/O (alignment: 4096)
            number of outstanding I/O operations: 32
            thread stride size: 0
            threads per file: 8
            using I/O Completion Ports
            IO priority: normal

System information:
    computer name: EPYC
    start time: 2020/02/04 08:37:00 UTC

Results for timespan 1:

actual test time:       16.00s
thread count:           8
proc count:             64
Read IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
 0 |<PHONE_NUMBER> |       774645 |     189.12 |   48414.09 | #1 (476GiB)
 1 |<PHONE_NUMBER> |       776500 |     189.57 |   48530.02 | #1 (476GiB)
 2 |<PHONE_NUMBER> |       780051 |     190.44 |   48751.95 | #1 (476GiB)
 3 |<PHONE_NUMBER> |       778216 |     189.99 |   48637.27 | #1 (476GiB)
 4 |<PHONE_NUMBER> |      1554811 |     379.58 |   97173.23 | #1 (476GiB)
 5 |<PHONE_NUMBER> |      1555006 |     379.63 |   97185.41 | #1 (476GiB)
 6 |<PHONE_NUMBER> |      1554257 |     379.45 |   97138.60 | #1 (476GiB)
 7 |<PHONE_NUMBER> |      1554754 |     379.57 |   97169.66 | #1 (476GiB)

total:<PHONE_NUMBER>0 |      9328240 |    2277.34 |  583000.23`

I'll think about the fist case in a bit.
Re: the second case, a suspect pops out quickly.
The PM981 specification from Samsung shows 270K random read and 420K random write. The type of IO is very important. You're showing a random read result at 583K, which is then way higher (>2x) than Samsung's spec.
As to that spec: Samsung is providing no context on test conditions, but given the difference in behavior there is a clear suspect - device preconditioning and the amount/structure of the data that has been written to the device prior to the measured run.
What's probably happening is that the SSD has had very little data written to it (relative to capacity) since it was last cleared and the device can optimize away reading the SSD's NAND since it knows it hasn't been written - it just returns zeroes. This can obviously be done a lot faster since its just stressing NVMe, the speed of the device's FTL processor and transfers over the PCIe bus attachment. If you were to precondition the device with writes (random is usually the choice since that will randomize the SSD's physical<->logical block mapping tables) the read performance would probably settle close to where Samsung documents it: the data would actually be coming all the way from NAND.
Even if the SSD happens to be formatted with a filesystem and some data has been written to it (say, even an OS installation), its still quite possible for a fresh system device to have only had a small fraction of its capacity touched by the time of this test (you tell me?). If say only 50GB of data has been written out of the device's 512GB capacity (476GiB), ~90% of the reads could be optimized away.
The difference between IOMETER, FIO and DISKSPD in this case may simply be down to a somewhat more Windows-native IO path in DISKSPD's case.
Some references you can dip into, and/or simply search for "ssd device preconditioning" and you'll find a lot of discussion.
SNIA SDC 2011 talk slides
SNIA SSD Testing Specification
Bottom line is that this isn't simple on modern devices. It is easy to fall into behavioral traps or, in another sense, learn a lot about how your system and storage works.

I'll think about the fist case in a bit.
Re: the second case, a suspect pops out quickly.
The PM981 specification from Samsung shows 270K random read and 420K random write. The type of IO is very important. You're showing a random read result at 583K, which is then way higher (>2x) than Samsung's spec.
As to that spec: Samsung is providing no context on test conditions, but given the difference in behavior there is a clear suspect - device preconditioning and the amount/structure of the data that has been written to the device prior to the measured run.
What's probably happening is that the SSD has had very little data written to it (relative to capacity) since it was last cleared and the device can optimize away reading the SSD's NAND since it knows it hasn't been written - it just returns zeroes. This can obviously be done a lot faster since its just stressing NVMe, the speed of the device's FTL processor and transfers over the PCIe bus attachment. If you were to precondition the device with writes (random is usually the choice since that will randomize the SSD's physical<->logical block mapping tables) the read performance would probably settle close to where Samsung documents it: the data would actually be coming all the way from NAND.
Even if the SSD happens to be formatted with a filesystem and some data has been written to it (say, even an OS installation), its still quite possible for a fresh system device to have only had a small fraction of its capacity touched by the time of this test (you tell me?). If say only 50GB of data has been written out of the device's 512GB capacity (476GiB), ~90% of the reads could be optimized away.
The difference between IOMETER, FIO and DISKSPD in this case may simply be down to a somewhat more Windows-native IO path in DISKSPD's case.
Some references you can dip into, and/or simply search for "ssd device preconditioning" and you'll find a lot of discussion.
SNIA SDC 2011 talk slides
SNIA SSD Testing Specification
Bottom line is that this isn't simple on modern devices. It is easy to fall into behavioral traps or, in another sense, learn a lot about how your system and storage works.

I actually have the same thought, that's one of the reason why I insist to test the drive AFTER formating and preconditioning, and test the file directly. Because even you've preconditioned the RAW drive in advance the SSD's GC mechanics could still get in the way. My friend seems to be too busy to do that so I did the test myself, that spawns the first result.

OK, lets consider the second case resolved then.
The specifics of preconditioning can be all important (they were in the other issue you initially commented on, too). Can you describe what you do?
Your comment of "preconditioned the RAW drive in advance" ... and then formatting with a filesystem raises the potential of a big trap. It won't affect your read-after-write test ordering, but you should be aware that formatting re-TRIM's the device, tossing out all of the lba-pba preconditioning.
As far as the (PM1725a)[https://www.samsung.com/semiconductor/global.semi.static/Samsung_PM1725a_NVMe_SSD-0.pdf], the 2.5" U.2 form factor is rated for 800K random read. I'm not sure DISKSPD getting closer to the rated device capability than IOMETER or FIO is strictly speaking strange - I'm confident DISKSPD is actually running that IO in and out of the device. It may be some aspect of the more Windows-native IO path in DISKSPD relative to them.
If this is all side-by-side one after the other runs on the same file, the couple ideas I have are related to worker thread affinity and alternation on NUMA domains. How many logical numa nodes does the EPYC2-7702 expose? See if FIO/IOMETER exercise the same one run to run, they may alternate if they aren't being careful (at least by default) about how the platform will round-robin processes between NUMA domains to level load around the system complex. DISKSPD defaults to hard thread affinitization starting at socket 0 core 0, and ascending through the topology from there.
If the Intel processor only enumerates as a single NUMA it might suggest something about the relative attachment of the PM1725a to the topology, or some other aspect of the AMD design. I'm not immediately familiar with what that could be.

p.s. I don't see the 8275 (CL = Cascade Lake?) at Intel Ark spec site. 8270 and three variants of 8276 (8276, 8276M and 8276L).

OK, lets consider the second case resolved then.
The specifics of preconditioning can be all important (they were in the other issue you initially commented on, too). Can you describe what you do?
Your comment of "preconditioned the RAW drive in advance" ... and then formatting with a filesystem raises the potential of a big trap. It won't affect your read-after-write test ordering, but you should be aware that formatting re-TRIM's the device, tossing out all of the lba-pba preconditioning.
As far as the (PM1725a)[https://www.samsung.com/semiconductor/global.semi.static/Samsung_PM1725a_NVMe_SSD-0.pdf], the 2.5" U.2 form factor is rated for 800K random read. I'm not sure DISKSPD getting closer to the rated device capability than IOMETER or FIO is strictly speaking strange - I'm confident DISKSPD is actually running that IO in and out of the device. It may be some aspect of the more Windows-native IO path in DISKSPD relative to them.
If this is all side-by-side one after the other runs on the same file, the couple ideas I have are related to worker thread affinity and alternation on NUMA domains. How many logical numa nodes does the EPYC2-7702 expose? See if FIO/IOMETER exercise the same one run to run, they may alternate if they aren't being careful (at least by default) about how the platform will round-robin processes between NUMA domains to level load around the system complex. DISKSPD defaults to hard thread affinitization starting at socket 0 core 0, and ascending through the topology from there.
If the Intel processor only enumerates as a single NUMA it might suggest something about the relative attachment of the PM1725a to the topology, or some other aspect of the AMD design. I'm not immediately familiar with what that could be.

Both processors are running with 1 numa(AMD NPS1) mode, the EPYC2 7702 have 2 processor groups in windows, but the test threads are strictly running in the first group during the all the read tests, as shown in the task manager.
By preconditioning the raw drive I also mean run the write i/os directly into the raw drive for some time, in this case the drive may still think these are useless data and run garbage collection over them.

p.s. I don't see the 8275 (CL = Cascade Lake?) at Intel Ark spec site. 8270 and three variants of 8276 (8276, 8276M and 8276L).

correction: after update the 7702 at NPS1 is running with a single processor group in windows.
8275CL is a CLX 24C48T processor with 3.0 base clock and 3.6-3.9 turbo and 240w tdp.

Try to get to this view.
You'll see my 2950x surfaces 1 socket but 2 NUMA domains. From the hot cores in your Task Manager view, I strongly suspect you have 2 as well and its running in the second on this run. It may alternate back to the first if you stop/restart IOMETER/dynamo.

Try to get to this view.
You'll see my 2950x surfaces 1 socket but 2 NUMA domains. From the hot cores in your Task Manager view, I strongly suspect you have 2 as well and its running in the second on this run. It may alternate back to the first if you stop/restart IOMETER/dynamo.

Indeed there are still 2 groups. Tried directly binding the core affinity to cpu 0-31, the results are still almost the same.
By the way, by setting the ioengine to windowsaio the FIO should also use windows-native io path right?

I just noticed that on intel platform the result generated by diskspd is a bit lower than the other 2(this is consistent).

Testing with enterprise level SSD, besides preconditioning the device to
achieve a steady state, there is the amount of DRAM cache inside the
device. Which will interfere in write testing mainly. So need to be aware
of that as well.
On the CPU side EPYC added different scheme for core to cache relation, and
it is also different on Gen1 and Gen2 EPYC system. On Zeppelin die the LLC
relation is different from XEON.
[image: image.png]
EPYC Naples introduced a 1:4:2 model, one socket, that contains four NUMA
domains, and each NUMA domain contains two LLC domains. Rome provides a
NUMA model similar to the XEON, with a single socket and single NUMA
domain. However, each chiplet contains two separate LLC domains. A Rome CPU
package contains eight chiplets, and thus, 16 different LLC domains exist
within a socket & NUMA domain.
So thread scheduling will have a bigger impact when moving from cores in
the same NUMA but with different CPU cache lines.
On Wed, Feb 26, 2020 at 3:37 PM 111alan<EMAIL_ADDRESS>wrote:

Try to get to this view.
You'll see my 2950x surfaces 1 socket but 2 NUMA domains. From the hot
cores in your Task Manager view, I strongly suspect you have 2 as well and
its running in the second on this run. It may alternate back to the first
if you stop/restart IOMETER/dynamo.
Indeed there are still 2 groups. Tried directly binding the core affinity
to cpu 0-31, the results are still almost the same.
By the way, by setting the ioengine to windowsaio the FIO should also use
windows-native io path right?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/diskspd/issues/131?email_source=notifications&email_token=AJK4JUK3X2G7CT5YX4WRFJTRE34M5A5CNFSM4K4NLZO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENCKA6I#issuecomment-591700089,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJK4JUMS4MXURL5L32QHUZLRE34M5ANCNFSM4K4NLZOQ
.

Testing with enterprise level SSD, besides preconditioning the device to
achieve a steady state, there is the amount of DRAM cache inside the
device. Which will interfere in write testing mainly. So need to be aware
of that as well.
On the CPU side EPYC added different scheme for core to cache relation, and
it is also different on Gen1 and Gen2 EPYC system. On Zeppelin die the LLC
relation is different from XEON.
[image: image.png]
EPYC Naples introduced a 1:4:2 model, one socket, that contains four NUMA
domains, and each NUMA domain contains two LLC domains. Rome provides a
NUMA model similar to the XEON, with a single socket and single NUMA
domain. However, each chiplet contains two separate LLC domains. A Rome CPU
package contains eight chiplets, and thus, 16 different LLC domains exist
within a socket & NUMA domain.
So thread scheduling will have a bigger impact when moving from cores in
the same NUMA but with different CPU cache lines.
…

I was only testing read performance, so there shouldn't be a drastic performance drop when entering transition state or steady state like a write test does.
I actually did a research on EPYC2 Rome's interconnection architecture, you can see the article with the conceptual model here:
http://www.ssdfans.com/blog/2020/02/22/epyc2-rome拓扑结构测试分析三大方法/
The intention of me binding the test tools to cpu 0-31 is to only use the first node in the north bridge. And the highly used cores didn't alternate between cores at all. That still doesn't explain the difference between diskspd and the other two tools.

Let me see if I restate where we are right now.
In the initial case DISKSPD is driving significantly higher read IOPS than IOMETER/FIO, closer to the stated capability of the PM1725a on AMD EPYC. This seems like something which comes down to triaging why IOMETER/FIO are running slower, not DISKSPD running faster. I can't speak to the windowsaio support in FIO and whether it matches well with DISKSPD's native implementation but I think this is a great question for the FIO community. DISKSPD's source is right here to compare and reuse as needed under the MIT license. Can you take this in that direction?
In the second case DISKSPD is driving modestly lower on an Intel platform, the 8275CL?
The other thing I noted in the task manager view of the EPYC case was that IOMETER seemed to be dividing affinitized threads in a simultaneous multi-thread (~intel hyper-thread) aware fashion. DISKSPD does not have this intelligence by default, but it is possible to do so with the -ag flag (-ag0,0,2,4,6 ... etc. to assign threads to kgroup 0 core 0, 2, 4, 6 ...). If you do that on the Intel platform there should be some improvement, and you should be able to easily spot how IOMETER/FIO lay out as compared to DISKSPD ahead of time.
Getting this intelligence into DISKSPD either as a new default (some risk in confounding old-new result comparability) or an easily accessible option (not requiring extensive specification of core masks with -ag) has been a thought for a while.

Thx for the reply, seems that there's something deep going on, I'll also post this on fio's page for more discussion.
Yes on intel platform diskspd get a bit lower results and on AMD platform diskspd get a bit higher results, I just disabled the SMT and HT on both processors, the results seem to stay the same.

Just to refresh the thread. Several people have reported this issue to me. Digging a little deeper, at least in FIO for Windows, one of the main reasons is regarding IOCP - IO Completion Ports - which FIO is either not using or creating Blocking situation with a synchronous pattern. To have Diskspd behave like FIO (downgrade performance) we need to use -x switch which does not use IOCP. For the meantime if there is a need to benchmark Storage on Microsoft Windows we should stick with diskspd. Check thread on FIO https://github.com/axboe/fio/issues/1073

One more thing... usually you should expect higher random read performance than random write, but most (if not all) Samsung Client SSD drives, incl. PM981 mentioned above, use "Intelligent Turbo Write" technology to speed-up short bursts of writes resulting in better write performance in the data sheet (notice they say "up to 420k IOPS" or something like that). In reality the drive is unable to sustain that performance level, so any benchmark longer than a few seconds should show much lower performance.


improving readthedocs manual

Read the docs manual (docs/index.rst) could be cleaned up. Also it's better if the menu on the left is structured properly in a way where subheadings are hidden until you are clicking them.

@pdey could you look at the current manual and give some critical feedback on how to make it (much) better. Because it's something that can be done well, I want to make sure that we present the gold standard in clarity and thoroughness in terms of user docs. One interesting challenge is different levels of users.
I think it will need a separate issue (or few) but the way I see this is that once the docs are 100% in line with the solution, then the next steps are:

code fragments
notebooks which have code fragments but also commentary
tutorials both text and video
zero-doubt code comments
clear non-python error messages

All feedback to this topic is most appreciated :)

This is going to the CONTRIBUTION.md in the next merge, but will post it here for now:
2.3. Documentation standards
Because Autonomio is to sombased on creating a very high level abstraction, and a lot of "magic" takes place without the user knowing about it, it's very important that we are elaborate in our documentation. This needs to take place to the extent that users like the one who made the following comment are happy:

Strong documentation and tutorials, with an emphasis on data prep. Don't abstract things at the cost of understanding what is going on behind the hood. I want to know how it all works, and what is being done to my data - I just don't want to have to code every step.

This is the kind of situation we are trying to create; a lot of mundane but hard or time consuming to do things are taken care of on behalf of the user, but if the user so desires, he or she can get a full (and clear!) picture of what's being done.
In order to pass code review (required for merging pull requests), together with your code in the same pull request you must provide the following updates to the documentation docs/index.rst.

What does it do (a high level overview)
1.1) some use examples
What data input is requires
2.1) some use examples
What it outputs
What parameters are there (a high level overview)
4.1) What options each parameter has
4.2) What is the function of each parameter option

In short summary, the goal is that the user can completely understand 100% of the functioning of the features. If something is done so that the user can’t see it when they are running the commands, we have to explain exactly and thoroughly what kind of automations are taking place.

New docs are here


GraphQL

Have you thought about using GraphQL in MERN?

This can be a good addition. Will be great if you can fork this repo and build a GraphQL server. In mern-cli we can pass a flag such as mern <yourApp> --graphQL which will clone the GraphQL based repo.

I agree. GraphQL is powerful and very flexible. It would be a very nice choice to implement on this stack.

I would love to see this in mern!

+1

Graphql is bit complex and may not be suitable for beginners. We want to keep this project beginner friendly. We are planning to create a separate project for advanced users in future.

@somus, Any update on support for GraphQL?


Use https protocol by default

Hello,
I noticed that all requests are not using https by default, this can cause issues on some browsers served on https because of mixed content.
Code reference:
https://github.com/woubuc/node-updown/blob/master/src/utils/url.ts#L21

Thanks for reporting this. I hadn't realised that I put the default on http, it should most definitely be https!


[fix][sql] Fix presto license issue

Motivation
When releasing 2.11, encounter license header issue :
netty-transport-native-epoll-4.1.86.Final.jar unaccounted for in lib/presto/LICENSE

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Codecov Report

Merging #18934 (e391201) into master (050b310) will decrease coverage by 10.41%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master   #18934       +/-   ##
=============================================
- Coverage     47.35%   36.93%   -10.42%
+ Complexity     9384     1963     -7421
=============================================
  Files           623      209      -414
  Lines         59104    14402    -44702
  Branches       6146     1569     -4577
=============================================
- Hits          27987     5320    -22667
+ Misses        28100     8500    -19600
+ Partials       3017      582     -2435

Flag
Coverage Δ

unittests
36.93% <ø> (-10.42%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...rg/apache/pulsar/client/impl/PulsarClientImpl.java
43.85% <0.00%> (-0.95%)
:arrow_down:

...va/org/apache/pulsar/broker/service/ServerCnx.java

.../pulsar/broker/service/persistent/SystemTopic.java

...java/org/apache/pulsar/proxy/stats/TopicStats.java

...er/service/persistent/PersistentMessageFinder.java

...roker/admin/impl/OffloaderObjectsScannerUtils.java

...n/java/org/apache/pulsar/PulsarVersionStarter.java

...schema/ProtobufNativeSchemaCompatibilityCheck.java

...ar/broker/stats/prometheus/ManagedLedgerStats.java

...che/pulsar/broker/stats/prometheus/TopicStats.java

... and 405 more


Applied JuliaFormatter to all of Javis.jl files

PR Checklist
If you are contributing to Javis.jl, please make sure you are able to check off each item on this list:

[ ] Did I update CHANGELOG.md with whatever changes/features I added with this PR?
[x] Did I update the TODO.md (if applicable)?
[ ] Did I make sure to only change the part of the file where I introduced a new change/feature?
[x] Did I cover all corner cases to be close to 100% test coverage (if applicable)?
[x] Did I properly add test dependencies to the test directory (if applicable)?
[x] Did I check relevant tutorials that may be affected by changes in this PR?
[x] Did I clearly articulate why this PR was made the way it was and how it was made?

Link to relevant issue(s)
#123
How did you address these issues with this PR? What methods did you use?
I ran JuliaFormatter.jl's format(".") function to recursively format each file to show how the formatting looks like. Though this PR would not fix everything for making a GitHub Action, it will allow a point for each piece of code to be formatted via JuliaFormatter from which we could make a GitHub Action.
Further, each piece of our code from this point forward could be in compliance with JuliaFormatter.
What do you think @Wikunia ?

Hey @Wikunia could you tell me what else you do/do not like to see if JuliaFormatter could take care of it for us? I made your changes that you first saw were wrong and fixed those correctly.

I think we could. So the way I was thinking this could be correctly implemented with this PR:

Roll back any changes on this branch
When you are done with any PRs you want to merge, let me know
Merge this branch with master so that it is even
After I make this branch equal with master, let you know that I am going to update the GitHub CI with JuliaFormatter
Merge to master.
Check that all tests pass on master and nothing was broken

From there, any branch that we have can then be made compatible with JuliaFormatter. Before I continue on this, could you double check the documentation about any particulars you may want to have modified? Meaning, kwargs to be activated or disabled, specific formatting, etc. We could even define our own behaviors if we wanted to.
Here is the command I am currently using to modify all files in a directory:
format(".",
annotate_untyped_fields_with_any=false,
always_for_in=true,
whitespace_in_kwargs=true,
whitespace_ops_in_indices=true)

What are your thoughts @Wikunia ?

Sounds good. I had a look at the modifiers earlier. I think the ones you applied are the ones I would use. I would like to not indent @testset inside @testset but that's probably just me and not worth going in to.
You can merge the documentation PR which is not linked any way and then do this one. When exactly is the formatting applied? Inside the PR, right? I think merge conflicts might be handled correctly by formatting the PR I work on before I push and later there shouldn't be a problem.

I think that this is ready for merging now @Wikunia . What do you think? I created the GitHub Action that will run JuliaFormatter anytime a new push is made to master.

Do you want to revert all changes other than the GitHub Action to see whether this actually works?


natural sort order implementation

I have chosen the best sort algorithm that I had found. It is false by default and could be activated by using natural options. Solution for #60 issue. Docs should be updated to desribe new option.

Thanks for the addition!
I've merged, added a unit test and updated the docs for v2.3.0.

Thanks! I have implemented another great improvement I believe.
Before creating new PR I want to make a short description and ask you about naming of the new option. I have described it here #115


Functional enhancement

Do you have any plan do below function?

Traffic restrictions,like ops ,connection
Blacklist,White-list
Copy and distribute traffic

Traffic restrictions,like ops ,connection
Blacklist,White-list

We have implemented auth for scenarios like that, which I think is sufficient enough.

Copy and distribute

No, we are not planning to support this. There is https://github.com/vipshop/redis-migrate-tool which may suit the role for replicating data.


Adding theia manager resync to TAD and restructuring TAD files

This PR works does the following:

TAD api and rest files are restructured as per the original design of theia-manager
Theia Manager resync is added for TAD that helps remove stale DB and spark apps.
Consistency of datatypes for thoughput and algocalc for TAD.
Add allowlist for namespaces as an argument for TAD

Partially solves #168

/theia-test-e2e

/theia-test-e2e


Error with Hardware Implementation

Hello!
I have been trying to implement the Clover package for my drone. I was successful in replicating the simulation part. But now I am having problems with the hardware implementation. The following are my device descriptions:

Holybro s500 V2 drone kit
PX4 Cube pilot - Orange Cube
Raspberry Pi 4 + Raspberry Pi camera module

I have setup the clover rpi image on the pi and was able to compile it properly. The web tools seems to work and the aruco detection is working fine.
The problem is with the FCU and Qgroundcontrol part. I have set the PX4 parameters as stated in the Clover documentation(I know the parameters for my drone is different but wanted to check it out anyway).
When I run the automated selfcheck I am getting a lot of errors. I have attached the shell script below.
Kindly help and guide me on what I should do differently so that I can finish this project successfully .
Thank you!
pi@clover-3808:~ $ rosrun clover selfcheck.py
[INFO]: Performing selfcheck...
[INFO]: Image: version: v0.23
[INFO]: Board: Raspberry Pi 4 Model B Rev 1.4
[WARN]: clover.service: WARN: ros.rosconsole_bridge.mavconn: udp1: resolve error: Host not found (non-authoritative), try again later
[WARN]: clover.service: FATAL: ros.mavros: GCS: DeviceError:udp: resolve:Bind address resolve failed
[WARN]: clover.service: WARN: ros.mavros.sys: VER: broadcast request timeout, retries left 4
[WARN]: clover.service: WARN: ros.mavros.sys: VER: broadcast request timeout, retries left 3
[WARN]: clover.service: WARN: ros.mavros.cmd: CMD: Command 520 -- wait ack timeout (4)
[WARN]: clover.service: WARN: ros.mavros.sys: VER: unicast request timeout, retries left 2
[WARN]: clover.service: WARN: ros.mavros.param: PR: request list timeout, retries left 2
[WARN]: clover.service: WARN: ros.mavros.param: PR: request list timeout, retries left 1
[WARN]: clover.service: WARN: ros.mavros.param: PR: request list timeout, retries left 0
[WARN]: clover.service: WARN: ros.mavros.sys: VER: unicast request timeout, retries left 1
[WARN]: clover.service: WARN: ros.mavros.sys: VER: unicast request timeout, retries left 0
[WARN]: clover.service: WARN: ros.mavros.sys: VER: your FCU don't support AUTOPILOT_VERSION, switched to default capabilities
[INFO]: Network: OK
[INFO]: FCU: no version data available from SITL
[INFO]: FCU: not Clover PX4 firmware, check https://clover.coex.tech/firmware
[WARN]: FCU: unable to retrieve PX4 parameter SYS_MC_EST_GROUP
[WARN]: FCU: unknown selected estimator: None
[WARN]: FCU: unable to retrieve PX4 parameter SENS_BOARD_ROT
[WARN]: FCU: unable to retrieve PX4 parameter CBRK_USB_CHK
[WARN]: FCU: set parameter CBRK_USB_CHK to 197848 for flying with USB connected
[WARN]: FCU: incorrect cell voltage: 6.53 V, https://clover.coex.tech/power
[INFO]: IMU: OK
[WARN]: Local position: no local position
[WARN]: Velocity estimation: no velocity estimation
[INFO]: Global position (GPS): no global position
[WARN]: Preflight status: no data from FCU
[INFO]: Main camera: camera is oriented downward, cable from camera goes backward
[INFO]: ArUco markers: aruco_detect/length = 0.33 m
[INFO]: ArUco markers: aruco_detector/known_tilt = map (ALL markers are on the floor)
[INFO]: ArUco markers: aruco_map/known_tilt =
[WARN]: ArUco markers: cannot read aruco_map/visualization topic
[WARN]: ArUco markers: no map detection
[INFO]: Simple offboard node: OK
[WARN]: Optical flow: no optical flow data (from Raspberry)
[WARN]: Vision position estimate: no VPE or MoCap messages
[WARN]: Rangefinder: no rangefinder data from Raspberry
[WARN]: Rangefinder: no rangefinder data from PX4
[INFO]: RPi health: OK
[INFO]: CPU usage: OK
[INFO]: Boot duration: OK

By "implement" you mean to run, I guess?
What firmware version do you run on PX4 Cube pilot?
I'm not sure also, is there is connection between RPi and the flight controller? What do you see in /mavros/state topic for a start (rostopic echo /mavros/state)?

mavros doesn't seem to be connected to the FC (although is received IMU messages based on the posted selfcheck report, which is weird).
Anyway, Clover’s packages by default connect to /dev/px4fmu device file, which is created by udev only for certain flight controller models listed here https://github.com/CopterExpress/clover/blob/master/clover/udev/99-px4fmu.rules.
Did this file appear on your system? if not, you can add a rule to udev file, or just change the usb_device argument in mavros.launch file to existing usb device file, namely, to /dev/ttyACM0.

Hello Mr. Kalachev!
I changed the usb_device argument to /dev/ttyACM0. The FCU seems to be connected to the RPi.
But still I am not getting the Local position and velocity estimation. I have studied in the documentation that only if the velocity estimation passes, simple offboard should be performed.
How do I fix this problem? Kindly help me with this issue.
Thank you!
Selfcheck output:
pi@clover-3808:~ $ rosrun clover selfcheck.py
[INFO]: Performing selfcheck...
[INFO]: Image: version: v0.23
[INFO]: Board: Raspberry Pi 4 Model B Rev 1.4
[INFO]: clover.service: OK
[INFO]: Network: OK
[INFO]: FCU: CUBEPILOT_CUBEORANGE
[INFO]: FCU: Release 1.13.3 (17630207)
[INFO]: FCU: not Clover PX4 firmware, check https://clover.coex.tech/firmware
[WARN]: FCU: unable to retrieve PX4 parameter SYS_MC_EST_GROUP
[WARN]: FCU: unknown selected estimator: None
[WARN]: FCU: unable to retrieve PX4 parameter SENS_BOARD_ROT
[WARN]: FCU: unable to retrieve PX4 parameter CBRK_USB_CHK
[WARN]: FCU: set parameter CBRK_USB_CHK to 197848 for flying with USB connected
[WARN]: FCU: incorrect cell voltage: 6.19 V, https://clover.coex.tech/power
[INFO]: IMU: OK
[WARN]: Local position: no local position
[WARN]: Velocity estimation: no velocity estimation
[INFO]: Global position (GPS): no global position
[INFO]: Preflight status: OK
[INFO]: Main camera: camera is oriented downward, cable from camera goes backward
[INFO]: ArUco markers: aruco_detect/length = 0.33 m
[INFO]: ArUco markers: aruco_detector/known_tilt = map (ALL markers are on the floor)
[INFO]: ArUco markers: aruco_map/known_tilt =
[WARN]: ArUco markers: cannot read aruco_map/visualization topic
[WARN]: ArUco markers: no map detection
[INFO]: Simple offboard node: OK
[WARN]: Optical flow: no optical flow data (from Raspberry)
[WARN]: Vision position estimate: unable to retrieve PX4 parameter SYS_MC_EST_GROUP
[WARN]: Rangefinder: no rangefinder data from Raspberry
[WARN]: Rangefinder: no rangefinder data from PX4
[INFO]: RPi health: OK
[INFO]: CPU usage: OK
[INFO]: Boot duration: OK

I guess we've continued in the support chat, so I can close this?

Continued in support chat.

Hello Mr. Kalachev!
Sorry for the late reply. I was on vacation sonI couldn't reply.
Yes you can close the issue.
Thank you very much for your support.
Best regards,
Prathap.
On Tue, 13 Jun, 2023, 1:41 pm Oleg Kalachev, @.***>
wrote:

Continued in support chat.
—
Reply to this email directly, view it on GitHub
https://github.com/CopterExpress/clover/issues/489#issuecomment-1589136030,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOBF6JLRXBLDGKUAADYBWRTXLBGYDANCNFSM6AAAAAAYWVSKAE
.
You are receiving this because you authored the thread.Message ID:
@.***>


Infinite loop Issue

This issue is Infinite loop Issue in AFURLConnectionOperation.

significant changes have been made to the SSE and WS transports including adding support for new AFNetworking which resolved https://github.com/AFNetworking/AFNetworking/pull/2496.  if this still reproduces please provide logs or a minimal sample


Catastrophic forgetting

I used Lora  to fine tune my own dataset, but the model only replied to the content I had trained on, and I didn't know any other common sense content but Bunny-v1_0-2B-zh is ok
Do you have any training tricks？
self model

Bunny-v1_0-2B-zh

Hi @Gary2018X ,
Great thanks for your interest in Bunny!
Basically, when  finetuning Bunny on datasets with large domain gap from laion (pretrain set) and Bunny_695k (finetune set), you can try:

Pretrain and Lora finetune Bunny on laion and Bunny_695k
Merge Lora weights with frozen LLM
Add a new Lora module and finetune Lora on your custom dataset.

Currently in this GitHub repo, some modifications to our codes are needed if you want to add a new Lora and finetune. It’s schedules that such training pipeline to the main branch, or replied under this issue. Stay tuned!
Feel free to comment on this issue if you have further questions or share your inspiring idea about it. Thank you again for your question!
Regards
Russell
BAAI

Thank you very much for your reply!

model_name_or_path If I specify this parameter as  Bunny-v1_0-2B-zh, Do I still need to do the first and second steps?
or how can i  finetune  based on Bunny-v1_0-2B-zh?

Regards
Gary

If you want to finetune Bunny-v1_0-2B-zh by adding a new lora to merged Bunny-v1_0-2B-zh (only the new lora is trainable, and there are two loras in tota), you may follow:

download mm_projector and lora weights
Merge Lora with LLM:

python script/merge_lora_weights.py \
 --model-path /path/to/bunny_lora_weights \
 --model-base /path/to/base_llm_model \
 --model-typer qwen1.5-1.8b \
 --save-model-path /path/to/merged_model

in finetune_lora.sh, change model_name_or_path to /path/to/merged_model
in bunny_arch.py:

try:
    self.mm_projector.load_state_dict(get_w(mm_projector_weights, 'mm_projector'))
except:
    pass

(mm_projector has been loaded, so load_state_dict here throws an error, just ignore it. An update may solve this thoroughly.)
or you may also comment this:
if hasattr(config, "mm_vision_tower"):
            self.vision_tower = build_vision_tower(config, delay_load=True)
            self.mm_projector = build_vision_projector(config)

Please note that:
(a) Two loras doen't guarantee to work in your case. We don't have sufficient experimental data to support that two lora will finetune Bunny to data with large domain gap.
(b) You may also want to try using 1 lora but unfreeze vision tower and mm_projector. This is done by:

in train.py:

    #  unfreeze projector
    for p in model.get_model().mm_projector.parameters():
        p.requires_grad = True

    # unfreeze vision encoder
    for p in model.get_model().vision_tower.parameters():
        p.requires_grad = True

When evaluating, in bunny/model/builder.py:

if 'lora' in model_name.lower():
            print('loading unfrozen vision tower with lora')
            assert model_base is not None
            vision_non_lora_trainables = {k[19:]: v for k, v in non_lora_trainables.items() if
                                          k.startswith('model.vision_tower.')}
            vision_tower.load_state_dict(vision_non_lora_trainables, strict=False)
        else:
            print('loading unfrozen vision tower without lora')
            assert model_base is None
            from safetensors.torch import load_file
            vision_weights = {}
            for file_name in os.listdir(model_path):
                if file_name.endswith('safetensors'):
                    vision_weights.update(
                        {k[19:]: v for k, v in load_file(os.path.join(model_path, file_name)).items() if
                         k.startswith('model.vision_tower.')})
            vision_tower.load_state_dict(vision_weights, strict=True)

Just keep in mind that all these things are experimental and we can't guarantee that they will work. The motivation is that: a lora module is not enough so more parameters need to be set trainable.
Please comment on this issue if you have probleming implenmenting, or you would like to share your thoughts.
Regards
Russell
BAAI

Thank you very much for your professional answer
There is no problem with the training process
But there was a problem when I merged the models
  File "/root/siton-glusterfs-eaxtsxdfs/xts/projects/Bunny/script/merge_lora_weights.py", line 26, in <module>
    merge_lora(args)
  File "/root/siton-glusterfs-eaxtsxdfs/xts/projects/Bunny/script/merge_lora_weights.py", line 10, in merge_lora
    tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name,
  File "/root/siton-glusterfs-eaxtsxdfs/xts/projects/Bunny/bunny/model/builder.py", line 53, in load_pretrained_model
    model = BunnyQwenForCausalLM.from_pretrained(model_base, low_cpu_mem_usage=True, config=lora_cfg_pretrained,
  File "/opt/conda/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3531, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/opt/conda/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3958, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/opt/conda/lib/python3.9/site-packages/transformers/modeling_utils.py", line 812, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/opt/conda/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 348, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([151936, 2048]) in "weight" (which has shape torch.Size([151646, 2048])), this look incorrect.```

Regards
Gary

Hi @Gary2018X ,
You may want to share your srcipts for merging and your model configs so we can help you debug.
Currently, QWen has some bugs in vocab size. From our experience:

After training QWen, vocab_size is 151936, as shown in config.json
After merging lora, vocab_size is 151646 (This is weird...)
We train the merged model with an additional lora, it works
If we want to merge the second lora into it, an error is throwed: ValueError: Trying to set a tensor of shape torch.Size([151646, 2560]) in "weight" (which has shape torch.Size([151936, 2560])), this look incorrect.

I have double checked our uploaded lora weights, it has vocab_size 151936, so it's not expected to get errors when merging the first lora. However, my error in merging the second lora is different from yours. May you please share more details with us?

Quick answer: your first merging script and finetune_lora.sh was good, but the second merging script should be:
python script/merge_lora_weights.py \
  	--model-path ./checkpoints-qwen1.5-1.8b/bunny-lora-juzao-base-qwen1.5-1.8b \
  	--model-base /root/siton-glusterfs-eaxtsxdfs/xts/projects/Bunny/base_model \
  	--model-type qwen1.5-1.8b \
  	--save-model-path ./juzao_model_base

In the second merging, you are trying to merge a new lora with the previously merged LLM+lora, so --model-base should be set to where you saved the LLM+lora (as in finetune_lora.sh).
Why all these things happen are mentioned (here)[https://github.com/QwenLM/Qwen1.5/issues/29]. It's padding things in tokenizer. If you encounter similar errors in the future, please check  vocab_size to 151646 in config.json in your --output_dir.
Reach out to us if you still have difficulty using Bunny in your project!
Regards

Thank you very much for taking the time to answer my question.
I have successfully merged the models.
so sad
the model output is not good

q1 你是谁
base_model:I am an AI language model.
after lora model:我是AI语言模型，没有实体，没有生命，没有自我意识，没有情感，没有行动，没有意义，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有成功，没有变化，没有发展，没有进展，没有结果，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有成功，没有变化，没有发展，没有进展，没有结果，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有成功，没有变化，没有发展，没有进展，没有结果，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有成功，没有变化，没有发展，没有进展，没有结果，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有成功，没有变化，没有发展，没有进展，没有结果，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有成功，没有变化，没有发展，没有进展，没有结果，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有成功，没有变化，没有发展，没有进展，没有结果，没有目的，没有动机，没有喜好，没有恐惧，没有焦虑，没有挫折，没有失败，没有
q2 你是一名消化内镜领域专家，请合并重复内容，尽可能不要删除已有信息，简化以下描述，合并成一句话:黏膜光滑，未见充血水肿;黏膜光滑，可见少量黄绿色胆汁;黏膜粗糙，可见黏膜呈裂纹样可见散在发红凹陷灶，相互融合，边界清晰:黏膜粗糙，红白相间，以白为主，可见散在发红凹陷灶，相互融合，边界清晰。
base_model:消化内镜检查结果显示黏膜光滑，未见充血水肿，黏膜粗糙，可见黏膜呈裂纹样，散在发红凹陷灶，相互融合，边界清晰。 better answer
after lora model:可见一处发红、平坦型病变，形态规则，表面糜烂，边界清晰，未见出血，综合判断为非瘤变。

Is this related to my inference code?
num_beams=1, temperature=0.1, max_new_tokens=300
text_input = f"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\n{prompt} ASSISTANT:"
text_chunks = [tokenizer(chunk).input_ids for chunk in text_input.split('<image>')]
input_ids = torch.tensor(text_chunks[0] + text_chunks[1], dtype=torch.long).unsqueeze(0)
output_ids = model.generate(
            input_ids,
            max_new_tokens=max_new_tokens,
            temperature=temperature,
            num_beams=num_beams,
            do_sample=True,
            use_cache=True)[0]
llm_message = tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip()

There are a wide variety of training data and parameter settings, so I'm not entirely certain how much assistance I can offer you on this case. Here are some basic insights:

For Q1, the model doesn't stop generation correctly, so you may want to check eos_token.
Try loading the model in cli, and see if the model works well.
Check learning rate and the loss curve (you can draw it from log.txt).
Double check the training dataset to see if it's correct configed for MLLM training? In a lot of cases, introducing new datasets in finetuing makes the model underperform. You will need to wisely choose datasets.
Try to ask some special instructions on domain-specific datasets.

Hope it helps!
Regards

I will close this issue since we have reached a consensus on codings.


Fixed: security group position for the logged in user and added missing import for IonSpinner (#84)

Related Issues

Related Issue #84
Short Description and Why It's Useful

Fixed the positioning of the security group displayed for the loggedin user.
Added missing import for the IonSpinner

Screenshots of Visual Changes before/after (If There Are Any)

Before

After

Contribution and Currently Important Rules Acceptance

[x] I read and followed contribution rules

Issue verified and working fine in release v1.9.0.
https://jam.dev/c/5de5578c-68d0-4fb2-b3de-b18dc7770913


Proof of Concept: Add cache busting and auto-reloading to service workers

This is an attempt to address issue #2554 SWI3. Issues with service worker cache invalidation.
It also addresses SWI1 & SWI2 by auto reloading updated/different pages without user interaction.
To avoid confusion to the user, while the page is checking if the service worker is changed / exists, it displays a blank page.
Using the technique we damage a few of the advantages of PWA in return for 'easy of use' for developers and end users.

By default with PWA/SW, we can load the page instantly with the old/cached content. Using this technique, we must wait for a response from the server before showing the page. (Still works offline)

If we we find the page has changed, we do a force reload which means we've already hit the server once and waited for a response, and we then have to do the whole thing over again. If we'd just not used service workers we would have loaded the page faster.

This is just a POC. It is not a solid way to solve issues due to the issues above. Hopefully it might inspire some other ideas about how we could invalidate SWs and alternative ways to update the app without asking the user to refresh.

This looks excellent @ro-savage ! I've been testing your solution and it works perfectly! I've also been playing around with foreignFetch.
Is there anything pressing that you need eyes on? I've been getting more familiar with the intricacies of this project, and I can see that help is needed. I'd be happy to assist, and again great work! :rocket:

Thanks @sethbergman
Its really just a POC. If you run with this, you may as well just remove PWA/SW.
What really needs to be done is probably a way to identify a service worker with some sort of id/version number internally. So that we can always fetch service-worker.js, check an id and know if we should re-render or just show whats already rendered.
Currently having to wait for a request to return, then if it fails, performance a full refresh isn't ideal. We end up making the site take significantly longer to load on a slow connection.
Happy to hear suggestions, if someone else has an idea. Or feel free to PR to my repo.
The goal isn't to be merged but hopefully inspire a different way / better way.

Hello @ro-savage: I think https://github.com/facebookincubator/create-react-app/issues/2440#issuecomment-305896286 applies here as well.
Moving away from some of the offline-first serving strategies means reintroducing network latency into the critical path of getting the page on the screen.
If the decision is made to move away from defaulting to the fastest offline-first experience, it would make more sense to me to just disable service worker registration by default rather ship a less-effective service worker.

I agree with @jeffposnick here. The main purpose is to avoid latency, but that comes with tradeoffs. I guess we should just be more explicit about those tradeoffs (and maybe change the default), but this is giving us the worst of both worlds IMO. Appreciate the effort though!


Show an error when one of the network RPC urls are out

The following display on the room block shows the login because the RPC url has hit a request limit. An error should show so support can update the PRC url.

Details
When an RPC url for a network is rate limited or monthly limit has been reached the join room/create room buttons will not appear on the connect modal.
Steps to Reproduce Issue

Rate limit a RPC url
Switch to the network with the poor RPC connection
View the lack of join room/create room buttons

Acceptance Criteria
An error modal should pop up telling the user that the network RPC is limited

There have been many changes to this application since this issue was created. Will close this


Rendering files

Populates the body of the post with markdown for an image. Retrieves the image when displaying the post.
Also contains the markdown changes which would address #87 and #146

It seems like something has happened to the markdown. When I add a photo and go to the blog homepage the file appears in markdown text. Moreover, it doesn't make a lot of sense to include the image or image text in the excerpt. It looks like the following:

Also, when I click on the post and go to the post's page the markdown syntax is no longer visible. Instead there is a small dot, which is neither the image or the syntax. This probably has to do with the markdown missing but it's a little weird why its not showing the syntax either.

I am unable to add the same file more than once without adding another file in between. For example, I tried to add the image 5f732f9.png three times but it only added the image once until I added snip1.png and then I added the image again. You should be able to add the same image multiple times without having to add a different file in between.

Need to make sure that the added file(s) don't appear in the excerpt.

In the BlogTemplate.csproj file can you change the reference to markdig 0.13.0 to 0.13.3


Memory leak in ApplicationInsights

Task<IReadOnlyList<Windows.Storage.StorageFile>> leaks a lot.

Fixed by ApplicationInsights v2.0.0


temporarily fixes #91.

we should still investigate the root cause of this: people not going
through the autocomplete dropdown to add things to their wishlist
(whether by accident or their own malicious designs).

Coverage remained the same at 76.92% when pulling 478fe20e8edb41249f689b64b067f04596475b15 on wishlist-fix into 8bc824ff80ac7eb62c10ef08f0f8d787cc2f3d83 on master.

:+1:


Table aggregation - get Snorre tables to work

[x] Fix expiration of token when running in radix
[x] Rewrite to limit time usage on huge tables
[x] Have a look at the faults, seems wrong
[x] Include Parameters.txt results
[x] integration  test for table aggregation code
[x] Merge latest changes from add_psutil
[ ] Merge service and aggregation repos

sumo-tableaggregator will be merged into the aggregation service in a separate issue


[FEA] Improve PTDS performance

Is your feature request related to a problem? Please describe.
CUDA per-thread default stream (PTDS) is enabled for the plugin, but benchmark results haven't shown a big jump compared to the legacy default stream. Need to figure out why and try to improve the performance.
Describe the solution you'd like
At the moment there is no clear solution, but here are some ideas:

[ ] Profiling using a smaller dataset on the desktop to better understand performance bottlenecks
[ ] Benchmark with the shuffle manager and UCX enabled to ameliorate the effects of I/O
[ ] Fix synchronous cudaMemcpy and cudaMemset calls
[ ] Get rid of cudaDeviceSynchronize calls, most likely caused by thrust::device_vector/rmm::device_vector initialization
[ ] Improve CUDA event handling in RMM's pool_memory_resource

Describe alternatives you've considered
It's possible some of the benchmark queries are too I/O bound that increasing GPU concurrency does not help with reducing the wall clock time.
Additional context
Original issue to enable PTDS: #15

but benchmark results haven't shown a big jump compared to the legacy default stream.

There's an important option you really should consider: a single stream might not be the bottleneck?
I think your first checkbox is crucial (simple standalone benchmark profiling).

@rongou
"but benchmark results haven't shown a big jump compared to the legacy default stream"
what benchmark? and how much?
thanks

We are using TPCx-BB. When I/O is very saturated, PTDS is only slightly faster, but probably not statistically significant. On a single GPU there are 10-20% improvements. Still investigating.

Tried with the shuffle manager enabled. Looks like there is more memory pressure/fragmentation, so had to increase the number of shuffle partitions/reduce gpu concurrency. Seems to perform better than without the shuffle manager, but it's not clear PTDS gives bigger gains.

This should be considered done.


Important changes in the variant index

tl;dr
The current variant index has several shortcomings that affect our production infrastructure, our ability to update it to a newer GnomAD version and our ability to consider rare variants within the Genetics Portal. Here, we present a proposed plan to enhance the way we represent more complex and diverse disease-related variants in a scalable way. The plan includes some compromises that we will need to consider moving forward.
The current variant index is a process defined in the genetics-variant-annotation repository. At the moment, we pull all variants from GnomAD 2 and apply a MAF > 0.1%. The dataset results in 72M+ "common" variants. We have a dataset usually referred to as variant-annotation which includes richer information including all variant consequences (vep) and a post-processed dataset generally referred to as variant-index that stores less metadata for the purpose of the web application.
Current problems
Some problems associated with the current setup are:

GnomAD version. The version of GnomAD is lagging behind. @DSuveges made an attempt in 2021 (#491) to update it, but we struggled to make a decision on what variants to include and the downstream consequences. The full GnomAD dataset contains ~800M variants. A MAF >0.1% filter will reduce it to ~300M but still way higher than the current ~70M. As of today, this will have terrible consequences for some of the downstream processes as explained below.
The size of the data. While the size of the variant-annotation and the variant-index datasets it's still manageable. There is an explosion of data in the v2g and v2g2d datasets just because of the fact that we have so many variants. v2g is built so it calculates all variant-to-gene information as long as we have some features about the relationship. Because one of the features is vep this implies that we are effectively storing any variant-to-gene pair as long as they are within a window. This results in 1,031,401,898 records. The v2g2d is then a further explosion of this dataset. Overall, these datasets demand a large amount of computation, egress, ingestion time, etc. The latest ingestion process in ES/CH took 17h which is really a burden on any process improvement (https://github.com/opentargets/platform/issues/1961). There is also an associated financial cost on required big machines to deal with this data.
Absence of rare variants. The definition of a MAF threshold implies the genetics portal can't serve rare variants in any form. While according to opentargets/issues#2073 this has very little impact on GWAS associations lost (n=834), this threshold prevents us from using the same index to store other disease-associated variants (e.g. ClinVar, UKBB exome seq, etc.). Ultimately, we would like the same index to be used by the Platform variants and this is not possible at this stage.

Proposed plan
Next, we describe the series of actions, we are planning to execute in order to tackle the current problems.
Universe of variants
This will be the main change. We are planning to reduce the set of variants in the variant-annotation / variant-index to only those variants with disease-associated information. The hope is that by better scoping the purpose of the genetics portal we would streamline the process by at least one order of magnitude.
This is an important change that will imply that variants without any link to a disease/trait will not be available for search and will not have their corresponding page in the genetics portal. The understanding is that the Genetics Portal does not become a catalogue of canonical variants, but instead captures variants that are - directly or indirectly - associated with disease, independently of their allele frequency. As mentioned above, we are at the moment not providing a good atlas for variants, since we only have 70M out of the 800M variants that GnomAD currently catalogues.
The inclusion criteria we would require is to include all lead or tag variants in the genetics portal. This is a composite of several tables covering:

top loci
Fine-mapping
LD
Credible set

The intersection of all variants in these studies is calculated here and accounts for 5,388,925 unique variants (way less than the current 70M+).
After consideration we decided not to include in the variant index, variants that are captured only by some of the V2G features (e.g. eQTL, pQTL, pHIC, etc.). We understood GWAS-significant associations will still be accessible in the studies included in the Portal and the V2G data (e.g. eQTL included at a lower threshold) is there for the purpose of explaining the V2G relationship for the relevant V2D signals.
Although, we haven't quantified the impact this change will have in all datasets and processes is expected that the reduction of the variant-index from 70M to 5M will give us more than an order of magnitude gains in the v2g dataset (currently 1B), the v2g2d dataset (currently 1.5B) and the sumstats filtered dataset (currently 600M).
For the first stage of implementation, we are not considering the inclusion of any new variants outside the scope of the current Genetics Portal (e.g. rare variants), but we could expect this will be the case in the near future.
Annotations
In order to annotate the universe of variants, we will (left) join our variants to GnomAD and bring the same metadata we currently have. By left-joining implies that we could start having variants that are not present in GnomAD, therefore lack some of the metadata. This can potentially affect downstream processes (e.g. web application, L2G) which we will need to review.
As another layer of annotations, we want to include extra information about the consequences of coding variants. We are in conversations with the OTAR2048 team to bring Uniprot positions, AA changes, and - eventually - some extra annotation on the conservation or the consequence of the mutation in the protein. This could further assist downstream processes such as L2G.
Dependencies
There are currently some dependencies of the variant-annotations in the V2D pipeline that we will need to resolve. At the moment, it's used to map variants to rsIds.
For further iterations
There are other improvements that we have not yet discussed and will probably require further scoping down the road. Some of these are the inclusion of alternative variant identifiers (e.g. G4GH) or structural variants.
@DSuveges and @JarrodBaker will be leading the implementation plan

As mentioned above, we are at the moment not providing a good atlas for variants

Would that not be a valuable goal for the roadmap since it has a lot of downstream use cases and the union of gnomAD, UKB, ClinVar, etc. variants are not on their own large enough in number to be unmanageable yet (<1B)?  I'm thinking of projects like MARRVEL [1] and VarSome [2] which sort of do that but it seems like OT is in a great position to create something better, with comparable utility to the target index you all are already maintaining (as an outer join from multiple sources instead of a more limited, use-case specific subset).

Would it be feasible to have a comprehensive variant annotation table (e.g. all 800 M variants) that is used only when needed?

This might also be valuable in supporting future experiments to define what indirect disease association means (as @d0choa mentioned limiting support to).  I can see that being hard to iterate on if there's always something like a mult-hour join via Hail required first.  My two cents, fwiw.

We were totally missing that dependency.

Do you think association with disease from co-occurrence of literature mentions is another source that might substantially shrink the gap between disease associated variants and canonical variants?  I'm thinking of something like LitVar, and wondering if you all have considered anything like it given that you are tagging genes, diseases, and drugs in literature (variants seem a natural extension).

We are already addressing all this in https://github.com/opentargets/genetics_etl_python.
Please keep in touch if you want to know more


Terminated EC2 instances are showing in the toolkit.

Describe the bug
I have terminated a few EC2 instances. But it's still displayed in "Publish to existing Target". If I try to use and publish to one of the terminated instances it gives an error.
To Reproduce
Publish to a new EBS.
Terminate the EBS and EC2 instances.
This terminated instance won't disappear from the toolkit list.
If we try to upload again to a terminated instance it is giving an error.
Error: The stack named fujideveloper failed to deploy: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [RecipeBeanstalkEnvironment83CC12DE]. ): Resource handler returned message: "No Application named 'xxxxxxx' found.
Expected behavior
Terminated EC2 or EBS instances should disappear from the toolkit list after a certain time.
or should have a delete button to remove it from the toolkit listing.
Computer (please complete the following information):

Windows Version: Windows 10
Visual Studio Version: 17.4.1
AWS Toolkit for Visual Studio Version:<IP_ADDRESS>

Additional context

The deployment may have been managed with a CloudFormation stack. Can you look for a stack named fujideveloper? You will want to delete this stack in order to remove all resources associated with your deployment. Once the stack has been deleted, it will no longer appear in the "Publish to existing Target" list.

Thank you so much for your help. I have deleted the stacks. Now everything is perfect.


Time Picker with Left Nav

when selecting time the left nav is slide also.

Could you check with the lastest release. This should be fixed.

@alonronin I'm closing this issue. feel free to reopen is the issue still persists ^_^


Using DECS to display code snippets in the tutorial

fixes #41

@moxiegirl any update on this PR?

@CS76 Apologies for the delay. I should have added a message here.
I did take a look at this PR. Unfortunately, we would not want to embed files from an outside source on our site. It isn't good practice. S
o,  put this on hold while I finish some other work required for a forthcoming release. Of course, my assumption is I could create these links myself.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


How to setup refactoring for vue file?

i see that there is implementation for react, how to achieve this for vue?

Someone would need to create a PR similar to #468 or #400 but defining the treesitter captures and the code generation functions for Vue

Thank you for the hint! Based on your commits, i will try to implement when I have free time

Hi @TheLeoP , i craeted PR #482  for Vue. Maybe you can take a look? especially tree-sitter queries.
Somehow class and function scopes are not getting detected, in the coming PRs it can be improved, any improvement suggestions are welcome :)


Take dependency on the common repo.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: geeknoid
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [geeknoid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@geeknoid: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
94da8955ca53a8a9808d68c521d1b89ad5bbe791
link
/test istio-unit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@geeknoid: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-security-k8s-presubmit-tests.sh
e80581203b5792d7b496e523fa2407d21b883ccc
link
/test integ-security-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Making "Universal - Orientation Programs" come before "Universal - Calendar Items" on "Universal - Pages"

@teddybradford - we're going to have a few pages on the site look like this: https://risd-undergraduate-welcome.risd.systems/_wh_previews/universalpages/ddd79678-6979-2f81-9a91-2e839e90e640/ HOWEVER, we'd like for the image with text element:

to come before the header with dates element:

So, in order to do this, can you make "Universal - Orientation Programs" come before "Universal - Calendar Item" when they are both used on a "Universal - Pages"
Another example of a page that will be changed the same way, after this reordering will be: https://risd-undergraduate-welcome.risd.systems/_wh_previews/universalpages/d7ba5b01-8a68-8ea6-bd16-8e977036298e/
Here are the fields on the POSE page (we also use an image gallery here, but that falling at the bottom of the page is already good)

Thanks so much, and please just let me know if you have any questions at all. I'm back in tomorrow at 8:30. - Carrie

Completed! 🙂

All looks great - thanks @teddybradford ! Going to close this one out.


Better integration with jupyter/ipython

The following sequence now works:
    %import pythran
    %load_ext pythran.magic
    %%pythran -O3 -fopenmp
    #pythran export foo(in)
    def foo(n):
            ...

fixed and ready to be merged!


HiveTable glob pattern could match sub directories in the partition folder

This is not an issue as show with the test case in https://github.com/CommBank/permafrost/pull/20


Support both axum 0.6 and axum 0.7

Another solution for the support of axum 0.7 without dropping support for axum 0.6
linked to https://github.com/cdriehuys/axum-jwks/pull/28

Oh, this is interesting. Definitely something to keep in mind for the future. In this case, I don't see a reason to support both 0.6 and 0.7, so I went ahead and merged the other PR.


Support .sbss/.sdata and $gp relative data access

Most PS1 games have .sbss and .sdata for vars that have sizeof(X) <= -G 4. This also causes access to data in these sections in the same translation unit to use the $gp register.
Currently splat will output:
/* 5360 80014B60 4C00828F */  lw         $v0, 0x4c($gp)

But this need to be use a symbol e.g:
/* 5360 80014B60 4C00828F */  lw         $v0,  %gp_rel(D_deadbeef)

This will need a way to set what the GP value should be in the yaml, by default on PSX it is usually a pointer to the .sdata section. Also the generated linker script should output _gp = .data_start or similar.
For the sections .sbss can be handled the same as .bss (although .sbss tends to actually appear in the binary which I think is covered by another issue). And .sdata is the same as .data but with a different section name.
A final bonus might be a way to tell splat about the -G N size value. If splat knows this value to be 8 for instance then it will know not to output any data bigger than 8 bytes without splitting it.
Tasks:

[x] Update `($gp) handling when parsing ASM
[x] Add _gp = ... to linker file
[ ] Create sbss segment type; TODO: flag/option to NOT discard it from output?
[ ] Create sdata segment type; same as data just different section type
[ ] BONUS: -G N support (small_data_max_size: 4) ? warn if larger?

I feel like the support is good enough that this issue can be closed.. anyone think it should be kept open?


feat: add CRUD for user

We now have the functionality to create a user and log in.
It would be nice to also have a "my account" page or similar where you can change password, name etc and delete the account.

This could be complemented with a "card" on the top of the users dashboard where you see your account info, and have the option (a button on the card?) to go to the "my account" page.

signout button can also be located in the card

I think it would be nice if it were a new page for settings, where you can edit at least name and delete account. Update password?


fix: run the contentScript on all frames in the page

What: The contentScript will now run on all frames since when using iframes the contextmenu event doesn't bubble.

Why: Solves https://github.com/testing-library/which-query/issues/16

How: Add the all_frames: true flag to the manifest.

Checklist:

[ ] Documentation
[ ] Tests
[ ] Ready to be merged

Would love to hear your opinions @benmonro, @nickmccurdy, @kentcdodds, @smeijer

@MatanBobi I am so sorry for the delay.  I just started a new job and unfortunately won't be able to take on any work here.   If you'd like to take ownership fo the extension, i'd be happy to give you rights to publish it on the chrome dev portal.

alternatively if you want to move the features here over into https://github.com/testing-library/testing-playground that'd be fine too

@benmonro it looks like testing-playground already has this fix, so maybe we should generally talk about this project roadmap and whether we should deprecate it or keep maintaining it..
CC: @nickmccurdy @kentcdodds

I'd say that's up to the maintainers. Here's something to think about: https://kentcdodds.com/blog/favor-progress-over-pride-in-open-source
If it were up to me, I'd welcome the opportunity to deprecate.

Yeah I'm fine with that.
On Sun, Feb 7, 2021 at 6:39 AM Kent C. Dodds<EMAIL_ADDRESS>wrote:

I'd say that's up to the maintainers. Here's something to think about:
https://kentcdodds.com/blog/favor-progress-over-pride-in-open-source
If it were up to me, I'd welcome the opportunity to deprecate.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/testing-library/which-query/pull/19#issuecomment-774684814,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADBPBHFY2GSDOA42WWQPT3S52Q2ZANCNFSM4W4T7V4Q
.

--
Ben Monro
Software Developer

So should we archive this project with a message in the readme sending people to head over to testing-playground? :)

So should we archive this project with a message in the readme sending people to head over to testing-playground? :)

That's what I would do.

I've temporarily unarchived the repository so we can update the documentation to point to Testing Playground. I'm still closing this PR though, as it's not needed in Testing Playground.


Shift Selecting Systems doesn't let you merge systems

For some reason, merge systems button only works if you drag select, not if you shift select two systems.

On further research, it appears that for some reason, shift selecting doesn't trigger any observable actions, namely that it should trigger both the 'object:selected' and 'selection:created', and actually does neither. There doesn't seem to be any action triggered when using shift select, but it does get triggered only on drag select.


Add purge timer for archived entities

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1428595
Periodically purge all archived container entities that were deleted more than 6 months ago.
This change currently applies only to containers but it may be used generically for any other areas of the project similarly to ArchivedMixin
@simon3z @moolitayer PTAL
@miq-bot add_label providers/containers, wip
Edit: I tested this out locally in the console with:
MiqQueue.put_unless_exists(:class_name => "Container", :method_name => "purge_timer", :zone => MiqServer.my_zone, :priority => MiqScheduleWorker::Runner::SCHEDULE_MEDIUM_PRIORITY)

I expected to see the PurgingMixin somewhere, but I don't see it.  How does purge_scope get used?

@kbrock can you also take a look?  🙇

I expected to see the PurgingMixin somewhere, but I don't see it. How does purge_scope get used?

@fryguy i didnt understand purging correctly. I made some changes and mostly borrowed examples from app/models/event_stream/purging.rb.

Added tests

@miq-bot remove_label wip

@zeari
Ok, so this seems like it is "container" related not "archived" related.
Let's change the names accordingly.
Every model used to define different purging methods but we were able to consolidate most of those methods into PurgingMixin. Think we're down to just 2 patterns.
So, if you feel you are duplicating a lot of code from the ::Purging classes, consider moving a few of the methods into the mixin.

@zeari
Ok, so this seems like it is "container" related not "archived" related.
Let's change the names accordingly.

@kbrock
I named it ArchivedPurgibgMixin with this comment by @Fryguy in mind: https://github.com/ManageIQ/manageiq/pull/13686#issuecomment-281713367

I don't like that this is container specific, in particular the name of the mixin implies that it can only be used for containers, which isn't necessarily true.

I thought it would apply here too. How about unifying this new purging mixin with the pre-exisisting ArchivedMixin? btw we can also change archived_entities_purge_timer to go over all models that include the mixin instead of listing each of them by name

The variable is dealing with containers and job is dealing with containers. would like that renamed
A majority of the mixin seems like basic pruning functionality. Can we purge, purge_timer, and purge_queue into the parent class.
Can you use MiqReportResult::Purging as a basic implementation?
def purge_timer
  purge_queue(*purge_mode_and_value)
end

I'm pretty sure we only have 2 different versions of this method.
Ping me if you want me to create a PR for this

Sorry, @zeari - I did not forget about you here.
This got me looking at all the various purging examples out there. Put in #14676 to make the implementations more consistent.
There are 3 examples of purging that I like a lot:

Ones that want :remaining for purging DriftState::Purging and MiqReportResult::Purging.
simpler ones PolicyEvent::Purging

Think this should simplify your implementation greatly.
I think of PurgingMixin as the general implementation, and this module should have the name Container in it. For all other implementations, we implemented one perger per model. I understand the desire for the generalization here.

@kbrock should this wait until the normalization is merged in #14676?

@kbrock Or, do you want to get this PR merged and add a commit in #14676 to normalize this for containers?
I'm guessing you're probably best to make the change since you've been refactoring them all.

@zeari It is up to you. The changes to PolicyEvent::Purging are minimal.

So if you want to follow the current code in PolicyEvent, then I can delete those 2 methods in my other PR.
Or you can wait for that PR to be merged and use the newer version of PolicyEvent.

Of note, the PolicyEvent::Purging#purge_associated_records is a big performance hit. So avoiding that is good. You don't need it, so it is more of a side note.

I think of PurgingMixin as the general implementation, and this module should have the name Container in it. For all other implementations, we implemented one purge module per model. I understand the desire for the generalization here. So while out of the norm, I'm open to it.

Im starting to think that we should have a module per model for consistency with the rest of the project so ill break down that mixin.

Or you can wait for that PR to be merged and use the newer version of PolicyEvent.

I think ill go with this option so then ill be familiar with the code thats going in for this BZ :sweat_smile: .
Thanks @kbrock !

Added modules and specs for each entity
@kbrock @simon3z Please review

ping @simon3z can you approve the changes?

@miq-bot add_label fine/yes euwe/yes

@miq-bot add_label fine/yes, euwe/yes

Euwe backport details:
$ git log -1
commit 3cc76af9a8a66a1777120527858f0d166b43c867
Author: Keenan Brock<EMAIL_ADDRESS>Date:   Tue Apr 18 11:44:10 2017 -0400

    Merge pull request #14322 from zeari/purge_containers

    Add purge timer for archived entities
    (cherry picked from commit b8b9c81719111cdbc2f27e8fee74e572bbb9c128)

    https://bugzilla.redhat.com/show_bug.cgi?id=1448148

@zeari  @kbrock It looks like this PR depends on https://github.com/ManageIQ/manageiq/pull/14676.  Travis is now failing on Euwe after backporting this PR.  Not sure if it's ok to backport https://github.com/ManageIQ/manageiq/pull/14676 (it's currently not marked as euwe/yes) to Euwe branch, or if you'd rather create a separate Euwe PR to address the Travis failure.

Fine backport details:
$ git log -1
commit 45b315a1eee6980730cac7b24f35a165837aa3ce
Author: Keenan Brock<EMAIL_ADDRESS>Date:   Tue Apr 18 11:44:10 2017 -0400

    Merge pull request #14322 from zeari/purge_containers

    Add purge timer for archived entities
    (cherry picked from commit b8b9c81719111cdbc2f27e8fee74e572bbb9c128)

    https://bugzilla.redhat.com/show_bug.cgi?id=1458333


[Feature Request] - Dark Theme

It would be nice if there was an option to change the color scheme to something darker.

I would be happy to help. Pleases assign me.


Improve dev-tools for docker users

Simplify checks for docker container
Comment all important lines for better understanding
Reset translation file if only the date changed
Remove default environment variable (causes error for gettext and Django doesn't need it for default settings)

I just fixed the script to work for users who don't have the german locale configured on their machines.


Missing method error after updating

Just updated to 1.8.4 and get this with every command including stripe help:
/Users/ryanwood/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/specification.rb:2158:in `method_missing': undefined method `this' for #<Gem::Specification:0x3fc02894ca50 stripe-cli-1.8.4> (NoMethodError)
	from /Users/ryanwood/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/specification.rb:1057:in `find_active_stub_by_path'
	from /Users/ryanwood/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:64:in `require'
	from /Users/ryanwood/.rvm/gems/ruby-2.3.1@global/gems/stripe-cli-1.8.4/bin/stripe:3:in `<top (required)>'
	from /Users/ryanwood/.rvm/gems/ruby-2.3.1@global/bin/stripe:23:in `load'
	from /Users/ryanwood/.rvm/gems/ruby-2.3.1@global/bin/stripe:23:in `<main>'
	from /Users/ryanwood/.rvm/gems/ruby-2.3.1@moonclerk/bin/ruby_executable_hooks:15:in `eval'
	from /Users/ryanwood/.rvm/gems/ruby-2.3.1@moonclerk/bin/ruby_executable_hooks:15:in `<main>'

Seems like a similar issue to #20.

okay, so I dug into this a little bit and found this discussion
Can you confirm that your RubyGems version is prior to 2.6.3?  If so, I think that is the issue.
Please let me know what you find out.  Thanks!

That was it. I was on 2.5.1. Upgraded to the latest (2.6.7) and it fixed the issue. Thanks!


Reward card colour changes

The Chore
We are re-using some colours on the reward cards on the rewards ad competitions page.
Suggest we change it to be

[ ] Liquidity provision - blue
[ ] Price making - pink
[ ] Price taking - Green
[ ] Average position - Orange
[ ] Relative returns - purple
[ ] Returns volatility - yellow
[ ] Network / boring rewards - White

[ ] Staking rewards
[ ] Validator ranking
[ ] Market creation

Additional details / background info

Should we display "Ended" or something else for cards like that:

It says ended in -27 epochs, which is technically fine, but I think we could do better, e.g. "Ended", etc.


⚠ Hartman Foundation has degraded performance

In b1482a5, Hartman Foundation (https://gordonhartman.com/) experienced degraded performance:

HTTP code: 200
Response time: 2745 ms

Resolved: Hartman Foundation performance has improved in 795c171.


x-trim in strings will result in joi trim(true) injection

As the title says.
Working on a few new projects with existing code base I have found some mistakes relating to untrimmed input.
Or, in some cases, the code trims the input and then check string length.. if this else throw 422.. all as additional code within the domain method.
With this PR the following openapi spec:
type: object
required:
  - name
properties:
  name:
    type: string
    minLength: 1
    x-trim: true

Will result in the Joi validation looking like:
Joi.object({'name':Joi.string().trim(true).min(1).required()...

It is an improvement but.. as the majority of string inputs should be trimmed and only a few actually want the additional white-space, I think this PR would be better written so the dev opt's in to not trim whitespace on a given schema object. The api spec object might look somthing like:
type: object
required:
  - name
properties:
  name:
    type: string
    minLength: 1
    x-dont-trim: true

So that passing in x-dont-trim would result in the Joi validator output being:
Joi.object({'name':Joi.string().min(1).required()...

Currently the following yaml will pass the Joi validation if the user adds simple whitespace which is technically correct but very rarely desired:
type: object
required:
  - name
properties:
  name:
    type: string
    minLength: 1

Sounds good to me.
Wont "x-trim: false" be the same as "x-dont-trim: true" ?


Feature Request: Bypass the plugin and open the link in the external browser when clicked with a modifier (ctrl or alt or..)

Feature Requested
Hi, it would be nice to have a simple way to open a link in the external browser (there is already a way if you first open the link in surfing plugin and then ask it to open it in the external browser) using a keyboard modifier when you click (ctrl, alt ...) that would make the handler return early if it's pressed during the click therefore letting obsidian treat it in the default way and open it in the external browser. Apparently this plugin used to work like that too https://github.com/PKM-er/Obsidian-Surfing/pull/58 ?
Thanks.
Relevant Screenshot
No response
Checklist

[X] The feature would be useful to more users than just me.

Hi,
I also vote for this functionality at least, but it would be good  if 'Open link in default browser' also works well See: https://github.com/PKM-er/Obsidian-Surfing/issues/217#issuecomment-1961464091
Thank you for considering.


Input tag wrapped by empty class div

The datepicker component add an empty-class div as wrapper, but the real wrapper sould be the parent div (the one with class="vdp-datepicker").
In this way when I add a wrapper-class prop to datepicker, that class is applied to the outermost div.
In short, I define this datepicker
<datepicker :minimumView="'year'"
:format="'yyyy'"
:openDate="startingDate"
v-model="currentYear"
placeholder="Select year"
wrapper-class="ui action input">
<button slot="afterDateInput" class="ui button" @click="confirmYear">Confirm</button>
</datepicker>

I am using semantic-ui styling here and this is what I get

And this is the HTML generated

<input type="text" open-date="Sun Jan 01 1950 00:00:00 GMT+0100 (Ora standard dell’Europa centrale)" placeholder="Select year" readonly="readonly" autocomplete="off">
<button class="ui button">Confirm</button>

If I manually remove the "Not needed tag!!", all is working

Am I doing something wrong?
I have done a sandbox here https://codesandbox.io/s/n4rz2nwok4

I made a pull-request for this #689

Hacked it and fixed with the following, until the related pull request is merged.
.vdp-datepicker>div:first-of-type {
    display:inline;
}

I have same problem, help me please!


[WIP] Add the 'none' sort algorithm to OrderedImportsFixer

I need some feedback on the following matter.
It is now possible to configure the fixer this way:
'sortAlgorithm' => 'none',
'importsOrder' => null,

which basically disables the fixer.
Should it be allowed to use such configuration, or should I implement a check for it?
Closes #3543

Am I supposed to use "Fix #3545" in my commit message? Or in PR title?

no.
closes #3543 and similar in opening comment are enough

Should it be allowed to use such configuration, or should I implement a check for it?

There are other fixers that can be configured in a way that disables them. Though it doesn't make much sense to use it, such configuration is not considered invalid currently, so we don't prevent it.

Thank you @EvgenyOrekhov.


Improve chargeback performance

:question: Goal
We have in chargeback big query which is loading everything. This query is on MetricRollup table. Results (are array of big AR objects) is input to object of ConsumptionWithRollups which is responsible to get aggregated metric values from set of MetricRollups.
:mortar_board: Facts

we don't need array of big AR objects in object ConsumptionWithRollups - ConsumptionWithRollups just need one. (to get resource and his relations)

:point_right:  How
This big query will be replaced by SQL and which will be return only values of needed metrics columns, tag column,...
Object of ConsumptionWithRollups will receive only array of values and not array of AR objects.
:1234: Numbers
:leaves: Enviroment
VMs: 877

-
Objects
Allocated Memory(GB)
Queries
Time(s)

before

8,05

922.44

after (without  https://github.com/ManageIQ/manageiq/pull/17591  )
108_696_653
0,46
17_614
113,78

Next Step

[ ] WIP https://github.com/ManageIQ/manageiq/pull/17591 (optionally)

Links
https://bugzilla.redhat.com/show_bug.cgi?id=1566452

[x] https://github.com/ManageIQ/manageiq/pull/17538
[x] https://github.com/ManageIQ/manageiq/pull/17552
[x] https://github.com/ManageIQ/manageiq/pull/17558
[x] https://github.com/ManageIQ/manageiq/pull/17558
[ ] https://github.com/ManageIQ/manageiq/pull/17560

@gtanzillo @JPrause @simaishi

@kbrock @NickLaMuro


Editing a persistent highlighter's name creates a duplicate record

Steps to repro

Create a new persistent highlighter
Click "edit"
Update the highlighter's name
Press "save"

Expected
highlighter name is modified
Actual
The old highlighter exists. A second highlighter with the new name is created.
https://user-images.githubusercontent.com/693981/154815511-9eb0e3e7-1ff4-4651-b5df-8adcc23042e7.mov

Sorry, can't fix this... too busy with https://github.com/nothingislost/obsidian-embedded-query-control/issues/2 ;)

I accept this 🙇 🙇