Slow accuracy metric

🐛 Bug

Accuracy class from metrics.classification is too slow, compared to pytorch vanilla implementation.
To Reproduce
Try the following demo code.

Code sample

from time import time
import torch
from pytorch_lightning.metrics.classification import Accuracy

def test_acc(pred, targ, get_acc, steps=1000, rep=10):
    tsum, mean_res = 0, []
    for i in range(rep):
        res = []
        ts = time()
        for j in range(steps):
            res.append(get_acc(pred, targ))
        te = time()
        tsum += te - ts
        mean_acc = torch.stack(res).mean().item()
        mean_res.append(mean_acc)

    return (tsum / rep), mean_res

def demo(batch_size=32, num_classes=10, device='cuda', steps=100, rep=5):
    print("Testing  bs: %d,  num_cls: %d,  device: %s,  steps: %d,  rep: %d" % (batch_size, num_classes, device, steps, rep))

    pred = torch.randint(0, num_classes, (batch_size, )).to(device=device)
    targ = torch.randint(0, num_classes, (batch_size, )).to(device=device)

    pl_acc = Accuracy(num_classes=num_classes)
    def pt_acc(pred, targ):
        return pred.eq(targ).to(dtype=torch.float).mean()

    pl_time, pl_res = test_acc(pred, targ, pl_acc, steps, rep)
    pt_time, pt_res = test_acc(pred, targ, pt_acc, steps, rep)

    if pl_res != pt_res:
        print("Results mismatch!")
    print("Lightning: %2.6fs" % pl_time)
    print("Vanilla  : %2.6fs" % pt_time)
    print("")

if __name__ == "__main__":
    demo(32, 10)
    demo(512, 10)
    demo(32, 100)
    demo(512, 100)
    demo(32, 1000)
    demo(512, 1000)

    demo(32, 10, 'cpu')
    demo(512, 10, 'cpu')
    demo(32, 100, 'cpu')
    demo(512, 100, 'cpu')
    demo(32, 1000, 'cpu')
    demo(512, 1000, 'cpu')

On my machine, the results are like the following.
Testing  bs: 32,  num_cls: 10,  device: cuda,  steps: 100,  rep: 5
Lightning: 0.378017s
Vanilla  : 0.002766s

Testing  bs: 512,  num_cls: 10,  device: cuda,  steps: 100,  rep: 5
Lightning: 0.372391s
Vanilla  : 0.002776s

Testing  bs: 32,  num_cls: 100,  device: cuda,  steps: 100,  rep: 5
Lightning: 3.346141s
Vanilla  : 0.002812s

Testing  bs: 512,  num_cls: 100,  device: cuda,  steps: 100,  rep: 5
Lightning: 3.335049s
Vanilla  : 0.002660s

Testing  bs: 32,  num_cls: 1000,  device: cuda,  steps: 100,  rep: 5
Lightning: 32.751692s
Vanilla  : 0.002666s

Testing  bs: 512,  num_cls: 1000,  device: cuda,  steps: 100,  rep: 5
Lightning: 32.967594s
Vanilla  : 0.002762s

Testing  bs: 32,  num_cls: 10,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.128131s
Vanilla  : 0.001317s

Testing  bs: 512,  num_cls: 10,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.128481s
Vanilla  : 0.001403s

Testing  bs: 32,  num_cls: 100,  device: cpu,  steps: 100,  rep: 5
Lightning: 1.117403s
Vanilla  : 0.001275s

Testing  bs: 512,  num_cls: 100,  device: cpu,  steps: 100,  rep: 5
Lightning: 1.166188s
Vanilla  : 0.001330s

Testing  bs: 32,  num_cls: 1000,  device: cpu,  steps: 100,  rep: 5
Lightning: 10.927701s
Vanilla  : 0.001229s

Testing  bs: 512,  num_cls: 1000,  device: cpu,  steps: 100,  rep: 5
Lightning: 11.560013s
Vanilla  : 0.001310s

100x ~ 10000x more time is needed for pl's accuracy, depending on the num_classes argument.
Expected behavior
Pytorch-lightning's accuracy function should take similar time compared to the pytorch vanilla implementation, and it should not depend on num_classes too much.

Environment
* CUDA:
	- GPU:
		- GeForce GTX 1050 Ti
	- available:         True
	- version:           10.2
* Packages:
	- numpy:             1.19.0
	- pyTorch_debug:     False
	- pyTorch_version:   1.5.1
	- pytorch-lightning: 0.8.1
	- tensorboard:       2.2.2
	- tqdm:              4.47.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.6.9
	- version:           #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020

I tested the same code from two other machines (each with 2080 ti and with V100), and they give similar results.
Additional context

I'm not sure this issue falls into the bug report or the feature request category. Please relocate this as you like. :)

Eh, now this issue looks a lot like a enhancement issue. Sorry :sweat_smile:
I saw the implementation of the accuracy function, and I noticed a few metric functions depend on the stat_scores_multiple_classes function.
I think this function is way too slow, as it doesn't run parallel on num_classes. It would be great if we can reimplement this function only with vanilla pytorch operations, but I can't find a way for that.
Considering the huge complexity gap, I think implementing on ad-hoc basis for each metric would be better, though it might be a bit ugly.

Hi @Diuven , I was able to reproduce your results.
I agree, that this is a huge slowdown and we should fix it. Unfortunately, I am not quite sure, why this happens.
I have profiled the code and by far the most time is taken to compute the stat scores.  Although I don't know why this takes that long in PyTorch, I am afraid, that we need a more general version. Do you have a proposal on how to parallelise the computation of stat scores among classes?
To be honest I don't like to duplicate stuff there. But I agree, that we should speed the current implementation up :)

Hi @justusschock , thanks for your reply.
Yeah, I agree on not having duplicate codes in pl. It would be great if we simply replace stat_scores_multiple_classes with faster version.
Though I haven't profiled the code, I think the main reason is that there is no parallelism on num_classes. In current implementation of stat_scores_multiple_classes, we simply run for loops to compute states for each classes.
In the results of the demo code, the elapsed time also heavily depends on num_classes, so it supports the claim above.
So if we replace the for loop, for c in range(num_classes):,  in the function with the parallelized code, it would be much faster. Or at least it wouldn't depend on num_classes.
Making a CUDA library just for this function would work, but that would be our last option, considering the pl's philosophy and current state. (I think there's no CUDA code in this repo, so it would be an overkill to make a CUDA library only for this)
I'm not sure about whether this will work well or not, but I think parallelized scatter function in pytorch might help.

You're right that this would be great. Mind putting together a draft? I would for sure help you with that.
Also it's worth mentioning, that PLs version will always be a bit slower than the pure pytorch version, since it does a bit more. I also checks all incoming and outgoing arguments for device and type and converts them if necessary :)

Here's my draft:
def stat_scores_multiple_classes(pred: LongTensor, targ: LongTensor):
  match_true = (pred == targ).long()
  match_false = match_true.logical_not()
  # tps, fps, tns, fns, sups initialized as same
  pred_true = pred * match
  pred_false = pred - pred_true

  tps.scatter_add_(dim=0, pred, match_true)
  fps.scatter_add_(dim=0, pred, match_false)
  fns.scatter_add_(dim=0, targ, match_false)
  tns = pred.size(0) - tps - fps - fns
  sups.scatter_add_(dim=0, targ, torch.ones_like(match_true))

  return tps, fps, tns, fns, sups

Please check if I missed something big there. If it seems right, I'll work on it and make pull request.

Also it's worth mentioning, that PLs version will always be a bit slower than the pure pytorch version, since it does a bit more. I also checks all incoming and outgoing arguments for device and type and converts them if necessary :)

Yeah, that's a really great part of PL, and why I like it. :smile_cat: I agree on compromising a bit of speediness to get huge convenience. All we need is just comparable performance to vanilla case, in my opinion.

I checked the code below with the demo code at the top, and it was 10x faster than the previous code, but still slower than the vanilla code and still depends on the num_classes.
def stat_scores_multiple_classes(
        pred: torch.Tensor,
        target: torch.Tensor,
        num_classes: Optional[int] = None,
        argmax_dim: int = 1,
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:

    #num_classes = get_num_classes(pred=pred, target=target,
    #                              num_classes=num_classes)

    #if pred.ndim == target.ndim + 1:
    #    pred = to_categorical(pred, argmax_dim=argmax_dim)

    placeholder = torch.zeros((num_classes,), device=pred.device)

    match_true = (pred == target).float()
    match_false = 1 - match_true
    # tps, fps, tns, fns, sups initialized as same

    tps = placeholder.scatter_add(0, pred, match_true)
    fps = placeholder.scatter_add(0, pred, match_false)
    fns = placeholder.scatter_add(0, target, match_false)
    tns = pred.size(0) - (tps + fps + fns)
    sups = placeholder.scatter_add(0, target, torch.ones_like(match_true))

    return tps, fps, tns, fns, sups

I think if we don't give up the (num_classes, ) sized tensor results, this is as far as we can get. If we make another version of stat_scores_multiple_classes returning only sums of the stats, it would be far better, in my opinion.
I'm not sure about whether we can bear with those two similar codes (sum version and tensor version of stat_scores_multiple_classes function). What's your thoughts?
Results:
Testing  bs: 32,  num_cls: 10,  device: cuda,  steps: 100,  rep: 5
Lightning: 0.053924s
Vanilla  : 0.002690s

Testing  bs: 512,  num_cls: 10,  device: cuda,  steps: 100,  rep: 5
Lightning: 0.052600s
Vanilla  : 0.002591s

Testing  bs: 32,  num_cls: 100,  device: cuda,  steps: 100,  rep: 5
Lightning: 0.234717s
Vanilla  : 0.002604s

Testing  bs: 512,  num_cls: 100,  device: cuda,  steps: 100,  rep: 5
Lightning: 0.237395s
Vanilla  : 0.002614s

Testing  bs: 32,  num_cls: 1000,  device: cuda,  steps: 100,  rep: 5
Lightning: 1.996236s
Vanilla  : 0.002550s

Testing  bs: 512,  num_cls: 1000,  device: cuda,  steps: 100,  rep: 5
Lightning: 2.008798s
Vanilla  : 0.002579s

Testing  bs: 32,  num_cls: 10,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.019533s
Vanilla  : 0.001189s

Testing  bs: 512,  num_cls: 10,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.020818s
Vanilla  : 0.001329s

Testing  bs: 32,  num_cls: 100,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.080327s
Vanilla  : 0.001167s

Testing  bs: 512,  num_cls: 100,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.081321s
Vanilla  : 0.001278s

Testing  bs: 32,  num_cls: 1000,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.682839s
Vanilla  : 0.001193s

Testing  bs: 512,  num_cls: 1000,  device: cpu,  steps: 100,  rep: 5
Lightning: 0.704203s
Vanilla  : 0.001301s

That's a great improvement :)
As long as we only end up with these two alternatives and not one version per metric this should be fine, I think. But doesn't that also depend on the reduction type?

Yeah you're right.
But as long as we use only mean, sum and none reduction methods, it the stat_scores_multiple_classes function won't be too long.
If we want to support additional reduction methods in the future, we probably can use none version of the function (i.e. function returning (num_classes, ) sized tensors) and simply reduce after that. But if we have a lot faster version for certain reduction types, such as sum and mean, it would be very useful to have faster version of it.

Great, do you want to just open a PR with your version and the even faster one?

Yeah, sure. Small notes before that:
I changed and benchmarked it again in my forked repo: changed function
In that file, note that I also slightly changed accuracy function too. Please check if it has any wrong parts.
This gives around 1000x faster results than before, and it's only 10x slower than the vanilla implementation.
By the way, I curious about the difference of expected behaviour of Accuracy class when reduction = 'elementwise_mean' and  reduction = 'sum'. I think they are supposed to give same results. Am I right?
 Thanks a lot. From what I saw just now, your implementation and changes seem to be fine. can you open a PR to see if all our tests are still passing?

Actually, I was running the existing tests in machine, but some tests are not passing.
I made a PR to share the results. Here's the link for the PR.
I think failed tests are the case when num_classes does not match with actual number of classes in the tensor. the scatter_add function throws an Error.
If we want to keep this functionality, rather than forcing the num_classes to be the actual number of classes, I'll make additional changes for that. Cutting of the out-of-bound values and some small fixes will do.

Cool! perhaps in the future the metrics package could include parity tests for speed, not only correctness.


Update spring-security.version to v5 (major)

This PR contains the following updates:

Package
Type
Update
Change

org.springframework.security:spring-security-ldap (source)
compile
major
3.1.7.RELEASE -> 5.7.4

org.springframework.security:spring-security-cas (source)
compile
major
3.1.7.RELEASE -> 5.7.4

org.springframework.security:spring-security-web (source)
compile
major
3.1.7.RELEASE -> 5.7.4

org.springframework.security:spring-security-config (source)
compile
major
3.1.7.RELEASE -> 5.7.4

org.springframework.security:spring-security-core (source)
compile
major
3.1.7.RELEASE -> 5.7.4

⚠ Dependency Lookup Warnings ⚠
Warnings were logged while processing this repo. Please check the logs for more information.

Release Notes

spring-projects/spring-security
v5.7.4
Compare Source
:star: New Features

automatically manage docs version (with collector) #11955

:beetle: Bug Fixes

AuthenticationEventPublisher bean is not picked up if no UserDetailsService bean #11729
Build fails with missing project property cloneOutputDirectory #11979
GitHubMilestoneApiTests due_on Should Use LocalDate #11707
HttpSecurity Bean does not set DefaultAuthenticationEventPublisher #11727
NamespaceLdapAuthenticationProviderTests Should Use Dynamic Port #11711
RemoteJwkSet is not refreshed when encountering an unknown KID #11723
RequestRejectedHandler does not reliable prevent Internal Server Error #11744

:hammer: Dependency Upgrades

Update Gradle Enterprise plugin to 3.11.1 #11830
Update hibernate-entitymanager to 5.6.10.Final #11745
Update hibernate-entitymanager to 5.6.12.Final #12016
Update io.projectreactor to 2020.0.22 #11743
Update io.projectreactor to 2020.0.24 #12012
Update io.rsocket to 1.1.3 #12014
Update jackson-bom to <IP_ADDRESS>21012 #12008
Update jackson-databind to <IP_ADDRESS> #12009
Update jackson-datatype-jsr310 to 2.13.4 #12010
Update jsonassert to 1.5.1 #11741
Update mockk to 1.12.8 #12011
Update org.eclipse.jetty to 9.4.48.v20220622 #11740
Update org.eclipse.jetty to 9.4.49.v20220914 #12015
Update org.springframework to 5.3.22 #11739
Update org.springframework to 5.3.23 #12017
Update org.springframework.data to 2021.1.6 #11742
Update org.springframework.data to 2021.2.4 #12018
Update reactor-netty to 1.0.24 #12013

v5.7.3
Compare Source
:star: New Features

Add Kotlin example showing integration with WebTestClient #9998
Set permissions for GitHub actions #11642
Update javadoc of EnableWebSecurity to reflect deprecation of WebSecurityConfigurerAdapter #11650

:beetle: Bug Fixes

Add Deprecated annotation to WebSecurity#securityInterceptor #11637
Check saganCreateRelease saganDeleteRelease Required Permissions #11425
org.springframework.security.saml2.provider.service.authentication.DefaultSaml2AuthenticatedPrincipal fails to return more than one "attribute" #11605
RequestAttributeSecurityContextRepository.loadContext(HttpServletRequest) should never return null SecurityContext #11606
RequestRejectedHandler does not reliable prevent Internal Server Error #11672
Sources and javadocs missing in latest snapshots #11628
Spring Security Bcrypt with strength/log rounds = 31 results in 'Bad number of rounds' error although 31 should be ok #11484
Update javadoc of HttpSecurity, WebSecurityConfiguration and WebSecurity to reflect deprecation of WebSecurityConfigurerAdapter #11651

:hammer: Dependency Upgrades

Update hibernate-entitymanager to 5.6.10.Final #11694
Update io.projectreactor to 2020.0.22 #11691
Update jsonassert to 1.5.1 #11696
Update mockk to 1.12.5 #11690
Update org.eclipse.jetty to 9.4.48.v20220622 #11693
Update org.jetbrains.kotlinx to 1.6.4 #11695
Update org.springframework to 5.3.22 #11697
Update org.springframework.data to 2021.2.2 #11698

v5.7.2
Compare Source
:star: New Features

Consider updating testing examples to use JUnit Jupiter #11293

:beetle: Bug Fixes

Some Security Expressions cause NPE when used within @Query #11289
CsrfWebFilter null save content-type check #11341
Docs example uses access(String) with authorizeHttpRequests() #11296
Fix typo in BasicLookupStrategy Javadoc #11339
KeyInfo missing in AuthnRequest when using OpenSaml4AuthenticationRequestResolver #11358
OidcClientInitiatedLogoutSuccessHandler url-encodes PostLogoutRedirectUri twice #11384
SAML request encoding: on redirect binding, base64 encoded message contains CRLF #11284
SecurityContextRepository.loadContext(HttpServletRequest) cache result #11390
Should SAML metadata EntityDescriptor tag have the md: prefix? #11311
Update opaque-token.adoc #11303

:hammer: Dependency Upgrades

Update aspectj-plugin to <IP_ADDRESS> #11402
Update hibernate-entitymanager to 5.6.9.Final #11405
Update io.projectreactor to 2020.0.20 #11403
Update jackson-bom to 2.13.3 #11399
Update jackson-databind to 2.13.3 #11400
Update jackson-datatype-jsr310 to 2.13.3 #11401
Update org.jetbrains.kotlinx to 1.6.3 #11406
Update org.opensaml:opensaml-core4 to 4.1.1 #11410
Update org.springframework to 5.3.21 #11407
Update org.springframework.data to 2021.2.1 #11408
Update reactor-netty to 1.0.20 #11404
Update spring-ldap-core to 2.4.1 #11409

:heart: Contributors
We'd like to thank all the contributors who worked on this release!

@andrelugomes

v5.7.1
Compare Source
:beetle: Bug Fixes

StrictHttpFirewall incorrectly rejects valid CJKV characters  #11266

v5.7.0
Compare Source
:star: New Features

Check Samples should run against the current artifacts #11199
Consider replacing an inner loop with Set of authority strings in AuthorityAuthorizationManager#isAuthorized #11188
Remember me should detect UserDetailsService bean #11170
WebSessionServerSecurityContextRepository provides Mono.cache option #8422
X509 should detect UserDetailsService bean #11174

:beetle: Bug Fixes

@EnableMethodSecurity doesn't resolve annotations on interfaces through a Proxy #11177
Add shouldFilterAllDispatcherTypes to Kotlin DSL #11153
Fix setServletContext not being called for AuthorizationManagerWebInvocationPrivilegeEvaluator #11165
Multiple .requestMatchers().mvcMatchers() override previous one #11185

:hammer: Dependency Upgrades

Update aspectj-plugin to 6.4.3 #11218
Update com.nimbusds to 9.35 #11217
Update htmlunit to 2.61.0 #11222
Update htmlunit-driver to 2.61.0 #11224
Update io.projectreactor to 2020.0.19 #11220
Update mockk to 1.12.4 #11219
Update org.jetbrains.kotlin to 1.6.21 #11223
Update org.springframework to 5.3.20 #11225
Update org.springframework.data to 2021.2.0 #11228
Update reactor-netty to 1.1.0-M2 #11221
Update spring-data-jpa to 2.7.0-RC1 #11226
Update spring-ldap-core to 2.4.0 #11227

:heart: Contributors
We'd like to thank all the contributors who worked on this release!

@evgeniycheban

v5.6.8
Compare Source
:star: New Features

automatically manage docs version (with collector) #11943

:beetle: Bug Fixes

Add rncToXsd task description to CONTRIBUTING.adoc #11935
AuthenticationEventPublisher bean is not picked up if no UserDetailsService bean #11730
Build fails with missing project property cloneOutputDirectory #11969
GitHubMilestoneApiTests due_on Should Use LocalDate #11708
HttpSecurity Bean does not set DefaultAuthenticationEventPublisher #11728
NamespaceLdapAuthenticationProviderTests Should Use Dynamic Port #11712
RemoteJwkSet is not refreshed when encountering an unknown KID #11724
Updated reference to architecture page #11778

:hammer: Dependency Upgrades

Update Gradle Enterprise plugin to 3.11.1 #11827
Update hibernate-entitymanager to 5.6.12.Final #12005
Update io.projectreactor to 2020.0.24 #12001
Update io.rsocket to 1.1.3 #12003
Update jackson-bom to <IP_ADDRESS>21012 #11997
Update jackson-databind to <IP_ADDRESS> #11998
Update jackson-datatype-jsr310 to 2.13.4 #11999
Update mockk to 1.12.8 #12000
Update org.eclipse.jetty to 9.4.49.v20220914 #12004
Update org.springframework to 5.3.23 #12006
Update org.springframework.data to 2021.1.8 #12007
Update reactor-netty to 1.0.24 #12002

:heart: Contributors
We'd like to thank all the contributors who worked on this release!

@rwinch
@underground-hill
@mojavelinux
@jprinet
@Kehrlann

v5.6.7
Compare Source
:star: New Features

Add Kotlin example showing integration with WebTestClient #11612
Set permissions for GitHub actions #11644

:beetle: Bug Fixes

Add Deprecated annotation to WebSecurity#securityInterceptor #11636
Fix saganCreateRelease saganDeleteRelease Required Permissions #11426
org.springframework.security.saml2.provider.service.authentication.DefaultSaml2AuthenticatedPrincipal fails to return more than one "attribute" #11608
RequestRejectedHandler does not reliable prevent Internal Server Error #11673
Sources and javadocs missing in latest snapshots #11629
Spring Security Bcrypt with strength/log rounds = 31 results in 'Bad number of rounds' error although 31 should be ok #11485

:hammer: Dependency Upgrades

Update hibernate-entitymanager to 5.6.10.Final #11683
Update io.projectreactor to 2020.0.22 #11680
Update jsonassert to 1.5.1 #11684
Update mockk to 1.12.5 #11679
Update org.eclipse.jetty to 9.4.48.v20220622 #11682
Update org.springframework to 5.3.22 #11685
Update org.springframework.data to 2021.1.6 #11686
Update reactor-netty to 1.0.22 #11681

v5.6.6
Compare Source
:star: New Features

Consider updating testing examples to use JUnit Jupiter #11292

:beetle: Bug Fixes

CsrfWebFilter null save content-type check #11342
Docs example uses access(String) with authorizeHttpRequests() #11297
Fix typo in BasicLookupStrategy Javadoc #11340
OidcClientInitiatedLogoutSuccessHandler url-encodes PostLogoutRedirectUri twice #11385
SAML request encoding: on redirect binding, base64 encoded message contains CRLF #11285
Should SAML metadata EntityDescriptor tag have the md: prefix? #11310
Some Security Expressions cause NPE when used within @Query #11290

:hammer: Dependency Upgrades

Update hibernate-entitymanager to 5.6.9.Final #11416
Update io.projectreactor to 2020.0.20 #11414
Update jackson-bom to 2.13.3 #11411
Update jackson-databind to 2.13.3 #11412
Update jackson-datatype-jsr310 to 2.13.3 #11413
Update org.opensaml:opensaml-core4 to 4.1.1 #11420
Update org.springframework to 5.3.21 #11417
Update org.springframework.data to 2021.1.5 #11418
Update reactor-netty to 1.0.20 #11415
Update spring-ldap-core to 2.3.8.RELEASE #11419

v5.6.5
Compare Source
:beetle: Bug Fixes

StrictHttpFirewall incorrectly rejects valid CJKV characters #11267

v5.6.4
Compare Source
:star: New Features

Check Samples should run against the current artifacts #11200

:beetle: Bug Fixes

Fix setServletContext not being called for AuthorizationManagerWebInvocationPrivilegeEvaluator #11166
Multiple .requestMatchers().mvcMatchers() override previous one #11186

:hammer: Dependency Upgrades

Update io.projectreactor to 2020.0.19 #11207
Update mockk to 1.12.4 #11206
Update org.springframework to 5.3.20 #11209
Update org.springframework.data to 2021.1.4 #11210
Update reactor-netty to 1.0.19 #11208

v5.6.3
Compare Source
:beetle: Bug Fixes

AuthorizationManagerWebInvocationPrivilegeEvaluator should grant access when AuthorizationManager abstains #10951
Change HashSet to LinkedHashSet for RelyingPartyRegistration credentials #10916
Fix saml2 authentication-requests documentation #11047
Remove "Hi servlet/authentication/architecture there" from docs #10963

:hammer: Dependency Upgrades

Update hibernate-entitymanager to 5.6.8.Final #11124
Update io.projectreactor to 2020.0.18 #11119
Update io.rsocket to 1.1.2 #11121
Update jackson-bom to <IP_ADDRESS>20328 #11115
Update jackson-databind to <IP_ADDRESS> #11116
Update jackson-datatype-jsr310 to 2.13.2 #11117
Update logback-classic to 1.2.11 #11114
Update mockk to 1.12.3 #11118
Update org.aspectj to <IP_ADDRESS> #11122
Update org.eclipse.jetty to 9.4.46.v20220331 #11123
Update org.springframework to 5.3.19 #11125
Update org.springframework.data to 2021.1.3 #11126
Update reactor-netty to 1.0.18 #11120
Update spring-ldap-core to 2.3.7.RELEASE #11127

v5.6.2
Compare Source
:rewind: Breaking Changes

Saml2 metadata includes SingleLogoutService even if saml2 logout is disabled / not configured #10734

:star: New Features

Document Authorize HTTP Requests for Reactive Security #10801
Introduce AuthorizationManagerWebInvocationPrivilegeEvaluator #10682

:beetle: Bug Fixes

add Kotlin examples for Spring Data Integration of servlet application #10848
commons-logging:commons-logging is a transitive dependency of some modules #10772
Do not rely on javax. group ids #10770
Fix broken link to SAML2 login example #10806
Getting Spring Security Reference Docs have a error #10796
Make source code compatible with JDK 8 #10699
Replace StringUtils class of oauth2-oidc-sdk completely #10824
RequestMatcherDelegatingWebInvocationPrivilegeEvaluator doesn't provided access to the ServletContext #10792
WebInvocationPrivilegeEvaluator Bean should support multiple SecurityFilterChains #10680

:hammer: Dependency Upgrades

Update hibernate-entitymanager to 5.6.5.Final #10873
Update io.projectreactor to 2020.0.16 #10867
Update io.spring.javaformat to 0.0.31 #10870
Update logback-classic to 1.2.10 #10865
Update mockk to 1.12.2 #10866
Update org.aspectj to 1.9.8 #10871
Update org.eclipse.jetty to 9.4.45.v20220203 #10872
Update org.slf4j to 1.7.36 #10874
Update org.springframework to 5.3.16 #10875
Update org.springframework.data to 2021.1.2 #10876
Update r2dbc-h2 to 0.8.5.RELEASE #10869
Update reactor-netty to 1.0.16 #10868
Update spring-ldap-core to 2.3.6.RELEASE #10877

v5.6.1
Compare Source
:star: New Features

Document authentication helper method in WebClient integration #10468
Document authentication helper method in WebClient integration for Servlet Environments #10120
Document parameters converter in oauth2 client servlet docs #10469
Document parameters converter in oauth2 client servlet docs #10467

:beetle: Bug Fixes

AuthorityAuthorizationManager incorrectly compares GrantedAuthority #10595
clockSkew Javadoc is not consistent with implementation #10535
Invalid_request failures in JwtTokenValidators are always turned into invalid_token errors #10560
Kotlin DSL examples in reactive oauth2 docs call build twice #10591
StaticServerHttpHeadersWriter should work with case-insensitive header names #10581

:hammer: Dependency Upgrades

Update cas-client-core to 3.6.4 #10654
Update hibernate-entitymanager to 5.6.3.Final #10653
Update io.projectreactor to 2020.0.14 #10651
Update jackson-bom to 2.13.1 #10647
Update jackson-databind to 2.13.1 #10648
Update jackson-datatype-jsr310 to 2.13.1 #10649
Update junit-bom to 5.8.2 #10656
Update logback-classic to 1.2.9 #10646
Update mockk to 1.12.1 #10650
Update org.jetbrains.kotlin to 1.5.32 #10655
Update org.junit.jupiter to 5.8.2 #10657
Update org.springframework to 5.3.14 #10658
Update reactor-netty to 1.0.14 #10652
Update spring-ldap-core to 2.3.5.RELEASE #10659

:heart: Contributors
We'd like to thank all the contributors who worked on this release!

@sjohnr

v5.6.0
Compare Source
:star: New Features

DaoAuthenticationProviderTests#avg function doesn't return fraction #10426
Docs Should Use Section Summary #10449
MissingCsrfTokenException message is misleading when not storing the CSRF tokens in the session #10436
Revamp OAuth 2.0 Login/Client reactive documentation #8174
Revamp Reactive OAuth 2.0 Login documentation #10479
Split up Documentation Further #10367
Support Structure 101 License Id in Package Tangle Check #10443

:beetle: Bug Fixes

Adding keyInfo section to LogoutRequest from RP side #10450
In saml2 LogoutRequest from RP doesn't contain KeyInfo  #10438
Oauth2 Resource Server will not retry on first failure with Multi-tenancy  #10444
Port Missing Integration Docs #10465
SAML 2.0 JUnit Tests are being skipped #10215
Various build time javadoc warnings fix #10423

:hammer: Dependency Upgrades

Update aspectj-plugin to 6.2.0 #10445
Update com.nimbusds to 9.19 #10491
Update hibernate-entitymanager to 5.6.1.Final #10495
Update hsqldb to 2.6.1 #10496
Update io.projectreactor to 2020.0.13 #10493
Update logback-classic to 1.2.7 #10490
Update org.springframework to 5.3.13 #10497
Update reactor-netty to 1.0.13 #10494

:heart: Contributors
We'd like to thank all the contributors who worked on this release!

@sjohnr
@larsgrefer
@Emkas
@surkoff-v

v5.5.8
Compare Source
:beetle: Bug Fixes

StrictHttpFirewall incorrectly rejects valid CJKV characters #11268

v5.5.7
Compare Source
:hammer: Dependency Upgrades

Update com.fasterxml.jackson.core to <IP_ADDRESS> #11211
Update com.fasterxml.jackson.datatype to 2.13.2 #11212
Update io.projectreactor to 2020.0.19 #11213
Update org.springframework to 5.3.20 #11215
Update org.springframework.data to 2021.0.11 #11216
Update reactor-netty to 1.0.19 #11214

v5.5.6
Compare Source
:beetle: Bug Fixes

AuthorizationManagerWebInvocationPrivilegeEvaluator should grant access when AuthorizationManager abstains #10952
Change HashSet to LinkedHashSet for RelyingPartyRegistration credentials #10917

:hammer: Dependency Upgrades

Update com.fasterxml.jackson.core to <IP_ADDRESS> #11130
Update com.fasterxml.jackson.datatype to 2.13.2 #11131
Update io.projectreactor to 2020.0.18 #11132
Update io.rsocket to 1.1.2 #11134
Update jackson-bom to <IP_ADDRESS>20326 #11129
Update logback-classic to 1.2.11 #11128
Update org.aspectj to <IP_ADDRESS> #11135
Update org.eclipse.jetty to 9.4.46.v20220331 #11136
Update org.springframework to 5.3.19 #11137
Update org.springframework.data to 2021.0.10 #11138
Update reactor-netty to 1.0.18 #11133
Update spring-ldap-core to 2.3.7.RELEASE #11139

v5.5.5
Compare Source
:star: New Features

Introduce AuthorizationManagerWebInvocationPrivilegeEvaluator #10683

:beetle: Bug Fixes

Add Kotlin examples for Spring Data Integration of servlet application #10847
Replace StringUtils class of oauth2-oidc-sdk completely #10825
Getting Spring Security Reference Docs have a error #10797
RequestMatcherDelegatingWebInvocationPrivilegeEvaluator doesn't provided access to the ServletContext #10791
Make source code compatible with JDK 8 #10700
WebInvocationPrivilegeEvaluator Bean should support multiple SecurityFilterChains #10681

:hammer: Dependency Upgrades

Update spring-ldap-core to 2.3.6.RELEASE #10863
Update org.springframework.data to 2021.0.9 #10862
Update org.springframework to 5.3.16 #10861
Update org.slf4j to 1.7.36 #10860
Update org.eclipse.jetty to 9.4.45.v20220203 #10859
Update org.aspectj to 1.9.8 #10858
Update io.spring.javaformat to 0.0.31 #10857
Update r2dbc-h2 to 0.8.5.RELEASE #10856
Update reactor-netty to 1.0.16 #10855
Update io.projectreactor to 2020.0.16 #10854
Update logback-classic to 1.2.10 #10851

v5.5.4
Compare Source
:beetle: Bug Fixes

Documentation has wrong code example in the 'Customizing OpenSAML’s AuthnRequest Instance' section #10527
Invalid_request failures in JwtTokenValidators are always turned into invalid_token errors #10561
MissingCsrfTokenException message is misleading when not storing the CSRF tokens in the session #10531
Multi-tenancy Documentation - com.nimbusds.jwt.proc.JWTProcessor does not have a setJWTClaimSetJWSKeySelector  method #10520
Multi-tenancy Documentation - JwtDecoder sample has multiple errors #10516
Oauth2 Resource Server will not retry on first failure with Multi-tenancy  #10484
StaticServerHttpHeadersWriter should work with case-insensitive header names #10582
WebInvocationPrivilegeEvaluator does not provide a way to pass a ServletContext #10435

:hammer: Dependency Upgrades

Update cas-client-core to 3.6.4 #10637
Update hibernate-entitymanager to 5.4.33 #10635
Update hsqldb to 2.6.1 #10636
Update io.projectreactor to 2020.0.14 #10633
Update io.spring.javaformat to 0.0.29 #10411
Update jackson-bom to 2.12.6 #10630
Update jackson-databind to 2.12.6 #10631
Update jackson-datatype-jsr310 to 2.12.6 #10632
Update logback-classic to 1.2.9 #10629
Update org.jetbrains.kotlin to 1.5.32 #10638
Update org.springframework to 5.3.14 #10639
Update org.springframework.data to 2021.0.7 #10640
Update reactor-netty to 1.0.14 #10634
Update spring-ldap-core to 2.3.5.RELEASE #10641

v5.5.3
Compare Source
:star: New Features

Allow defining custom SAML 2.0 Assertion Signature Validator #10317
Add Documentation for Static Methods Classes for mockJwt() and jwt() #10265

:beetle: Bug Fixes

ClaimAccessor#getClaimAsMap doesn't return null as documented #10371
5.5.X only works with spring-security-5.4.xsd schema (XML-based config) #10369
SecurityNamespaceHandler: update schema version to 5.5 #10348
JwtTimeStampValidator uses wrong error on token expiration #10328
Fix typo #10313
Saml2LoginConfigurer relyingPartyRegistrationRepository method does not return correct type #10257
ACL docs refer to nonexistent sample apps #10237
SAML 2.0 Login should allow loginProcessingUrl without {registrationId} when providing an AuthenticationConverter #10176

:hammer: Dependency Upgrades

Update org.springframework.data to 2021.0.6 #10417
Update org.springframework to 5.3.11 #10416
Update org.jetbrains.kotlinx to 1.5.2 #10415
Update org.jetbrains.kotlin to 1.5.31 #10414
Update org.eclipse.jetty to 9.4.44.v20210927 #10413
Update io.spring.nohttp to 0.0.10 #10412
Update r2dbc-spi-test to 0.8.6.RELEASE #10410
Update reactor-netty to 1.0.12 #10409
Update io.projectreactor to 2020.0.12 #10408
Update jackson-datatype-jsr310 to 2.12.5 #10407
Update jackson-databind to 2.12.5 #10406
Update jackson-bom to 2.12.5 #10405
Update logback-classic to 1.2.6 #10404

:heart: Contributors
We'd like to thank all the contributors who worked on this release!

@Emkas
@marcusdacoregio

v5.5.2
Compare Source
:star: New Features

Consider adding springFrameworkVersion property #10068
Introduce samplesBranch property #10036
Use the new springFrameworkVersion property in docs' links #10067

:hammer: Dependency Upgrades

Update com.nimbusds to 9.9.1 #10186
Update io.projectreactor to 2020.0.10 #10187
Update jackson-bom to 2.12.4 #10183
Update jackson-databind to 2.12.4 #10184
Update jackson-datatype-jsr310 to 2.12.4 #10185
Update logback-classic to 1.2.5 #10182
Update org.aspectj to 1.9.7 #10189
Update org.eclipse.jetty to 9.4.43.v20210629 #10190
Update org.jetbrains.kotlin to 1.5.21 #10191
Update org.jetbrains.kotlinx to 1.5.1 #10192
Update org.slf4j to 1.7.32 #10193
Update org.springframework to 5.3.9 #10194
Update org.springframework.data to 2021.0.4 #10195
Update reactor-netty to 1.0.10 #10188

v5.5.1
Compare Source
:star: New Features

Consider adding a link checker to build #9972
Use Job Outputs to Transmit Error #9928
Store one request by default in WebSessionOAuth2ServerAuthorizationRequestRepository #9917
Combine different OS Build in one CI Job #9798
Use GPG_PRIVATE_KEY directly #9778

:beetle: Bug Fixes

Update links to point to migrated samples #9971
Add messaging to documentation about sample migration #9970
Fix broken links in docs #9969
CORS section is missing in Reactive reference documentation #9952
RSocket documentation mentions non-existent class #9950
Disabling logout keeps LogoutPageGeneratingWebFilter registered at /logout #9941
Missing log of "caused by" exception when OP document metadata cannot be reached  #9939
Missing support for private_key_jwt in ClientRegistrations #9936
Allow client registration from issuer uri with no authorize_endpoint  #9935
Missing support for urn:ietf:params:oauth:grant-type:jwt-bearer in ClientRegistrations #9934
Using the SecurityMockServerConfigurers.java requires the com.nimbusds oauth2-oidc-sdk on the classpath #9929
Jwt client authentication converter should detect new key #9927
Adding filters relative to custom ones is broken #9906
SEC-3139: Anonymous authentication token not passed to Controller #9890
Clarify quick start section in README #9885
RSocket and WebClient with Security refCount: 0 #9870
spring-security-config kotlin-stdlib-jdk8 dependency isn't optional #9864
Client credentials not correctly encoded in Basic Auth #9858
Docs should state default value for Resource Server validation clock skew is 60 seconds #9849
OidcClientInitiatedLogoutSuccessHandler url-encodes PostLogoutRedirectUri twice #9819
DefaultSpringSecurityContextSource can't handle spaces in baseDn #9806
OAuth2ErrorResponseErrorHandler throws IllegalArgumentException for a nonstandard HTTP status code response #9805
NPE in HttpSessionSecurityContextRepository.isTransientAuthentication #9801
Fix Build Scan in Build Windows CI Job #9797
GitHub Actions only Activated for main #9777
Artifactory missing mavenJava publication #9774
spring-security-core depends on s

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about these updates again.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

Renovate Ignore Notification
As this PR has been closed unmerged, Renovate will ignore this upgrade and you will not receive PRs for any future 5.x releases. However, if you upgrade to 5.x manually then Renovate will reenable minor and patch updates automatically.
If this PR was closed by mistake or you changed your mind, you can simply rename this PR and you will soon get a fresh replacement PR opened.


Change BEP numbers to include MRS

See #89
Change BEP numbers to include MRS as BEP022 and so update BEP numbers of PET derivatives and Computed Tomography.

Thanks!


[Deprecation Notice] python 3.8 is end of life

Python Version Support for pyjpboatrace<=0.3.1
The current version of pyjpboatrace<=0.3.1 requires python >= 3.8. However, Python 3.8 reached its end-of-life on October 7, 2024.
As Python 3.8 is no longer maintained, it may become increasingly difficult for users to work in environments restricted to this version. Supporting Python 3.9 and later would enhance the usability and longevity of this package.
Suggested Actions:
Update the package to explicitly support python >= 3.9.
Update pyproject.toml or setup.py to indicate that Python 3.8 is no longer supported.
Reference:

PEP 569 – Python 3.8 Release Schedule

These updates would ensure that users can continue to use this package with newer versions of Python, benefiting the community.
Thank you for your attention to this matter!

It may be better to issue a warning if you are using Python 3.8 or lower, which has less impact.

I guess that the following failure related to this issue:
https://github.com/hmasdev/pyjpboatrace/actions/runs/12528321821/job/34942809132

We plan to raise the version to 0.4.0 once this issue is resolved.


[PWS] Fixed webui form validation

My previous commit didn't handle correctly webui form validation.
This patch does.

I didn't even know you could do this, cool!


Not An Issue, more an piece of info.

I was put onto your work here by someone on Blender Artist. Brief background, I have developed a set of nodes for MID integration with Blender, with live input and animation from MIDI files, this lead to integration of sound, using Sounddevice, then this guy told me about your work here.
Seeing the other comments, I guess you weren't going to develop it further, so I have taken the AUD stuff and built a set of nodes to work in Animation Nodes. For Blender 2.79 at the moment, but I will convert them over to Blender 2.8 AN 2.1 in time, after I have finished testing everything.
You can see what I have been up to here: https://blenderartists.org/t/were-in-the-midi-audio-daw-real-time/1142160
Updated link to correct one...
Hope you don't mind me playing with this, but it all seems to work very well in AN, if you want more info, just let me know and I will be happy to talk about it. All the sound nodes are on my AN fork, under the master/nodes/daw directory.
Thanks for all you did to get me started on this project.
Cheers, Clock.
My test node tree in B2.79 all the yellow nodes are my new ones:

Hi! This looks great! Unfortunately, I can't access the blenderartists thread to read more about it:

Sorry, you don't have access to that topic!

So sorry, wrong link, here is the good one:
https://blenderartists.org/t/were-in-the-midi-audio-daw-real-time/1142160
Just one more question, how does the threshold function work, any number I put in threshold other than 0 seems to do the sane thing...

Thanks for the link. Thresholding is pretty easy [1], basically:
if sample >= threshold
  sample = 1
else if sample <= -threshold
  sample = -1
else
  sample = 0

[1] https://github.com/neXyon/audaspace/blob/master/src/fx/Threshold.cpp#L22

Yes, I understand that, but if I put a value in of 0.001, or a value of 1, I get the same sound played, i.e it always sounds the same whatever value I enter.
This is my node code:
import bpy
import aud
import os
from bpy.props import *
from ... base_types import AnimationNode
from ... events import propertyChanged

class thresholdSound(bpy.types.Node, AnimationNode):
    bl_idname = "an_thresholdSound"
    bl_label = "SOUND Threshold"
    bl_width_default = 150

    def create(self):
        self.newInput("an_FloatSocket","Threshold","threshold",minValue=-1,maxValue=1)
        self.newInput("an_SoundSocket","Sound I","snd")
        self.newOutput("an_SoundSocket","Sound O","sound")

    def execute(self,threshold,snd):
        self.use_custom_color = True
        self.useNetworkColor = False
        self.color = (1,0.8,0.5)
        if 'aud.Sound' in str(snd) and threshold > 0:
            snd = snd.threshold(threshold)
        return snd

Ignore this on Threshold, now it is working if I use low values for threshold, like 0.16 for example.
I have no idea why it did not work earlier.

So next question then, if you don't mind, is there any way I can make an LFO (say at 20Hz) and multiply the sound by this new LFO sound to modulate it?

Unfortunately not, there is currently no modulation effect. It should be quite simple to implement it though. Do you know a bit of C++ and want to give it a try?

What I know about C++ could be written in 100mm letters on a postage stamp... Never even looked at it. I suppose I could learn, but at my age....
Somewhere recent I read that you ADD the normal sound to an LFO sound to get this effect, not multiply it. So now I am also confused. BTW I now have a Drum Machine working, see test post on my BA thread.
Thanks for the reply!

If it was adding, then just playing them back at the same time would do the trick, because adding is exactly what happens when you play two sounds at the same time. I added a modulate() effect to the python API which does multiplication instead of addition, maybe that's what you're looking for. It should be available in any blender 2.8 build made from the master branch starting today.

I will try the "adding" method with an LFO and see what it sounds like, then get a build of Blender with your new "modulate" function, when its available.
BTW, I have now built an automatic Arpeggio node that sequences sounds together in runs from 3 to 9 notes, using frequency lists from my "chord" function. So for up to 5 counts it just rises, then it rises and falls. For example a count of 8 for C major plays C E G B C B G E. The node uses your  join() function to add all the notes into one sound and limit() to set the durations.
My sequencer automatically sets one frame to be say 1/16th of a Beat, so frames are always multiples of Beat divisions. My setup node sets Beats Per Minute and smallest note division (up to 1/64th) to set parameters in the project. To get decimal fps of 29.33, for example, I set fps to 2933 and fps base to 100.
Thanks again for all your efforts and help with AudaSpace. If you want any of the node files, none are on my GitHub yet for 2.8, just let me know, and I will provide them. I intend to get the nodes on my GitHub as soon as reasonable testing has been completed.

Simply Brilliant! Thank you so much.

So pleased with Modulate function - works really well with frequencies less/equal to 30Hz.

This takes it to a whole new level, adding LFO, Mod frequency (either by frequency, or note name) and lets you mix the three things together:

🥇 to you!

I'll close this now, thanks for all your help. I now have an FM Synthesiser working in Blender 2.8 thanks to your new modulate() function. BTW I noticed you have changed the square() function so it is now a separate oscillator rather than a wave "squarer" - this is also a good development.
Cheers, Alan.

Could you explain this in a bit more detail please:

I am not sure what values to enter here as integers are required for format, container, codec, etc.

@neXyon A video for you, shows animations built directly from MIDI csv file and sound generated by AudaSpace and my extension nodes to AN.
https://vimeo.com/329301448
Cheers, Alan.

Aha! I think I have got it:

Command was this:
>>> snd = snd.write("/Users/mac/Music/Test.mp3",aud.RATE_16000,aud.CHANNELS_STEREO,aud.FORMAT_FLOAT32,aud.CONTAINER_MP3,aud.CODEC_MP3,16,2048)
That produced a test sound! Probably are by luck than skill...

Thank you!

Sorry to bother you, but I have another issue:
When I write a file out from my Generator Node, all is well, this is the code line:
sndW = snd.write(fileName,aud.RATE_16000,aud.CHANNELS_STEREO,aud.FORMAT_FLOAT32,aud.CONTAINER_FLAC,aud.CODEC_FLAC,16,2048)
However, when I try to write sound out from my FM Synth, which uses four oscillators, the first is modulated by the second, third and forth, Blender hangs when I try to write out the file, that might contain 30 modulated notes in sequence.
Is there something I have go wrong in the "write" code, like is the buffer too small, etc?
If you want the "FM Synth" code, I can supply it. Basically it generates four sounds, modulates the first by the others, then mixes them together. In "List" mode it takes all the notes from a Track and then mixes them together, having applied timing delays so they occur in sequence and then tries to save this.
I don't really know what the buffer size is doing, or what values it expects. The same system works well with single oscillator configuration for the same 30 notes.
I am writing Flac files now so I can reverse them...
Thanks, Alan.

Hi Alan,
yes, please provide a minimal code sample that runs in stock blender 2.8 to reproduce the problem.
From what you're writing everything looks fine in this one line of code. The buffer size is ok, you can just not set it to get a default value (which is 1024). I would also not set the bitrate which for flac isn't used anyway.
Cheers

So this is the code in the Output Player node, which saves the sound file:
            if 'aud.Sound' in str(snd):
                snd = snd.volume(volM)
                if self.fileOut is not "" and writeF:
                    fileName = self.fileOut.split(".")[0] if ".flac" in self.fileOut else self.fileOut
                    fileName = fileName+"_"+str(frameC)+".flac"
                    my_file = Path(fileName)
                    if my_file.is_file():
                        self.message = "File Exists, Enter New Name/Delete it"
                    else:
                        sndW = snd.write(fileName,aud.RATE_16000,aud.CHANNELS_STEREO,aud.FORMAT_FLOAT32,aud.CONTAINER_FLAC,aud.CODEC_FLAC,16,2048)
                        self.message = "Written "+fileName
                try:
                    play = dev.play(snd)
                    return snd
                except:
                    self.color = (0.7,1,0.7)
                    return None

This is the code in the FM Synth node, which merges the sounds together:
            sndO = snd1.mix(snd2)
            sndO = sndO.mix(snd3)
            sndO = sndO.mix(snd4)
            sndO = sndO.volume(soundV)
            if startT > 0:
                sndO = sndO.delay(startT)
            sndO = sndO.rechannel(2)
            if i == 0:
                sndM = sndO
            else:
                sndM = sndM.mix(sndO)
            if startT + durT > limit:
                limit = startT + durT
        self.message = "Playing Input List "+str(limit)+"s"
        sndM = sndM.limit(0,limit)
        sndM = sndM.rechannel(2)
        return samples, sndM

Since I wrote the last post I have added the bit near the bottom (limit function), which applies a limit to the sound before outputting it to the next node. This seems to have done the trick, although I am a bit confused since the incoming sounds, snd1, snd2, snd3 & snd4 already had limits applied. What this actually does is to mix four sounds per note of the incoming array, then mix all the occurrences of this note into one sound for the whole song, or part song.
So I think I solved it, it was not an error in the library, just me not limiting the final output sound.
I will alter the write function to take out the last two arguments, I have noticed that the output sound from the FM Synth is a little distorted, whereas the output from the Generator is not, but I will work at that, it may be a volume issue?
Thanks, Alan.

This works fine for me:
snd = aud.Sound.sine(440).limit(0, 5)
snd = snd.mix(aud.Sound.sine(880).volume(0.8).limit(0, 5))
print(snd.length)

In general, you can check snd.length for any snd to see if it's properly limited, maybe you can debug and find a sound that is not properly limited before you mix it this way. That should  then be the cause for your file writing hanging, because it tries to write an infinitely long sound.

New Question, can I plot the Sound Data array as an image?
I tried:
import matplotlib.pyplot as plt
plt.imshow(snd.data())
plt.show()
But only got this:

What I really want is a plot of the Sound's waveform. I could make a Blender Cruve object from the array, I think. bt I thought this approach might be better.

OK I sorted it:

This is the code:
            if 'aud.Sound' in str(snd):
                fig1 = plt.figure(figsize=(20,2))
                fig, ax = plt.plot(snd.data(),scalex=True,scaley=True,color='#008080')
                ax = plt.gca()
                ax.set_facecolor('#80F0F0')
                fig.set_figure(fig1)
                plt.tight_layout()
                plt.savefig(fileName)

It's amazing that as soon as one asks a question one thinks of the answer! Maybe if my code is not good you can let me know.

Now I have been asked to look at "Side-Chaining" where the volume of one sound affects the volume of another, so for example I can do this:

I extract the data of sound 1 and manipulate this with the data of sound 2, I get this waveform for sound 1:

The manipulated Array looks like this:

But altering the Array does not affect the sound it came from. Is this a limitation, or can I generate a new sound from the modified Array? Or do we need a new function that allows us to modify the sound in this way, i.e. a Side-Chain function?
This is the code in the new Side-Chain node:
            if 'aud.Sound' in str(snd):
                frameC = bpy.context.scene.frame_current
                if frameC == tFrame:
                    sndArrayI = snd.data()
                    max = 0
                    for i in sndArrayI:
                        if i[0] > max:
                            max = i[0]
                    self.message = "SOUND Side-Chain "+str(max)
                    sndArrayA = sndC.data()
                    for i in range(0,len(sndArrayI)):
                        if i < len(sndArrayA):
                            sndArrayI[i][0] = sndArrayI[i][0] * (max - abs(sndArrayA[i][0]))
                            sndArrayI[i][1] = sndArrayI[i][1] * (max - abs(sndArrayA[i][1]))
                    # Write out sound Waveform
                    fig1 = plt.figure(figsize=(20,2))
                    fig, ax = plt.plot(sndArrayI,scalex=True,scaley=True,color='#008080')
                    ax = plt.gca()
                    ax.set_facecolor('#80F0F0')
                    fig.set_figure(fig1)
                    plt.tight_layout()
                    plt.savefig(fileName)

Sorry to keep throwing these things at you, just say if this is getting too much. I may need to modify the maths for the Array manipulation, I will do that if I can get the sound to be the same as the modified Array.
Thanks, Alan.

The data() method reads the sound and stores the results in the array that you get. This array is not used by the sound itself in any way. You have to create a sound from your final array using the buffer() method. Note that any subsequent changes to the array after buffer() has been called will also not change the sound created with the buffer() method.

Thank you for that, here is the code:
        if 'aud.Sound' in str(snd):
            sndArrayI = snd.data()
            sndArrayA = sndC.data()
            self.message = "SOUND Side-Chain"

            for i in range(0,len(sndArrayI)):
                if i < len(sndArrayA):
                    if sndArrayI[i][0] > 0:
                        sndArrayI[i][0] = sndArrayI[i][0] - abs(sndArrayA[i][0])
                    else:
                         sndArrayI[i][0] = sndArrayI[i][0] + abs(sndArrayA[i][0])

                    if sndArrayI[i][1] > 0:
                        sndArrayI[i][1] = sndArrayI[i][1] - abs(sndArrayA[i][1])
                    else:
                         sndArrayI[i][1] = sndArrayI[i][1] + abs(sndArrayA[i][1])

            sndO = aud.Sound.buffer(sndArrayI,samples).rechannel(2)
            return sndO

It seems to work well! I have also made a white noise generator, code is here:
    def whiteNoise(self):

        def band_limited_noise(min_freq, max_freq, samples, samplerate=1):
            freqs = np.abs(np.fft.fftfreq(samples, 1/samplerate))
            f = np.zeros(samples)
            idx = np.where(np.logical_and(freqs>=min_freq, freqs<=max_freq))[0]
            f[idx] = 1
            return fftnoise(f)

        def fftnoise(f):
            f = np.array(f, dtype='complex')
            Np = (len(f) - 1) // 2
            phases = np.random.rand(Np) * 2 * np.pi
            phases = np.cos(phases) + 1j * np.sin(phases)
            f[1:Np+1] *= phases
            f[-1:-1-Np:-1] = np.conj(f[1:Np+1])
            return np.fft.ifft(f).real

        x = band_limited_noise(self.minFreq,self.maxFreq, self.samples, self.samples)
        x = np.int16(x * (2**15 - 1))
        wavfile.write("white_noise.wav", self.samples, x)

It uses "wavefile" from "scipy.io" to store the sound, I am going to see if I can use the Array (x) as an input to a aud.Sound.buffer() function next... I'll let you know how this goes. This is the node tree to generate, modulate and pitch bend the sound:

The green node, when connected, saves the waveform image, the Calculator node gets a harmonic frequency for the modulation, amongst other functions and the Pitch Bend node does what it says on the node name! This is an example of the sound produced:
white_noise_6.flac.zip
I could use this to generate the sound from a steam engine animation maybe?
Thanks again for all the help, Alan.

That's nice work you're doing. Keep it up!

Thank you!
I have modified the Sound Generator node, so now it has a "White Noise" mode:

The code for this mode in the node is:
    def executeW(self,minFreq,maxFreq,volume,duration,samples):
        if duration > 0 and minFreq > 0 and maxFreq > minFreq:
            def band_limited_noise(min_freq, max_freq, samples, samplerate=1):
                freqs = np.abs(np.fft.fftfreq(samples, 1/samplerate))
                f = np.zeros(samples)
                idx = np.where(np.logical_and(freqs>=min_freq, freqs<=max_freq))[0]
                f[idx] = 1
                return fftnoise(f)

            def fftnoise(f):
                f = np.array(f, dtype='complex')
                Np = (len(f) - 1) // 2
                phases = np.random.rand(Np) * 2 * np.pi
                phases = np.cos(phases) + 1j * np.sin(phases)
                f[1:Np+1] *= phases
                f[-1:-1-Np:-1] = np.conj(f[1:Np+1])
                return np.fft.ifft(f).real

            array = band_limited_noise(minFreq,maxFreq,samples,samples)
            array = np.float32(array)
            sndO = aud.Sound.buffer(array,samples)
            loop = int(duration)
            if loop > 0:
                sndO = sndO.loop(loop)
            sndO = sndO.limit(0,duration).rechannel(2).volume(10*volume)
            return samples, duration, sndO
        else:
            return samples, duration, None

Seems to work well, I hd to change the array to a Float32 type for Audaspace to use it. These white noises are very quiet, so I multiply the normal volume input by 10 to get them at the same sort of level.

I have been struggling with random "clicks" at the end of sounds. Here is the code snippet from my "Doppler" node:
        if 'aud.Sound' in str(snd):
            if maxOff != 0:
                self.label = "SOUND Doppler"
                # Incomming sound length in seconds:
                sliceT = snd.length / samples
                refT   = sliceT / resN
                for i in range(0,resN):
                    strT = refT*i
                    endT = (refT*i)+refT
                    slice = snd.limit(strT,endT)
                    pVal = 1+(cos(i*(pi/resN)) * maxOff)
                    sndS = slice.pitch(pVal).resample(samples,False)
                    if i == 0:
                        sndO = sndS
                    else:
                        sndO = sndO.join(sndS)
                sndO = sndO.rechannel(2)
                if mixO:
                    sndO = sndO.mix(snd)
                return sndO

This is used in a node tree like this:

Where the Calculator node is used to produce an increase in pitch based upon the number of semitones set. So, the sound is pitched up by this amount at the start, then the pitch is changed according to the cosine of a value increasing from 0 to pi over the time of the sound. In this example the sound starts of 3 semitones higher than the note given and ends up 3 semitones lower. Using a negative value for semitone causes the opposite effect in that the note starts pitched down and ends up pitched up.
This nearly always produced a "click" at the end of the "Doppler" sound, I have also noticed clicks at the end of generated sounds, sometimes, and came to the conclusion that the sound waveform is being cut abruptly at the end, this interrupts the wave pattern and causes the click, I think, this is because the waveform does not end at 0 amplitude.
To stop this, I have introduced this code to the generator node:
snd = snd.volume(volume).rechannel(2).fadeout((duration*0.95),(duration*0.05))
return snd

I am wondering if you think this is a good "fix", or should I try something else? Values of 0.95 & 0.05 seems to work well, I have also tried valets of 0.98 and 0.02, which work most of the time.

Fading is exactly the correct solution. Not however, that you also should do a fade in at the beginning, not just a fade out at the end, to avoid any clicking. If you do this, you should be able to keep the duration of the fade quite short!

Thank you!
This works well for every sound so far, including some crap drum samples I got of the 'net:
    try:
        duration = snd.length / snd.specs[0]
        snd = snd.limit(0,duration).fadein(0,duration*0.03).fadeout((duration*0.97),(duration*0.03))
        play = dev.play(snd)
        return snd
    except:
        self.color = (0.75,1,0.75)
        return None

This is my new drum machine, it stores sound identifiers in a dictionary, written to disc so it can be recalled at Blender startup, then makes a list of sounds from these and uses the coloured notes to play them:

Still looking for some decent drum samples, it would be nice to get the ones out of my Reason package, but they stopped us doing that now....

I have found an issue, for me, when trying to mix sounds together. If I take two sounds of different lengths and mix them, the output sound is always the same length as the shortest sound. If I try to pad the shortest one with silence, using the join() function, it won't mix because the silence seems to be always 48000 samples. Rightly so Aud will not mix different spec sounds. If I resample the silence, I get clicks everywhere!
Then I tried Sequences, but could not get a sequence of notes that overlap in time, so both start at time 0 and go for different lengths, I presume this is planned as I can happily use sequences for consecutive sounds, as I do in my Phaser and Flanger nodes.
I needed this for my Drum Machine, as the drum samples are not the same length and more than one needs to be played at once. In the end I did it by altering my Player node to take a list of sounds, the drum machine now looks like this:

The Player node takes the list of sounds from the Sound File List node (I may rename this) and uses this simple piece of code:
    dev = aud.Device()
    for s in soundList:
        if 'aud.Sound' in str(s) or 'aud.Sequence' in str(s):
            handle = dev.play(s)

This works well, but I wonder if there is another way to do this, or have I hit on the best way? I am checking to make sure the incoming list does contain Aud Sounds, or Sequences. This does allow the same drum sounds to be played at say 0.5s intervals when its length is 1 second, without any issues.
Just like to say that I really appreciate all the help you have given me here. this is all new to me and I have had to learn so much!

You're right, silence is always at 48 kHz, I could change that. However, resampling it should not give you clicks, it doesn't for me:
d = aud.Device()
s = aud.Sound.silence().resample(44100, False) # no high quality needed, silence will stay silence
h = d.play(s)

Maybe the clicking comes from another problem?
Also, mixing two sounds plays until both sounds are done, for example:
s1 = aud.Sound.sine(440).limit(0,1)
s2 = aud.Sound.sawtooth(320).limit(0,5)
s = s1.mix(s2)
h = d.play(s)

Again, maybe you have another problem there?
A sequence makes sense, if you're building a sequencer ;)
It also works fine for me:
seq = aud.Sequence()
se1 = seq.add(s1, 1, 2, 0)
se2 = seq.add(s2, 0, 5, 0)
h = d.play(seq)

Note that the parameters of the add function are: sound, start time, end time and skip. So if your sound is 5 seconds long and you want it to start at second 3 and not skip anything either at the beginning or end, you should call it with start time = 3, end time = 8 and skip = 0. You can use these features to avoid having to limit your synthesized sounds before:
s1 = aud.Sound.sine(440)
s2 = aud.Sound.sawtooth(320)
seq = aud.Sequence()
se1 = seq.add(s1, 1, 2, 0)
se2 = seq.add(s2, 0, 5, 0)
h = d.play(seq)

Lastly, there is currently no API to get audio input from a microphone or similar, sorry.

This code works for me:
# sndList is a list of indices for all sounds played per frame
# self.Store is a dictionary of saved sounds generated from sound files (e.g. drum samples)
    sndO = aud.Sequence()
        for i in range(0,len(sndList)):
            r = sndList[i]
            snd = self.sndStore[name][r]
            durT = snd.length / snd.specs[0]
            fp = fadP / 100
            if durT >= maxL:
                snd = snd.limit(0,maxL).fadeout((maxL*(1-fp)),(maxL*fp))
            else:
                maxL = durT
            e = sndO.add(snd,0,maxL,0)
        return sndO

I think this is right now. So, this is fed with a list of note indices played by the Drum Sequencer. The node stores a dictionary of sounds relating to the channels in my drum Machine (for example). I have inputs for the maximum sound length (float) and percentage fade (Integer). If the sound is longer than the max value, it gets limited and faded. This operates per frame at the moment and stops sounds that are longer than the input max length ending abruptly.
I will investigate creating a sequence for the entire song, writing that to a file and then adding that to the VSE as a further development. My Player node already writes incoming sounds to a file and adds them to the VSE as an option on a per frame basis, but this would allow me to write the entire track out to a file, then add all the track to the VSE for playing, or writing the complete song to a file.
One of the things I was doing wrong was not using correct variables and structure for adding Sequence Entries, so your code really helped me. On the sound mixing front, I have re-worked my code so that it doesn't keep the duration of the first sound in the "for" loop and apply it to the others, D'Oh!
I think I will use Sequences far more now that I understand them better. They seems to be the best option for mixing many sounds of differing lengths, with delays and limits applied as there seems to be no need for some of the stages I am using currently. I can just generate a sound, then effectively delay and limit it when adding it to the Sequence.
This is the code in my Player node that fades sounds to get rid of clicks, it does not do this for sequences as these will be sorted already. I have already checked that I am only getting Aud Sounds or Sequences.
    try:
        if 'aud.Sound' in str(snd):
            #Don't process Sequences
            duration = snd.length / snd.specs[0]
            snd = snd.fadein(0,duration*0.05).fadeout((duration*0.95),(duration*0.05))
        play = dev.play(snd)
        return snd
    except:
        self.color = (0.75,1,0.75)
        return None

Thank you!

Just to let you know I can now reproduce the mix problem every time with this code:
        if len(listI) > 0:
            self.inputs[5].value = 0
            # Process input list from Array format [note,start,vol,duration] repeated in blocks of 4
            rangeN = int(len(listI) / 4)
            for i in range(0,rangeN):
                n = i * 4
                noteIdx = int(listI[n])
                freq    = getFreq(noteIdx)
                startT  = listI[n+1]
                volM    = listI[n+2]
                durT    = listI[n+3]
                if i == 0:
                    sndM = osc(self.mode,freq,samples,durT,volM).fadein(0,durT*0.05).fadeout(durT*0.95,durT*0.05)
                else:
                    sndI = osc(self.mode,freq,samples,durT,volM).fadein(0,durT*0.05).fadeout(durT*0.95,durT*0.05)
                    sndM = sndM.mix(sndI)
            duration = sndM.length / sndM.specs[0]
            return 0, duration, samples, sndM

Here is what is feeding this node:

The two notes are of duration 0.2084 and 0.8336, the first values per block of 51 and 55 relate to notes C4 and E4, but you can try this with any two frequencies. When I mix them together the resultant sound is always 0.2084 whatever I do in the code, i.e always the shorter sound. This is the problem I was wrestling with and one I need to cure. I suspect I am doing something wrong, but just cannot see it.
The function "osc()" is this in my functions file, I call this in a lot of nodes, so I put it in a separate .py file:
def osc(mode,freq,samples,duration,volume):
    if mode == 'Sine':
        snd = aud.Sound.sine(freq,samples)
    elif mode == 'Triangle':
        snd = aud.Sound.triangle(freq,samples)
    elif mode == 'Sawtooth':
        snd = aud.Sound.sawtooth(freq,samples)
    elif mode == 'Square':
        snd = aud.Sound.square(freq,samples)
    elif mode == "Silence":
        snd = aud.Sound.silence().resample(samples,False)
    snd = snd.limit(0,duration).rechannel(2).volume(volume)
    return snd

Do you have any suggestions to cure this please?

Yep, this is a bug in the length reporting of the mixed sound. It reports the shorter length, although it is actually longer as you can hear if you try playing it back. I fixed this, so if you use a build that is newer than right now it should work.
I also added a sample rate parameter to silence, so you don't have to resample anymore.
And one more tip: to check the type of an object in python you should use type(snd) is aud.Sound rather than 'aud.Sound' in str(snd). Alternatively you can use isinstance(snd, aud.Sound) which also works with inheritance, so isinstance(aud.Sequence(), aud.Sound) gives True, because the Sequence class is derived from the Sound class so that you can use it just like a Sound which seems to be what you want in some cases.

Thank you, I was reading the length to do other processes in later nodes, this is what didn't work with mixed sounds. I'll get a later build in a day or so. Thanks also for the tips on checking Aud Sounds, I had tried just isinstance(sound) which of course didn't work as that is taken by some other form of sound.

Hmmm, I have just found that I cannot mix Sounds and Sequences:

Now, I can convert Sounds to Sequences, if I know the length of the sound, but can I convert a Sequence to a Sound?

Please read the error message. Sequences are stereo by default, so if you make sure the specifications (channels and sampling rate) are equal, it works.
import aud
d = aud.Device()
s = aud.Sound.sine(440)
seq = aud.Sequence(aud.CHANNELS_MONO)
mixed = s.mix(seq)
h = d.play(mixed)

Eeeek! Sorry, didn't spot that one...

So, I will tell the whole story:
I was getting this error when trying to mix a Sequence and a Sound:
  File "/Users/mac/Library/Application Support/Blender/2.80/scripts/addons/animation_nodes/nodes/daw/sound_mixer.py", line 104, in execute6
    snd = snd.mix(snd2)
TypeError: Object is not of type Sound!

One was a Sequence and one was a Sound, but, the real problem was that some of my sound files are not the same sample rate as everything else! Now, I resample all sound files when I load the sound file dictionary to stop that - this is a one-off operation when I open a project.
Oh well, one lives and learns! This now works fine:

Thank you again.

I have now used my new-found knowledge to create a node which records any sound output from anywhere on the node tree and for any frame range, to a file and then adds this to the VSE:

In this example I have recorded the entire Drum track. The Switch node inputs either the drum objects, or nothing, if nothing the drum objects are not used to make a sound, but the track in the VSE will play. I have tried this for the entire project's sound and it seems to work OK - I will test this further though.
Now I can output the entire project to a sound file Yippeeee!


Erlang

Previous discussion: https://gitlab.com/gasche/gc-latency-experiment/merge_requests/14

Hey @filipevarjao, I would like to get this merged, but I don't have Erlang experience, and there's a couple of things I'm not sure about.

When I run the benchmark I don't get any output, do you know why this is? I'm using Docker to run it (see latest commit)
It looks like this benchmark isn't measuring the same thing as the other benchmarks: it looks like it is just measuring the time to allocate a single "message". Since the message is discarded on each call to push_message, there is only a single "message" object being referenced at a given moment in time. This is  in contrast to the other benchmarks that allocate an array of references to messages, and at each iteration an existing reference is replaces by a reference to a new message. Am I misunderstanding something about your benchmark?

cool, I'll take a look and get back to you

hey, @WillSewell, I tried to push my changes, but I don't have permission. Could you give me the rights? or do you prefer that I fork the project to open a new PR?

Hey, sorry for the delay @filipevarjao. That's great you have made some changes :)
Unfortunately it looks like the only why I can give you permission is to give your write access to the whole repo. It's probably best if you create a fork and create a new branch of this branch, and submit that as a new PR with this branch as the base branch.


change comments to // style

valid css comments were getting through sass compilation and causing some issues, e.g.  /* @import scss/_debug */, particularly in django's collectstatic, but it could potentially be an issue elsewhere

Great, thanks!

Published to npm as<EMAIL_ADDRESS>


add 2228z as an alternative to 2228o

Resolves #734
An L10s Ultra from 6 months ago worked fine (2228o), but a newly bought one, from the very same Amazon listing, same make and model, turned out to be a 2228z

This single change does not seem to be enough. "Add Entry" now works, but the newly added entry just shows this:

I take that back. After waiting a few hours the vacuum seems to work fine. Could you please consider merging this?

Could you elaborate a bit? Locally this PR works for me:

If there is some data you need me to extract from the new vacuum or from the app, let me know, happy to attempt to help with that.

@Tasshack any chance you will be adding this model soon?

Also interested. New L10s Ultra but isn't found in my account

Also interested. New L10s Ultra but isn't found in my account.
The Name in Fritzbox is listed as "dreame-vacuum-r2228z".

Thanks for the great work with the integration so far, crossing my fingers I might be able to use it in due time! =)
I just replaced my RoboRock with one of these Dreame's and it turns out mine was also this "r2228z" model so unfortunately I also get this error that the device can't be found in my account.


Owned Object Display Improvements

This improves the display of Owned Objects in three ways:

Images now display in the Preview Cards
Preview Cards are now vertically aligned
When a row has two Preview Cards these align with the first two cards above

Before:

After:

Without compromising on small screen appearance:

Looks like there's a regression in the types


Remove error reporting from rustup-init.sh

In response to #987 I made rustup-init.sh terminate unsuccessfully instead of printing a message in case there is an error running rustup-init.exe.

Looks like run() is also used by ignore(), but I don't think the latter should exit if an error occurs.

I see, so it would make more sense in this case to remove the conditional block completely in order to take out error reporting without influencing the behaviour of ignore()?

Yes, I think so.

Thank you for your feedback. I changed this.

@bors r+

:pushpin: Commit b223bb7 has been approved by Diggsey

run() this way is basically a pass-through, it might make sense to not use it at all.

I agree, removing it would also re-establish the intended documentation factor of ignore(), as stated in the comments.

@bors r+

:pushpin: Commit 176d874 has been approved by brson

:hourglass: Testing commit 176d874 with merge d9e6167...

:sunny: Test successful - status-appveyor, status-travis
Approved by: brson
Pushing d9e616716adc02253192c2c193e5f6e47ad42a32 to master...


Users who do not have permission for consent management, can delete the purposes through REST API.

Steps to re-create

Login to Management Console and create 3 purposes as Marketing/ Sales and Stat.
Create a role which has only has login permission and assign a user to it.

username<EMAIL_ADDRESS>password: Test123

List the purposes list and get a purpose ID. (Assume purpose ID=3)
Get the base64 value of (user1@wso2.com:Test123)
Execute the following curl command.

curl -k -v -H "Authorization: Basic base64encodedvalue" -H "Content-Type:application/json" -X DELETE "https://localhost:9443/api/identity/consent-mgt/v1.0/consents/purposes/3

Expected: 401
Actual: 200
Purpose gets deleted using encoded user credentials which only has login permission.
Type/Bug
Affected/5.5.0-Alpha
Severity/Critical
Priority/High

This is fixed with https://github.com/wso2/carbon-identity-framework/pull/1389/files


Dev 6.0 fixes

Fixed gc related issues
Commented out functions which are declared but not defined.

@kisvegabor should review the undefined functions. Other than that it looks fine to me.

@amirgon
Thank you!  These function can be removed. Can you do it?

@kisvegabor removed the functions on a9f807a

Great, thank you!


on_behalf_of missing in Conversion

When trying to call a conversion via the SDK, we are unable to find the on_behalf_of field.

None of the Entities are supplied with an on_behalf_field directly. There is a function on the Client that will execute operations on behalf of another contact. Hence you create a conversion in the normal way, but then execute the creation on behalf of the desired user
see
https://github.com/CurrencyCloud/currencycloud-net/blob/b26709b621c1ad656a34bdd19ad42d195b7325c7/Source/CurrencyCloud/Client.cs#L251


added contribution docs

In this Pr, i added a contribution guide for first time contributors, tips on how to keep clean branches and solutions to some blockers I faced on mac OSX

@manwar Kindly review and give feedback, thanks

This is very handy, thanks for your contribution. @kemsguy7


Adding jsonschema to requirements

This adds missing dependency jsonschema to requirements.txt.
Closes #187.

cc: @veghdev

@sugatoray Thank you for reporting the issue and your contribution.


Enabling sinfo printing and disabling CONFIG_SYSLOG_TIMESTAMP_REALTIME caused a recursive calling bug?

I enabled sinfo printing:

I found that sinfo called nx_vsyslog, then called clock_gettime
  if (OSINIT_HW_READY())
    {
#if defined(CONFIG_SYSLOG_TIMESTAMP_REALTIME)
      /* Use CLOCK_REALTIME if so configured */

      clock_gettime(CLOCK_REALTIME, &ts);

#else
      /* Prefer monotonic when enabled, as it can be synchronized to
       * RTC with clock_resynchronize.
       */

      clock_gettime(CLOCK_MONOTONIC, &ts);
#endif

But clock_gettime called sinfo again, which makes a recursive bug?
Here`s the backtrace from gdb:

@GooTal already fixed by https://github.com/apache/nuttx/pull/8897

@anchao Thanks a lot


Compiled sql should show all sql run for a model

Presently, compiled output in the target/ directory only shows the main statement in a dbt resource. This is undesirable for mulit-step resources like archives, incremental models, or table models with flags applied (eg. --non-destructive). Instead, dbt should show every executed SQL statement involved in creating a resource, as well as pre- and post- hooks.
dbt writes rendered sql to the target/ directory twice: once during parsing and then again during model running.
Both "parsing" and "running" invoke the statement block. In this same Statement block, there's a line that says if name == 'main', then write the compiled output to a file. Instead, this block should write the compiled output to a file for every statement. The "main" statement should still be used to determine the overall "status" for the resource.
The write function exists in the dbt context, and is responsible for writing compiled sql out to a file. This method should be changed (or a new method should be created) which writes the output of multiple statements into the same file. This could either be implemented by 1) writing statement sql to a file using append mode or 2) buffering the entire compiled sql output for a resource, then writing the whole thing to a file when the resource has completed running.

add comments to each statement explaining what it is

Would it be possible to output all the compiled statements during dbt compile?
Currently only the raw select statements are written out during compile, so at the moment there isn't really any way to check what's going to run before actually running it.
Alternatively something like a --dry-run flag would be useful to run the parsing stage only, this might be preferable as compiling all the materialization macros, checking for existing tables etc, is possibly going to trigger a lot more DB calls than just a regular compile!

hey @lewish! Yeah - I really want to do this. We prioritized it and deprioritized it for 0.9.1 because it proved to be a bigger chunk to bite off than we could manage at the time. This is scheduled for our next minor release, 0.11.0.
dbt previously had a --dry-run flag, but it proved to be pretty ineffective in our initial implementation. Our current thinking for --dry-run might look more like running an explain on the SQL to get some quick validation. More info on that here if you're interested

@lewish thinking about this further, it's not exactly clear how dbt should handle introspective adapter functions during compilation. If you have a model like:
select *
from events

{% if adapter.already_exists(this.schema, this.table) %}
  where created_at > (select max(created_at) from {{ this }}
{% endif %}

I think our only real option here is to run the query. Ideally, we'd be able to cache this query so that it's only executed once during the whole parsing stage. Just wanted to point out that there's a bit of a challenge here, and there's sort of a tradeoff between speed and correctness.

Example branch: https://github.com/fishtown-analytics/dbt/compare/investigate/compiled-output?expand=1
This is very doable, but will require non-trivial changes to the materialization code. Closing this as the "investigation" is complete. We should prioritize an issue in the future to actually implement the functionality described here.


chore(docs): Update docs to mention new protoc requirements in tonic >= 0.8.0

This PR seeks to make the docs clear on new protoc requirements in tonic >= 0.8.0

Motivation
This PR is for issue https://github.com/hyperium/tonic/issues/1049

Solution

Update docs to include clear mention of new protoc requirements in tonic >= 0.8.0

@LucioFranco This draft adds mention of protoc requirements to main README. In addition to that I will now look over the other docs to see if there are any places where I think this should be mentioned.

@LucioFranco the only other place in the docs where I see it as obviously useful to mention the dependency on protoc is in the README of the examples subdirectory, so I added text about it there as well.
IMO this PR is ready for merging, let me know what you think.


Main file NOT added to list of $compiler->parsedFiles

To maintain consistency with lessphp and the intent of cached compiling the following needs to be added to Compiler.php around line #164 (since there is no 'addParsedFile() method in scssphp)
        if (!empty($name)) {
            $realPath = realpath($name);
            $this->parsedFiles[$realPath] = filemtime($realPath);
        }

Ref #334 & #321

Sure. Submit a PR.


Shifting down the content which is under the banner

Hi Devin,
Loving your work. Would it be easy to allow the content which is behind the content banner to also slide down as the content banner appears ? I would like to display some banners which cannot be closed and stick as the user changes state. For this to be possible, I cannot hide any content.

got ya!  It is possible, I'll just need to find some time to look into it ha!  Going to be pretty busy over the next few weeks but I'll get to it thanks!


Having trouble searching for partial words

Hello -
I'm new to this, so I'm not having the easiest time using the package. I managed to get it running in the application, but I'm a little confused on how to do this.
I'm trying to search a field that has values like<PHONE_NUMBER>4 for 600005044 or 5044 and I'm receiving 0 results. If I search for the whole value (with leading zeros), I get the intended result back.
I have tried messing around with the mapping properties.
protected $mappingProperties = array(
   'testField' => array(
        'type' => 'string',
        'analyzer' => 'standard'
    )
);

and
protected $mappingProperties = array(
   'testField' => array(
        'type' => 'string',
        'analyzer' => 'wildcard'
    )
);

and
protected $mappingProperties = array(
   'testField' => array(
        'type' => 'integer',
        'analyzer' => 'standard'
    )
);

It seems like everything I try gives me the same results. Also, I've noticed that after changing those and running rebuildMappings() it always returns the same analyzer.
Any help would be greatly appreciated.

Hey @brockwddb,
Have you tried performing a wildcard query as per the documentation on the subject (wildcard and regexp Queries)
GET /my_index/address/_search
{
    "query": {
        "wildcard": {
            "postcode": "W?F*HW"
        }
    }
}

I have been trying to understand that documentation for a few days now, to no avail. Where exactly does that query string go in relation to the laravel package?
Is that the parameter you pass to the search function of the model?

@brockwddb check out this: https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/_search_operations.html
{
    "query": {
        "wildcard": {
            "serial": "*5044"
        }
    }
}

To run this query (in a manual way) using Elasticquent, do the following:
$model = new Books();

$params = $model->getBasicEsParams();
$params['body'] = ['query' => ['wildcard' => ['serial' => "*5044"] ] ];

$results = $model->complexSearch($params);

That appears to have fixed my issue. Thank you very much for your help!

Hey @timgws
I have one problem with searching in multi fields ,
This code work for me but i want search in multi filed like name and family:
`$model = new Person();
$params = $model->getBasicEsParams();
$params['body'] = ['query' => ['wildcard' => ['name' => "*mas"] ] ];
$results = $model->complexSearch($params);`
can you help me for optimize this query for searching in name and family fields,
Thanks


Uncaught ReferenceError: React is not defined

Problem Description
Hello, I'm having this exception in the console when builiding in the Visual Builder:
Uncaught ReferenceError: React is not defined
As far as I know, It's happening in the latest versions of Divi.
This is the source code, we are importing React as usual:
// External Dependencies
import React, { Component } from 'react';

It this happening to anyone else, or is it just me? Thank you!
Steps To Reproduce

A  React is not defined exception is thrown when using a third-party module in the VB.

Error
Uncaught ReferenceError: React is not defined
    at Object.<anonymous> (builder-bundle.min.js?ver=1.0.0:1)
    at n (builder-bundle.min.js?ver=1.0.0:1)
    at Object.<anonymous> (builder-bundle.min.js?ver=1.0.0:1)
    at n (builder-bundle.min.js?ver=1.0.0:1)
    at Object.<anonymous> (builder-bundle.min.js?ver=1.0.0:1)
    at n (builder-bundle.min.js?ver=1.0.0:1)
    at Object.<anonymous> (builder-bundle.min.js?ver=1.0.0:1)
    at n (builder-bundle.min.js?ver=1.0.0:1)
    at Object.<anonymous> (builder-bundle.min.js?ver=1.0.0:1)
    at n (builder-bundle.min.js?ver=1.0.0:1)

Same issue here. @lots0logs

Hi, could you bring this up in our Divi Devs Slack channel? You can gain access using this form: https://docs.google.com/forms/d/e/1FAIpQLSfe_iclu50e5laCcfdzsS1NR4KKqnvyFRWCRPLHkEUtdIPPYQ/viewform

Same here. As a result, all modules are broken built on top of this 'create-divi-extension'.
Tested on Divi v3.29.3.


[#3404] Update paver.sh from the python-package repo

Problem
paver.sh has been updated several times in salt-master and python-package.
Solution
Updated it to the latest version from python-package.
How to test
Please review the change.
Run the test_review tests.
reviewer: @adiroiban

needs-review

all good. thanks!
changes-approved


[PE-D][Tester A] Partnercourse info not functioning

When clicking any partnercourse display detail does not change

Labels: severity.Medium type.FunctionalityBug
original: Chrainx/ped#8

will be fixed in https://github.com/AY2324S1-CS2103T-W10-2/tp/pull/236


Technology section is missing for user

https://sourcerer.io/rrethy
Ideally, his profile should reflect his interests - machine learning, android, kotlin. we only show kotlin. where is machine learning? where is android?
Comments from luba in slack:
@android - he has android.app imports that are not supported currently because the lib doesn't have open source code. Will do it.
@machine learning - the code is in Matlab in which we don't detect libs now.

Profile have tech section now and libs for android. But we should cover more machine learning techs in future.


Does not build with standard library 2.0

I tried to build this with standard library 2.0 and it seems it doesn't:
Checking Everything (/build/source/Everything.agda).
 Checking Generics.Prelude (/build/source/src/Generics/Prelude.agda).
/build/source/src/Generics/Prelude.agda:9,51-10,49
warning: -W[no]ModuleDoesntExport
The module Relation.Binary.PropositionalEquality doesn't export the
following:
  Extensionality
  ∀-extensionality
when scope checking the declaration
  open import Relation.Binary.PropositionalEquality public
                                                    hiding ([_]; Extensionality; ∀-extensionality)
/build/source/src/Generics/Prelude.agda:22,45-23,87
warning: -W[no]ModuleDoesntExport
The module Reflection doesn't export the following:
  return
when scope checking the declaration
  open import Reflection public
                         hiding (var; return; _>>=_; _>>_; assocˡ; assocʳ; visibility;
                                 relevance; module Arg)
/build/source/src/Generics/Prelude.agda:24,1-27
Failed to find source of module Reflection.Argument in any of the
following locations:
  /nix/store/1yxiwwy44xxxgzdmvyjizq53w9cfinkn-standard-library-2.0/src/Reflection/Argument.agda
  /nix/store/1yxiwwy44xxxgzdmvyjizq53w9cfinkn-standard-library-2.0/src/Reflection/Argument.lagda
  /build/source/src/Reflection/Argument.agda
  /build/source/src/Reflection/Argument.lagda
  /build/source/examples/Reflection/Argument.agda
  /build/source/examples/Reflection/Argument.lagda
  /build/source/Reflection/Argument.agda
  /build/source/Reflection/Argument.lagda
  /nix/store/r3cpdqcxl9hna5xwgdgd57bazvfjbivq-Agda-<IP_ADDRESS>-data/share/ghc-9.6.5/x86_64-linux-ghc-9.6.5/Agda-<IP_ADDRESS>/lib/prim/Reflection/Argument.agda
  /nix/store/r3cpdqcxl9hna5xwgdgd57bazvfjbivq-Agda-<IP_ADDRESS>-data/share/ghc-9.6.5/x86_64-linux-ghc-9.6.5/Agda-<IP_ADDRESS>/lib/prim/Reflection/Argument.lagda
when scope checking the declaration
  import Reflection.Argument

See https://github.com/NixOS/nixpkgs/pull/153757/checks?check_run_id=26884760539 and https://github.com/NixOS/nixpkgs/pull/153757.

Hello, thanks for the issue, I haven't kept in touch with stdlib changes for a while. Can you please let me know if current master builds as expected on your side?
If so, I'll update the README with versionning requirments and make a proper release with a bumped version number.

Yes, indeed it does build, thanks!

Thanks for checking! Released under 1.0.1.
Let me know if you encounter any more issues.


Upgrade to bluebird 3.0

The new version of bluebird has a few breaking changes. They are pretty minor and this will probably amount to just updating package.json. However, this will sort of be a breaking change the chaining API since you can use the bluebird methods in the chains, so we should probably wait and do this as a part of v3.0.

Sounds good.

Closed in 3.0


CB-9529: Revert "CB-8969 Expand B/R flow to cover full backup/restore instead of just database"

There are multiple issues in the change merged in CB-8969.
Here is the list of issues

Database operation is set to failure when there is a failure in fetching overall backup-restore status.

Wrong id is used while polling overall data lake backup-restore status so the status of the database backup will always be set to failure/

Currently the flows end if the database backup-restore operation fails. Instead, polling for over-all status should be performed.

Currently, only backup-id is passed to restore flow but we also need the restore-id to perform a status lookup in datalakeFullResoreInProgress state. This needs changes to API and more testing.

I made code changes to fix most of them but one of the fixes needs API change which I will continue working on. As this issue reported in CB-8969 would break datalake-dr functionality which is already promoted to stage/prod, I prefer to first revert CB-8969 and commit again with appropriate changes.

As this issue reported in CB-8969 would break datalake-dr functionality which is already promoted to stage/prod, I prefer to first revert CB-8969 and commit again with appropriate changes.

If you want to do a HOTFIX for 2.32 you should target this PR to CB-2.32.0, and once it get's merged then create a PR for master. This is how we handle Prod hotfixes.


fs mount on local mk8s

While we wait for a general-purpose Pebble FUSE fs, this jhack allows you to open an admin file explorer in a live charm unit (the charm container or any of the workload containers).
WARNING:
horrible hackery involved.

todo:

derive PVC name from kubectl so we can drop the find hackery
mount the separate container mounts into one another so that dirs don't appear empty (using metadata.yaml to know the locations?)


Initial add of http-php template

This adds a PHP template that uses the Wagi executor.
It uses the latest PHP build from VMware Wasm Labs: https://github.com/vmware-labs/webassembly-language-runtimes/releases
Signed-off-by: Matt Butcher<EMAIL_ADDRESS>

The reason for the test failure is that at the moment the template unit tests have a constant for how many templates they expect to find in the Spin repo.  So it needs to be updated every time we add a new template.  It's a bit of a maintenance pain but it is what it is for now.
https://github.com/fermyon/spin/blob/1439b95fea89a4863c05ae489f1a9c0de17f27b9/crates/templates/src/manager.rs#L417

Okay, I think I got it all fixed up. Thanks @itowlson.

I will leave it to someone else to merge, since I have given up the pretense of being a maintainer. 😀


Anerdw: Watch heavenlyhimalayan.com

Anerdw requests the watch of the watch_keyword heavenlyhimalayan\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

heavenlyhimalayan\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Nisse Engström in Charcoal HQ


Fix issue :Failed to open file 'ETL_DDL/kafka_tracking.sql

ERROR at line 26: Failed to open file 'ETL_DDL/kafka_tracking.sql', error: 2

Coverage remained the same at 28.099% when pulling d2d84a60162d3e8d5334e76b4970cbe98682c7b0 on kiranalda:Kafka-Error-Resolved into d5eccf409c6e692f0b87155ab1f90bf7d4836f04 on linkedin:master.


Set project dock repo root to current working directory

It should be safe to assume that dock can operate whereever it
is executed as long as the current directory meets the general
configuration requirements (i.e. is a part of a git repository
and contains a .dock file - for at least specifying how the Dock
development container is constructed).
So, rather than default a project's repo root to the top level
git repository root, default to the current working directory and trust
that it and the user adhere to the previously mentioned requirements.

Looks good.


CMake unable to find GLFW3_LIBRARY

Hello I am stuck on step 10) of "Compiling for Windows with Visual Studio 2012". Specifically when I press configure on CMake, it can find the GLFW3_INCLUDE_DIR but cannot find GLFW3_LIBRARY. If it helps, cmake can find GLM_INCLUDE_DIR. I have placed all the files in "ProgramFiles(x86)" as stated in the instructions.

Are you pointing the GLFW3_LIBRARY path to the path of the shared object or DLL files on CMake?

Just updated the CMAKE files.  Please give it another try.  Use CMAKE_PREFIX_PATH to point to the location of your depencencies.

No response from original reporter.  Closing.


[Oppia Web] Give guidance to developers for how to test emails

It would probably be useful for developers to have a section explaining how to test functionality that involves emails, since these aren't set up to actually send an email anywhere except the prod environment (and we shouldn't use that for testing). In general, the basic idea is to make sure the relevant flags are turned on and look for the email text showing up in the server logs, but this flow needs to be verified/validated, probably using a step-by-step example.
(See https://github.com/oppia/oppia/issues/12062#issuecomment-1523300149 for an example of a developer looking for help with this.)

Hiii @seanlip please assign this issue to me

@vishisht-dubey  Per the guidance at https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia#choosing-a-good-first-issue, please provide an explanation of what your PR will do (with names of files you're changing, what you plan to change in each file, etc.). If it looks good, we can assign you to this issue.
Please also follow the other instructions on that wiki page if you have not yet done so. Thanks!

Take a look at core/platform/email/dev_mode_email_services.py. This is the file that handles emails in dev mode -- as you can see, it logs messages to the terminal using logging.info().
Now see if you can figure out how to do a real example that involves, e.g., sending the email that a user gets on signup. (Note that you might need to turn on various flags in the code like ENABLE_EMAILS etc. in order to do that, but working through this example should help you figure out what to change.)


AlfredKat not using the recommended location for the workflow cache

The cache location should be implemented through the alfred_workflow_cache environment variable. See: https://www.alfredapp.com/help/workflows/script-environment-variables/

as far as i remember i'm not caching anything in Alfred Kat. did you see any place in the code that is overriding this?

This following is created by AlfredKat and is located at ~/Library/Caches/AlfredKat.

oh. this is just me snooping on all the porn downloaded through Alfred Kat so that i'm getting some inspiration.
ok, in real i guess it may then be the SwiftSoup library used to parse HTML that may do some caching then. will check, thanks. quite a feat you caught that.

yeah so the files in fsCachedData are downloaded pages. will check with SwiftSoup if there's a way to configure where this is cached. will keep you updated.

wait. there's more. so after a quick research i'm finding that it's Apple's URL (https://developer.apple.com/documentation/foundation/url#) that is caching by default. had no idea. very cool thanks, learning a bit more about Apple's APIs. will investigate more and hopefully a fix within this week.

done in v7.0.0: https://github.com/godbout/AlfredKat/releases/tag/7.0.0
if you were using 6.2.0, you should see the update show up in Alfred when using KAT. thanks for reporting! learned quite a bit.