Eventually the circuit breaker will trip if enough minor transient errors happen

Hi,
The circuit breaker doesn't handle the scenario where we get a small amount of errors over an extended period of time.  The circuit breaker state keeps just a simple count and breaks for a hard set amount of time once it's seen the set number of exceptions.
Say we're using Polly to wrap a call an external web service that errors out once an hour.  After three hours we'll lock up hard, even though we probably don't want to break the circuit in this scenario.
Does it make sense to have some sort of expiration on when an exception happens so that their effect dampens over time?

Agreed, I was thinking that you could specify another timespan that would reset the count if an exception hadn't occurred in that period of time.
Thoughts?

Good idea, however could you have some internal checks on the handle exception to get round it....? You can argue that to keep it simple, you should implement it yourself in this layer instead, instead of adding the override to the circuit breaker?
Tbh, I would like to see the self-implementation of circuit breaker fork merged instead if that is in-line with your design, as that sort of covers this, ie do it yourself in a custom implementation, and also allows us to handle circuit breaks in a central place across several out-of-process services........ Though maybe my solution to tims request is the same for me... Haven't thought it through fully yet...😏

I'm not sure from @TimGebhardt's original question, if he was assuming there were successes in-between the failures across the three-hour period or not.  My reading of the codebase was that the breaker state is Reset() after any successful call to the supplied action.  (Have I got this right?). So, if [A] a service failed intermittently (with successes in-between), the existing CircuitBreakerPolicy implementation wouldn't trip.  If on the other hand [B] the service only received three calls across a three-hour-period and all failed, the existing CircuitBreakerPolicy would trip. (Seems to make sense - all calls have failed - but not sure how long one would choose to trip a circuit for on a service only averaging a call per hour?)
@TimGebhardt's question made me think tho about a frequency-based circuit-breaker (which Michael Nygard also hints at in his book).  Something like: "If the policy experiences N exceptions in any period T, break the circuit for a given timespan" (eg "Break the circuit if we receive 3 exceptions in 30 seconds").  I have just firmed up a sketch I put together a few days ago (when first saw Tim's q) and committed to https://github.com/reisenberger/Polly/tree/FrequencyBasedCircuitBreaker - is this an avenue worth pursuing at all?
It essentially maintains a FIFO queue of the timestamps of the last N exceptions, and uses this quite simply to determine if N have occurred within the given timespan.

Or for an alternative mathematical averaging approach? (doesn't maintain a FIFO queue)
https://github.com/reisenberger/Polly/tree/FrequencyBasedCircuitBreaker2
(I can see merits and demerits to all of the alternatives ...)

Regardless of the mathematics, I think your approach to the problem is fine as it makes more sense to what you would want a circuit breaker to be in practice.
However, to refer back to the other part of my note, all of these circuit breakers are great for the simple one-process approach. IMO, the type of things you are circuit breaking (external services) are highly likely to be used across processes. Unless I've completely missed it, I don't think Polly handles this scenario.
So, should Polly implement this ? (I think not, as the backing store to hold this data, and the appropriate trigger logic for the breaker, is probably very system specific) but I'm not sure of the best way to go about implemented this outside of Polly and injecting the functionality into it's fluent based framework...

Hi @bunceg .  If I understand you rightly, the scenario is that if services A, B and C (say) all use another service M, then if service A breaks its circuit to M, it could be useful for services B and C to know about this and break their calls to service M too?  We went through a similar line of thinking, but concluded - is there a need?  If service A cannot reach service M for a reason that would also prevent service B, won't service B just discover that of its own accord (and also circuit-break) in due course?  Is the overhead of inter-process co-ordination (which might also fail) worth it?  (And A could also fail to reach M for a reason which doesn't affect B ...)
We did, however, go through a similar line of thinking - that it could be useful to be able to inject state (as you say) into a CircuitBreaker - for manually breaking a circuit.  Sam Newman's (highly recommended) book on microservices suggests one might manually break all circuits to a downstream service in order to upgrade that downstream service.  I fretted for a while over the fact that we couldn't inject state into a Polly CircuitBreaker to achieve this.  For now, we have however gone with the same pragmatic above logic - do we need to?  If we build our processes robust enough to be able to survive downstream unavailability (whether unexpected or 'expected' eg due to upgrade), then that should be enough.  But @michael-wolfenden - could being able to manually break a circuit (inject state) perhaps be a useful addition sometime?

HI,
Yes you do understand, and we are following the same principle in that A, B or C will find out of their own accord eventually. I take on board your comment about the co-ordinator could fail, which is a real concern to our situation too, but IMO that is a decision for the implementing team based on the services involved and the consumers of those services.
However, your approach of injecting state may well solve the problem anyway - something else can check the shared state first and manually break the circuit and I like the way of doing this. Our implementation uses a master switch to avoid calling the service anyway but your option sounds a bit cleaner.
ps: thanks for the book recommendation, I'll add it to my collection with "release it!" :)

How about using a MemoryCache for success/failure rates by default, and getting a failure ratio from that? We prefer to use ratios rather than hard numbers (and likewise, ratios when the circuit is in the half-open state). Then just a simple x% of requests in n minutes have failed, trip it. TTL on the cache can easily give you the n minutes automatically.
Also agree re not having a centralised way to notifying other services about failures. What if the connection from just Serivce A -> M goes down/degrades, but the rest are OK? Now you're telling services B & C that they can't talk to M even though they can. Dangerous and lots of code/systems overhead. They'll figure it out.

Hi @reisenberger. Thanks to the team for developing Polly. It is a nice library.
In regards to this issue, I concur it would be nice with a feature that would allow the circuit to break if N exceptions occur over a period of T.
One example, where I would like a circuit to break is in the case, where a client is communicating with a server that is overloaded. The server would once in a while successfully handle a request, but most of the time return timeouts (As it will not be able to handle a request within the alloted time). Breaking the circuit for a little while, might help the server get it self up again.
Another example is the same as @TimGebhardt talked about where, during slow periods (Maybe nighttime), a single request is being retried and failing every once in a while, which breaks the circuit and can keep the circuit open, when good requests begin to filter in.
I looked at both implementations you made and think that both would solve the two examples above. The first implementation is easier to read. The second implementation requires are bit more thought to convience that there isn't an outlying case that could result in improper behavior.
Is this a feature that the team wants into Polly?
Anything I can help with?

@kristianhald Thank you for your interest in Polly - and offer of help!
More sophisticated approaches to circuit-breaking are very much on the Polly roadmap (see towards end of doc) (for reasons TimGebhardt and you mention, and others)
Right now AppvNext are reflecting further on the best forward path for circuit-breaker: whether to go for the frequency-based and proportion-based circuit-breakers already mooted in the roadmap, and/or whether something more sophisticated along the lines of Hystrix’s time-slicing approach (somewhat similar to the suggestion from @DarrellMozingo above).  Community views?
(may set out further views/options for feedback, later in this thread)

( and @kristianhald - we may yet take you up on your offer of help! )

Looked through the source for Hysterix regarding their Circuit Breaker implementation. As I read it, they use a bucket, where each item is a slice of the '_durationOfExceptionRelevance'. The precision is then defined by the size of the slice.
I think that the 'Frequency-based' solution is the same that Hysterix has implemented (Except that they look at percentages instead of the total number within the frequency given). The hubris in me, would state that the first implementation that @reisenberger has made with the queue of exception times, is time slicing where the window is very small and each window only contains a single information that an error occurred. :smiley:
I also like that the Hysterix implementation, before triggering the circuit breaker, checks if the number of requests received during the period is above a certain threshold (This is probably necessary, as else a single error will have a large impact on the error percentage, if the throughput is low).
// check if we are past the statisticalWindowVolumeThreshold
if (health.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
    // we are not past the minimum volume threshold for the statisticalWindow so we'll return false immediately and not calculate anything
    return false;
}

Personally, the few times I have used circuit breaker, it has mostly been leaning towards the frequency based. It is probably a personal preference, because I always like comparing stuff that happens with when they happened.

@kristianhald Many thanks for your input!
The Polly circuit-breaker internals have evolved since last August, and I pushed some new prototypes last couple days to https://github.com/reisenberger/Polly/tree/CircuitBreakerSpikeMarch16 . This includes a new sketch closer to the Hystrix approach.  @kristianhald / wider community: very happy for any feedback on these.  (New variants are also optimised for performance over the previous, and factor out metrics controlling the circuit more clearly.)
ConsecutiveCountCircuitController: the existing Polly circuit-breaker logic.
FrequencyCircuitController: as discussed earlier in this thread and in the Polly roadmap.  Triggers on >N exceptions in a period T.
ProportionCircuitController: as in the Polly roadmap.  Triggers on >N exceptions in T calls.
Commentary: An advantage of the FrequencyCircuitController and ProportionCircuitController is that both are simple to understand.  Some disadvantages:

a pure proportion/ratio-based circuit breaker may not adapt well to the kind of ‘slow period’ referred to earlier in this thread.  7-in-10 failures occurring widely spaced in a slow period –> you might not want to break circuit.  A minimum throughput could help here.

a pure frequency-based circuit breaker potentially has problems at the opposite end of the scale – it doesn’t adapt well to sudden increases in load.  For instance, say you initially set your circuit-breaker to trip on >5 failures per T.  That might have been a significant threshold at the time of configuration.  If the system in question goes through a serious load increase (say, peak sale period), suddenly you may be getting (and managing to handle) 75 requests per T, of which only 5 are failing (a small proportion) – you might not want to circuit-break on only 5 failures at that throughput (and kill off the other 70 requests which could have got through successfully).  One could argue this was misconfiguring the tripping threshold in the first place, but the concept remains: pure error frequencies (N per time T) as a measure, are vulnerable to not scaling with load increases or decreases.

... which brings us to...
TimesliceCircuitController: This is an early cut at a ratio-within-timeslice approach.  @kristianhald, it brings in the minimum throughput threshold idea too.  Comments welcome!
Commentary: Initially harder to understand.  But the approach deals well with both quiet period and high densities (combining as it does elements of both ratio and frequency measures).  The approach may be better suited to higher throughput systems because of the time-slicing, tho user has control over precision of slice.

@kristianhald If you are interested in helping/contributing, I can set out what would be needed to bring one or more of the above variants to production in Polly (wiring up the configuration; unit tests), and you can state what you might be interested in taking on?  ... The unit tests for any of the above time-based policies will be quite fun :smiley_cat:, using Polly’s SystemClock concept to manipulate time to simulate scenarios.
(AppvNext efforts at the moment are focused on #14 (a major step to unlocking some of the other elements on the roadmap), and I will have extra time challenges from April.)
As a first step also, the wider AppvNext team must decide how many of the above variants to bring to market.  As ever, balance to be struck between power/options and simplicity.
Community feedback welcome again: would all of the circuit-breaker variants in previous post be useful?

Further development effort has been made on TimesliceCircuitBreaker .  The https://github.com/reisenberger/Polly/tree/CircuitBreakerSpikeMarch16  spike now wires up the TimesliceCircuitBreaker syntax and adds unit tests.  The implementation does not at this time (in the hystrix manner) subdivide the timeslice into further statistical buckets, for 'smoothing' of the statistics at timeslice rollover.  Comment welcome.

Woaw, nice progress!! A bit faster than I expected. Here I come and say that I want to help and then I am nowhere to be seen. I apologise for that. Enough selfpity.
I think the TimesliceCircuitController looks good. The OnActionSuccess not checking, if the failureThreshold has been reached got me thinking. There might be a situation, where 3 failures will occur followed by a success, which would result in 75% errors and the throughputThreshold of 4 having been reached, which would mean that it should fail.
However, after a bit of thought, I came to the conclusion that it did not make sense to open the circuit on a success even if the failureThreshold has been reached.
What do you think?
Also I am a bit uncertain if the rollover is necessary and how hard it will be to understand. Without it will mean that in worst case, it will take up to 2*timesliceDuration, before the circuit opens.
A scenario could be where the timesliceDuration is set to 5 minutes and 1 minute into the slice, failures begin to occur, but just enough to be below the failureThreshold. The slice resets and the failures continue to occur.
I think that it will depend on what the timesliceDuration is going to be set to (seconds, minutes or hours) and how fast the circuit breaker should react.
For what I work on, rolling slices is not critical, as it will just mean that Polly will react on the failure state a bit later. Being able to specify ratio and timesliceDuration is already a long way.
I know I said it before, but I have more time on my hand now. If there anything you need fixed before this can be merged, then please throw it to me.

@kristianhald  Thanks for your interest and involvement with this feature!  If you have time available, any contribution to add the rolling slices/buckets within timesliceDuration (making it more a sliding statistical window) would be valuable.  Shout if you can work on that - and if any thoughts/questions around it.  (We could operate/operate initially without it, but it would improve the responsiveness of the breaker.)
Nice observation about OnAccessSuccess: will comment further shortly!
Re:

what the timesliceDuration  is going to be set to (seconds, minutes or hours)

... I have thoughts exactly about this - about the most appropriate way to configure this kind of circuit-breaker.  I will add thoughts either here or in draft documentation (review will be welcome...), in a few days' time.
Thanks!

@kristianhald Here are some specific thoughts around adding the rolling slices/buckets within timesliceDuration ... if you are able to work on it!
[1] The number of buckets that timesliceDuration is divided into internally, need not (I am thinking) be user-configurable at this stage.  For example, we could just pick a number (4? 5? 8? 10?) which significantly improves the responsiveness/fidelity of the circuit-breaker to its configuration.  eg 5 = rolling-throw-away 20% of the statistics at a time (a great improvement on 100%-throw-away; likely to cover most fidelity situations; strikes a balance between fidelity and extra computation).  What do you think?
Regarding not user-configurable:  Simplicity of configuration is a Polly goal.  My feeling is the extra configurability of number of internal buckets would not significantly add feature benefit versus the extra comprehension load.  Our aim is just improve breaker fidelity-to-configuration to acceptable tolerance, any sensible value that achieves this is good ...  (tho open to community feedback if anyone can make a case for configurable buckets ...)
(Hystrix probably cares about configuring this more, because they emit stats at bucket frequencies)
[2] Should we consider some minimum timesliceDuration below which we do not subdivide into further buckets?  Or (alternative/similar) some minimum size for a bucket? (ie use fewer buckets - or none at all - if they would be smaller than that).  What do you think?
Rationale: At some point there will be diminishing returns in the trade off between more fidelity/a more responsive circuit-breaker, versus higher computation.  We could performance-measure to determine exactly that point ... it might even be quite small ... But keeping pragmatic, is there likely to be a need to fine-tune breaker responsiveness (what buckets affects) at sub-half-second or sub-quarter-second levels?  (And if that fine-tuning is needed, reducing the overall timesliceDuration to those levels, without buckets, is probably equally effective.).  I would be comfortable with a minimum half-second or quarter-second bucket-within-timeslice size.
(In summary ... Usually I do not like to impose decisions from within a library on the user.  But for this already-quite-complex feature, I propose starting with some sensible/pragmatic decisions to reduce configuration complexity, and await community feedback (always welcome!) if refinement is needed... ... )

@reisenberger I can work on placing the health metrics in slices(buckets) to provide a rolling window for the circuit breaker.
The other day I made a quick POC based on the code you did, which includes a single test:
https://github.com/kristianhald/Polly/commit/0b0bb45d28a0e7fe983a003c68ce6628740d792b
The POC hardcodes the bucket size to 10, but this was only done because of the POC.
Would that be a way to go or were you thinking on going in a different direction?
Do you have existing tests that measure performance, which I could use for inspiration?

eg 5 = rolling-throw-away 20% of the statistics at a time (a great improvement on 100%-throw-away; likely to cover most fidelity situations; strikes a balance between fidelity and extra computation). What do you think?

Any number higher than than 1 would indeed improve responsiveness. My initial thought was to have a quick calculation (Is only done once at the creation of the circuit breaker), which would give a larger bucket slice duration for larger windows with a minimum of 0.x seconds per slice (And maybe also a maximum). Something like 4Log(windowDuration * 500) which would give the following results:

Window duration
No of buckets(rounded)
Bucket timeslice

10 seconds
8
1.25 seconds

1 minute
13
4.62 seconds

10 minutes
23
26.087 seconds

1 hour
37
97.297 seconds

The reason for choosing a calculation like the above, is to state that if the window duration is long, then the responsiveness of the breaker is allowed to be lower. However, choosing a calculation that provides a good number for any input might be hard?

Regarding not user-configurable

I agree with the difficulty of using the feature, if the developer has to choose the bucket size or the bucket duration. I always like having the library providing me with reasonable defaults as I believe the developers of the library to have better knowledge of the area, than I do. Also if the default is based on the window duration provided, then a developer wanting more responsiveness can lower the window duration.
I do not believe it is necessary for a 1.0 version of this feature(because the feature goes a long way and the bucket duration can be controlled by the window duration), but it can be hard for a library developer to anticipate all usages of their library(Having worked on a few software projects, where a core library had to be built upon and extension was only possible at the hooks they provided) and therefore there might be cases, where a developer will need to override the default.

[2] Should we consider some minimum timesliceDuration below which we do not subdivide into further buckets? Or (alternative/similar) some minimum size for a bucket?

I cannot imagine (Not that someone probably would need it) needing a bucket duration less than quarter- or half-a-second. We are talking about an application that needs to cut off as soon as enough errors are encountered and cannot wait quarter- to half-a-second before opening the breaker. In most applications I do not think that this is an issue.
We also have to remember that we are using DateTime, which only has a resolution of 1-16ms depending on hardware and OS settings. I believe talking about bucket durations that are so low is an advanced topic.
I still believe that in the first version, the library should provide reasonable defaults as this will be the setting mostly used. Also adding this feature will require some thought into what would be allowed and what should not be allowed.

Hi @kristianhald.  Thanks for all this thorough and detailed thinking; all very useful!

I made a quick [buckets] POC

Thanks - will look at this shortly!
I made a draft documentation (wiki page) earlier today at https://gist.github.com/reisenberger/d0ed99101634b0388dd7e3b92fbfadac .
@kristianhald Re the proposed logarithmic calculation of bucket size from stat window duration, see my comments within the draft doco about the downside of this kind of circuit-breaker with long statistical windows: the responsiveness of the circuit-breaker seems (do you agree?) essentially proportional to the stat window duration, however finely you divide it into buckets, because of the 'long tail of successes' problem described.  Is the logarthimic calculation of bucket size still worthwhile in light of this?  I cannot imagine anyone really wants a circuit-breaker that doesn't react for 5 minutes, 10 minutes, 30 mins, whatever, to an 100% failure problem.  Given the relatively narrow range of times this leaves for which the timesliceDuration probably makes sense (a few seconds to a minute or two), I wonder if it is adequate just to adopt a fixed number of buckets (unless makes buckets too small as prev discussed).  (While I like the elegance of the logarithmic approach, we have to consider also that we have perhaps to explain it in doco; that somebody has to maintain this in future, or ask/seek to understand why it was done this way, etc).  @kristianhald, perhaps really this is the same as you are already saying, in saying: choosing a calculation that provides a good number for any input might be hard... :smiley:
@kristianhald You'll see in the doco that I have varied the terms slightly (AdvancedCircuitBreaker; samplingDuration).  However, for now, in the forks we are working on, stick to current variable names if poss - this will keep forks easier to merge while work-in-progress.  When we are feature complete, then we rename to final naming before making the PR to master.  Thanks!  And: feedback on choice of terminology also v welcome!

@kristianhald To state more precisely part of my previous thinking: varying the size/number of buckets can increase the fidelity (fidelity-at-all-times) of the circuit-breaker to its configured thresholds, but cannot increase its responsiveness (overall speed of response) to the theoretical 'long-tail of successes' problem stated. (which also, after all, might not only be a theoretical problem in a lot of cases: some systems will behave exactly like this 100->0%! :smile_cat: )
I will continue thinking about this, but would be interested in your reaction to this problem.
Finally, regarding my comments about configuration values that are suitable / not-suitable: this is at this stage to share thinking and explore the contours of the problem, not suggesting yet to disallow values in code.  However, we should indeed consider (later) the configuration limits for each parameter.
Again: very many thanks for the productive contributions!

Draft documentation (https://gist.github.com/reisenberger/d0ed99101634b0388dd7e3b92fbfadac)  updated to be less prescriptive about which timesliceDurations make sense, because, as @kristianhald you rightly point out, we cannot anticipate all usages of the feature.
However, relationship between timesliceDuration and breaker responsiveness kept clear, because it is important that users seeking a responsive breaker do not fail to understand the possible implication of configuring a breaker in minute or hour periods.

@reisenberger Read the document and I entirely agree. I think that the naming is much better than what we have used before. I especially like samplingDuration.

This is because of the way a 'long tail' of successes affects statistics. Consider a circuit configured with a sampling duration of 5 minutes (not recommended), set to break at 50% failure threshold. Imagine everything has been working perfectly (100% success rate) for a while. 100% failures then start occurring (at roughly the same throughput). Such a circuit will take at least 2.5 minutes to reach 50% failure rate (to 'work off' the statistics from the 100% success era).

I did not take that into account, when looking at the value given to the samplingDuration variable. Its a bit interesting what happens, when both the failure threshold and the sampling duration can be changed.
Setting the sampling duration low, but the failure threshold high means that the breaker will break if there is a small hiccup in the network. However, setting the sampling duration high, but the failure threshold low means small hiccups in the network does not affect the breaker, but issues like packet loss can trigger the breaker. If both are set high, then it just means that it should only break in case the other side is really down and setting both low means you are using a service, where you really do not want to hammer in case something is wrong and you want it to not do it quickly (Reminds me somewhat of scraping protocols :smile:).

the responsiveness of the circuit-breaker seems in the worst case (do you agree?) essentially proportional to the stat window duration, however finely you divide it into buckets, because of the 'long tail of successes' problem described.

Yes.

I wonder if it is adequate just to adopt a fixed number of buckets (unless makes buckets too small as prev discussed).

I think that a fixed number of buckets would be sufficient for most cases and if someone needs a more finegrained or coarsegrained bucket number, then the easiest and probably best solution, would be to overload the AdvancedCircuitBreaker syntax with an additional field for specifying bucket number.
If a developer has this need, then it probably means the developer has enough knowledge about the circuit breaker to know the implications of this, I would assume.

To state more precisely part of my previous thinking: varying the size/number of buckets can increase the fidelity (fidelity-at-all-times) of the circuit-breaker to its configured thresholds, but cannot increase its responsiveness (overall speed of response) to the theoretical 'long-tail of successes' problem stated.

I was about to write, that I didn't agree, but having thought through different examples, the only example I can think of is the case where no communication has gone through the breaker, when suddenly communication begins and it fails. This is a very poor example.
I think that the number of buckets influence the circuit-breaker more or less depending on what the failure threshold and minimum threshold are set to (As you write). However, if the failure threshold is set low (< 50%) and the minimum threshold is set higher, then having only 1 bucket will reset the throughput counter every samplingDuration requiring the throughput counter to be met again, before the circuit-breaker acts.
I believe that the minimum threshold is the threshold that is the most affected by the bucket numbers, than the two other thresholds. Is that correct?

However, relationship between timesliceDuration and breaker responsiveness kept clear, because it is important that users seeking a responsive breaker do not fail to understand the possible implication of configuring a breaker in minute or hour periods.

Agreed. Just thinking about the scenarios using the current three variables, shows how difficult it is to select correct values for ones need. Selecting a bad combination of numbers can either result in the circuit-breaker opening too soon or too late.
Would it be an idea to describe in the documentation what happens, when varying the failure threshold and sampling duration. Thinking here about the four combinations that are the outer cases (high, low), (low, high), (high, high) and (low, low) and what would happen in the circuit-breaker?

@kristianhald Lots more useful thoughts - thank-you!  I think we are shaping up what are the important practical limits on each parameter. I will summarise these shortly for review.
Re the POC sketch, I see no problems with this (nice refactor).  I can see possible points-to-check later around possible/minor micro-optimisations [to consider if necess] and naming, but that best done after all the major conceptual issues and boundary issues (in discussing now) are resolved, implemented and tested/proved through testing.
Thanks for all the great contribution!

@kristianhald  Just additional forewarning that I will be away-from-keyboard most of today (house guests), so less responsive ...

@reisenberger Made a new implementation, which is a bit more cleaned up version of the POC.
Its located here: https://github.com/kristianhald/Polly/tree/CircuitBreakerSpikeMarch16_AddedWindows
I did not try to optimize the implementation, like as you said, we can do that when the fundamentals on done.
I went from using 'Bucket' to using 'Window' instead, as I felt it was a better name for it.
The implementation looks alot like the POC. Also I added a few tests to ensure that the windows are being used by the implementation correctly.
Do you see some test cases that are missing, which I should add?
I was thinking, in regards to the issue with a low sampling duration and windows, maybe for simplicity only use rolling windows if the sampling duration is set high enough. It could be documented by stating that if sampling duration is set to x or higher, then rolling windows are used else a single window is used for the entire sampling duration.
What do you think?

Hi @kristianhald

new implementation [...]
https://github.com/kristianhald/Polly/tree/CircuitBreakerSpikeMarch16_AddedWindows

Super; I will review further in the next few days.

Hi @kristianhald. Re:

I was thinking, in regards to the issue with a low sampling duration and windows, maybe for
simplicity only use rolling windows if the sampling duration is set high enough. It could be
documented by stating that if sampling duration is set to x or higher, then rolling windows are used
else a single window is used for the entire sampling duration.

Yes, let's do this.  I suggest herewith some decisions to keep us moving (tho comments certainly welcome if anyone sees other angles).  Let us declare in the TimesliceCircuitBreaker some internal consts such as:
internal const int DefaultNumberOfInternalBuckets = 10; // or 'windows' ... align to preferred terminology
internal const long ResolutionOfCircuitTimer = TimeSpan.FromMilliseconds(20).Ticks;

and run some decision code as you suggest, something like
if (timesliceDuration < ResolutionOfCircuitTimer * DefaultNumberOfBuckets) { /* don't divide into windows */ } else { /* do divide into windows */ }

Can the operational code be done so that it doesn't look too messy / branching depending on whether it's using further buckets/windows or not?
Rationale: Per DateTime documentation, DateTime resolution is around 15-16ms, as you commented earlier.  Let's round this up to TimeSpan.FromMilliseconds(20).  There isn't any point in creating slices (or windows within slices) smaller than this, as it will just lead to empty slices/windows.  As we are defining as a well-named const ResolutionOfCircuitTimer, it'll be easy to change later if needed, and easily visible if any user later has higher resolution requirements and investigates the code.
Similarly, in TimesliceCircuitBreaker/Async, let's change:
if (timesliceDuration <= TimeSpan.Zero) throw new ArgumentOutOfRangeException("timesliceDuration", "Value must be greater than zero.");

for something like:
if (timesliceDuration.Ticks <= TimesliceCircuitBreaker.ResolutionOfCircuitTimer ) throw new ArgumentOutOfRangeException("timesliceDuration", String.Format("Value must be greater than {0} milliseconds.  This is the minimum resolution of the CircuitBreaker timer.", TimesliceCircuitBreaker.ResolutionOfCircuitTimer /* converted to millisecnds! */));

These approaches could be refined later (for example, between 200ms and 20ms we could adopt an  algorithm which provided the maximum number of buckets which kept bucket size over 20ms).  But let's start instead from a premise 'keep things as simple as possible until we know they need to be more complicated'.
Regarding descending as far as TimeSpan.FromMilliseconds(20) rather than some arbitrary limit 1 second or 1/4 second, well: given we have the resolution, we may as well permit it: as you say, we can't foresee all users of the library.
Does this sound like a good way to proceed?
Tomorrow hopefully I will add my promised comments/responses on the practical limits on each parameter and how they may interact. Have one more observation to add, but otherwise believe that is fairly thoroughly thought through now.
Thanks!

@kristianhald I made a timing test harness also for the circuit breakers here: https://gist.github.com/reisenberger/92dc8d73b4df127b296ed8daee3ed93d
The results on my current development machine are here: https://gist.github.com/reisenberger/a6fab34402731333a61600dc0f06d7b0
Later, we can use this for performance tuning, if/as necessary.  At this stage, the intent was to determine that the impact of using-versus-not-using CircuitBreaker or TimesliceCircuitBreaker (both are tested) was negligible compared to the other tolerances we have been discussing.  It is not a surprise to know this, but good to have it confirmed.  The impact of using either circuit-breaker is (on my machine) about 3 to 6 millionths of a second per call (if anyone wants to double check the code/calculation I will be happy!).  While other machines/servers etc may vary, this is clearly orders of magnitude different than typical network latency etc.  And orders of magnitude faster than the dimensions we are setting on slices/buckets/windows (important to know no interference there).
[Results based on my original spike of the TimesliceCircuitBreaker; we should run this against the variants with buckets/windows also - but reassuring to know we are orders-of-magnitude insulated from interferance at mo]
If anyone can spot any thinking mistakes in the timing test harness, do shout!
Thanks

This is awesome stuff, you two! Sorry I've been unable to engage in the conversation, due to very tight commitments at the moment. I have been watching from the sidelines, but don't want to interject my opinions without thoroughly looking through the code and thinking through the scenarios.
However, I will take @reisenberger's tests for a spin and offer up the results, as well as any suggestions that could possibly be helpful.
You guys rock!

Addendum: my performance stats test only the synchronous versions of the circuit-breaker.  We should also test the async versions, as there'll be the extra async/await overhead.  It may be significant, but hopefully/probably not enough extra to touch the other configuration limits we have proposed.

@kristianhald Briefly re my previous comment:

Can the operational code be done so that it doesn't look too messy / branching
depending on whether it's using further buckets/windows or not?

(BTW, this was just a general thought - not based on any reading of the code). Thanks!

@kristianhald I now feel relatively clear on the effects of setting different parameters near boundaries, as follows.
failureThreshold: Very low (eg 10%) will clearly cause the breaker to behave like a 'hair trigger', to break on almost the slightest error.  Very high (eg 90%+) will cause the breaker to break almost only on underlying system completely unresponsive - similar to the original Polly breaker, but with the added benefit of samplingDuration cut-off.  I am assuming users can deduce these effects of failureThreshold; in the interests of keeping documentation concise, don't plan to document.
samplingDuration: Low will hit circuit resolution (already documented). Proportional responsiveness of breaker means that high will cause slow responsiveness (already documented).
minimumThroughput: Low will cause coarse initial throughput resolution (already documented). Propose code forbids the value 1; minimum permitted is 2.   @kristianhald A good observation where you alluded to the issue of possible miscalibration with minimumThroughput set high such that it might struggle to be reached within the samplingDuration.  Have documented this more generally as keeping minimum throughput away from typical throughput.
Effect of buckets/windows versus not: I believe the effects are much as we have discussed, but - given the very low 200ms boundary, which few users are likely to work near - my instinct is keep things simple (thinking of the majority of users): state the boundary but not document elaborately.  Most circuits governing downstream network calls will likely be working in timescales (eg seconds), clear away from the boundary.  If higher frequency requirements emerge, we can refine as-and-when.
As a general comment: @kristianhald as you say, we cannot predict all ways that people will use the circuit-breaker.  And performance characteristics of the circuit in question (for example whether more sssssssssssssssfffffffffffffffffffffffff [or] sssfsffsffsffsfsfssf) will also be a significant factor in the interaction between configuration and results.  I think it is to be expected that users using a circuit of this sophistication (and with sufficient throughput to merit such a circuit) should expect to have to engage in some performance tuning in light of the real-life characteristics of their given system, and those characteristics are not something we can predict.  However, we can warn users away from configurations which we know are likely to give unhelpful results.
This is my summary of configuration characteristics I see as worth documenting.  Is this missing something major, some more complex interaction?

I have updated my fork with some additional features. Below I will go through the changes as they all relate to comments you have provided @reisenberger.
The code can be found here: https://github.com/kristianhald/Polly/tree/CircuitBreakerSpikeMarch16_AddedWindows

Rationale: Per DateTime documentation, DateTime resolution is around 15-16ms, as you commented earlier. Let's round this up to TimeSpan.FromMilliseconds(20). There isn't any point in creating slices (or windows within slices) smaller than this, as it will just lead to empty slices/windows. As we are defining as a well-named const ResolutionOfCircuitTimer, it'll be easy to change later if needed, and easily visible if any user later has higher resolution requirements and investigates the code.

I have created the internal constants and updated the syntax to use the resolution of the circuit as the minimum allowed timesliceDuration (sampleDuration).

Can the operational code be done so that it doesn't look too messy / branching depending on whether it's using further buckets/windows or not?

I have done circa the same, that you have done with the ICircuitController using a strategy based approach. I believe it is the cleanest and I thought that any performance issues with using an interface instead of the class directly could be measured before not selecting it.
There are currently two implementations (RollingHealthMetrics and SingleHealthMetrics). The selection is based on timesliceDuration < ResolutionOfCircuitTimer * DefaultNumberOfBuckets, selecting one or the other. There is a performance overhead for using interfaces. See test results in the bottom of this comment.
At the moment the decision on using one or the other strategy is happening in the controller, but depending on your view, the choice can easily be done where the circuit controller is being created and then injected. I have a preference, but in this case I think that its better you make that decision :smiley:

These approaches could be refined later (for example, between 200ms and 20ms we could adopt an algorithm which provided the maximum number of buckets which kept bucket size over 20ms). But let's start instead from a premise 'keep things as simple as possible until we know they need to be more complicated'.

I agree. Should not be hard to add. If we keep the current implementation, then its just another implementation of the interface and then doing a selection of when to use it (Probably when timeslice duration is between X * Resolution and Resolution * MaxNumberOfWindows.
@reisenberger @joelhulen Did a run with the code from the fork I am working on using my development host.
First result is where RollingHealthMetrics are used without inheriting from an interface:
https://gist.github.com/kristianhald/534ed0540e8030060f1c49da04100e4a
Second result is still with RollingHealthMetrics, but it inherits an interface and the interface is used in the circuit controller(The strategy based implementation):
https://gist.github.com/kristianhald/81d00afc036001afaac3c7d8927b6864
Using an interface decreases the performance, but only with around 10 ticks per iteration.
I checked that on my machine there goes about '10000' ticks per millisecond, which means that a difference of 10 ticks is very low.

I now feel relatively clear on the effects of setting different parameters near boundaries, as follows.

I agree with every variable comment.

To take the documentation away from too abstract discussion, I may add a 'suggested typical starting configuration' for downstream network calls.

I looked at the draft documentation again and the part about configuration recommendations for the sampling duration requires some thought to it. I think that beginning the documentation with suggested configuration, which can allow the user to quickly be up an running and then having more detailed information later is a very good way of going :smile_cat:

@kristianhald Re:

updated my fork with some additional features

Thank you for this extremely productive contribution!  Hoping to review and then we can pull this together for release very shortly (keen altogether to get this feature out this week or next, due to other commitments) (just need find time to review :smile: ).

@kristianhald

the part about configuration recommendations for the sampling duration requires some thought [...]
beginning [...] with suggested configuration [...] then more detailed information later

Completely agree!  Will adjust ...

@kristianhald Merged your work on RollingHealthMetrics to my local fork.  Thanks for the great contribution!
@kristianhald @joelhulen Remaining to do (on my side) before this ready to PR against master:

minor tidies
rename TimesliceCircuitController etc to preferred terminology
remove breaker implementations (proportion, frequency) not currently intended for release
add readme.md doco [brief]
yaml etc

Hopefully in the next day or so ...

@kristianhald  To offer some commentary on final decisions taken on previous points you raised:

At the moment the decision on using [RollingHealthMetrics and SingleHealthMetrics]
is happening in the controller

Left this where you had it.  Encapsulates the concern.

on my machine there goes about '10000' ticks per millisecond

See const TimeSpan.TicksPerMillisecond

after a bit of thought, I came to the conclusion that it did not make sense to open the circuit
on a success even if the failureThreshold has been reached

I reviewed and agree with this.  Breaking on a success seems counterintuitive.  The circuit may have 100% recovered for now (sssssssssss...), in which case breaking would be counterproductive.  If the circuit hasn't 100% recovered and is still intermitting (ssfsfsfsfsf), circuit will receive a failure soon enough within the same period, to break on.
@kristianhald Please feel free to review the final changes to  https://github.com/reisenberger/Polly/tree/CircuitBreakerSpikeMarch16 if you have an interest.  I intend to PR this to master later today.

See const TimeSpan.TicksPerMillisecond

Ahh, did not know that the ticks per millisecond for DateTime and TimeSpan is constant. Nice to know.
Normally ticks per millisecond is defined by the CPU and OS based on the frequency they work on, which is why I provided the number of ticks :smile:

Please feel free to review the final changes to https://github.com/reisenberger/Polly/tree/CircuitBreakerSpikeMarch16 if you have an interest

Did a quick lookthrough the changes and I think they are good.

I intend to PR this to master later today

Cool. Out of curiosity, what is the procedure for nuget packaging the master branch?

what is the procedure for nuget packaging the master branch?

Merging to App-vNext/master (and earlier, creating the PR against it) automatically runs an AppVeyor verification build.  We could push packages to nuget automatically on merging any PR, but opt to push them to Nuget manually.  (Manual gives us the ability to occasionally merge PRs (eg readme doco fixes) without pushing a package and bumping rev number.)
@joelhulen owns this part of the process, and can correct me if any of that is wrong / out of date.

@kristianhald PR to master in process.  Thanks for your awesome contributions to this feature in code and thought.

@kristianhald Yes, @reisenberger has it right. The AppVeyor build process generates release-level nuget packages when we merge a PR. We control the version number via a file named GitVersionConfig.yaml, which is used for applying the release version for the signed and unsigned versions of the packages. We also update the Polly.nuspec file to include the change notes that are added as part of the package, as well as on the nuget.org site. Once the files are generated by the build server as artifacts, I manually upload them to nuget.org. As Dylan rightly states, this is to control the deployment of the packages.

@kristianhald The Nuget packages are live!

A further post to this closed issue just to document an issue that was discussed between @kristianhald and @reisenberger during merging down on to @reisenberger's fork.  This post copies parts of the discussion from there (https://github.com/reisenberger/Polly/pull/1) in case that is ever lost:
[The issue has no bearing on the operation of the v4.2 AdvancedCircuitBreaker (proved by unit tests), but would come into play if the Polly AdvancedCircuitBreaker were to want to evolve to emit health statistics at regular intervals.]
@reisenberger wrote:

Hey @kristianhald. Here is a conceptual issue. I observe that, from our implementation, the
windows we create within the timeslice are not necessarily adjacent. For instance, imagining
windows of size 1, if the first action records at t = 0, and the second action at t = 1.5, we will be
running two windows from 0->1, and 1.5->2.5, with a 0.5 gap in between them. Reasoning
suggests this has no effect on the statistics, as the gaps only represent 'empty' times with no
results.

I considered an outside case: imagining windows of size 1 in slice size 10, we could have a window
at t = 0 and another at t = 9.9, this latter in theory extending +1 to t = 10.9 ... Again, reasoning
suggests that if any statistic comes in at t >= 10, the t = 0 window will be dropped, so at no point
will we be considering statistics for a period >10. However, it might be nice (for regression value, in
case anyone refactors...) to have a test proving this.

It looks like this could be done by duplicating the test
Should_not_open_circuit_if_failure_threshold_exceeded_but_throughput_threshold_not_met_before_timeslice_expires__even_if_timeslice_expires_only_exactly()
and adding SystemClock.UtcNow = () => time.Add(timesliceDuration.Subtract(TimeSpan.FromTicks(1))); just before the third exception?
Would this cover it? It should cause it to create a new bucket which overlaps into the following
timeslice ... but the t = 0 window should be dropped when the fourth exception comes in?

In general: Do you agree that the non-adjacent windows issue has no effect on statistics, or am I
missing something?

Perhaps controversially, my first instinct is not to code to make windows adjacent before merging. It
adds complication and doesn't add anything to solving the problem at hand. Following the mantra
'Keep the code as simple as possible to deliver the job needed', we don't need to correct for this,
and the current creation/disposal of window code reads intuitively. However, there would be future
scenarios (eg emitting regular statistics, as Hystrix does) where it would become important. And
interested in your thoughts...

@kristianhald :

@reisenberger Don't have such a test and I believe that you are correct that we should have one to
ensure that it works as expected. I think that the test would ensure the outer case is valid.
I will add it immediately.

[this test was added and proves correct operation of the current statistics]
@kristianhald :

I am under the impression, like you state, that as long as we remove buckets/windows that are
older than Now() - timesliceDuration then it will not have an impact. Also I felt that the
implementation would be smaller and easier to read.

Did a little thinking and it might actually be quite easy to add the logic for adjacent
buckets/windows.
If we need the buckets to align adjacent to each other, then I think that the easiest solution would
be to decide on a 'start time' (maybe construction time of the class or just zero) and when we
create a new bucket, we move the 'StartedAt' timestamp backwards in time to where the bucket
should have been created.

Lets take the following scenario:
Start time: Could be decided at constructor time or just be zero (Beginning of the universe
:smile_cat:). Lets say it is 23 seconds (from when the universe began).
Bucket/Window duration: 0.7 seconds

At the current moment in time, DateTime.Now() reads 26.457 seconds and an error occurs.
From 23 seconds and forward until 26.457 the buckets StartedAt reads:
23, 23.7, 24.4, 25.1, 25.8, 26.5

As the 26.5 bucket is in the future, then this cannot be created. Therefore the bucket we are in is
25.8. The question is then, how do we get from 26.457 to 25.8 without having to create all buckets
from 23 seconds. The answer (I think :smile:. Just throwing a quick idea here) is the following
calculation: StartedAt = Now() - ((Now() - StartTime) MOD BucketDuration)

If we take the above numbers we should get:
StartedAt = 26.457 - ((26.457 - 23) MOD 0.7)
= 26.457 - (3.457 MOD 0.7)
= 26.457 - 0.657
= 25.8

It will still have empty buckets as holes, but it should align the buckets to each other.

Question: Would aligning the buckets actually solve the statistics issue?

eg emitting regular statistics, as Hystrix does

When you say 'regular' do you mean bucket/window duration regular, every 1 second regular or a
delegate call, when an event occurs (Success or Error)?

@reisenberger:

At regular intervals (every call = too frequent). But the frequency of emitting these statistics would
absolutely be aligned to either timesliceDuration or windowDuration. Definitely don't want some
different system of timing to interact with ;~) Vague memory, Hystrix emits stats at one of these
intervals I think.

Options for adding this (later) to Polly:

+emit a HealthMetric when dequeueing it from front of the queue;
OR (more timely, fractionally more code)
+emit a metric when it ceases being the _current

.
This essentially documents the issue, tho there is further implementation discussion at
https://github.com/reisenberger/Polly/pull/1.
Implementation would be as @kristianhald suggested with:
StartedAt = Now() - ((Now() - StartTime) MOD BucketDuration)
An alternative could be:
StartedAt = Now() - ((Now() - StartTimeOfLastBucket) MOD BucketDuration)
but this opens up complications across a circuit reset, when there isn't a last bucket.

Noting against this closed issue for future reference, possible optimisations that could be made in the AdvancedCircuitBreaker implementation in future:
Any optimisation decisions should be driven by need, and by running before-and-after performance stats, for example with the test harness here: https://gist.github.com/reisenberger/92dc8d73b4df127b296ed8daee3ed93d
Current performance:
Performance analysis of the AdvancedCircuitBreaker across one-hundred-thousand (100000) iterations, including frequent circuit-breaking, suggests introducing the AdvancedCircuitBreaker costs around 30 ticks (3 millionths of a second) per call.  Performance for the original Polly CircuitBreaker broadly matches.
Possible optimisations:
[a] Replacing the use of Queue with an internal fixed-size array we manage ourselves.  While this has little benefit in itself (the implementation of Queue is backed by a similar array...), it may open up [b].
[b] Avoid the generation (and subsequent garbage-collection once out of scope) of HealthCount instances.  This could be done by retaining a fixed set of instances ourselves, in an array [a] we own, and blanking-and-reusing existing instances rather than letting them go out of scope and creating new.
[c] Maintain a running total of successes/failures across the timeslice, rather than re-summing it each failure  (to be checked by performance analysis whether this optimises or not; it’s swings and roundabouts; the change would add a ++ to each success/fail; avoids larger summing each fail)


No error on too many grams / unexpected behavior

As mentioned in https://github.com/Bookworm-project/BookwormDB/issues/105, querying for an ngram greater than the n that are indexed returns full summary statistics. This should give an error or, at the very least, return zeroes.
The error handling can be done with the new "method":"data", "format":"json" response type.

Is there a already a way to recall the number of words that are indexed, or does it need a new method to check with the DB? I have a simple response here that raises an error if there are more than 2 words, but for unigram-only BW or event >2grams, the better version is to compare against the index parameters.

Currently there's no way to check. The ticket is open at https://github.com/Bookworm-project/BookwormDB/issues/88.
I think this requires a whole new table in the MySQL database to use as a key-value store.
For short-term, the big question is what happens when you execute a bigram query on a unigram-only bookworm. I'm not sure, but there's probably some way to catch an error?


documentation about recursive functions (CREATE FUNCTION)

What happens?
It seems that currently functions that are defined recursively will either not compile or will result in a max expression depth limit error, e.g. using the snippet shown below.
It would be helpful if the documentation could not only clarify what should be expected of the current versions of DuckDB, but also indicate the path forward.
Please feel free to interpret this documentation-bug report as a functional-enhancement request :-)
To Reproduce
create or replace function f(n) as (select 1);

create or replace function f(n) as (
  case when n <= 1 then 1
  else f(n-1)
  end
);

select f(3);

OS:
MacOS
DuckDB Version:
any
DuckDB Client:
CLI
Hardware:
No response
Full Name:
Peter Koppstein
Affiliation:
Princeton University
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a source build
Did you include all relevant data sets for reproducing the issue?
Not applicable - the reproduction does not require a data set
Did you include all code required to reproduce the issue?

[X] Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

[X] Yes, I have

@soerenwolfers asked:

do you have an example that doesn't use the CASE clause?

I'm not sure the following will be helpful, but since you asked:
create or replace function f(n) as (select null);

create or replace function f(n) as (
  select coalesce(n, f(n))
  );

select f(1);

In this case, of course, one might be forgiven for expecting the result of f(1) to be 1, but it yields a "Binder error".


Using this for ARM packages

Hi, I am maintaining currently the ARM fork of meteor at https://github.com/4commerce-technologies-AG/meteor.
There are often packages on atmosphere that are not ready deployed for ARM architecture. In this cases you have to download the package source (mostly from github) and append it locally to your meteor project.
I would like to spend an ARM stratosphere server to provide all packages also on ARM devices.
Could / would you like to spend some time helping on this?
In any way, great to see this project!
Tom

Well, I guess this could be used to that purpose.
Although I can't spend much time towards that right now, we can discuss the subject and look at what the best approach would be.
Note that this project is only the package server, but not the build server. So the build server will have to support builds for ARM (Do the meteor build servers support that?) If not you will have to set up your own build server.
Anyway what you'd probably have to do is:

When importing packages from upstream, make sure they do not get published to the client if they do not have an ARM build. You could do so by setting a flag when importing them (here: https://github.com/sebakerckhof/stratosphere/blob/master/packages/stratosphere_synchronizer/server/synchronizer.js#L131 ) and not publishing those packages to the clients (here: https://github.com/sebakerckhof/stratosphere/blob/master/packages/stratosphere_core/server/rpc/meteortool/syncNewPackageData.js#L113 )

Schedule builds for those packages and after they are build, remove the flag so they get published to the clients.

Hi Seba ( @sebakerckhof )
thanks for feedback. I have already setup a build farm on ARM processors to create the meteor tools bundles and some packages. So building up the ARM packages is not the problem, I am just looking for the package server.
Well, I will setup a package server on our ARM server farm and get familiar with your development. I will come back with some explicit questions.
Tom


JAXB XSD annotations?

Hi, thanks for creating jsonschema2pojo! It's very useful!
Would it be possible to include JAXB annotations to the generated POJO classes so that an XSD can be generated from them? We need to be able to convert from XML to JSON using the JSON schema from an external provider and our service needs to provide a contract (XSD) to another service that gives us XML to send to the REST API.
So:-
JSON REST API <--JSON--->  Us <--XML--> Internal service

You could write your own annotator to do this. The annotator could add the jaxb annotations you need. Pass this to the plugin as customAnnotator.
If you can live without the XSD (or create it some other way) then you might find Jackson's XML support useful:
https://github.com/FasterXML/jackson-dataformat-xml/blob/master/README.md
It can take a Jackson annotated class and serialize/deserialize to/from XML.


bibutils: 6.6 -> 6.7

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/bibutils/versions.
meta.description for bibutils is: '"Bibliography format interconversion"'.

Checks done (click to expand)

built on NixOS
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/bib2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/biblatex2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/copac2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/ebi2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/end2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/endx2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/isi2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/med2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/nbib2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/ris2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/wordbib2xml passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2ads passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2bib passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2end passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2isi passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2nbib passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2ris passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2wordbib passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/bibdiff passed the binary check.
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/modsclean passed the binary check.
20 of 20 passed binary check by having a zero exit code.
0 of 20 passed binary check by having the new version present in output.
found 6.7 with grep in /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7
directory tree listing: https://gist.github.com/ed775298befddb629f9b11edd02c5e10
du listing: https://gist.github.com/ec275d2c3f07c078ebee878ee85450ba

Instructions to test this update (click to expand)

One-time optional setup to skip building using Cachix:

Install cachix from https://cachix.org/
Use r-ryantm's cache:cachix use r-ryantm

Test this update by entering a nix shell, seeing what is inside the
result, and if applicable, running some binaries:
nix-shell --pure -I nixpkgs=https://github.com/r-ryantm/nixpkgs/archive/ad3af8a7064a752b830f3f0241e02d1c70512731.tar.gz -p bibutils
ls -la /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7

cc @garrison for testing.

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: bibutils
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: bibutils
Partial log (click to expand)
shrinking /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/bibdiff
shrinking /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/ebi2xml
shrinking /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/xml2nbib
shrinking /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/endx2xml
shrinking /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin/isi2xml
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/lib  /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7/bin
patching script interpreter paths in /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7
checking for references to /build in /nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7...
/nix/store/crpkl9ypd5c80s7qcf4k3355ic395pz7-bibutils-6.7

Success on aarch64-linux (full log)
Attempted: bibutils
Partial log (click to expand)
shrinking /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7/bin/ebi2xml
shrinking /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7/bin/copac2xml
shrinking /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7/bin/biblatex2xml
shrinking /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7/bin/bib2xml
shrinking /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7/lib/libbibutils.so.6.7
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7/lib  /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7/bin
patching script interpreter paths in /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7
checking for references to /build in /nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7...
/nix/store/dbh21y33ilky3c7hkaw27pjl5bd7qyyg-bibutils-6.7


update rules_js

Ah. I already had this already covered in #8. Did you mean to land it separately?

yes I totally borrowed your fixes from there, we do want to land this separately because we can :)

because it fixes https://github.com/aspect-build/rules_ts/issues/15


Remove needless uses of PyString::intern

Change Summary
While working on #1085 I noticed that we have some calls to PyString::intern in the validator build step. This isn't great; it's plain and simple a memory leak because:

These strings are already held onto by the validators, no need to intern them
Interning them means they will outlive the validators if they're destroyed

It's only a very minor issue, as validator build normally happens only on startup, but still worth cleaning up.
Related issue number
N/A
Checklist

[ ] Unit tests for the changes exist
[ ] Documentation reflects the changes where applicable
[ ] Pydantic tests pass with this pydantic-core (except for expected changes)
[ ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

please review


oneof values with nil

Package version v10:
Question:
Is possible validate oneof value with nil like when I need 2 values or value not send? Like:
AcceptTerms   *string `json:"terms" validate:"isdefault,oneof=yes no"`

please close.... I found the solution :flushed:
AcceptTerms   *string `json:"terms" validate:"omitempty,oneof=yes no"`


docs: update user faq page

Removed an old question and added one that is still relevant.

@davidumea You must have banged your head quite a bit to discover this problem and come up with this solution. :smile:
Can you send me the upstream issue? Do they see it as a bug or a feature?
Right now, it feels like we're caught between two bad choices:

either we recommend our users to (selectively) use a kubectl version which is EOL;
or we disable PostgreSQL TLS.

I'll read the issue upstream and will get back to you.

@davidumea Nevermind, I just saw the ck8s-arch issue you just created.

@OlleLarsson Since we won't have a fix for this in place & applied in clusters for a while, I guess we should still merge this?

@OlleLarsson Yes, we should merge this, but somewhat reworded. We should:

recommend disabling TLS;
explain that this does not compromise security, since:

traffic between kubectl and the Kubernetes API is encrypted, and;
in-cluster network is trusted.

point to the upstream kubectl bug.

Since we won't have a fix for this in place & applied in clusters for a while

Well, they need to request that we should disable tls and we can disable it, so I would say that everything is in place :)


[SDP-945] serve/httphandler/cmd: post tenants endpoint

What
This PR adds the POST /tenants API to provision new tenants through HTTP requests.
Why
We should be able to provision a new tenant through HTTP requests.
Known limitations
N/A
Checklist
PR Structure

[x] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR does not mix refactoring changes with feature changes (split into two PRs otherwise).
[x] This PR's title starts with the name of the package, area, or subject affected by the change.

Thoroughness

[x] This PR adds tests for the new functionality or fixes.
[x] This PR contains the link to the Jira ticket it addresses.

Configs and Secrets

[x] No new CONFIG variables are required -OR- the new required ones were added to the helmchart's values.yaml file.
[x] No new CONFIG variables are required -OR- the new required ones were added to the deployments (pr-preview, dev, demo, prd).
[x] No new SECRETS variables are required -OR- the new required ones were mentioned in the helmchart's values.yaml file.
[x] No new SECRETS variables are required -OR- the new required ones were added to the deployments (pr-preview secrets, dev secrets, demo secrets, prd secrets).

Release

[x] This is not a breaking change.
[ ] This is ready for production.. If your PR is not ready for production, please consider opening additional complementary PRs using this one as the base. Only merge this into develop or main after it's ready for production!

Deployment

[ ] Does the deployment work after merging?

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr97.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr97.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr97.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr97.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr97.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr97.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr97.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr97.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr97.previews.kube001.services.stellar-ops.com


Remove old graphics, extra checkout, secret search and more

Removed old graphics dir
Added **/dist/** to .gitignore
Removed unnecessary checkout from pr-close
Dialed down backend allocations
Removed secret search (now handled by Trivy)

Thanks for the PR!
Any successful deployments (not always required) will be available below.
Backend available
Frontend available
Once merged, code will be promoted and handed off to following workflow run.
Main Merge Workflow

Just got a question, for using GitHub secrets, I didn't see we specify the environment in each step, how the step knows if it should get secrets from "dev" or "test" or "prod" environment  from GitHub secrets?

We shouldn't be providing an environment for DEV, only TEST and PROD.  Using an environment for DEV removes PR-level secrets from Dependabot, which causes a break.  So use no environment for DEV and Dependabot, let TEST and PROD override them when called.


vaex dataset cannot be read in bqplot example

ERROR:MainThread:vaex.file:could not read 4 bytes from '/Users/scorlay/.vaex/data/helmi-dezeeuw-2000-10p.hdf5'
Downloading http://vaex.astro.rug.nl/data/helmi-dezeeuw-2000-10p.hdf5 to /Users/scorlay/.vaex/data/helmi-dezeeuw-2000-10p.hdf5
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-67d039f6bcd7> in <module>()
      1 ds = vaex.example()
      2 N = 2000 # for performance reasons we only do a subset
----> 3 x, y, z, vx, vy, vz, Lz, E = [ds.columns[k][:N] for k in "x y z vx vy vz Lz E".split()]

<ipython-input-4-67d039f6bcd7> in <listcomp>(.0)
      1 ds = vaex.example()
      2 N = 2000 # for performance reasons we only do a subset
----> 3 x, y, z, vx, vy, vz, Lz, E = [ds.columns[k][:N] for k in "x y z vx vy vz Lz E".split()]

AttributeError: 'NoneType' object has no attribute 'columns'

Thanks, which OS is this?

OS X

Ok, vaex uses wget, which osx does not have by default, have fixed this in ipyvolume, until fixed try installing wget.


A fix and optimization for the HTTP interface to riak_kv

The fix is just dropping the ability to have the user set "content-encoding" and subsequently stop storing it on the riak_object.  It seems to have possibly be previously conflated as another sort of "content-type" metadata, but the HTTP RFC disagrees with us.
The optimization is to avoid the current code path on PUT that does a full client replica GET request at the coordinator before performing the actual PUT operation.  Testing done by @deanproctor showed that on a 100% PUT load we could cut CPU, network, and disk utilization almost in half.

Can one of the admins verify this patch?


Newrelic Agent <IP_ADDRESS> Not honoring HTTP_PROXY env vars

I have two identical services deployed. One is using <IP_ADDRESS> the other is using <IP_ADDRESS>.
The one on version 6 is not registering with newrelic when in a network with no default route out only proxy. It does however register when given a default route without proxy...
The one on version 5 works as expected registering with newrelic using the proxy configured in HTTPS_PROXY env var.

Interesting, that's most likely related to this change which defers loading of HTTP logic until the agent activates.
Can you provide further detail about your deployment? Are you using a WSGI server, for example? If so, which server? Which framework? Are you using popen or multiprocessing?
Can you provide debug logs showing the log line where the agent is activating -- log line: Application was activated from:? (you can redact most information aside from this line)

Interesting, that's most likely related to this change which defers loading of HTTP logic until the agent activates.
Can you provide further detail about your deployment? Are you using a WSGI server, for example? If so, which server? Which framework? Are you using popen or multiprocessing?
Can you provide debug logs showing the log line where the agent is activating -- log line: Application was activated from:? (you can redact most information aside from this line)

Oh, also - how are you setting the HTTPS_PROXY env var? Are you doing this via python code modifying os.environ, or other?

Oh, also - how are you setting the HTTPS_PROXY env var? Are you doing this via python code modifying os.environ, or other?

@a-feld we are setting them in our entrypoint.sh (this is all running in a docker container)
Using FastAPI and gunicorn + uvicorn. No popen.
#!/usr/bin/env bash

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
export SSL_CERT_DIR=/etc/ssl/certs/

generatecerts.sh

echo "Active version: ${ACTIVE_VERSION}"

if [[ -n "$ACTIVE_VERSION" ]]; then
  proxy_host_port=http://squid-proxy:3128
  export http_proxy=$proxy_host_port
  export https_proxy=$proxy_host_port
  export no_proxy=<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>,localhost,<REDACTED>
  export HTTPS_PROXY=$https_proxy
  export HTTP_PROXY=$http_proxy
  export NO_PROXY=$no_proxy

  NR_KEY=$(get_newrelic_key.py)
  newrelic-admin generate-config $NR_KEY newrelic.ini
  sed -i "s/Python Application/e4p-notification-decorator-<REDACTED>" newrelic.ini
  NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program gunicorn --certfile=/usr/local/share/certs/selfSigned.crt --keyfile=/usr/local/share/certs/selfSigned.key --worker-tmp-dir /dev/shm -b <IP_ADDRESS>:443 --access-logfile - --error-logfile - --workers=2 --worker-class=uvicorn.workers.UvicornWorker --keep-alive 300 run:APP
else
  gunicorn --worker-tmp-dir /dev/shm --certfile=/usr/local/share/certs/selfSigned.crt --keyfile=/usr/local/share/certs/selfSigned.key --bind <IP_ADDRESS>:443 --access-logfile - --error-logfile - --workers=2 --worker-class=uvicorn.workers.UvicornWorker --keep-alive 300 run:APP
fi

@a-feld we are setting them in our entrypoint.sh (this is all running in a docker container)
Using FastAPI and gunicorn + uvicorn. No popen.
#!/usr/bin/env bash

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
export SSL_CERT_DIR=/etc/ssl/certs/

generatecerts.sh

echo "Active version: ${ACTIVE_VERSION}"

if [[ -n "$ACTIVE_VERSION" ]]; then
  proxy_host_port=http://squid-proxy:3128
  export http_proxy=$proxy_host_port
  export https_proxy=$proxy_host_port
  export no_proxy=<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>,localhost,<REDACTED>
  export HTTPS_PROXY=$https_proxy
  export HTTP_PROXY=$http_proxy
  export NO_PROXY=$no_proxy

  NR_KEY=$(get_newrelic_key.py)
  newrelic-admin generate-config $NR_KEY newrelic.ini
  sed -i "s/Python Application/e4p-notification-decorator-<REDACTED>" newrelic.ini
  NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program gunicorn --certfile=/usr/local/share/certs/selfSigned.crt --keyfile=/usr/local/share/certs/selfSigned.key --worker-tmp-dir /dev/shm -b <IP_ADDRESS>:443 --access-logfile - --error-logfile - --workers=2 --worker-class=uvicorn.workers.UvicornWorker --keep-alive 300 run:APP
else
  gunicorn --worker-tmp-dir /dev/shm --certfile=/usr/local/share/certs/selfSigned.crt --keyfile=/usr/local/share/certs/selfSigned.key --bind <IP_ADDRESS>:443 --access-logfile - --error-logfile - --workers=2 --worker-class=uvicorn.workers.UvicornWorker --keep-alive 300 run:APP
fi

My peer also noted that some endpoints have been broken by this latest newrelic change in fastapi. Rolling back the version of NewRelic resolved the issue

My peer also noted that some endpoints have been broken by this latest newrelic change in fastapi. Rolling back the version of NewRelic resolved the issue

Thanks for the info! I'll take a closer look once I get into the "office" (AKA open my work machine 😛 ) tomorrow.
Re: asgi changes - it's possible some instrumentation we added around asgiref may have also broken some fastapi stuff, although we didn't see a regression in the tests.  That may be a separate issue. If you have any stack traces / information around that, we'd also love to take a look!

Thanks for the info! I'll take a closer look once I get into the "office" (AKA open my work machine 😛 ) tomorrow.
Re: asgi changes - it's possible some instrumentation we added around asgiref may have also broken some fastapi stuff, although we didn't see a regression in the tests.  That may be a separate issue. If you have any stack traces / information around that, we'd also love to take a look!

@a-feld sounds great! Thanks for being so responsive. If it makes it easier we can hop on a call in the AM and go over it and I can share more of the code, it's a very simple smaller app 😄
Traceback (most recent call last):
  File "/app/e4p_notification_decorator/main.py", line 34, in catch_exceptions_middleware
    return await call_next(request)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "<string>", line 5, in wrapper
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 201, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 25, in decorate_notification_request
    notification_schema = decorate_event_data(notification_schema, push_events_request.dict())
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 37, in decorate_event_data
    decorate_event_data(schema_value, push_event_request)
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 46, in decorate_event_data
    update_with_hotel_information(notification_schema, schema_key, location_keys, picard_response)
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 56, in update_with_hotel_information
    picard_response = picard.get_hotel_information()
  File "/app/e4p_notification_decorator/client/picard.py", line 17, in get_hotel_information
    response = client.get(URL)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 907, in get
    return self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 733, in request
    return self.send(
  File "/usr/local/lib/python3.9/site-packages/newrelic/hooks/external_httpx.py", line 85, in sync_send_wrapper
    response = wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 783, in send
    hook(response)
  File "/usr/local/lib/python3.9/site-packages/newrelic/hooks/external_httpx.py", line 26, in newrelic_event_hook
    tracer.process_response(getattr(response, "status_code", None), headers)

@a-feld sounds great! Thanks for being so responsive. If it makes it easier we can hop on a call in the AM and go over it and I can share more of the code, it's a very simple smaller app 😄
Traceback (most recent call last):
  File "/app/e4p_notification_decorator/main.py", line 34, in catch_exceptions_middleware
    return await call_next(request)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "<string>", line 5, in wrapper
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 201, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 25, in decorate_notification_request
    notification_schema = decorate_event_data(notification_schema, push_events_request.dict())
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 37, in decorate_event_data
    decorate_event_data(schema_value, push_event_request)
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 46, in decorate_event_data
    update_with_hotel_information(notification_schema, schema_key, location_keys, picard_response)
  File "/app/e4p_notification_decorator/routers/decorator_router.py", line 56, in update_with_hotel_information
    picard_response = picard.get_hotel_information()
  File "/app/e4p_notification_decorator/client/picard.py", line 17, in get_hotel_information
    response = client.get(URL)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 907, in get
    return self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 733, in request
    return self.send(
  File "/usr/local/lib/python3.9/site-packages/newrelic/hooks/external_httpx.py", line 85, in sync_send_wrapper
    response = wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 783, in send
    hook(response)
  File "/usr/local/lib/python3.9/site-packages/newrelic/hooks/external_httpx.py", line 26, in newrelic_event_hook
    tracer.process_response(getattr(response, "status_code", None), headers)

Ah, that looks like some sort of bug in our new httpx instrumentation 😞
What is the exception at that traceback?

Ah, that looks like some sort of bug in our new httpx instrumentation 😞
What is the exception at that traceback?

Hi @evanlurvey - I just realized that you're setting the HTTPS_PROXY environment variables, but the agent uses NEW_RELIC_PROXY_HOST.
Give export NEW_RELIC_PROXY_HOST=$https_proxy a try 😄

Hi @evanlurvey - I just realized that you're setting the HTTPS_PROXY environment variables, but the agent uses NEW_RELIC_PROXY_HOST.
Give export NEW_RELIC_PROXY_HOST=$https_proxy a try 😄

@a-feld I will give that a try once the httpx instrumentation is fixed. Is that something new with version 6?  I know we haven't had to do this in prev versions.
As far as the exception message it got swallowed somewhere. Once I get some time I can try and step through and find it.

@a-feld I will give that a try once the httpx instrumentation is fixed. Is that something new with version 6?  I know we haven't had to do this in prev versions.
As far as the exception message it got swallowed somewhere. Once I get some time I can try and step through and find it.

@a-feld So to make it more interesting we added the NEW_RELIC_PROXY_HOST env var and that resovled the proxy issue AND the httpx issue. But this worries me that if there is no connection to newrelic that we might get that exception.

@a-feld So to make it more interesting we added the NEW_RELIC_PROXY_HOST env var and that resovled the proxy issue AND the httpx issue. But this worries me that if there is no connection to newrelic that we might get that exception.

@evanlurvey There's definitely a bug in the httpx instrumentation that we're fixing -- it's not you 😄
We're trying to get that fix out today.

@evanlurvey There's definitely a bug in the httpx instrumentation that we're fixing -- it's not you 😄
We're trying to get that fix out today.

@evanlurvey We just put out a new agent release. Could you give the new version a try and let us know how that works for you?

@evanlurvey We just put out a new agent release. Could you give the new version a try and let us know how that works for you?


Host poweroff: Removed extra slash on URL

While @JPalacios14 and I were trying some of the examples, specifically the "Host hard power off" curl example, we kept getting a "Not Found" error. This was due to the fact that there was two "//" after "${bmc}" i.e. https://${bmc}//xyz/openbmc_project/state/chassis0/attr/RequestedPowerTransition. This change removes the extra '/' and we are able to get an OK status response.

@loerac , @JPalacios14 - Thanks for identifying this problem.
Two comments:

We don't actually use Github for development but use a Gerrit server at https://gerrit.openbmc-project.xyz/q/status:open and we automatically mirror content from Gerrit to Github.  We cannot accept PRs from Github because the sync is one-way (Gerrit -> Github).

https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server-to-openbmc

I don't recognize either of you as having contributing to OpenBMC before (Great to have new people!).  The organization of the project requires Contributor License Agreements from contributors.  There are corporate and individual CLAs depending on if you are doing work on behalf of your employer or as individuals.

For a small documentation change like this, I would probably allow it in without a CLA, but for any code of significance we currently need one.  It is possible you are already covered by your employer's CLA and I simply don't recognize your names.
(Instructions on CLA are also covered in the CONTRIBUTING link above.)

@loerac - Any update on the above?

We don't acccept PRs.  No response from contributor in a year.


Setting is_superuser and is_staff on Registration

Hello, I am trying to set the is_superuser and is_staff fields when a user Registers using the /rest-auth/registration/ (POST)  or the /rest-auth/user/ (PUT) endpoints. Are these fields available in Django-Rest-Auth to be updated, or do I need to create custom code to do so? If so, does anyone have any guidance on how?

Any news on that @SRomero321?

Hi,
this repo is not maintained anymore, so the development moved to dj-rest-auth. (reference: #568)
It may be best, if you move this PR there. (and upgrade to using dj_rest_auth)
new repo link: https://github.com/jazzband/dj-rest-auth (I'm not the upkeeper of that repo, it just makes sense for me to help you write where it brings value.)
Best,
Barney


Feature request

Would it be possible to make a version of the mod that works on servers/realms (a purely client-side version of the mod)?

Would it be possible to make a version of the mod that works on servers/realms (a purely client-side version of the mod)?

It's already client side. You don't need to install it on the server to use it.


ability to configure menu bar font

Although it is currently possible to configure the font used in the editor and file trees, it would also be nice to be able to modify the font used in the menu bar.

Indeed, as @TheKiteEatingTree says. Please reopen if you have problems.


editor decoration

hello,how can l  widen the red zone marked on picture above and add text there

Please ask your question on StackOverflow. We have a great community over there. They have already answered thousands of questions and are happy to answer yours as well. See also our issue reporting guidelines.
Happy Coding!


[BugFix] Verify the truncate operation of iceberg

At present, Starrocks does not support Iceberg's truncate operation and requires additional verification, otherwise the error of 'ERROR 1064 (HY000): Unexpected exception: Unknown database xxx' will be reported
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[x] Yes, this PR will result in a change in behavior.
[ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

[ ] This is a backport pr

Bugfix cherry-pick branch check:

[ ] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

@linfey90 what's your use case of iceberg's truncate in starrocks


GitHub: Invite member

Please invite the following user
GitHub User:
birgitZF
Vouching person:  ChristopherWinterZF

Hey @ChristopherWinterZF! Invite sent. After accepted, the user can be added to a specific team.


Spring reactive documentation deliverables

This PR contains the collective changes introduced by PRs #1300, #1306 and #1307 with additional required fixes. @tradej
Please DO NOT MERGE yet, as the additional fixes and changes must undergo SME and Peer review first.
Delivery date is set for Tuesday 23/7.
@huntley Thank you for the feedback on earlier PRs, I have addressed some of your comments. Please feel free to re-review. If possible, please review the conceptual parts and let me know if you find anything that needs further improvement. I'll keep pushing fixes to the procedural parts as I address additional comments.
@gytis Please feel free to review the conceptual and procedural parts and let me know if you find anything that needs to be changed or clarified further. Thank you.

@tradej @huntley Thank you for the feedback on all related pull requests. I have incorporated your comments and suggestions in this PR. Please feel free to re-review and let me know if you spot additional issues with this content.

Hi, @snelluli1 would you please have time to look at the content in this PR and provide feedback? Thank you.

@inoxx03 any chance you have a rendered html version of this?

@gytis I expected there'd be a rendered preview provided by Netlify, but it seems to have failed for some reason. I'll have to fix that, but I can upload a static HTML preveiw in the meantime.

[test]

@gytis The rendered preview via netlify now works: please feel free to view the rendered output here: https://deploy-preview-1309--launcher-documentation.netlify.com/docs/spring-boot-runtime.html#developing-reactive-applications-using-spring-boot-vertx_spring-boot
Please let me know what you think. Thank you.

@snelluli1 Thank you for the review comments. I have incorporated your suggestions. Please feel free to let me know if there is anything else you see that should be fixed.  @tradej Please let me know what you think.

@tradej Responding to multiple comments above: All superfluous markup artifacts have been removed, Factual correction to concepts carried out as requested by ENGG. Peer Review and ENG ACK received. Automated checks passed. Manual build check successful on local machine. Merging.


Knife Bootstrap Improperly Detecting Existence of Chef Installation

When bootstrapping a fresh Ubuntu 14.04 node with chef, the exists function improperly returns true and the boostrap script believes there is a chef-client executable in place:
Connecting to <IP_ADDRESS>
<IP_ADDRESS> -----> Existing Chef installation detected
<IP_ADDRESS> Starting first Chef Client run...
<IP_ADDRESS> sh: 333: chef-client: not found

Initial attempts to replace the script with bashisms such as if [[ ! -f ... all of which result in the bootstrap script either constantly believing that a chef client is available, or always downloading a new one. I haven't spent any time digging the history here past current open issues in GH, but i assume someone has already beaten their head against the wall with this.

Ditto on debian 7.8

Got same error on Debian 8.1 with chef-12.4.0.rc.2.
Works ok when using chef-12.3.0.

@mcquin ran into this internally and her issue was with the if ! exists /usr/bin/chef-client line at https://github.com/chef/chef/blob/master/lib/chef/knife/bootstrap/templates/chef-full.erb#L169 . She gets exists: command not found.
This is shell magic that I don't wield - so I'm going to follow the advice at https://github.com/chef/omnibus-chef/blob/master/package-scripts/chef/postinst#L8 and ping reviewers of #2877
@btm @lamont-granquist @thommay @juliandunn @danielsdeleo

@mcquin what's your platform? what does sh --version return?

We're probably missing this:
https://github.com/chef/opscode-omnitruck/blob/master/views/install.sh.erb#L36-L44

@lamont-granquist no that's here: https://github.com/chef/chef/blob/master/lib/chef/knife/bootstrap/templates/chef-full.erb#L14-L21

@lamont-granquist the one in https://github.com/chef/chef/blob/master/lib/chef/knife/bootstrap/templates/chef-full.erb#L14-L22 is very slightly different, but i can't see why it'd cause this problem

Well you can use if test -f "/usr/bin/chef-client" and reverse the conditional.  You don't have to check through the path so that the command -v check is unnecessary and you can avoid the if ! completely.

@thommay &> /dev/null is a bash-ism, that needs to be fixed to >/dev/null 2>&1


Why there is no method like withContext() in 3.0.0?

I need to check permission in my custom view. In 2.x, it works fine. But when I update to 3.0.0, I found it can't work because there is only context in the view, no activity to initialize the Dexter.
And I see the initialize() method in Dexter.java, which only use the context, not the activity.
So why there is no method like withContext() in 3.0.0?

Hi @dzbx
The only reason (and I personally think it's a big one) is that we need an Activity instance to check for the permissions state, otherwise, we are forced to create our own and deal with issues such as #47, #81 and #84. We decided to go for the most limited approach so that behavior is more consistent and predictable.
You can always pass the activity or the Dexter builder instance (with an activity already injected) to your views so that you only have to call withPermissions(...).check()
Hope that makes sense.
Thanks for reporting!