Load balancer sample using Consul and Least Connection [TCP]

Closes #1811

#1811

LoadBalancer Sample using Consul and Least Connection type.
I am creating this PR as a handy guide for local setup and running so any contributor can see loadbalancer in action with minimal repeated setup.
I think this can also help me / anyone as always present setup so just quickly go and make required changes to debug any requested changes / bug.
Proposed Changes

Under Sample folder, created LoadBalancerDemo
This folder have 3 Services (Instance1, Instance2 - to demo 2 instances & API Gateway).
When we hit - /hostname of API Gateway, it loadbalancer will come into picture and with least connection method it will return to which instance it is serving request.
Ocelot.Sample.Loadbalancer.Common for setup of Consul with Instance 1 and instance 2 (Services)

@raman-m / @ggnaegi / @RaynaldM  I am raising clean PR again. There was lot of changes and some how lot errors I use to get while working on docker on that branch.
Please add required reviewers again.
I am adding more queries on based on this PR @ggnaegi / @raman-m. Please help me when you get chance.

@ggnaegi I am trying to understand how I can run multiple instances of an Image using docker compose.
I am facing an issue here, when my instance starts in container - I am able to reach to one port assigned in container but unable to reach to another port which is assigned for same image.
here is my experimental branch where I tried to do this thing. Can you quickly go through my code and check what I am missing in docker-compose or any other thing?

Tanmay, why did you close #1810 ?

Code and changes got into mess and my solution started giving lot of local errors. so I had to create branch again & solving them was taking more than creating & chery picking again.

You've got the perfect code review of #1810 by @ggnaegi
Seems you have to take into account old code review issues too...

You've got the perfect code review of #1810 by @ggnaegi Seems you have to take into account old code review issues too...

Yes I am trying to clean referring to same. but I am struggling with docker instances. I need help here.

@ggnaegi Please help me with https://github.com/ThreeMammals/Ocelot/pull/1810#discussion_r1414529390 comment also & how I can set my host name to consul while running on docker? It gets hosted with localhost.

@ks1990cn commented

Route JSON ServiceName

@raman-m example also suggest for host as localhost, but @ggnaegi suggesting for Consul https://github.com/ThreeMammals/Ocelot/pull/1810#discussion_r1414540043

Is he want to explain something different?

Wait for our Consul expert, @ggnaegi ! 😉 👑 aka King of the England!

#1810 (comment)
@ggnaegi I am trying to understand how I can run multiple instances of an Image using docker compose. I am facing an issue here, when my instance starts in container - I am able to reach to one port assigned in container but unable to reach to another port which is assigned for same image.
here is my experimental branch where I tried to do this thing. Can you quickly go through my code and check what I am missing in docker-compose or any other thing?

@ggnaegi Here I can understand is first docker image builds and creates port and I am trying to assign another port on same built, which is causing this issue. I am waiting for your reply what kind of approach you are suggesting to use only 1 project.
https://github.com/ThreeMammals/Ocelot/pull/1810#discussion_r1414528706
cc: @raman-m

Not even king at home, sometimes in my office, if the dog agrees.

@ks1990cn commented on Dec 9

Have you ran all your Docker containers for each image?
Seems you need to learn Docker more...
Which Docker software do you use except Visual Studio Docker support?

@ks1990cn commented on Dec 9

Have you ran all your Docker containers for each image? Seems you need to learn Docker more... Which Docker software do you use except Visual Studio Docker support?

Same container is running all images which are defined in compose file.
I use visual studio only, I saw only it as an example in Microsoft tutorials.
Not much expert here, I am learning.

Not much expert here, I am learning.

Aha! Good luck in learning!
Install Docker Desktop if you use Windows OS...
This is Docker engine management GUI tool. It will allow you to manage Docker artifacts easily...
Also it is possible to use Docker CLI... But it is a bit complicated thing for Docker newbies who learn this software... Just use Docker Desktop UI.
Visual Studio can run one Docker container for current default startup project, but for projects with multiple Docker images you have to run multiple Visual Studio sessions... It is not required if you use Docker Desktop for objects managements!
Docker Compose project just allows to create Docker Network for all Docker images in the solution. They can communicate to each other via Docker Network of composing project. But Compose project doesn't manage all Docker objects of the solution, it complies all Docker filed as images and nothing more. You have to manage Docker objects manually somehow...
I will return to this PR back after fixing the problem of placing all this code into the root repo folder...
So, work on this issue now ❗

Not much expert here, I am learning.

Aha! Good luck in learning!
Install Docker Desktop if you use Windows OS... It installs also Docker engine, CLI, virtual machine aka WLS, and UI app on top of them. This is Docker engine management GUI tool. It will allow you to manage Docker artifacts easily... Also it is possible to use Docker CLI... But it is a bit complicated thing for Docker newbies who learn this software... Just use Docker Desktop UI. Visual Studio can run one Docker container for current default startup project, but for projects with multiple Docker images you have to run multiple Visual Studio sessions... It is not required if you use Docker Desktop for objects managements! Docker Compose project just allows to create Docker Network for all Docker images in the solution. They can communicate to each other via Docker Network of composing project. But Compose project doesn't manage all Docker objects of the solution, it complies all Docker filed as images and nothing more. You have to manage Docker objects manually somehow...
I will return to this PR back after fixing the problem of placing all this code into the root repo folder... So, work on this issue now ❗

Ah! I have docker desktop, I thought you are asking about tool to write docker code.

@ks1990cn I will provide you a sample implementation during the week. Be patient, i'm a bit busy at the minute. Thanks!

Tanmay, you 've created the solution and projects in the root folder! That's wrong!!! We have special folder for all samples: samples
You must move solution/projects into a folder inside of the samples folder!

Made respective changes

@ks1990cn commented on Dec 12:

Made respective changes
https://github.com/ThreeMammals/Ocelot/pull/1842/commits/b6257233d80370982eeaf7be39d76529a53a7695

Is that your change? Are you kidding?
Please, work on major code review issue aka root folder problem
After that, as a team, we will perform classic code review: C#, design etc...

Your feature branch: https://github.com/ks1990cn/Ocelot/tree/lbsample
You've moved NOTHING!

I have last drop of my patience...

I've rebased the branch.
Probably you did the fix in local repo.

Delete local branch in local repo.
Checkout the feature branch once again
Apply the fix
Push

Your feature branch: https://github.com/ks1990cn/Ocelot/tree/lbsample You've moved NOTHING!  I have last drop of my patience...

Your feature branch: https://github.com/ks1990cn/Ocelot/tree/lbsample You've moved NOTHING!  I have last drop of my patience...

You don't need to loose patience, I am not sure whats wrong, let me figure out.

@raman-m Steps you mentioned didn't worked for me. Let me try again, maybe I need to pull through this fork and remove my local folder too.

Closing this PR, cause this had folder structure issue. Clean PR is raised again.

Tanmay, there is no need to close a PR each time when the code reviewer found an issue.
You just need update your feature branch! Nothing more! The changes will be reflected to a PR...
This is the 2nd closing... I have no words...

Also, creation of new PR based on old feature branch has no sense, because files in the feature branch remain the same!

I agree on you @raman-m , but I figured out sample folder structure is little different thing. On File explorer it different and on visual studio naming and structuring is different.
I already had project file into it which was created to wrong path. I had no chance for it. In my new code, I've made some more docker changes which @ggnaegi was suggesting (Just facing another issue with cerficates).

Correct


Question about variable scoping

In the chatbot example the messages variable is declared at the script level. Does that mean that all requests would share the same set of messages, or is FastHTML doing something to limit the messages scope to a user session somehow? I'm trying to understand how FastHTML works under the hood. Thank you. The framework looks interesting.

This example shares the messages across requests to keep it simple.


Adding Peer Comparison Tool.

The program allows users to input data for multiple students, calculating their average study hours, participation rates, and assignment completion. It identifies outliers in these metrics and visualizes the results with a bar chart comparing individual student performances to the averages.
key features:-
Data Input, Average Calculation, Outlier Detection, Data Visualization, User-Friendly Output, Interactive Experience
Overall, the tool is designed to help users evaluate and visualize student performance metrics in a comparative manner.
This tool can be using in future video detection. and then generating a bar graph.
Note:- Add all labels added to the issue #9. So that GSSoC team can easily recognize the commit!

why are you not merging the pull request??
is there any problem??

I need this code to be endpoint ready, for enabling api in future...


Ensure knative serving and go conventions followed in project's structural organization

See conversation here:
https://github.com/knative-sandbox/async-component/pull/41#pullrequestreview-531834547

Did a comparison and we seem to be following conventions:

[x] Small mains, unit tests for functionality (ingress/controller)
[x] Using latest knative release dependencies for E2E tests in this PR https://github.com/knative-sandbox/async-component/pull/105
[x] cmd/vendor/test/config/etc and general project layout follow standard Golang conventions
[x] Pre submit tests are working and test-infra has unit/integration tests marked as required
[x] No dependency on ytt for E2E tests

I couldn't find a document outlining other particular Knative conventions.  A new contributor guide is being discussed here here and here which may help add some detail to this in the future.
I think this looks good to close unless you have other input @maximilien

Thanks for verifying @AngeloDanducci
Closing


experiment result

Hi,
Thanks for your awesome work!  I have some question about the experiments as following:

How to get a similar result as you post

I run your code and set the same dataset and epochs for every algorithm, but the result is different from yours.

Do you think this is a good result ？

`'appliances': ['washing machine'],
'methods': {
 'WindowGRU': WindowGRU({'n_epochs':30}),
 'RNN': RNN({'n_epochs':50}),
 'DAE': DAE({'n_epochs':50}),
 'Seq2Point': Seq2Point({'n_epochs':50}),
 'Seq2Seq': Seq2Seq({'n_epochs':50}),
 'Convlstm': Convlstm({'n_epochs':30,}),

},
'train': {
'datasets': {
'1': {
'path': 'C:/Users/Jia/Desktop/NILM/ukdale.h5',
'buildings': {
1: {
'start_time': '2014-01-05',
'end_time': '2014-02-05'
},
}
    }
}

},
'test': {
'datasets': {
'Datport': {
'path': 'C:/Users/Jia/Desktop/NILM/ukdale.h5',
'buildings': {
1: {
'start_time': '2014-04-1',
'end_time': '2014-04-7'
},
}
}
},
'metrics': ['mae','rmse' ]
}
`

Thanks！

I would like to point out that I have the exact same problem as you, regarding your first question. In fact, I wasn't able to reproduce any result of the article.
I believe there is a bug somewhere in the API, I therefore have no clue where to look.

Hello @Geckao @Hessen525
I was busy during the past few days, so I couldn't get back to you. Recently, I made a change in the API, that was the reason for this issue. Give me sometime, I'll get the issues resolved soon!

Hello @Rithwikksvr
Thank you for your quick reply. Do you know if there is a version I can revert to in order to remove this issue ? If so, which one ?
Thank you !


Include jQuery, Popper.js, and Bootstrap JavaScript assets

pelican-alchemy ✨ was set out to work without JavaScript. But that decision has it's drawbacks. According to https://getbootstrap.com/docs/4.0/getting-started/introduction:

Many of our components require the use of JavaScript to function. Specifically, they require jQuery, Popper.js, and our own JavaScript plugins.

Curious which components explicitly require jQuery, Bootstrap's JS, and Popper.js?

Alerts for dismissing
Buttons for toggling states and checkbox/radio functionality
Carousel for all slide behaviors, controls, and indicators
Collapse for toggling visibility of content
Dropdowns for displaying and positioning (also requires Popper.js)
Modals for displaying, positioning, and scroll behavior
Navbar for extending our Collapse plugin to implement responsive behavior
Tooltips and popovers for displaying and positioning (also requires Popper.js)
Scrollspy for scroll behavior and navigation updates

Hi;
thanks for your prompt anwser...
BTW : I like the fact you don't use JS
so it is possible to make it through a if / optional ?
Regards!!
Jonathan

Opt-in support for JavaScript assets has been added. By default we still use no JS.


Run time error '91':  Object Variable or With block variable not set

I installed XLRisk per the instructions at "Download and Install."  I opened the spreadsheet I want to use XLRisk on and when I clicked on XLRisk / Setup I got the error message above.  I clicked on Debug and it pointed me to the line of code in the attachment.
I am on a Windows laptop (Dell G5 SE), Windows 10 Home, Microsoft Office Professional Plus 2016 with Excel and other office products.
Any advice?
Thanks,
Vin Underwood

No idea!  Does it happen consistently? Say if you close and start again Excel?  Does it happen with an empty workbook?

My next step is to either uninstall and re-install, or to rebuild my workbook from scratch, omitting a lot of fluff and hidden sheets.  I think the second option is more promising.

Can you post your workbook here (or a  link to it), so that I can have a look?

sure - attached.
oops - wouldn't attach.  Got an error message "We don't support that file type".  Maybe the problem is that it is a xlsm file??

Put it in a zip file. Would that work?  Otherwise place in a sharable cloud storage (onedrive, google drive etc.) and post a link.

I copied it to OneDrive.   It wouldn't let me copy the link and share.   I use 7-zip, which is not easily converted to zip format.  Let me download zip right now and I will create a zip file.

before i create a zip file, I copied it to google drive.  Here is the link https://drive.google.com/file/d/1deSmgu0Luyt43cxLxejWdWnhDgjUMgYd/view?usp=sharing

the file is Provenir Decision Engines - Queuing Perspective v3.0.xlsm.   I want to assign normal distributions or lognormal distributions to the blue cell on tab "Peak Hour", and the three blue cells on the tab "Final Calc".

I want to disable this link as soon as you have looked at the file.  I don't want to share it with the whole world.

I have downloaded the file, so you can delete the link.

When I open your file VBA macros are disabled due to security issues.
I could reproduce the error.
Then I saved your file as an xlsx file (without the macros) opened it and XLRisk worked as expected.

I did exactly as you suggested.  It solved the problem for me as well.  Thank you for debugging this for me!  Now I just have to figure out how your software works - I'm sure I won't have a problem - your doc looks pretty good and I worked with Crystal  Ball many years ago.

Good.

correction to doc at https://github.com/pyscripter/XLRisk/wiki/RiskFunctions.  Your lognormal distribution formula is RiskLogNorm, not RiskLogNormal.
Also, what is the value meand in the function =RiskLogNorm(meand, stdev)?  Is it the mean or something else?  Maybe these parameters are the mean and standard deviation of my variable's natural logarithm?
I coded =RiskLogNorm(1.5, 0.025) expecting to get a confidence interval approximately centered on 1.5, but I'm getting values like 4.34, 4.58. etc.

The correction to the docs was done.
This is the source fode for RiskLogNorm
Public Function RiskLogNorm(Mean As Double, StDev As Double, Optional Corrmat As Long = 0) As Double
'   Random sample from a Log Normal distribution

    Dim RndValue As Double
    Application.Volatile (ProduceRandomSample)

    If ProduceRandomSample Then
        If gSimulation Is Nothing Then
            RndValue = Rnd()
        Else
            RndValue = gSimulation.GetRndSample(Application.Caller)
        End If
        RiskLogNorm = WorksheetFunction.LogNorm_Inv(RndValue, Mean, StDev)
    Else
        RiskLogNorm = Exp(Mean + 0.5 * StDev ^ 2)
    End If
End Function

As you can see it uses the Excel function LogNorm_Inv.
From the documentation of LogNormInv you can see that Mean is the mean of Ln(x) and stdev the standard deviation of ln(x).
The mean of x is Exp(Mean + 0.5 * StDev ^ 2)

yep - that was my guess.  thanks


Updating Readme.md file

updating Readme.md file
Fixes #50

please rebase your PR against main
the UI changed, so please use newer screenshots

Hi @fbricon,  as  i have raised PR from main branch , i am planning to reset all the changes done in the main branch of my forked repository ,so there might be a chance to affect this PR as well , i will raise new PR from my feature branch after all the changes


remove empty page

Fixes #5272

I'd like @brentschmaltz to confirm this is the right target.

He confirmed offline.


Update CMake version

LLVM requires at least CMake 3.20 on next. Bumping the CMake version to unblock work.
CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
  CMake 3.20.0 or higher is required.  You are running version 3.19.6

@swift-ci please test

Made it past the CMake version failure and into issues in LLVM that are fixed in https://reviews.llvm.org/D154664.
(though half of the below is also due to a leaky architecture)
/Users/ec2-user/jenkins/workspace/swift-PR-macos@2/branch-next/llvm-project/llvm/include/llvm/Support/Format.h:220:3: error: no template named 'optional' in namespace 'std'; did you mean 'Optional'?
  220 |   std::optional<uint64_t> FirstByteOffset;
      |   ^~~~~~~~~~~~~
      |   Optional
/Users/ec2-user/jenkins/workspace/swift-PR-macos@2/branch-next/swift/stdlib/include/llvm/ADT/Optional.h:251:29: note: 'Optional' declared here
  251 | template <typename T> class Optional {
      |                             ^

We can close this PR, I have merged https://github.com/apple/swift/pull/67018


Angular issues with 1.10 rc versions

Problem:
Hi @owen-m1 and Co,
The angular users report, that the library does not work after upgrade to rc versions.
Are there known breaking changes? What could be the issue that the code that worked previously (1.7 -> 1.9) stopped working now
See https://github.com/SortableJS/ngx-sortablejs/issues/163
JSBin/JSFiddle demonstrating the problem:
Please see the attached issue

Before you create an issue, check it:

Try master-branch, perhaps the problem has been solved;
Use the search, maybe we already have an answer;
If not found, create an example on jsbin.com (draft) and describe the problem.

Bindings:

Angular

2.0+: https://github.com/SortableJS/angular-sortablejs/issues
legacy: https://github.com/SortableJS/angular-legacy-sortablejs/issues

React

ES2015+: https://github.com/SortableJS/react-sortablejs/issues
mixin: https://github.com/SortableJS/react-mixin-sortablejs/issues

Polymer: https://github.com/SortableJS/polymer-sortablejs/issues
Knockout: https://github.com/SortableJS/knockout-sortablejs/issues
Meteor: https://github.com/SortableJS/meteor-sortablejs/issues

I took a look at it, and I am not sure. There aren't any error messages, so there must be an issue with ngx-sortablejs interacting with Sortable somehow. There aren't any breaking changes that I know of between 1.9.0 and 1.10.0-rc2.
I think you are in a better position to figure this out than me. The fact that it works on StackBlitz and not locally makes me think it has something to do with the way angular is building it. But again, I have no idea :/
The only thing I can think of is that this is now the entry point of the ESM that is imported when import Sortable from 'sortablejs' is called: https://github.com/SortableJS/Sortable/blob/master/entry/entry-defaults.js
But it still has Sortable as the default export, so I don't know why that would affect it.

Thanks, the latter could be very useful! I’ll check the ways how it can be imported differently

Here is the output of require('sortablejs')
1.9.0 (and former):

1.10.rc-2

That's a breaking change still... I have nothing against, just want to know whether it will be kept like that.
Doesn't this version deserve to be 2.0.0?

@smnbbrv I never thought about anyone using the require syntax.
It is backwards compatible for import Sortable from 'sortablejs';, because the default export is the same, but it seems the require function does not respect default exports.
Is this what is causing ngx-sortablejs to break?

Yeppers, here is a permalink
https://github.com/SortableJS/ngx-sortablejs/blob/337eb0d3ab81cd144d8e35ca815c1e2b0886aea0/projects/ngx-sortablejs/src/lib/sortablejs.directive.ts#L8
As I can remember formerly I was forced to use require syntax because of the broken typescript typings. Now I can rework it in a way it uses default export, so it is fine for me.
However this is going to be closely connected to the https://github.com/SortableJS/Sortable/issues/1547 . The typings are messy now.
I would close this issue then and raise a newer (probably a major version) on ngx-sortablejs in the near future.

@owen-m1 @smnbbrv Is this actuall allowed in the sortablejs package?

import { Sortable } from 'sortablejs';

I'm in the middle of a PR and can add this in if it works.

@waynevanson Yes you are able to import Sortable that way

@owen-m1 Can you link to source code for this? I wanna make sure I get the correct typings #1547

@waynevanson https://github.com/SortableJS/Sortable/blob/5a8603d6ff892ea28e6e2aa42e68170c3250a718/entry/entry-core.js#L10


Support pagination in projects and sessions listing API

Support pagination on WorkflowsPage, ProjectsPage and ProjectPage(#1106 is related)
I made the docker image, so you can check the ui with the following command.
docker pull tosametal/digdag-pagination:0.0.1
docker run -d -p 65432:65432 tosametal/digdag-pagination:0.0.1

→ access to http://localhost:65432

I also created an image to show the loading when fetching sessions.
tosametal/digdag-pagination:loading
Please check this too and compare which looks better.

Hello, @tosametal
Thank you for creating this PR.
Does the difference between tosametal/digdag-pagination:loading3 and
tosametal/digdag-pagination:0.0.1 is that whether show loading image or not?
If so, I like Loading version.
I confirmed the following steps.
no loading version
docker pull tosametal/digdag-pagination:0.0.1
docker run -d -p 65432:65432 tosametal/digdag-pagination:0.0.1

digdag init -t sh fuga
cd fuga
digdag push fuga

# macOS
for i in `seq 0 120` ; do
  d=`date -v+${i}d -j -f "%Y-%m-%d" "2019-01-01" +"%Y-%m-%d"`
  digdag start fuga fuga --session $d
done

Loading version
docker pull tosametal/digdag-pagination:loading3
docker run -d -p 65432:65432 tosametal/digdag-pagination:loading3

digdag init -t sh fuga
cd fuga
digdag push fuga

# macOS
for i in `seq 0 120` ; do
  d=`date -v+${i}d -j -f "%Y-%m-%d" "2019-01-01" +"%Y-%m-%d"`
  digdag start fuga fuga --session $d
done

@hiroyuki-sato Thank you for confirmation.

Does the difference between tosametal/digdag-pagination:loading3 and
tosametal/digdag-pagination:0.0.1 is that whether show loading image or not?

Yes, that's the only difference.

If so, I like Loading version.

OK. I will add a commit that shows the loading image to this PR.
When the number of pages is more than 10, the button display changes. Please try it.

Hello, @tosametal
This is just my(not committer) opinion.
What do you think? @yoyama, @muga, and @szyn?

Hello, @tosametal

When the number of pages is more than 10, the button display changes. Please try it.

It seems to work well.
docker pull tosametal/digdag-pagination:loading3
docker run -d -p 65432:65432 tosametal/digdag-pagination:loading3

digdag init -t sh fuga
cd fuga
digdag push fuga

# macOS
for i in `seq 0 1100` ; do
  d=`date -v+${i}d -j -f "%Y-%m-%d" "2016-01-01" +"%Y-%m-%d"`
  digdag start fuga fuga --session $d
done

In addition I propose split this ticket to UI part and API part for our review.

@hiroyuki-sato Thank you for confirmation. I confirmed it as follows.
# sample1.dig
timezone: 'Asia/Tokyo'
schedule:
  minutes_interval>: 1
+task:
  sh>: echo hoge

for i in `seq 2 100`
do
  cp sample1.dig sample$i.dig
done

digdag push project -e http://localhost:65432

and wait 10 minutes...

@yoyama Thank you for your review. I will check the comments and fix them.

In addition I propose split this ticket to UI part and API part for our review.

Does this mean I should close  this PR and create two new PRs?

@tosametal You don't need to close this PR.
I suppose that remove UI codes from this PR and then create follow up PR which include UI code.
How about this way?

@yoyama I agree with your suggestion.I will do that.

@tosametal If you finished update, please ping me.

@yoyama Since the update is finished, please review again.

@tosametal  I changed the title, if you feel it does not fit, please let me know.

@yoyama I fixed the point that you commented,  and added tests. Please confirm.

@tosametal Could you check my comments?

@tosametal I commented to your question in the code, please check it. Sorry for my late response.

@yoyama I checked your comments, and fix the code.Please check the commits below.
ff63080
2fecfdf

@tosametal Thanks for you fix. I will check it later.

I updated the title. I misunderstood this PR.
This PR try to add page_number parameter to '/api/sessions/{id}'  and '/api/projects/{id}/sessions'.
Both are list of sessions.

@yoyama I added validatePageNumber to QueryParamValidator and fixed variable and method name.Please check it.

@tosametal Sorry for my late response.
I just started to review your latest PR.
Please wait for a moment.
Your PR is large, so I need to check carefully.

@yoyama Sorry for my too late response. I fixed the below.Please check them.

switch the order of pageNumber and lastId
add site_id condition and add test

@tosametal I would like to close this ticket once because it is too old and code is conflicted and we are working on v0.10.0 release.
In addition if we add these functionality, it is better to support not only sessions but also attempts and other for the integrity of API design.
If you want to continue, feel free to reopen after v0.10.0 released.


caddy plugins help

1. What version of Caddy are you using?
(Caddy 0.10.10 (+5cca9cc) (unofficial)
2. What are you trying to do?
Import a plugin: caddy.upload per these instructions:
https://github.com/mholt/caddy/wiki/Plugging-in-Plugins-Yourself
38     // This is where other plugins get plugged in (imported)
39     _ "blitznote.com/src/caddy.upload"

4. How did you run Caddy (give the full command and describe the execution environment)?
% sudo caddy -conf /usr/local/www/Caddyfile
Activating privacy features... done.
2017/11/11 19:16:57 no action found for directive 'upload' with server type 'http' (missing a plugin?)

I have performed the following:
go get blitznote.com/src/caddy.upload
and I see it here:
~/bin/gopath/src/blitznote.com/src/caddy.upload
What am I doing wrong?
Also opened an issue here:
https://github.com/wmark/caddy.upload/issues/28

Resolved by using the correct binary!
https://caddy.community/t/middleware-builder/1951/5

Glad you figured it out!


Create ISSUE_TEMPLATE.md

this is a proposal for a githib-issues-template.

Merged https://github.com/nicolaiarocci/cerberus/commit/c27da3197b2ad9cfa6081844f175ed97233b880c


Sometimes click is not triggered

Hello I found what seems another issue, the click on on the targetet element seems nerver be triggered. It works fine if I connect to remote chrome with dev tools, but not if clicking by code.
Steps to reproduce
Tell us about your environment:

Puppeteer version: next
Platform / OS version: Debian (Dockerized in node:10-stretch-slim)
URLs (if applicable): https://www.kusmitea.com/vente-the/the-mate.html
Node.js version: v10.15.1

What steps will reproduce the problem?
Open the website
Emulate a mobile environnment
Click on button.addToCart.btn[data-sku="DETO125"]
Wait for the ajax request ajax/cart/addToCart to be triggered after clicking the button
Please include code that reproduces the issue.
const browser = await puppeteer.launch();

const page = await browser.newPage();
                await page.emulate({viewport: { width: 360, height:  640, deviceScaleFactor: 3, isMobile: true, hasTouch: true, isLandscape: false }, userAgent: 'Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3679.0 Mobile Safari/537.36'});

                await page.goto('https://www.kusmitea.com/vente-the/the-vert.html', {waitUntil: ['networkidle0', 'load', 'domcontentloaded'], timeout: 30000});
                console.log(await page.url());

                const element = await page.$('button.addToCart.btn[data-sku="DETO125"]');
                await element.screenshot({path: 'button.png'});
                await page.screenshot({path: 'screenshot.png'});

                await Promise.all([
                    element.click(),
                    page.waitForRequest(request => {
                        const test = /ajax\/cart\/addToCart/;
                        return test.test(request.url());
                    })
                ]).catch(error => {
                    console.log(error);
                });

                console.log(await page.url());

What is the expected result?
The ajax request ajax/cart/addToCart should be run
What happens instead?
Nothing happens and the wait for request ends with a timeout error

TimeoutError: Timeout exceeded while waiting for event

Here are some info after some try it seems react when using page.tap instead of element.click
It's definitively not a bug from puppeteer, but maybe you can consider to use throw tap events when use click method in a touch context.

It's definitively not a bug from puppeteer, but maybe you can consider to use throw tap events when use click method in a touch context.

@Kalliser in fact, click events are very much legit in a touch context. One can connect a mouse to an android, and certain accessibility tools might emit clicks in touch contexts - so in general websites should react on clicks even when opened on touch-enabled screens.
Puppeteer works as expected; the website doesn't :-)  Thanks for reaching out and filing bugs!


Week 5: Quantum Circuit Simulation

Week 5: Submit a Pull Request to the Selection Round Project

Topics: Add Section 3.iii to Tutorial 4 describing your investigations and further exploration of quantum computing circuit design / simulation.
Tasks:

Create a feature branch that adds a section with pictures to the tutorial content
Submit a PR

Thanks @Jay5454 for catching the typo and assisting us in maintaining the content!

Good afternoon Mr Nyameko and team
@mpofukelvintafadzwa
@nyameko
@mmhashatsi
I have completed week 4 for Quantum computing, please look at the  page labelled Week4: Documentation of Qiskit , please do have a look at our documentation  pertaining to the design of a Qiskit quantum circuit and provide valuable feedback on it as this fosters our learning experience as  a team.
I am currently making progress on week 5 in which I will be collaborating with my colleague Basiel , expect completion before Friday 22 November 2024
Thank you
Have a great day ahead 😊

Good afternoon mr Nyameko  @nyameko
Re: Week 5 :Quantum computing  simulation
I am  a bit confused with regards to the instructions of week 5 quantum computing simulation instructions
According to my understanding , I need to refer to section 3 of tutorial 4 which provides information on qiskit . I need to then have it in a branch which points to tutorial 4 then I should create a pull request of this from tutorial 4 in a new branch
Please correct me and guide me I am new to pull requests on github , it would be valuable for me and my team
Thank you sir
Have a wonderful day ahead

To assist you in editing markdown and preparing your Pull / Merge Requests, you are strongly advised to review:

The first page from Selection Round Content Contribution to the Project, just that you will be creating a fork.
The section just below that, with the links to GitHub Markdown Syntax and Style.

Hi @MavishDevOps ,

You will fork the SCC repo,
You will clone your forked repo, onto your local machine / workstation,
You will create a feature-branch - called something useful like qiskit-experiments,
You will edit the tutorial4/README.md file, "very briefly and succinctly" adding your experiments to section 3.iii, as if you were explaining to one of your university coursemates, who is not familiar with HPC or QC.
You must try to match the style and syntax of the existing SCC documentation.

Good Luck!

Thank you @nyameko

Good Afternoon Mr Nyameko @nyameko
I have done steps 1 and 2 of the steps you had suggested to complete the task for week 5 however after cloning the repository onto my machine I get this message please guide me
Thank you sir
Have a wonderful day ahead
😊

Dear Mr Nyameko @nyameko
I have resolved the error I was using Windows 11 ,I switched to Ubuntu Linux and have created a feature branch called qiskit experiments .
I am now going to edit tutorial4/README in the forked repo add section 3 iii then I will create a pull request
Thank you sir

Good afternoon @nyameko we are good with ssh key part. However we are having issues installing Open Blas
after these commands.

SSH into the VM we are good

Error the value of NB is less than 1
Carefully study the contents of your HPL.dat input file. Alternatively use HPL Calculator

Good morning @nyameko are the slides and the git work done on 4-December. Or do we need to be done with the slides before the 1 December.

Good morning @nyameko are the slides and the git work done on 4-December. Or do we need to be done with the slides before the 1 December.

Hi @thandiwe-01, Slides are to guide and assist you during presentations to judges and delegates at the conference. So you will be updating it right through as you get results.


Errors compiling

Running into an issue around invalid conversions. Not sure if I have a newer and more strict version of Arduino? Or is there a permissive flag I can set to make this work?
Arduino: 1.8.19 (Mac OS X), Board: "LOLIN(WEMOS) D1 R2 & mini, 80 MHz, Flash, Disabled (new aborts on oom), Disabled, All SSL ciphers (most compatible), 32KB cache + 32KB IRAM (balanced), Use pgm_read macros for IRAM/PROGMEM, 4MB (FS:2MB OTA:~1019KB), v2 Lower Memory, Disabled, None, Only Sketch, 921600"

/Users/dalan/repos/AirGradientHomeAssistant/AirGradientHomeAssistant.ino: In function 'void loop()':
AirGradientHomeAssistant:173:22: error: invalid conversion from 'int' to 'const char*' [-fpermissive]
  173 |  MQTT_PM2_5.setValue(PM2);
      |                      ^~~
      |                      |
      |                      int
In file included from /Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/ArduinoHA.h:19,
                 from /Users/dalan/repos/AirGradientHomeAssistant/AirGradientHomeAssistant.ino:43:
/Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/device-types/HASensor.h:33:31: note:   initializing argument 1 of 'bool HASensor::setValue(const char*)'
   33 |     bool setValue(const char* value);
      |                   ~~~~~~~~~~~~^~~~~
AirGradientHomeAssistant:176:20: error: invalid conversion from 'int' to 'const char*' [-fpermissive]
  176 |  MQTT_CO2.setValue(CO2);
      |                    ^~~
      |                    |
      |                    int
In file included from /Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/ArduinoHA.h:19,
                 from /Users/dalan/repos/AirGradientHomeAssistant/AirGradientHomeAssistant.ino:43:
/Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/device-types/HASensor.h:33:31: note:   initializing argument 1 of 'bool HASensor::setValue(const char*)'
   33 |     bool setValue(const char* value);
      |                   ~~~~~~~~~~~~^~~~~
AirGradientHomeAssistant:179:28: error: cannot convert 'float' to 'const char*'
  179 |  MQTT_temperature.setValue(caliTemp);
      |                            ^~~~~~~~
      |                            |
      |                            float
In file included from /Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/ArduinoHA.h:19,
                 from /Users/dalan/repos/AirGradientHomeAssistant/AirGradientHomeAssistant.ino:43:
/Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/device-types/HASensor.h:33:31: note:   initializing argument 1 of 'bool HASensor::setValue(const char*)'
   33 |     bool setValue(const char* value);
      |                   ~~~~~~~~~~~~^~~~~
AirGradientHomeAssistant:180:32: error: invalid conversion from 'int' to 'const char*' [-fpermissive]
  180 |  MQTT_humidity.setValue(result.rh);
      |                         ~~~~~~~^~
      |                                |
      |                                int
In file included from /Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/ArduinoHA.h:19,
                 from /Users/dalan/repos/AirGradientHomeAssistant/AirGradientHomeAssistant.ino:43:
/Users/dalan/Documents/Arduino/libraries/home-assistant-integration/src/device-types/HASensor.h:33:31: note:   initializing argument 1 of 'bool HASensor::setValue(const char*)'
   33 |     bool setValue(const char* value);
      |                   ~~~~~~~~~~~~^~~~~
exit status 1
invalid conversion from 'int' to 'const char*' [-fpermissive]

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

Installed libraries
Name                                               Installed    Available Location              Description
AirGradient Air Quality Sensor                     2.0.2        -         LIBRARY_LOCATION_USER -
ESP8266 and ESP32 OLED driver for SSD1306 displays 4.3.0        -         LIBRARY_LOCATION_USER -
home-assistant-integration                         2.0.0        -         LIBRARY_LOCATION_USER -
PubSubClient                                       2.8          -         LIBRARY_LOCATION_USER -
SGP30                                              0.1.5        -         LIBRARY_LOCATION_USER -
U8g2                                               2.33.15      -         LIBRARY_LOCATION_USER -
WiFiManager                                        2.0.14-beta  -         LIBRARY_LOCATION_USER -

I'm so sorry I did not get notified by your issue...
Did you manage to get it work? Or should I have a look into it?


Correct handling of term updates during the election for Candidate/Follower

According to the raft paper
"Current terms are exchanged whenever servers communicate; if one server’s current term is smaller than the other’s, then it updates its current term to the larger value. If a candidate or leader discovers that its term is out of date, it immediately reverts to follower state. If a server receives a request with a stale term number, it rejects the request."
This pull request implements this rules for election phase of Follower/Candidate states. It also fixes the #46 and #47 issues.

After running whole tests few times I've discovered that the multi jvm tests are failing from time to time and for now I have no idea why. I need to investigate this test cases deeper and update my pull request accordingly.

Random guess: I bet that the multi-jvm tests are inherently somewhat non-deterministic.

@colin-scott Hi, Colin. I guess you're right but it's not good anyway. The problem is that previously term updates that are handled by this PR weren't reflected in the code and this new changes can cascade and affect other parts of code that depends on it because everything is linked to some extend.

Just realized: I think you should also have the leader step down if term > currentTerm.
It currently stay()s:
https://github.com/ktoso/akka-raft/blob/master/src/main/scala/pl/project13/scala/akka/raft/Leader.scala#L28

Can be an issue though. Leader should also have the same term update rules as candidate. Will check that later.

After investigating test failures for some time I've found that election tests are failing from time to time on one of three JVM nodes with the following output:
[JVM-1] [WARN] [08/07/2015 15:50:22.389] [RaftClusterSpec-akka.remote.default-remote-dispatcher-5] [akka.tcp://RaftClusterSpec@<IP_ADDRESS>:33394/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FRaftClusterSpec%40<IP_ADDRESS>%3A58897-4] Association with remote system [akka.tcp://RaftClusterSpec@<IP_ADDRESS>:58897] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
[JVM-1] [INFO] [08/07/2015 15:50:22.389] [RaftClusterSpec-akka.actor.default-dispatcher-19] [akka://RaftClusterSpec/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FRaftClusterSpec%40<IP_ADDRESS>%3A51937-4] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://RaftClusterSpec/deadLetters] to Actor[akka://RaftClusterSpec/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FRaftClusterSpec%40<IP_ADDRESS>%3A51937-4#-372792519] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Same error messages can be observed in the failed build log of travis. Don't actually know what can cause this problem but probably it's not related to the bugs in code itself.

Gating happens when a connection to another node fails (i.e. in multi node tests when a node is killed) as a node notices it cannot talk to the other node and "gates it" for a while, trying to re-establish the connection later on.

@ktoso Is there any way to prevent it or its an expected behavior? I can retrigger the build and it will eventually succeed but it's not the best solution I guess.

I'll need to look at the tests where exactly it happens, some tests could be purposefully killing nodes – then it's totally expected. Sadly now prepping and delivering a webinar so will only get to it rather later.
Thanks for all the work btw! I hope to get to reviewing soon.

Changes make sense, thanks a lot @dmitraver!
I looked at the failures and would say they're timing problems due to too small await timeouts for the travis box (it's pretty slow).
I'll merge the changes and work a bit in the infra / timeouts to make tests not as fleaky.

For the next contrib @dmitraver please use 2-spaces though... :wink:

@ktoso Thx for the hint ;) Will do.


Unparsed hex

Hi there,
Another one not parsed by threefive, extracted from a MediaPackage stream: 0xFC302C00000003289800FFF005068000F528F000160214435545490000000100D100002932E00C002200006349D893.
It has a 0x22 BreakStart descriptor with empty UPID (tsduck did not let me insert one without a UPID)

Here's the thing, USE threefive to encode! Why are you using tsduck?

Segmentation descriptors have UPIDs, that is kind of the point.Declare a UPID, but make it  zero length,, I believe that will work. Give me a little while, and I'll do one for you.

Here you go
jason ='''{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 59,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment_ticks":<PHONE_NUMBER>,
        "pts_adjustment": 11162.385333,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 5,
        "splice_command_type": 6,
        "descriptor_loop_length": 37,
        "crc": "0x592280ed"
    },
    "command": {
        "command_length": 5,
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time": 58.52,
        "pts_time_ticks": 5266800
    },
    "descriptors": [
        {
            "tag": 2,
            "descriptor_length": 35,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x02",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": true,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": true,
            "no_regional_blackout_flag": false,
            "archive_allowed_flag": false,
            "device_restrictions": "Restrict Group 1",
            "segmentation_duration": 120.04,
            "segmentation_duration_ticks": 10803600,
            "segmentation_message": "Break Start",
            "segmentation_upid_type": 18,
            "segmentation_upid_type_name": "Unknown",
            "segmentation_upid_length": 0,
            "segmentation_upid": "",
            "segmentation_type_id": 34,
            "segment_num": 0,
            "segments_expected": 0,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        }
    ]
}'''
from threefive import Cue
cue=Cue()
cue.load(jason)
cue.encode()
cue.show()
print(cue.encode_as_hex())

You can omit the info secrtion, if you have a pts adjustment, it's just as easy to  add it to the time signal
jason ='''{
    "command": {
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time_ticks":<PHONE_NUMBER>
    },
    "descriptors": [
        {
            "tag": 2,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x02",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": true,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": true,
            "no_regional_blackout_flag": false,
            "archive_allowed_flag": false,
            "device_restrictions": "Restrict Group 1",
            "segmentation_duration": 120.04,
            "segmentation_duration_ticks": 10803600,
            "segmentation_message": "Break Start",
            "segmentation_upid_type": 18,
            "segmentation_upid_type_name": "Unknown",
            "segmentation_upid_length": 0,
            "segmentation_upid": "",
            "segmentation_type_id": 34,
            "segment_num": 0,
            "segments_expected": 0,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        }
    ]
}'''
from threefive import Cue
cue=Cue()
cue.load(jason)
cue.encode()
cue.show()
print(cue.encode_as_hex())

output:

a@slow:~/cuei$ python3 ../load.py
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 44,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment_ticks": 0,
        "cw_index": "0x0",
        "tier": "0xfff",
        "splice_command_length": 5,
        "splice_command_type": 6,
        "descriptor_loop_length": 22,
        "crc": "0x3d698a6b"
    },
    "command": {
        "command_length": 5,
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time": 11220.905333,
        "pts_time_ticks":<PHONE_NUMBER>
    },
    "descriptors": [
        {
            "tag": 2,
            "descriptor_length": 20,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x02",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": true,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": true,
            "no_regional_blackout_flag": false,
            "archive_allowed_flag": false,
            "device_restrictions": "Restrict Group 1",
            "segmentation_duration": 120.04,
            "segmentation_duration_ticks": 10803600,
            "segmentation_message": "Break Start",
            "segmentation_upid_type": 18,
            "segmentation_upid_type_name": "Unknown",
            "segmentation_upid_length": 0,
            "segmentation_upid": "",
            "segmentation_type_id": 34,
            "segment_num": 0,
            "segments_expected": 0,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        }
    ]
}
0xfc302c00000000000000fff00506fe3c3191880016021443554549000000027fd10000a4d99012002200003d698a6b

threefive will create an info section for you and it also figures out the Command length and Descriptor length for you,, if you have values there ,it will correct them, same with the crc32.

Media Package is Amazon , right?  Those cats are super cool, they use threefive. Me and tunein.com, and NBC and Amazon all had a meeting back in December to make sure we were all in sync doing SCTE-35 stuff, this is actually one of topics we discussed, how to handle this very situation.

The reason I'm using tsduck is that I'm preparing a TS file with embedded SCTE message, which is used as a source in MediaLive -> MediaPackage -> our SSAI solution. To check how our solution reacts to various scenarios.
I use tsduck out of habit for the TS preparation.
I then use threefive alongside to check what the output of MediaPackage is and what was the input to our solution

Don't get me wrong, I like tsduck, but there is a lot of room for interpretation in the SCTE-35 spec, and the SCTE-35 xml fields aren't the same as the SCTE-35 data fields, which baffles me. I don't understand that logic at all.
I don't usually have compatibility issues with tsduck,
I think the xml doesn't have upid length.
the proper way to not have a upid is to set the upid type to 0 and upid length to 0 , which works with threefive.
I forgot until I looked it up.
if you would have specified upid type 0 and upid length 0 , it would have parsed correctly.
jason ='''{
    "command": {
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time_ticks":<PHONE_NUMBER>
    },
    "descriptors": [
        {
            "tag": 2,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x02",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": true,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": true,
            "no_regional_blackout_flag": false,
            "archive_allowed_flag": false,
            "device_restrictions": "Restrict Group 1",
            "segmentation_duration": 120.04,
            "segmentation_duration_ticks": 10803600,
            "segmentation_message": "Break Start",
            "segmentation_upid_type": 0,
            "segmentation_upid_length": 0,
            "segmentation_type_id": 34,
            "segment_num": 0,
            "segments_expected": 0,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        }
    ]
}'''
from threefive import Cue
cue=Cue()
cue.load(jason)
cue.encode()
cue.show()
print(cue.encode_as_hex())

output:

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 44,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment_ticks": 0,
        "cw_index": "0x0",
        "tier": "0xfff",
        "splice_command_length": 5,
        "splice_command_type": 6,
        "descriptor_loop_length": 22,
        "crc": "0x2ca14b7d"
    },
    "command": {
        "command_length": 5,
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time": 11220.905333,
        "pts_time_ticks":<PHONE_NUMBER>
    },
    "descriptors": [
        {
            "tag": 2,
            "descriptor_length": 20,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x02",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": true,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": true,
            "no_regional_blackout_flag": false,
            "archive_allowed_flag": false,
            "device_restrictions": "Restrict Group 1",
            "segmentation_duration": 120.04,
            "segmentation_duration_ticks": 10803600,
            "segmentation_message": "Break Start",
            "segmentation_upid_type": 0,
            "segmentation_upid_length": 0,
            "segmentation_type_id": 34,
            "segment_num": 0,
            "segments_expected": 0,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        }
    ]
}

0xfc302c00000000000000fff00506fe3c3191880016021443554549000000027fd10000a4d99000002200002ca14b7d


EES-4642 Default and max limit for LT lists

Do not allow unlimited lists by default.

Would be nice to have this specified on the documentation

Can you add this new env var to helm chart pls?

@diegomarangoni this change has nothing to do with helm chart. Currently we do not have docs with all the available env vars. Will create a task to create it.

+1


[Part 2: MyHypervisorDriver] Multiple errors when compiling with with Visual Studio 2019

Sorry for commenting on an already closed issue, but I had this same issue when I I realized I was compiling in x32 bit mode (MASM seems to work differently on x32 and x64 bit, requires different directives..etc.)

Hello
Hypervisor From Scratch is an x64 project by design, you can't use it in 32-bit machines. Please try to compile it with x64 configuration.


Share input encoder at training and tagging time

I wanted to evaluate for the future Latin model I am training (with 8 different models for lemma and each morpho-syntactic feature).
Here is a small snippet I created to evaluate this which shows that we could win by sharing things accross on tagging large corpora. It does not make a huge change to the code.
Current situation
import time
import glob
import os.path

import tqdm

import pie
from pie.tagger import simple_tokenizer, pack_batch

# file = "PathToAModel"
# base_encoder = pie.model.BaseModel.load(file).label_encoder

example = list(simple_tokenizer("""Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Phasellus dolor sapien, laoreet non turpis eget, tincidunt commodo magna. Duis at dapibus ipsum.
Etiam fringilla et magna sed vehicula.
Nunc tristique eros non faucibus viverra.
Sed dictum scelerisque tortor, eu ullamcorper odio.
Aenean fermentum a urna quis tempus.
Maecenas imperdiet est a nisi pellentesque dictum.
Maecenas ac hendrerit ante. Vestibulum eleifend nulla at vulputate sagittis.
Maecenas sed magna diam. Sed facilisis tempus ipsum, nec mattis elit tincidunt lobortis.
Phasellus vel ex lorem. Nulla nunc odio, tempor non consequat in, luctus elementum dolor.
Nullam tincidunt purus vel lorem placerat, ac pulvinar turpis sodales.
Sed eget urna ac quam cursus porta.
Pellentesque luctus aliquet sem, a egestas purus finibus ac.
Mauris nec mauris non metus tempor faucibus non in est.
Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos.
Proin tristique nulla nec purus iaculis, eu scelerisque mi egestas.
In hac habitasse platea dictumst.
Ut placerat a neque eget aliquet. """))

print(f"Sentences : {len(example)}")
print(f"Words : {len([x for s in example for x in s])}")

runs = 1000
encoders = 8  # Latin has 8 tasks

records = []
for run in tqdm.tqdm(range(runs)):
    start = time.time()
    for encoder in range(encoders):
        pack_batch(base_encoder, example, "cpu")
    records.append(time.time() - start)

print(f"Average encoding time with {encoders} encoders {sum(records) / len(records)}")

Sentences : 22
Words : 194
100%|██████████| 1000/1000 [00:21<00:00, 46.80it/s]
Average encoding time with 8 encoders 0.02126287913322449

Sharing encoders
records = []
for run in tqdm.tqdm(range(runs)):
    start = time.time()
    pack_batch(base_encoder, example, "cpu")
    records.append(time.time() - start)

print(f"Average encoding time with a single encoders {sum(records) / len(records)}")

100%|██████████| 1000/1000 [00:02<00:00, 352.26it/s]
Average encoding time with a single encoders 0.002821772575378418

Hey, I don't get the difference between this and #72

The set-up for the test is the same, but this apply to different features:

#72 applies to not encoding wemb when unecessary
#70 applies mostly to Tagger (but also to training with a smaller impact) where you use different models (say, for Latin, one for Case, one for Lemma, one for Tense, etc.). Instead of encoding the input N times (where N is the number of models), if they have the same input translation table (label_encoder.table) and settings, it should be possible to not recode the input. In this context, sharing the input encoder would give, for the Latin tagger, a 10 time smaller encoding time (~20 seconds for 194k tokens vs 2 seconds). This is a small QOL that for inference is a nice improvement.

I think that'd make the code much harder to maintain and I am not convinced the gains are worth it. Even when considering your estimate, we are talking about 20 seconds for tagging a whole dataset. This encoding is never going to be the bottleneck, because it's just the conversion from strings to integer form. And even if you were referring to sharing the input embedding across models trained independently, the refactoring needed to make that work isn't going to be worth the small speed improvement you'd get (getting word-level embeddings is very cheap).

You can actually look at the implementation in #71 , the refactor is really really small... I know this is not a bottleneck, but the cost was quite small...

The only changes are the followings :
https://github.com/emanjavacas/pie/pull/71/files#diff-df0f6f3f56e1cc328ee783bb55949adeR129-R143
All other changes are new feature to accompany this change, including the test. I still feel like this is worth it :/

It's just a high cost to maintain: making sure older models work, new
models train exactly as old ones, etc.. but especially it's another edge to
keep in mind every time you want to tweak anything related to input
processing (for example the transformer input). And really speed is not an
issue here. Look at the speed up in terms of relative improvement and you
see it's really not that important. I am sorry because I know you have
already worked on the fix, but I have already run into a couple of issues
in the past where a small change actually introduced totally unexpected
bugs. It's just a headache that I'd rather not have.
On Tue, Aug 4, 2020 at 4:29 PM Thibault Clérice<EMAIL_ADDRESS>wrote:

The only changes are the followings :
https://github.com/emanjavacas/pie/pull/71/files#diff-df0f6f3f56e1cc328ee783bb55949adeR129-R143
All other changes are new feature to accompany this change, including the
test. I still feel like this is worth it :/
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/emanjavacas/pie/issues/70#issuecomment-668630335, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPIPI3AQELYQYDBBQ5OTQLR7ALMXANCNFSM4PUG42FA
.

--
Enrique Manjavacas

Hey :)
Technically, I do not touch models or label encoders. The only thing I add (and I really mean it) is a way to provide (optionally) at tagging time shared encoders. Models are not changed, tagging is not changed per se. I would completely agree with you but here I do not touch to anything in the models :/
As for bugs, there is a reason I carefully checked and provided tests with it :/

It's nice you wrote out some tests but unit testing isn't definitely ever
100% bulletproof. Still, my issue with this kind of small-scale patches is
the mental overhead they create. For example, right now if we need to
update the labelencoder we will have to make sure to also update this
functionality to make sure it still works. That's just an overhead I would
assume if the gains were substantial, but I don't think they are in this
case.
On Tue, Aug 4, 2020 at 4:40 PM Thibault Clérice<EMAIL_ADDRESS>wrote:

Hey :)
Technically, I do not touch models or label encoders. The only thing I add
(and I really mean it) is a way to provide (optionally) at tagging time
shared encoders. Models are not changed, tagging is not changed per se.
I would completely agree with you but here I do not touch to anything in
the models :/
As for bugs, there is a reason I carefully checked and provided tests with
it :/
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/emanjavacas/pie/issues/70#issuecomment-668636729, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPIPI6SBHWDEW6LQHL27HDR7AMVJANCNFSM4PUG42FA
.

--
Enrique Manjavacas

I agree unit testing is not bullet proof. I just think that I disagree that
the change will have impact down the road, as they are not at the level of
the label encoder. The things I added at the label encoder level are
helpers but all changes could have been kept with this single code snippet
I showed you. The rest is just there for helping users use this feature
Le mar. 4 août 2020 à 5:01 PM, Enrique Manjavacas<EMAIL_ADDRESS>a écrit :

It's nice you wrote out some tests but unit testing isn't definitely ever
100% bulletproof. Still, my issue with this kind of small-scale patches is
the mental overhead they create. For example, right now if we need to
update the labelencoder we will have to make sure to also update this
functionality to make sure it still works. That's just an overhead I would
assume if the gains were substantial, but I don't think they are in this
case.
On Tue, Aug 4, 2020 at 4:40 PM Thibault Clérice<EMAIL_ADDRESS>wrote:

Hey :)
Technically, I do not touch models or label encoders. The only thing I
add
(and I really mean it) is a way to provide (optionally) at tagging time
shared encoders. Models are not changed, tagging is not changed per se.
I would completely agree with you but here I do not touch to anything in
the models :/
As for bugs, there is a reason I carefully checked and provided tests
with
it :/
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/emanjavacas/pie/issues/70#issuecomment-668636729,
or
unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ABPIPI6SBHWDEW6LQHL27HDR7AMVJANCNFSM4PUG42FA
.

--
Enrique Manjavacas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/emanjavacas/pie/issues/70#issuecomment-668649169, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOXEZWABHUCL4ND7ZRHC6DR7APFDANCNFSM4PUG42FA
.

I am closing this issue. I understand your argument, and I really do not think it's worth going further in the debate :) Worst case scenario, my code and reflection stay around and we can go back to it. It definitely is less impactful than #73


Critical dependency: the request of a dependency is an expression when using App dir.

Using Next-Translate v2.0.5 with Next.js v13.3.
Node v19.9, npm v9.6.4
While trying to setup Next-Translate following the instructions in this post: https://dev.to/aralroca/i18n-translations-in-nextjs-13s-app-dir-for-serverclient-components-4ek8 and copying exactly the code form the CodeSanbox example, I'm getting this error in the server console:
Critical dependency: the request of a dependency is an expression
I've been using Next-Translate for several projects but this is the first time with the new App router.
My next.config.js file is as follows:
const nextTranslate = require('next-translate-plugin')
module.exports = nextTranslate({
experimental: { appDir: true },
})

middleware.js:
import { NextResponse } from 'next/server'
import i18n from './i18n'

export function middleware(request) {
  const locale = request.nextUrl.locale || i18n.defaultLocale
  request.nextUrl.searchParams.set('lang', locale)
  request.nextUrl.href = request.nextUrl.href.replace(`/${locale}`, '')
  return NextResponse.rewrite(request.nextUrl)
}

I18n.js:
module.exports = {
  locales: ['es', 'en'],
  defaultLocale: 'es',
  pages: {
    '*': ['common'],
  },
}

While trying to navigate to the main page I just get a 404 error. /es or /en doesn't change a thing.
Any ideas?

The 404 error went away after placing both layout.jsx and page.jsx inside a [locale] folder after reading Next's docs.
It kinda works, but Node seems to leak memory and spawning several instances. The server console still shows the error.

I am having the same issue.

Critical dependency: the request of a dependency is an expression

Same!
The 404 error is occurring because Next.js 13.3 has made some changes to routing (13.2 works perfectly fine).
like @guillermo-avalos said. After move to inside [locale] folder, it does work, but it doesn't seem to be stable.
Btw, to use next-translate 2.0.5, I believe we should stick with Next.js 13.2 for now.

Same problem here. next 13.4 & next-translate 2.0.5.
The i18n loads only on the client, and not on the server.
Also the code to sync date and lang cause runtime error:
Unhandled Runtime Error
TypeError: Cannot read properties of undefined (reading 'length')

"use client";

import { useEffect } from "react";
import dayjs from "dayjs";
import "dayjs/locale/en";
import timezone from "dayjs/plugin/timezone";
import utc from "dayjs/plugin/utc";
import useTranslation from "next-translate/useTranslation";

interface Props {
  children: JSX.Element;
}

dayjs.extend(utc);
dayjs.extend(timezone);
dayjs.tz.guess();

export const DateProvider = ({ children }: Props) => {
  const { lang } = useTranslation();

  useEffect(() => {
    dayjs.locale(lang);
  }, [lang]);

  return children;
};

Is there a way to HACK it somehow (even losing ssr) until we have a fix?

This is a warning from webpack. It seems webpack doesn't like imports with unpredictable paths at build time used by the default loader in next-translate-plugin. I don't know if it's really the case but it seems omitting overwriteLoadLocales still works.

Related https://github.com/aralroca/next-translate-plugin/issues/9

I'm also facing the same issue I'm currently using "next": "13.4.7" and "next-translate": "2.4.4" don't have a codesanbox prepared yet but here are the things I did:
This is what appears on my terminal:

And in the browser:

I tried to follow the docs and this article: https://dev.to/aralroca/i18n-translations-in-nextjs-13s-app-dir-for-serverclient-components-4ek8
middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
import i18n from './i18n';

// /es/page-name -> rewrites to -> /es/page-name?lang=es
export function middleware(request: NextRequest) {
    const locale = request.nextUrl.locale || i18n.defaultLocale;
    request.nextUrl.searchParams.set('lang', locale);
    return NextResponse.rewrite(request.nextUrl);
}

i18n.js
module.exports = {
    locales: ['en', 'pt', 'fr'],
    defaultLocale: 'en',
    pages: {
        '*': ['common'],
    },
};

next.config.mjs
// @ts-check
import bundleAnalyzer from '@next/bundle-analyzer';
import nextTranslate from 'next-translate-plugin';

const analyzeEnabled = process.env.ANALYZE === 'true';

/** @type {import("next").NextConfig} */
const config = {
    reactStrictMode: true,
    swcMinify: true,
    webpack: (config, { isServer, webpack }) => {
        return config;
    },
};

const withBundleAnalyzer = bundleAnalyzer({
    enabled: analyzeEnabled,
});

export default withBundleAnalyzer(nextTranslate(config));

My app dir is currently like this:

The 404 probably is for the middleware, I need to update this article, but after next.js 13.2 the middleware with i18n has an issue reported in Next.js but we updated using dynamic router [lang]. Probably you can remove the middleware, is not needed now to add the lang param and rewrite of the middleware is causing a 404 for this Next.js bug

any solotion of this?????

@aralroca is there any solution for this problem? Can you help?
When I add loadLocaleFrom for the fallback, I get the same error: Critical dependency: the request of a dependency is an expression and can't find any solution for it. Is it a bug?
I am using next v12.2.5 and next-translate v2.4.4 and next-translate-plugin v2.4.4
This is what I added to the config. When I remove it, the error dissapears:
loadLocaleFrom: async (lang, ns) => {
    const defaultLocale = 'en-gb';
    const defaultNs = await import(`./locales/${defaultLocale}/${ns}.json`);
    if (lang === defaultLocale) return defaultNs;
    const localNs = await import(`./locales/${lang}/${ns}.json`);
    return { ...defaultNs, ...localNs };
}

I get the same error when adding this version too:
loadLocaleFrom: async (lang, ns) => {
    const defaultLocale = 'en-gb';
    const defaultNs = await import(
      `./locales/${defaultLocale}/${ns}.json`
    ).then((x) => x.default);
    if (lang === defaultLocale) return defaultNs;
    try {
      const localeNs = await import(`./locales/${lang}/${ns}.json`).then(
        (x) => x.default
      );
      return { ...defaultNs, ...localeNs };
    } catch (e) {
      return defaultNs;
    }
}

@rolandjlevy upgrading to the latest version of next-translate-plugin should fix the Critical dependency: the request of a dependency is an expression from  a custom loadLocaleFrom

I have the same problem, using "next" v13.4.9, and "next-translate" v2.5.2.
- wait compiling...
- warn ./node_modules/next/dist/pages/_app.js
Critical dependency: the request of a dependency is an expression

I don't have the 404 error though, since I moved the middleware.ts to the 'src' folder on the same level as 'app' folder.

I have the same problem. All pages inside the [lang] folder return a 404 and the warning "Critical dependency: the request of a dependency is an expression.
Is there already a solution how to fix it?
next 13.4.16
next-translate 2.5.3
next-translate-plugin 2.5.3

@aralroca I found a solution.
In case this helps someone else having this problem. The code below works for me and I am no longer getting the error: Critical dependency: the request of a dependency is an expression
I am using next v12.2.5 and next-translate v2.5.2 and next-translate-plugin v2.5.2
in i18n.js:
 loadLocaleFrom: (lang, ns) => {
    const defaultLocale = 'en-gb';
    const defaultNs = require(`./locales/${defaultLocale}/${ns}.json`);
    if (lang === defaultLocale) return Promise.resolve(defaultNs);
    const localNs = require(`./locales/${lang}/${ns}.json`);
    return Promise.resolve({
      ...defaultNs,
      ...localNs,
    });
  },

@aralroca is there any solution for this problem? Can you help?
When I add loadLocaleFrom for the fallback, I get the same error: Critical dependency: the request of a dependency is an expression and can't find any solution for it. Is it a bug?
I am using next v12.2.5 and next-translate v2.4.4 and next-translate-plugin v2.4.4
This is what I added to the config. When I remove it, the error dissapears:
loadLocaleFrom: async (lang, ns) => {
    const defaultLocale = 'en-gb';
    const defaultNs = await import(`./locales/${defaultLocale}/${ns}.json`);
    if (lang === defaultLocale) return defaultNs;
    const localNs = await import(`./locales/${lang}/${ns}.json`);
    return { ...defaultNs, ...localNs };
}

I get the same error when adding this version too:
loadLocaleFrom: async (lang, ns) => {
    const defaultLocale = 'en-gb';
    const defaultNs = await import(
      `./locales/${defaultLocale}/${ns}.json`
    ).then((x) => x.default);
    if (lang === defaultLocale) return defaultNs;
    try {
      const localeNs = await import(`./locales/${lang}/${ns}.json`).then(
        (x) => x.default
      );
      return { ...defaultNs, ...localeNs };
    } catch (e) {
      return defaultNs;
    }
}

@aralroca I found a solution 😀
In case this helps someone else having this problem. The code below works for me and I am no longer getting the error: Critical dependency: the request of a dependency is an expression
I am using next v12.2.5 and next-translate v2.5.2 and next-translate-plugin v2.5.2
in i18n.js:
 loadLocaleFrom: (lang, ns) => {
    const defaultLocale = 'en-gb';
    const defaultNs = require(`./locales/${defaultLocale}/${ns}.json`);
    if (lang === defaultLocale) return Promise.resolve(defaultNs);
    const localNs = require(`./locales/${lang}/${ns}.json`);
    return Promise.resolve({
      ...defaultNs,
      ...localNs,
    });
  },

did you solve this problem ? i got the same error ((

I'm also facing the same issue I'm currently using "next": "13.4.7" and "next-translate": "2.4.4" don't have a codesanbox prepared yet but here are the things I did:
This is what appears on my terminal:
And in the browser:
I tried to follow the docs and this article: https://dev.to/aralroca/i18n-translations-in-nextjs-13s-app-dir-for-serverclient-components-4ek8
middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
import i18n from './i18n';

// /es/page-name -> rewrites to -> /es/page-name?lang=es
export function middleware(request: NextRequest) {
    const locale = request.nextUrl.locale || i18n.defaultLocale;
    request.nextUrl.searchParams.set('lang', locale);
    return NextResponse.rewrite(request.nextUrl);
}

i18n.js
module.exports = {
    locales: ['en', 'pt', 'fr'],
    defaultLocale: 'en',
    pages: {
        '*': ['common'],
    },
};

next.config.mjs
// @ts-check
import bundleAnalyzer from '@next/bundle-analyzer';
import nextTranslate from 'next-translate-plugin';

const analyzeEnabled = process.env.ANALYZE === 'true';

/** @type {import("next").NextConfig} */
const config = {
    reactStrictMode: true,
    swcMinify: true,
    webpack: (config, { isServer, webpack }) => {
        return config;
    },
};

const withBundleAnalyzer = bundleAnalyzer({
    enabled: analyzeEnabled,
});

export default withBundleAnalyzer(nextTranslate(config));

My app dir is currently like this:

did you solve this problem ? i got the same error ((

the same issue!!!!

@aralroca I found a solution 😀
In case this helps someone else having this problem. The code below works for me and I am no longer getting the error: Critical dependency: the request of a dependency is an expression

Tested and doesn't work in my environment. Still the same error/warning.

@b-vetter can you share the contents of your i18n.js and next.config,js files?

@roland-cromwell
i18n.js
module.exports = {
    locales: ["en", "de"],
    defaultLocale: "en",
    pages: {
      "*": ["common"],
      "/[lang]": ["home"]
    },
    loadLocaleFrom: (lang, ns) => {
      const defaultLocale = 'en';
      const defaultNs = require(`./locales/${defaultLocale}/${ns}.json`);
      if (lang === defaultLocale) return Promise.resolve(defaultNs);
      const localNs = require(`./locales/${lang}/${ns}.json`);
      return Promise.resolve({
        ...defaultNs,
        ...localNs,
      });
    }
  };

next.config.js
/** @type {import('next').NextConfig} */

const nextTranslate = require('next-translate-plugin')

const dns = require("dns");
dns.setDefaultResultOrder("ipv4first")

const nextConfig = nextTranslate({
  images: {
    domains: ['avatars.githubusercontent.com', 'avatar.vercel.sh']
  },
  experimental: {
    appDir: true,
    serverComponentsExternalPackages: ['@tremor/react']
  },
  async headers() {
    return [
      {
        source: "/api/:path*",
        headers: [
          // some header configuration
        ],
      }
    ]
  },
});

module.exports = nextConfig;

I got the same error in https://github.com/aralroca/next-translate/tree/master/examples/with-app-directory when using next 13.4.19
next 13.4.7 works fine in the example

Don't know whats happening there, but just do this:

It works fine somehow?! Even with 13.4.19

Don't know whats happening there, but when I just do this:  It works fine somehow?! Even with 13.4.19

It doesn't work for me

I got the same error in https://github.com/aralroca/next-translate/tree/master/examples/with-app-directory when using next 13.4.19
next 13.4.7 works fine in the example

Works also for me with 13.4.7. So is it a problem at next-js? When can we expect a fix on newer versions?

I created a fix https://github.com/aralroca/next-translate-plugin/pull/72
Here is a patch if somebody needs it right away https://github.com/bluelakee02/next-translate-plugin-patch

@aralroca OK, now this can be probably closed

@aralroca OK, now this can be probably closed

Sure 😊 people, now is already solved in 2.6.1 thanks to @bluelakee02

@bluelakee02 can u give access to https://github.com/bluelakee02/next-translate-plugin-patch ?


feat(ci): Add changelog workflow

PR Standards
Opening a pull request should be able to meet the following requirements

This PR add auto changelog workflow as the following:
When new commit pushed/PR merged to main branch then
Check if a changelog PR already opened:

(Yes) merge from main branch (with ort strag) if needed, then new changelog entry will be append to changelog.md and update to already open PR then exit.
(No) checkout from main branch, generate changelog entry and create PR.

Branch should delete automatically on next ci run, or you can delete it manully.
NOTE
Do not create branch "auto-changelog-update-do-not-create-manually" manually.
For Author:

[ ]  Targeted PR against correct branch
[ ]  included the correct type prefix in the PR title
[ ]  Linked to Github issue with discussion and accepted design
[ ]  Targets only one github issue
[ ]  Wrote unit and integration tests
[ ]  All CI checks have passed
[ ]  Added relevant godoc comments

For Reviewer:

[ ]  confirmed the correct type prefix in the PR title
[ ]  Reviewers assigned
[ ]  confirmed all author checklist items have been addressed

After reviewer approval:

[ ]  In case targets main branch, PR should be squashed and merged.
[ ]  In case PR targets a release branch, PR should be rebased.

@omritoptix can we merge this


Management module overrides set parameters for policies

Problem statement
Tested with:
Terraform version: 0.14.10 and 0.15.3
AzureRM provider version: latest (2.58.0)
Module version: 0.2.0 and 0.3.0
Some ES policies, that are applied to the root with manually specified parameters, want to set the parameters to the default value originating from the management module, even if we don't deploy the management resources from the ES module.
It doesn't matter if we apply our parameters in the PolicyAssignment, ArchetypeDefinition, or archetype_config_overrides, it still choose the settings from the management module.
A workaround that almost work all way, is to configure configure_management_resources with a mockup where you can configure the es module with this property:
configure_management_resources = {
    settings = {
      log_analytics = {
        enabled = true
        config = {
          retention_in_days                           = 30
          enable_monitoring_for_arc                   = true
          enable_monitoring_for_vm                    = true
          enable_monitoring_for_vmss                  = true
          enable_solution_for_agent_health_assessment = true
          enable_solution_for_anti_malware            = true
          enable_solution_for_azure_activity          = true
          enable_solution_for_change_tracking         = true
          enable_solution_for_service_map             = true
          enable_solution_for_sql_assessment          = true
          enable_solution_for_updates                 = true
          enable_solution_for_vm_insights             = true
          enable_sentinel                             = true
        }
      }
      security_center = {
        enabled = true
        config = {
          email_security_contact             =<EMAIL_ADDRESS>          enable_defender_for_acr            = true
          enable_defender_for_app_services   = true
          enable_defender_for_arm            = true
          enable_defender_for_dns            = true
          enable_defender_for_key_vault      = true
          enable_defender_for_kubernetes     = true
          enable_defender_for_servers        = true
          enable_defender_for_sql_servers    = true
          enable_defender_for_sql_server_vms = true
          enable_defender_for_storage        = true
        }
      }
    }

    location = "westeurope"
    tags     = null
    advanced = {
      existing_resource_group_name                 = "custom-rg"
      existing_log_analytics_workspace_resource_id = "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/custom-rg/providers/Microsoft.OperationalInsights/workspaces/custom-la"
      existing_automation_account_resource_id      = "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/custom-rg/providers/Microsoft.OperationalInsights/workspaces/custom-aa"
      custom_settings_by_resource_type = {
        azurerm_automation_account = {
          name = "custom-aa"
        }
        azurerm_log_analytics_workspace = {
          name = "custom-la"
        }
      }
    }
  }

But there is still 1 hardcoded parameter in the management module for the policy Deploy-ASC-Defender which is:
ascExportResourceGroupName          = "${local.root_id}-asc-export"
This blocks an upgrade to the new versions, or creation of custom named resource group for that policy.
Problem reproduction
Create a ES setup with the following config:
module "enterprise_scale" {
  source  = "Azure/caf-enterprise-scale/azurerm"
  version = "0.3.0"

  root_parent_id = "e30cb998-45a0-4111-b2fd-092fed41f457"
  #library_path     = "${path.root}/lib"
  default_location = "westeurope"
  root_id          = "es-test"
  root_name        = "es-test"
}

Run a terrafrom plan with the configuration and it displays the default values.
Example from the Deploy-ASC-Defender policyAssignment:
 # module.enterprise_scale.azurerm_policy_assignment.enterprise_scale["/providers/Microsoft.Management/managementGroups/es-test/providers/Microsoft.Authorization/policyAssignments/Deploy-ASC-Defender"] will be created
  + resource "azurerm_policy_assignment" "enterprise_scale" {
      + description          = "Deploy-ASC-Defender."
      + display_name         = "Deploy-ASC-Defender"
      + enforcement_mode     = false
      + id                   = (known after apply)
      + location             = "westeurope"
      + metadata             = (known after apply)
      + name                 = "Deploy-ASC-Defender"
      + not_scopes           = []
      + parameters           = jsonencode(
            {
              + ascExportResourceGroupLocation      = {
                  + value = "westeurope"
                }
              + ascExportResourceGroupName          = {
                  + value = "es-test-asc-export"
                }
              + emailSecurityContact                = {
                  + value = "security_contact@replace_me"
                }
              + logAnalytics                        = {
                  + value = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/es-test-mgmt/providers/Microsoft.OperationalInsights/workspaces/es-test-la"
                }

Adjust the ES module config with a archetype_config_overrides so the module config looks like this:
module "enterprise_scale" {
  source  = "Azure/caf-enterprise-scale/azurerm"
  version = "0.3.0"

  root_parent_id = "e30cb998-45a0-4111-b2fd-092fed41f457"
  #library_path     = "${path.root}/lib"
  default_location = "westeurope"
  root_id          = "es-test"
  root_name        = "es-test"

  archetype_config_overrides = {
    root = {
      archetype_id = "es_root"
      parameters = {
        Deploy-ASC-Defender = {
          emailSecurityContact                =<EMAIL_ADDRESS>          logAnalytics                        = "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/custom-rg/providers/Microsoft.OperationalInsights/workspaces/custom-la",
          ascExportResourceGroupName          = "custom-rg"
          ascExportResourceGroupLocation      = "westeurope"
          pricingTierContainerRegistry        = "Standard"
          pricingTierAppServices              = "Standard"
          pricingTierArm                      = "Standard"
          pricingTierDns                      = "Standard"
          pricingTierKeyVaults                = "Standard"
          pricingTierKubernetesService        = "Standard"
          pricingTierVMs                      = "Standard"
          pricingTierSqlServers               = "Standard"
          pricingTierSqlServerVirtualMachines = "Standard"
          pricingTierStorageAccounts          = "Standard"
        }
      }
      access_control = {}
    }
  }
}

Run a terrafrom plan with the new configuration and it still displays the default values, instead of the ones that should have been overridden of the archetype_config_overrides config in root.
Example from the Deploy-ASC-Defender policyAssignment after the override:
 # module.enterprise_scale.azurerm_policy_assignment.enterprise_scale["/providers/Microsoft.Management/managementGroups/es-test/providers/Microsoft.Authorization/policyAssignments/Deploy-ASC-Defender"] will be created
  + resource "azurerm_policy_assignment" "enterprise_scale" {
      + description          = "Deploy-ASC-Defender."
      + display_name         = "Deploy-ASC-Defender"
      + enforcement_mode     = false
      + id                   = (known after apply)
      + location             = "westeurope"
      + metadata             = (known after apply)
      + name                 = "Deploy-ASC-Defender"
      + not_scopes           = []
      + parameters           = jsonencode(
            {
              + ascExportResourceGroupLocation      = {
                  + value = "westeurope"
                }
              + ascExportResourceGroupName          = {
                  + value = "es-test-asc-export"
                }
              + emailSecurityContact                = {
                  + value = "security_contact@replace_me"
                }
              + logAnalytics                        = {
                  + value = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/es-test-mgmt/providers/Microsoft.OperationalInsights/workspaces/es-test-la"
                }

Which is the same result and means that the archetype_config_overrides in the es module configuration is ignored.
Expected behavior
If ES variable deploy_management_resources is set to false (or not set), the archetype_config_overrides from the management module should not be applied.
We should be able to use the ES policies used in the management module and set the parameters for the policies with either PolicyAssignment, ArchetypeDefinition, or archetype_config_overrides, without the need to configure the management module.

Deploy-ASC-Defender
Deploy-LX-Arc-Monitoring
Deploy-VM-Monitoring
Deploy-VMSS-Monitoring
Deploy-WS-Arc-Monitoring
Deploy-AzActivity-Log
Deploy-Resource-Diag

I noticed that, but I have not been using that setting yet.
I think it would be part of the same kind of issue, since its enforcing settings from the module even if you don't use the module.

Closing issue as resolved by #98


[volk] Update dependencies to only depend on vulkan-headers rather than vulkan

Describe the pull request
The volk library doesn't need the entire vulkan package as a dependency. It can build with just vulkan-headers. This would be ideal as vulkan-headers is a more minimal dependency than vulkan which currently relies on the user to have an already installed Vulkan SDK.

What does your PR fix?
Minimizes the dependencies on the volk library

Which triplets are supported/not supported? Have you updated the CI baseline?
All

Does your PR follow the maintainer guide?

Yes

Does upstream approve of this modification?

The upstream CMakeLists.txt does have support for simply being given the path to the vulkan headers. Also it doesn't really make sense for volk to depend on the vulkan package. The Vulkan SDK actually contains a volk library out of the box now, so the current setup is a bit redundant.
I can certainly open an issue upstream to get input from @zeux if you think that's required but the change here feels pretty natural / minimal to me as someone who has worked with the volk library for a while now

Also of note is that upstream's CMakeLists.txt explicitly checks for this variable that I'm setting in the port: https://github.com/zeux/volk/blob/master/CMakeLists.txt#L71

Thank you!


Implement own authentication system with OpenID protocols

Hello!
I currently work at a company that has built an authentication API using Python/Flask and we are looking to make it a federated SSO such as Google+ and Facebook.
Would this be possible to implement using this library? If so, how can would I go about hooking in this library into our API so our authentication endpoints can support the OpenID protocol?
Thank you so much!

Hello! What you are building is an OIDC Provider (aka OP). PyOP is a library that helps you with that, but this particular library is not properly maintained. I would suggest to look at oidcendpoint that is building on top of what was learned when working on pyop.
(I'm pinging @rohe that is leading the oidcendpoint library.)

Hi!
Yes, as Ivan says you should look at oidcendpoint and possibly also oidc-op which gives you an example of an Flask based OP built using oidcendpoint.
I'm a bit occupied right now but hopefully I'll have time later today to give a more complete answer.

Thank you for the responses!
I'll be sure to look at both oidcendpoint and oidc-op.
Last thing, I am currently going through the OpenID Connect Core specification and implementing the main functionality from scratch for learning purposes. Do you guys know of any online community centered on OAuth/OpenID where we can ask/answer questions about the specification?
Thank you again!

Since you're using Python you could connect to the IndentityPython slack (identity-python.slack.com).
If I had your email I could have sent you an invite :-)

Regarding what you have to do to connect your authentication module to oidcendpoint have a look at
the class UserPassJinja2 in https://github.com/IdentityPython/oidcendpoint/blob/develop/src/oidcendpoint/user_authn/user.py
together with how this method is hooked in as defined in
https://github.com/IdentityPython/oidc-op/blob/93a995fb761ceaaff97a82de33d2ab7b4e013a7f/flask_op/config.yaml#L195
The configuration specifies an acr (authentication context class reference, tells you what type of authentication this is), the class that implements the authentication method and keyword arguments that are used when the class is initiated.
The class instance is then called using the call method where you can do whatever you want but that must end with the user being redirect to a website that calls the verify method.
The UserPassJinja2 class for instance constructs a web page that contains a form (with username and password boxes) that when activated sends the user to the verify method using this part of views.py .
https://github.com/IdentityPython/oidc-op/blob/93a995fb761ceaaff97a82de33d2ab7b4e013a7f/flask_op/views.py#L147
If you need more info don't hesitate to ask.

Since you're using Python you could connect to the IndentityPython slack (identity-python.slack.com).
If I had your email I could have sent you an invite :-)

That would be awesome! My email is<EMAIL_ADDRESS>Thank you!

See also django-oidc-op, It have a pure oidcendpoint in a Django environment (with template)
It comes with an example project


Deberta v3 not working

System Info
I started the server on runpod with deberta v3, but I got this and the model didn't download:
root@eb4c9177bc5e:/workspace# infinity_emb v2 --model-id microsoft/deberta-v3-large --port 8000
INFO     2024-06-01 20:01:40,608 datasets INFO: PyTorch version 2.3.0 available.                                                                    config.py:58
INFO:     Started server process [1091]
INFO:     Waiting for application startup.
INFO     2024-06-01 20:01:41,678 infinity_emb INFO: model=microsoft/deberta-v3-large selected, using engine=torch and device=None       select_model.py:54
INFO     2024-06-01 20:01:41,848 sentence_transformers.SentenceTransformer INFO: Use pytorch device_name: cuda                        SentenceTransformer.py:188
INFO     2024-06-01 20:01:41,849 sentence_transformers.SentenceTransformer INFO: Load pretrained SentenceTransformer:                 SentenceTransformer.py:196
microsoft/deberta-v3-large
WARNING  2024-06-01 20:01:41,934 sentence_transformers.SentenceTransformer WARNING: No sentence-transformers model found with name   SentenceTransformer.py:1298
microsoft/deberta-v3-large. Creating a new one with mean pooling.
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/transformers/convert_slow_tokenizer.py:560: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
warnings.warn(
INFO     2024-06-01 20:01:44,327 infinity_emb INFO: Adding optimizations via Huggingface optimum.                                             acceleration.py:25
WARNING  2024-06-01 20:01:44,329 infinity_emb WARNING: BetterTransformer is not available for model: <class                                   acceleration.py:36
'transformers.models.deberta_v2.modeling_deberta_v2.DebertaV2Model'> Continue without bettertransformer modeling code.
INFO     2024-06-01 20:01:44,330 infinity_emb INFO: Switching to half() precision (cuda: fp16).                                       sentence_transformer.py:73
Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
INFO     2024-06-01 20:01:44,944 infinity_emb INFO: Getting timings for batch_size=32 and avg tokens per sentence=1                           select_model.py:77
0.60     ms tokenization
17.30    ms inference
0.08     ms post-processing
17.99    ms total
embeddings/sec: 1778.96
INFO     2024-06-01 20:01:45,633 infinity_emb INFO: Getting timings for batch_size=32 and avg tokens per sentence=512                         select_model.py:83
14.28    ms tokenization
317.46   ms inference
0.26     ms post-processing
332.00   ms total
embeddings/sec: 96.39
INFO     2024-06-01 20:01:45,635 infinity_emb INFO: model warmed up, between 96.39-1778.96 embeddings/sec at batch_size=32                    select_model.py:84
INFO     2024-06-01 20:01:45,636 infinity_emb INFO: creating batching engine                                                                batch_handler.py:291
INFO     2024-06-01 20:01:45,638 infinity_emb INFO: ready to batch requests.                                                                batch_handler.py:354
INFO     2024-06-01 20:01:45,640 infinity_emb INFO:                                                                                infinity_server.py:56
     ♾  Infinity - Embedding Inference Server
     MIT License; Copyright (c) 2023 Michael Feil
     Version 0.0.39

     Open the Docs via Swagger UI:
     http://<IP_ADDRESS>:8000/docs

     Access model via 'GET':
     curl http://<IP_ADDRESS>:8000/models

INFO:     Application startup complete.
INFO:     Uvicorn running on http://<IP_ADDRESS>:8000 (Press CTRL+C to quit)
INFO:     <IP_ADDRESS>:42100 - "GET / HTTP/1.1" 307 Temporary Redirect
INFO:     <IP_ADDRESS>:42100 - "GET /docs HTTP/1.1" 200 OK
INFO:     <IP_ADDRESS>:49954 - "GET / HTTP/1.1" 307 Temporary Redirect
INFO:     <IP_ADDRESS>:49954 - "GET /docs HTTP/1.1" 200 OK
INFO:     <IP_ADDRESS>:49954 - "GET /openapi.json HTTP/1.1" 200 OK
INFO:     <IP_ADDRESS>:51264 - "POST /classify HTTP/1.1" 400 Bad Request
INFO:     <IP_ADDRESS>:34484 - "POST /classify HTTP/1.1" 400 Bad Request
INFO:     <IP_ADDRESS>:54440 - "POST /classify HTTP/1.1" 400 Bad Request
Information

[ ] Docker
[X] The CLI directly via pip

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
infinity_emb v2 --model-id microsoft/deberta-v3-large
Expected behavior
to work with classify endpoint since its classification model

Hey!
this model does not has a classification head , or id2label in its config. https://huggingface.co/microsoft/deberta-v3-large/blob/main/config.json
therefore, you maybe can use it for embeddings, but it should be not well performing. Essentially its a model for mask-filling, which is not a downstream task that is interesting to support in infinity.