ManimPango should be a plugin

There are a number of problems with our integration with Pango that are detrimental to library's code hygiene. Only one community member understands the code, how to build it, and how to deploy it. It's caused difficulty for installation and development from the time it was first added to as recently as #1075, but more than either than those it just isn't necessary for manim's use.
We developed the plugin system but continually find reasons not to use it, even in instances like this that it was designed for. I'm fine with keeping it in the pypi package, but leaving ManimPango in the main code doesn't make much sense to me.
As an aside, there are a number of additions that we should probably consider moving into plugins as well, such as jupyter integration and (more of) the webgl renderer now that #1075 is merged.

There are a number of problems with our integration with Pango that are detrimental to library's code hygiene.

Can you elaborate?

Only one community member understands the code, how to build it, and how to deploy it.

@PhilippImhof also has knowledge about that

It's caused difficulty for installation and development from the time it was first added to as recently as #1075

What were the problems you faced in #1075?
I don't see any reports on the problem in installation problems of ManimPango.

it just isn't necessary for manim's use.

Manim should have at least one Text Rendering Engine either LaTeX or Pango, and we are encouraging in our docs that LaTeX are optional so Pango should be included in the Core of Manim IMO, as in how it is now.

We developed the plugin system but continually find reasons not to use it, even in instances like this that it was designed for. I'm fine with keeping it in the pypi package, but leaving ManimPango in the main code doesn't make much sense to me.

Are you thinking of depreciating Text and remove it from the Core library and make a plugin to provide it? Will, it not break things? ManimPango being used in the upstream repo also, I think there was no issue there, other than https://github.com/3b1b/manim/issues/1391 which was fixed in the recent release(https://github.com/ManimCommunity/ManimPango/issues/27).
I would something like manim-rubikscube is fit to be a plugin because many people don't find it useful enough to have it in the Core Library. But, if you think about Text it is a thing which is used pretty much everywhere.

As an aside, there are a number of additions that we should probably consider moving into plugins as well, such as jupyter integration and (more of) the webgl renderer now that #1075 is merged.

In that case, what do you think should be part of the core library?

I fully agree with @naveen521kk: Manim should offer at least one way of integrating text without the need of a LaTeX installation. And this functionality should be included in Manim's core. For as much as I love LaTeX, we have to admit that it is "big" and that e.g. installing und using custom fonts with LaTeX is not easy for everyone.

When it comes to problems that users run into, we actually get a lot more reports about issues TeX than about Pango; the situation with Pango really has improved dramatically since ManimPango has been introduced.
While I'm not active in ManimPango development, I do have a basic understanding about how it works as well.
If you ran into issues with it in #1075 then I am sure we can fix them. But I am firmly against removing it as a core feature (apart from the fact that it is not clear to me which problem that would solve -- whatever you ran into in #1075 would have to be resolved even if support for Text was delegated to a plugin).

I agree with Naveen's assessment of when the plugin system is useful. I think it makes more sense to have things like jupyter (or in this case a text engine) in the core of manim. If it's (among other things) delegated to a plugin, chances are high that it will not receive much attention and therefore no longer work with later versions of manim. When something like the rubikscube is outside the scope of the manim core, that's when plugins make the most sense. But I think a lot of current and future features belong more in the library than in plugins.
I also reinforce behackl's observation that there are more issues reported with tex than pango as I've noticed the same.

There's a lot to go over here and text isn't the best medium, so I'd prefer to do this on discord or a voice call. To get started here, we can focus on this

Only one community member understands the code, how to build it, and how to deploy it.

@PhilippImhof also has knowledge about that

From looking at the commit history almost all contributions from @PhilippImhof have been minor bugfixes; nearly every line was written by @naveen521kk. It also involves use of the CPython and Pango APIs, both of which will be slow to work with since they use C and the project is primarily in Python. This is a single point of failure, which is indicative of poor code hygiene.

From looking at the commit history almost all contributions from @PhilippImhof have been minor bugfixes; nearly every line was written by @naveen521kk.

That's normal, he started the whole thing. There was a time nearly every line of Manim was written by Grant. @naveen521kk started ManimPango, because Pango was abandoning their Python bindings. We had discussions on Discord about what to do. Among the different alternatives, no one suggested removing text capabilities from Manim IIRC.
ManimPango was done somewhat in a hurry, in order to have something to get started with and keep Manim's text functionality running with no interruption. Now that things have settled a bit, @naveen521kk and I are doing a complete refactoring and cleanup. I am currently planning for that.

It also involves use of the CPython and Pango APIs, both of which will be slow to work with since they use C and the project is primarily in Python.

Slow to work with, as opposed to lightning fast LaTeX which we currently use for text and which would be the only way to include text into an animation once we remove ManimPango? Also, speed has never been an issue before and text (be it Cairo or Pango) has been in Manim for a while now. Sorry, but that sounds like a straw man.

This is a single point of failure, which is indicative of poor code hygiene.

That's true, this point is definitely yours.
However, one objective of the refactoring mentioned above is to make it easier for ManimPango to be used in other Python projects where people want to draw formatted text. Once this happens, ManimPango will probably get more attention and more people will be able to keep up its development.
Also, if we keep text functionality in Manim's core, the community will do its best to maintain the code if @naveen521kk and I should get hit by the well-known bus. After all, we all managed to get into Grant's Manim code which was a much bigger code base and, with all due respect, not extremely well documented.

There's a lot to go over here and text isn't the best medium, so I'd prefer to do this on discord or a voice call.

I can only come for a call next week.

From looking at the commit history almost all contributions from @PhilippImhof have been minor bugfixes; nearly every line was written by @naveen521kk.

PhilippImhof was the author of MarkupText see https://github.com/ManimCommunity/ManimPango/pull/1 the history shows like that because I recently moved files https://github.com/ManimCommunity/ManimPango/pull/31. Also, he reviews every code I write, see the PR section, other than the CI configuration.
Also, I was the one to write all the packaging scripts, which anyone can read and understand easily.

It also involves use of the CPython and Pango APIs, both of which will be slow to work with since they use C and the project is primarily in Python. This is a single point of failure, which is indicative of poor code hygiene.

ManimPango is a separate project and it isn't an extension build inside of Manim. Consider it like how you will do Pillow or pycairo or for the matter any C extensions. Would you say them slow and not use them? Or you would say, the project is primarily python so it shouldn't be used? (that looks crazy!).

Consider it like how you will do Pillow or pycairo or for the matter any C extensions.

These are maintained by their own teams and presumably have several people supporting them. The same isn't true of ManimPango.

Also, I was the one to write all the packaging scripts, which anyone can read and understand easily.

This is subjective, you could just as well say the entire library can be read and understood easily.

PhilippImhof was the author of MarkupText...

I consider a having part of library written in a more difficult language that most of the team doesn't use, and maintained by one person who does nearly everything and another who makes <1 contribution a month to be poor code hygiene. It's fine if you disagree with that, but that'd probably be a reasonable evaluation for most people.

Consider it like how you will do Pillow or pycairo or for the matter any C extensions.

These are maintained by their own teams and presumably have several people supporting them. The same isn't true of ManimPango.

Not yet. As I wrote above, other projects might be interested in using ManimPango, especially with the end-of-life of Pango's Python bindings.

I consider a having part of library written in a more difficult language that most of the team doesn't use, and maintained by one person who does nearly everything and another who makes <1 contribution a month to be poor code hygiene. It's fine if you disagree with that, but that'd probably be a reasonable evaluation for most people.

I still do not disagree (as you might have read in my comment above). However, I come to a different conclusion.
My starting point is that Manim should have text functions that do not depend on LaTeX. (Anyone is free to disagree.) Then we need something to do that. We used to have Pango with its Python bindings. Unfortunately, this is no longer maintained. We then looked whether something else was readily available. This was not the case. So @naveen521kk started ManimPango. If one day he cannot or does not want to maintain it anymore, we will do the same again: We will ask ourselves whether Manim needs text functions and if it does we will either find ways to maintain ManimPango or look for an alternative which might exist by then.
Removing ManimPango and all text functions from the core just for the fear that one day it might be unmaintained will definitely not help in having more people want to engage with that part of the code.
Also MarkupText has gotten more features and together with my recent contribution to Pygments it will soon improve code rendering in animations as well. Some people never use anything else than LaTeX for their texts. That's fine, but remember we are building a community tool and our own use patterns are not necessarily valid for everyone.

My starting point is that Manim should have text functions that do not depend on LaTeX.

I can't come even this far; everything that attracted people to manim was accomplished with only Latex, and most people who use manim don't use it with Latex. There's a substantial natural overlap between people who use Latex and people who use manim, and since it allows for rendering equations and manim is primarily used for STEM, most projects using manim will end up using it anyway.
The benefits that come with using vanilla text are nice to have, but given the choice between one of them to support in the main code I don't see why you would choose regular text.

everything that attracted people to manim was accomplished with only Latex

I've seen various projects on computer science-y stuff like algorithm visualization which often make use of Text (and in particular Code), and little to no Tex.

virtually everything*

Also, I was the one to write all the packaging scripts, which anyone can read and understand easily.

This is subjective, you could just as well say the entire library can be read and understood easily.

Anyone with some amount of knowledge on building C stuff can understand that.

I consider a having part of library written in a more difficult language that most of the team doesn't use, and maintained by one person who does nearly everything and another who makes <1 contribution a month to be poor code hygiene. It's fine if you disagree with that, but that'd probably be a reasonable evaluation for most people.

Well in that case, according to you, all the glsl stuff you added in #1075 and the WebGL(javascript) stuff are of poor code hygiene. Because AFAIK only @eulertour and Grant have knowledge about them and they do literally everything.
In that case shouldn't all that be a plugin because of "poor code hygiene"?
You can say that part of code is used by everyone so, it should be in Core of Manim, yeah true, but what if you aren't there to maintain it? That is the case you are talking about now.
In either case, there are two maintainers here on ManimPango, one myself and @PhilippImhof, and I also think @behackl should have some experience with Cython, because IIRC, is from sagemath which uses Cython.
Essentially, for me having something which will work without the installation of LaTeX seems to be better for me. LaTeX is huge and is slow, and we use to call them from the command line, where there could be a lot of problems.

I think this all should come with no bad feelings whatsoever. @eulertour is certainly right in saying that ManimPango needs more people involved (or at least being capable of maintaining it). This might be true for other parts of Manim, but that is not a reason to not address it.
IMHO we should maintain the planned course of action, i.e. refactor ManimPango as planned. Then we can give other community devs the opportunity to have a closer look at it (of course, they are already free to do so now, but we should probably make this more explicit) and give them the ability to maintain it if need be. Also, after the refactoring, other projects might start using it and this could lead to having even more people involved.
Removing Text and MarkupText from Manim's core would be a bad choice IMHO.

Well in that case, according to you, all the glsl stuff you added

I'm sure you know why that'd be a bad decision.
But the issue is deeper than simply putting more effort into ManimPango, since that effort would be better spent on facilitating the migration to OpenGL and the possibilities it makes available. When breakages are virtually guaranteed, our documentation coverage is dropping as the library's complexity is growing, and devs are becoming less and less familiar with how the library works, it doesn't make much sense to incur the opportunity cost of putting more effort into text rendering.
The fact that OpenGL opens up lots of possibilities through interaction and 3D rendering makes the problem that much worse.

Well as @PhilippImhof said already there are plans to provide a stable API that can be used outside of Manim as there isn't a project in PyPI like ManimPango to render text say to an Image or SVG file, and also the wheels which makes it so that it is easily installable. It can be used in Pillow and also in other Project which doesn't have these capabilities directly. As in what I meant to say ManimPango is a separate project which Manim depends on, and it should be fine to have in the Core.
About maintainability, I would say that after I and @PhilippImhof do a refactor, other projects could depend on it and other developers could also help in maintaining it.
Depreciating or removing a feature, that actually works without any issue, which opens a lot of possibilities that would make Manim generate videos in other languages easily, for example, Arabic or something isn't that great. You can argue that they can use LaTeX but it isn't a thing made for that, and it is heavy and slow, and people would need special knowledge on how to, I myself don't know that.

Well as @PhilippImhof said already there are plans to provide a stable API...

What do you mean by a "stable" API? I didn't know there was anything unstable about the API right now.

that actually works without any issue...

This just isn't true.

would make Manim generate videos in other languages easily, for example, Arabic

Even as someone who has never attempted to write Arabic in Latex, I'm going to guess that this isn't very difficult with Latex.

Well as @PhilippImhof said already there are plans to provide a stable API...

What do you mean by a "stable" API? I didn't know there was anything unstable about the API right now.

https://github.com/ManimCommunity/ManimPango/issues/28

that actually works without any issue...

This just isn't true.

I haven't heard of any problems that haven't been fixed.

I don't want or not have any interest in taking this discussion further.
If you want it as a plugin do the needs of making it a plugin.

I have not read the entire thing, but I do want to highlight this fact:

I consider a having part of library written in a more difficult language that most of the team doesn't use, and maintained by one person who does nearly everything and another who makes <1 contribution a month to be poor code hygiene. It's fine if you disagree with that, but that'd probably be a reasonable evaluation for most people.

Well in that case, according to you, all the glsl stuff you added in #1075 and the WebGL(javascript) stuff are of poor code hygiene.

I'm sure you know why that'd be a bad decision.

The first comment ends with a sentence that basically says "most people would agree with me". I highly dislike the tone of this message. @eulertour if your contributions are met with pushback, it is your job to convince us by using arguments. It does not matter whether "most people" would agree with you. When we are talking about commiting changes to the library, the only people that matter are the dev team. If you can say that "most of the dev team" agrees with you, then that's fine. Otherwise, "most people would agree" is an invalid argument in my book.
The second comment brings up a fact that I want to echo. All the JS renderer and the openGL renderer cannot currently be maintained by anyone other than @eulertour. (Please correct me if I'm wrong.) By @eulertour's own words, this is poor code hygiene.
The third sentence also rubs me the wrong way. I understand this sentence as "it is obvious you are wrong". @eulertour I must ask you in the future to provide arguments rather than say such things.
@naveen521kk I respect your decision to not take part of this discussion, but that alone doesn't solve the issue that this thread is supposed to address. I believe a consensus was reached by @eulertour and @naveen521kk on discord. Can you please update this thread with that consensus so that other interested parties can chime in?

The first comment ends with a sentence that basically...

There are three things to say about this.
First, you can replace the end of the sentence with "most of the dev team" and I still stand by it, I'd just rather make a more general solution than a more specific one. Second, the tone of the sentence is extremely mild, and I don't think it's possible even to have disagreements if they have to be worded more lightly than that.

I think even saying "most of the dev team would agree with me" is still tonally iffy. For starters, this issue shows that there is disagreement in the first place, and so that statement cannot be taken as true without further evidence, because as it is, it's just your estimation that most would agree with you, and I don't like how that frames the conversation into "Most devs agree with me and therefore disagree with you". Second of all, the original phrasing of "that'd probably be a reasonable evaluation for most people" implicitly conveys that the opposing evaluation is not reasonable, whether you intended to convey that or not. I think something like "I don't think my evaluation is unreasonable" would work just as well while being less tonally iffy.

The first comment ends with a sentence that basically...

There are three things to say about this.
First, you can replace the end of the sentence with "most of the dev team" and I still stand by it, I'd just rather make a more general solution than a more specific one.
Second, the tone of the sentence is extremely mild, and I don't think it's possible even to have disagreements if they have to be worded more lightly than that.
Third, I've said before that if every other member of the dev team decided to make it preventatively difficult to merge commits I'd continue to merge commits anyway, and I still stand by that. My fundamental goal in this project is to do what's best for manim, and convincing people comes downstream from that.

The second comment brings up a fact that I want to echo...

I agree with this.

The third sentence also rubs me the wrong way...

First, the argument is basically that failure to implement OpenGL rendering at the same time that Grant is pushing to make Manim more of a community tool constitutes an existential threat to ManimCE. I can elaborate more if you want me to.
Second, if this rubs you the wrong way then what @naveen521kk said prior should do the same:

Well in that case, according to you, all the glsl stuff you added in #1075 and the WebGL(javascript) stuff are of poor code hygiene.

what if you aren't there to maintain it? That is the case you are talking about now.

If someone else speaks to me with a harsh tone I'll generally take that as license to respond in kind. In this particular case the tone of my response is more mild than the tone I was responding to.

this issue shows that there is disagreement in the first place

To be crystal clear:

I don't think anyone disagrees with what I said there, and the statement isn't to say that anyone disagrees with anyone.
The statement I was referring to isn't the topic of this thread, it's a more general one about code hygiene. To address the later responses, I absolutely agree that the OpenGL code exhibits bad code hygiene for the exact same reasons.

On the point of the tone being too harsh, I don't think this level of tone moderation is sustainable.

Ok, so to summarise both sides' arguments here amidst all this bickering about tone and stuff (feel free to edit if I miss anything):
EulerTour's assertions:
ManimPango (and with it Text) should be removed from the core library and redelegated as an external optional dependency/plugin, because:

ManimPango caused problems in some installations, particularly Windows installations when it was first launched in both upstream and downstream installs
There are only 2 members of the team (namely naveen521kk and PhilippImhof) with knowledge of how MP works, which is bad code hygeine
Supporting Pango forces us to maintain our own set of bindings written in C, which most of the dev team is not well-versed with.
Text causes problems with the OpenGL renderer
Most of what can be done with Text can be done with LaTeX, hence supporting Text is somewhat redundant.

naveen521kk and PhilippImhof's Rebuttals:
ManimPango and Text should remain in the core library as is the current status quo, because:

The problems caused by Pango/ManimPango have been solved with subsequent releases, and both upstream and downstream installs work fine on a majority of platforms.
By that logic, only 1 person (EulerTour) is working on the WebGL/OpenGL renderers (EulerTour agrees with this point)
Same argument, most of the dev team is not well versed with OpenGL as well (potential solution: both sides educate us on their respective topics; suggested by leotrs)
(couldn't find a reply to this point :/)
LaTeX is not fit for every use case, such as right-to-left text, using system fonts etc. For such situations, Text (and its derivatives, such as MarkupText) are valuable alternative solutions, and hence it would be a bad idea to remove them from core manim.

Thank you for summarizing, @kilacoda!

Text causes problems with the OpenGL renderer

Given that Text, just as Tex mainly does stuff to provide a SVG which we then use via SVGMobject, I would think that as soon as we support rendering SVGMobject, both Tex and Text should work. I think this was also briefly discussed in Discord.

The first comment ends with a sentence that basically...

There are three things to say about this.
First, you can replace the end of the sentence with "most of the dev team" and I still stand by it, I'd just rather make a more general solution than a more specific one.

As @friedkeenan said, I simply don't see the evidence for this.

Second, the tone of the sentence is extremely mild, and I don't think it's possible even to have disagreements if they have to be worded more lightly than that.

Once again, like many times in the past, I find your response unacceptable. It seems like whenever you are met with pushback, your response is to deny there is anything to be discussed at all, and you default to assuming you are always right. In this particular instance, you say that the sentence is "extremely mild" right after I told you I have a problem with it. Instead of talking about it, or admitting that it could have been phrased more clearly, or showing me that I'm wrong, or even engaging with my complaint in good faith, all you are doing is saying "there is nothing wrong here" and trying to move on. This exchange basically boils down to me saying "this bothers me" and you replying with "it shouldn't bother you".

Third, I've said before that if every other member of the dev team decided to make it preventatively difficult to merge commits I'd continue to merge commits anyway, and I still stand by that. My fundamental goal in this project is to do what's best for manim, and convincing people comes downstream from that.

But this, more than anything, is what I disagree with the most. Here you are telling us two things. First, that you believe it is good form (hygiene?) to decide what changes to make, and to merge your changes in spite of what "every other member of the dev team" may think. Second, implied by the first, that you can single-handedly decide what's best for manim on your own. I wholeheartedly reject both notions.
But also, you say that convincing other people comes downstream. Downstream of what exactly may I ask? When in your workflow does it matter to work and communicate with your teammates? How long after you've commited your changes do we all have to wait for your explanation of YOUR wishes about OUR library?
Your stance here just tells me that we have fundamental differences about what it means to work together as a team on the same FOSS project, and I believe we have (finally) reached a place where we have to solve these differences before moving forward with the project.

The third sentence also rubs me the wrong way...

First, the argument is basically that failure to implement OpenGL rendering at the same time that Grant is pushing to make Manim more of a community tool constitutes an existential threat to ManimCE. I can elaborate more if you want me to.

The existence of 3b1b/manim has always been an existential threat to manim ce and I don't see why OpenGL changes or accelerates that. We got quite a bit of momentum while manim ce had no trace of OpenGL and 3b1b did. If all of this is coming from a place of existential angst, then perhaps we could have saved a lot of time by communicating what you intended to do before doing it.

Second, if this rubs you the wrong way then what @naveen521kk said prior should do the same:

Well in that case, according to you, all the glsl stuff you added in #1075 and the WebGL(javascript) stuff are of poor code hygiene.

what if you aren't there to maintain it? That is the case you are talking about now.

If someone else speaks to me with a harsh tone I'll generally take that as license to respond in kind. In this particular case the tone of my response is more mild than the tone I was responding to.

I do not sense a trace of harshness in that quote by @naveen521kk. However I am willing to be shown otherwise, as I admit that text communications over the internet are difficult.

On the point of the tone being too harsh, I don't think this level of tone moderation is sustainable.

Either we work civilly as a team, or we do not have a team. Working in teams requires moderation. If we cannot sustain the amount of moderation required, we cannot work as a team. What are you going to do in the future to work better as a team?

One thing I would like to add regarding the discussion on tone: everyone please keep in mind that many people here don't speak English as a first language. Not everyone has the same feeling for English as native speakers have, and choosing words in a way such that an unintentional tone is transported can happen rather easily.

Your stance here just tells me that we have fundamental differences about what it means to work together as a team on the same FOSS project, and I believe we have (finally) reached a place where we have to solve these differences before moving forward with the project. I am going to tag the team bc I need everybody's opinion on this point

I could not agree more, be it with the interpretation of what has been said or with the conclusion. We must find a way to take important decisions as a community. If we cannot manage to decide on our community project as a community, we … probably are not a community.

We've gone too long without a community dev meeting and we should really schedule them regularly a few days before the releases anyways just to ensure things are going well. Even if only a few people can show up, it'd be helpful to have talking points to refer to (like the one @kilacoda posted, or @PhilippImhof in Discord). Given the current circumstance/perceived heat from this issue, I move we meet this weekend, but I think we should regularly default to meeting the weekend before the next scheduled release. I'll link it in Discord as well.
It's important to keep in mind long term goals for Manim v1.0.0 and to some degree, I agree with the differing visions -- having of course my own vision -- of what a stable API in ManimCE looks like. A large number of things in Manim could be plugins (i.e. WebGL/Jupyter/ManimPango/OpenGL), but whether they should be is a team decision and actually doing the work to make the plugin (i.e. the real questions of WHO will do it by WHEN and WHY need to be considered in that decision) is time from the team spent refactoring/documenting/peer reviewing code without adding new useful functionality/features other than extra installation steps for the end-user.
I feel voice discussions with someone taking the minutes/writing notes would be quicker and hopefully save us any issues with perceived tone (I imagine we're all trying to behave well). We are a team with limited time so the more we can align our visions, cooperate, and ultimately get along, the quicker we can get to OUR community edition to stability.

From a third person perspective, based on the following assumptions:

Manim will move entirely from Cairo Based rendering to OpenGL rendering as a default, because of the advantages provided by OpenGL.
One will be able to switch between renderers via installing plugins and importing from them.
Usage of Non-LaTeX based Text rendering is important in Manim and not something which "isn't necessary for manim's use"

My views are as follows:
On Whether ManimPango should be a Plugin.
Cairo-manim relies on ManimPango to write text that is not LaTeX based. This is beneficial to the library and massively simplifies getting non-English text to be displayed without finagling about with LaTeX. Some people may find working with LaTeX easy, and yes, the target audience of Manim will probably have experience with LaTeX, but the ease with which one can use Text makes me feel that it's worth keeping ManimPango in the core library as long as we are still in the liminal stages of moving from Cairo to OpenGL.
We are moving to OpenGL anyway. I recall some discussion on Discord about having a Modular Renderer. I haven't been active enough in the community (exams and school, sorry!) to check if this has already been done, but if the rendering logic has completely been separated from the backend (cairo/opengl) itself, it should be easy to make plugins that change the renderer as the end user would require, that keep the entire communication that's done between the backend and manim in the plugin.
Once OpenGL becomes the default backend, we can strip Cairo completely from ManimCE and package all of it into a plugin, and call it something like "manim-cairo-rend", "manim-cairo-rend" will have those of Manim's current dependencies that exist only to facilitate use of Cairo and PangoCairo, such as ManimPango. We can then get rid of ManimPango as a dependency for core manim.
Basically, I say we should keep the status quo for as long as it takes for OpenGL to become the default, all the while making sure that we can move everything Cairo related into a plugin, which itself will have dependencies such as ManimPango. Once OpenGL is reasonably stable and we can completely switch to it as the default, we can move everything cairo over to a plugin.
On Tone and Comms over the Net.
Disagreements happen. I'm pretty liberal on my views with this, but I personally am fine with sarcasm etc.
However, for the sake of clear communication, I say we should be as explicit as possible and refrain from implying things without actually stating them. Statements like "I'm sure you know why that'd be a bad decision." come off kind kind of condescending, though I'm sure that wasn't the commenter's intent. Explicit communication at all times. It doesn't matter if you're repeating yourself several times over, just say what you want explicitly.
On the other hand, statements like this,this and this, on discord that, (to me) sound like they essentially boil down into ultimatums are completely unacceptable.

Once OpenGL becomes the default backend, we can strip Cairo completely from ManimCE and package all of it into a plugin, and call it something like "manim-cairo-rend", "manim-cairo-rend" will have those of Manim's current dependencies that exist only to facilitate use of Cairo and PangoCairo, such as ManimPango. We can then get rid of ManimPango as a dependency for core manim.

As I understand it, text rendering in OpenGL is possible through PyOpenGL. The text will be rendered as a bitmap instead of a vector format.
However I am not sure whether it is actually necessary to do text rendering in OpenGL. In order to keep LaTeX based text, we will need a way to import from SVG (or similar) and we probably want to keep the ability to import vector graphics anyway. In that case, I do not know what we have to gain from rendering the text in OpenGL as opposed to Pango which creates an SVG. Basically, I just see the advantage of having fewer dependencies. Speed maybe? But again, I am by no means an OpenGL expert.
In any case, we would still need another way of rendering text for people using the current (by then probably: legacy) backend, no? So we would have to maintain two ways of rendering text, OpenGL in the core and Pango as a plugin?

text rendering in OpenGL is possible through PyOpenGL

I think we're using ModernGL though, which while compatible with PyOpenGL, might complicate things, although I can't say for sure.

My bad, I did not know what package we were using. AFAICS at the first glance, ModernGL does not support text rendering: https://github.com/moderngl/moderngl-window/issues/25
That might complicate things, but as I said -- from my naive perspective as a non-expert -- that's not a big thing, as we can just import MarkupText and the like in the same way as we do for LaTeX. There will definitely be no LaTeX support in OpenGL ;-)

Maybe @einarf could help us out and provide some clarity on this matter.

Cross platform text rendering is mostly done through freetype-py. It mainly relies on building an atlas or texture array of glyphs. This is simple enough for latin1 text, but will add challenges for other characer sets. Pango is using harfbuzz (https://github.com/harfbuzz/harfbuzz) to get the text shaping right.
You get a lot of free stuff using a text shaping library. It's a very complicated affair.
(We're just adding basic glyph rendering in moderngl-window with freetype-py)

This discussion was so hot :fire: when I was busy for the whole day, sorry for this late reply
To explain all this, OpenGL by itself doesn't have or is a text rendering engine (as explained by @einarf, thanks!), and ManimPango is required for OpenGL renderer also. So, this thread isn't about removing ManimPango because we have the OpenGL renderer, but rather this is about the problems that caused the downstream (3b1b version), https://github.com/3b1b/manim/issues/1391 https://github.com/3b1b/manim/issues/1387, and thinking whether that same problem will happen here.
On the other hand, maintaining ManimPango isn't that simple and requires quite some work for my side and as well as from @PhilippImhof (thanks!), and doing all those for being as a plugin, which may not be used by everyone, seems too much to me, if that happens I would simply archive it and won't maintain it.

In any case, we would still need another way of rendering text for people using the current (by then probably: legacy) backend, no? So we would have to maintain two ways of rendering text, OpenGL in the core and Pango as a plugin?

You get a lot of free stuff using a text shaping library. It's a very complicated affair. Also, if you want bitmapped fonts to look crisp in any size you might need to delve into distance fields. Text shaping libraries renders the text to the specified size every time to make it perfect. I don't even want to imagine how much work is involved in all this. I'd rather use a text shaping library 😅

To explain all this, OpenGL by itself doesn't have or is a text rendering engine (as explained by @einarf, thanks!), and ManimPango is required for OpenGL renderer also.

I believe that we should attempt to reduce both our dependencies and the amount of things we have to actively maintain. If there's a tried and tested method for Text rendering in OpenGL, (like Pango's python bindings for Cairo (not ManimPango) before they stopped maintaining it) then we should use that alone instead of ManimPango.
This is mainly because I believe this community was made to improve manim; not for building text-layout libraries just because Manim offers the ability to include Text in animations or, for the sake of example, making GUI libraries just because we will build GUIs for Manim in the future. As such, our focus "should" (it's a pretty strong word but I can't think of another one) be on Manim and unless we absolutely have to, not on whatever it may depend upon.
As long as there's no better option for Text rendering in Manim, we can keep ManimPango in the library. If OpenGL Manim makes use of it as well, it must only be because there are no better options available.
@naveen521kk , I read your conversation with @einarf on this ModernGL issue which referenced this issue in ManimPango and your conversation with @PhilippImhof.  If you both really do plan on making ManimPango open to more use than than just for Manim, there's an argument to be made that it's not purely for Manim anymore and doesn't need to have Manim in its name/be related to manim at all. Furthermore, if you intend on doing this, I believe a better course of action would simply be for you to take over maintenance of cairocffi entirely from their current maintainers, or alternatively to fork it, make every change that you did to ManimPango on that fork (except the name change), and archive ManimPango because, like I said, it's not purely for Manim anymore and the rest of the ManimCommunity shouldn't be expected to maintain it (not that they currently are maintaining it, but you get my point). The advantages there are: more people already know about cairocffi (so there'll be a sort of demand) and will be more likely to become co-maintainers/contributors, and you won't have to deal with the demands of manim and whether it "needs" a feature or not before you implement it in cairocffi/ManimPango.
Also, @naveen521kk @PhilippImhof  I read on cairocffi's github page that they are a drop-in replacement for pycairo. We have pycairo as one of our dependencies, so why do we need ManimPango at all? Can't we just use purely pycairo?

As long as there's no better option for Text rendering in Manim, we can keep ManimPango in the library. If OpenGL Manim makes use of it as well, it must only be because there are no better options available.

I agree with that. We should use the best option available and although I have put in quite some work, I would not mind dropping something, if (after thorough analysis) we find a better option. It is generally bad advice to keep something just because of the effort that has been put into it. (Reminds me of sunk costs somehow.)

If you both really do plan on making ManimPango open to more use than than just for Manim, there's an argument to be made that it's not purely for Manim anymore and doesn't need to have Manim in its name/be related to manim at all.

Again, I agree. If current ManimPango eventually becomes a "Python library for converting text to SVG", I think it would be sensible to make it a standalone project and maybe even change its name in order to emphasize that it is a general-purpose tool ready to be used in other projects. Although it might be kept for historical reasons, of course. Bear in mind that IMO we are currently far from that point, though.

As long as there's no better option for Text rendering in Manim, we can keep ManimPango in the library. If OpenGL Manim makes use of it as well, it must only be because there are no better options available.

Again, OpenGL has nothing to do with ManimPango or rather Text. https://github.com/ManimCommunity/manim/issues/1102#issuecomment-802198553

If you both really do plan on making ManimPango open to more use than than just for Manim, there's an argument to be made that it's not purely for Manim anymore and doesn't need to have Manim in its name/be related to manim at all.

I agree. But still, there is a long way to go there.

Furthermore, if you intend on doing this, I believe a better course of action would simply be for you to take over maintenance of cairocffi entirely from their current maintainers, or alternatively to fork it, make every change that you did to ManimPango on that fork (except the name change), and archive ManimPango because, like I said, it's not purely for Manim anymore and the rest of the ManimCommunity shouldn't be expected to maintain it (not that they currently are maintaining it, but you get my point).

cairocffi is a replacement for pycairo and isn't binding to Pango. So, cairocffi and ManimPango are entirely different.

Also, @naveen521kk @PhilippImhof I read on cairocffi's github page that they are a drop-in replacement for pycairo. We have pycairo as one of our dependencies, so why do we need ManimPango at all? Can't we just use purely pycairo?

The previous implementation of Text was based on pycairo but later that didn't work out because it is just a toy API and we have to look out for other implementation. We found Pango as in pangocffi but that created a lot of other problems, installation on Windows was broken because of that. That was fixed by ManimPango.

To explain further, first, Pango is part of GNOME libraries and the only python binding they officially offer is pygobject and none of the cffi bindings like cairocffi and pangocffi is maintained by Gnome itself. About using pygobject, it can't be used directly on Windows, and it would be too much for people to install full GTK+ for just rendering a text in Manim, and those were the reasons for me not using it.
Second, about using the cffi bindings, it caused many DLL problems as we previously used that and have a specific section for troubleshooting, https://docs.manim.community/en/v0.1.1/installation/troubleshooting.html#windows-oserror-dlopen-failed-to-load-a-library-pango which many a time doesn't work properly on Windows.
Let me explain what is the difference between all the cffi and ManimPango itself. First, cairocffi and pangocffi use https://cffi.readthedocs.io/en/latest/ which is like ctypes which means it searches for shared objects like .dll in the case of Windows and when it is unable to find things it crashes and that was a reason ManimPango by itself becomes an actual C extension rather than a cffi binding. So, both cairocffi and ManimPango are entirely different.

Again, OpenGL has nothing to do with ManimPango or rather Text.

Yes, OpenGL by itself doesn't have a text-rendering engine, but there must be several third-party engines that work with OpenGl that we don't have to maintain. Freetype+Harfbuzz, like @einarf mentioned could be used in place of ManimPango, and that might be beneficial in the long term since they have a much wider developer base!
I also found https://github.com/rougier/freetype-gl which at first glance seems like exactly what we are looking for with regards to text rendering on OpenGL manim, albeit it's entirely in C and C++.

cairocffi is a replacement for pycairo and isn't binding to Pango. So, cairocffi and ManimPango are entirely different.

Alright, thanks. That was confusing me for a bit.

Yes, OpenGL by itself doesn't have a text-rendering engine, but there must be several third-party engines that work with OpenGl that we don't have to maintain. Freetype+Harfbuzz, like @einarf mentioned could be used in place of ManimPango, and that might be beneficial in the long term since they have a much wider developer base!

Pango already uses it. It uses Cairo+Harfbuzz to render Text in an SVG file. If you want to use Freetype+Harfbuzz then you would be implementing another version of Pango in python and that would be too much work. Currently what ManimPango does is using the C API which Pango provides, and make it possible to call these from Python using Python-C API.

I also found https://github.com/rougier/freetype-gl which at first glance seems like exactly what we are looking for with regards to text rendering on OpenGL manim, albeit it's entirely in C and C++.

Are you asking to create another C extension? We would anyway have a way to draw SVG files in OpenGL, in that case, it seems too much to use freetype-gl. ManimPango creates SVG files which can be shown on OpenGL.

Currently what ManimPango does is using the C API which Pango provides, and make it possible to call these from Python using Python-C API.

Right. If we find another library like Pango, compatible with OpenGL that has a wider user base or is generally "better", then we should look into moving to it, regardless of what ManimPango does.

Are you asking to create another C extension? We would anyway have a way to draw SVG files in OpenGL, in that case, it seems too much to use freetype-gl. ManimPango creates SVG files which can be shown on OpenGL.

Nope. I'm saying that similar to freetype-gl, if there's a better way to render text than using ManimPango, we should consider it. It doesn't have to be freetype-gl or any library I mentioned in particular, which I mentioned for the sake of giving examples.
This goes back to your comment here:

Again, OpenGL has nothing to do with ManimPango or rather Text.

Yes, OpenGL doesn't care about Text rendering engines or bindings that interface with text-rendering engines.
My point is that if there is a better, "preferred" way to render text via OpenGL, which is not ManimPango, we should implement that/use libraries that implement that.

Right. If we find another library like Pango, compatible with OpenGL that has a wider user base or is generally "better", then we should look into moving to it, regardless of what ManimPango does.

I don't understand why you are saying ManimPango can't be used with OpenGL. It can be used without any problem, what actually happens is ManimPango created an SVG file with the Text as SVG paths which is later parsed by Manim and displayed on the screen. See 3b1b's repo Text for a working example.

Yes, OpenGL doesn't care about Text rendering engines or bindings that interface with text-rendering engines.
My point is that if there is a better, "preferred" way to render text via OpenGL, which is not ManimPango, we should implement that/use libraries that implement that.

I would say once ManimPango has a stable API it can be used with other projects also and could be the preferred way to render text as @einarf already said. Currently, there is freetype-py but it is the same as how CairoText is currently is, no support for RTL texts etc.

I don't understand why you are saying ManimPango can't be used with OpenGL

That's not what I said. I said that if there is a better library that we do not have to maintain, then instead of ManimPango, that's the one we should use.
This is because this community was made to improve manim; not for building text-layout libraries just because Manim offers the ability to include Text in animations or, for the sake of example, making GUI libraries just because we will build GUIs for Manim in the future. As such, our focus "should" (it's a pretty strong word but I can't think of another one) be on Manim and unless we absolutely have to, not on whatever it may depend upon.

I would say once ManimPango has a stable API it can be used with other projects also and could be the preferred way to render text as @einarf already said. Currently, there is freetype-py but it is the same as how CairoText is currently is, no support for RTL texts etc.

And in your opinion, how long will it take before ManimPango gets a stable API?

That's not what I said. I said that if there is a better library that we do not have to maintain, then instead of ManimPango, that's the one we should use.
This is because this community was made to improve manim; not for building text-layout libraries just because Manim offers the ability to include Text in animations or, for the sake of example, making GUI libraries just because we will build GUIs for Manim in the future. As such, our focus "should" (it's a pretty strong word but I can't think of another one) be on Manim and unless we absolutely have to, not on whatever it may depend upon.

Again, ManimPango isn't a text-layout library, it is just a binding to Pango. I have done that search long back before implementing ManimPango, and I couldn't find anything in python, which can handle RTL texts. One thing, I found was libraqm but that is again, not for python, and has fewer features than Pango, as in you would have to pass the font file each time you render something as is impossible to render into an SVG directly. Again, that would mean a C binding.

And in your opinion, how long will it take before ManimPango gets a stable API?

That needs a lot of work and maybe should take around a month.

That needs a lot of work and maybe should take around a month.

Awesome. Given this info, I'm in agreement with what @behackl said over on discord.
Once ManimPango stabilises and allows use for other purposes, let's rename it to something more generic and Pango related instead of related to Manim.
We can keep ManimPango (or whatever it's called after its name changes) as a dependency if required.

Once ManimPango stabilises and allows use for other purposes, let's rename it to something more generic (maybe pypango or MPango, to acknowledge that it was related to manim?) and Pango related instead of putting the emphasis on Manim.
We can keep ManimPango (or whatever it's called after its name changes) as a dependency if required.

From an "outside" perspective:
Doesn't it make more sense to keep ManimPago as a plugin respositry that depends on a pypango package containing the core functionality (eventually...)? Come up with a standard api the text layout plugins needs to follow? I'm guessing the ManimPango repository is more like an evolving experiment that could turn into something great?
People are free to make other plugins following the standard api. They could even be selected based on platform if needed. What text layout plugin should be default can be based on the typical criterias:

Stability
Ease of installation for end users <- Often highly underestimated!
Performance
Platforms

Pick whatever is reasonable based on this. Ease of installation in development is secondary to easy of installation for the end users. The former can be solved by having a dev setup guide.
Also: Creating a library using freetype-py and harfbuzz I suspect is expontentially more work than just relying on a true and tried text layout library.

Doesn't it make more sense to keep ManimPago as a plugin respositry that depends on a pypango package containing the core functionality (eventually...)? Come up with a standard api the text layout plugins needs to follow? I'm guessing the ManimPango repository is more like an evolving experiment that could turn into something great?

The reason a fair few people in the Dev team think keeping ManimPango as a core dependency is worthwhile was discussed in the first few comments on this issue. The most important points were:
Manim should offer at least one way of integrating text without the need of a LaTeX installation. And this functionality should be included in Manim's core. For as much as I love LaTeX, we have to admit that it is "big" and that e.g. installing und using custom fonts with LaTeX is not easy for everyone. -> @PhilippImhof
LaTeX is not fit for every use case, such as right-to-left text, using system fonts etc. For such situations, Text (and its derivatives, such as MarkupText) are valuable alternative solutions, and hence it would be a bad idea to remove them from core manim. -> @kilacoda via his summary of that conversation between @eulertour and @naveen521kk.
Having something like Text work out of the box without dealing with LaTeX is what most of the people in this conversation want. Getting rid of it in a release would seem almost unnecessary.
Though now that I think of it, having a standard api for text layout plugins so that anyone can write their own ones does sound enticing. Maybe we could have a "standard" one as well, to make sure Text works OOTB if we go the plugin approach?

Doesn't it make more sense to keep ManimPago as a plugin respositry that depends on a pypango package containing the core functionality (eventually...)? Come up with a standard api the text layout plugins needs to follow?

Essentially there isn't a pypango package. ManimPango is a C extension that interacts with Pango C API. Also, we don't have a standard API for text layout plugins to follow. All we have for plugins is https://docs.manim.community/en/v0.4.0/installation/plugins.html where manim will automatically import a plugin if it is defined in a config file. The idea of coming up with a standard API for Text seems to be a nice idea. Essentially, the class need to inherit from SVGMobject and then, have a function text2svg, that's all it needs to.

I'm guessing the ManimPango repository is more like an evolving experiment that could turn into something great?

Yup! It all started 3 months ago when I was searching for a better text rendering system and found Pango, and now it is as good as works on every system and can be easily installed with pip.

Pick whatever is reasonable based on this. Ease of installation in development is secondary to easy of installation for the end users. The former can be solved by having a dev setup guide.

Currently, I would say there are three implementations of text-rendering where two are needed for the core library.

Tex, LaTeX
Text <- where ManimPango is used
CairoText, the previous implementation of Text using plain Cairo.

Out of these, there should be Tex and Text while the other isn't required. Now, this issue is about having only Tex and removing everything else from the Core library and making it a plugin.
We would possibly decide in a call on Discord soon :tm:(possibly this week?) about this issue. If you want you can join us on discord @einarf, most probably we will do an announcement before we do the call.

This issue was discussed in yesterday's meeting and agreed that for now, ManimPango will stay until the plugin system is made better than how it is currently and one day eventually every other Mobject will be as a plugin and the core contain only the basic Mobject and the Renderer, Camera, Scene setup.


vue cli bug report does not have "3.0.0" version in Version select

Version
3.0.0
Node and OS info
node 8.11.1 / yarn 1.7.0 / OS X
Steps to reproduce
Try to file an issue for 3.0.0 version for vue-cli
What is expected?
To have 3.0.0 listed in choices
What is actually happening?
There is not search choice

Wrong depot, I'll move it to vue-issue-helper, sorry


[eventseg] Fix binary module name issue

Python started complaining about brainiak.eventseg.utils:
'dynamic module does not define module export function (PyInit_utils)'
https://travis-ci.org/IntelPNI/brainiak/jobs/170847241
Renaming it brainiak.eventseg._utils, in accordance to the Cython source file
name, fixes the issue.

@yidawang Please have a look.


⚠ ClamAV has degraded performance

In 1970839, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 112 ms

Resolved: ClamAV performance has improved in edcaf34 after 5 minutes.


[BUG] Creating a BlobClientBuilder from an endpoint-url fails if the url has a port

Description
Creating an BlobClient from an non-ip authority with a port is still detected as IpStyle and will therefore fail, as no accountName is set in the resulting BlobClient
String url = "https://xxxxx.blob.core.windows.net:443/....";
BlobClient blobClient = new BlobClientBuilder().endpoint(url).buildClient();
blobClient.uploadFromFile(file.getPath()); //Will fail, as no accountName set!

The actual bug is inside ModelHelper.determineAuthorityIsIpStyle, as it detects the IpStyle wrongly by simply checking for the existence of a port inside the URL.
Currently the workaround is to remove the port number from the URL.
Expected behavior
URLs with port can be used to create BlobClients

@rickle-msft I'm using another MS Service called AppCenter, which provides me with an upload URL with the port already attached, so sadly also not a lot I can do about this on the other end.

@RiederAlex Thank you for that information. Unfortunately the pattern of having a port attached to a url which uses a host name as in this case conflicts with a pattern we need to support for the emulator. Consequently our recommendation here needs to be that you strip the port number before passing in the url. This should hopefully be fairly manageable by using Java's URL type but you can let us know if you need any assistance there.

@rickle-msft Ok I understand that and no worries, I can manage to strip the URL myself ;)
Thank you for your responses!

Hi @rickle-msft there's another reason why I need to attach a port number. I use MockWebserver to script server responses for testing purposes. I am unable to do this if I strip out the port. Here's how an example MockWebserver test would look like for some context:

Setup MockWebserver to listen to requests. It sets itself up to listen to localhost and a random port by default.
Configure your HTTP client to send requests to MockWebserver. A BlobClient configured with OkHttpAsyncHttpClient in this case.
Send request, exercising the logic of your app.
Capture and assert properties of the request in MockWebserver and send back canned responses for test purposes.

If I strip out the port, yes that works for the app but it makes testing really hard. In other words, how do I test my code whilst using a BlobClient without having to send real requests to real servers? I can do that nicely with MockWebserver but I am struggling with how to integrate that here.

Hi, @mezpahlan. Two things:
The first is to use your existing workflow, but use IP-style endpoints. That is to say, your URL should be of the format: https://<IP_ADDRESS>:<proxy port>/<account name>. This does allow you to still attach a port number to your URL.
To answer your broader question of how to test BlobClient without sending real requests, you could also consider using Azurite, which is a Storage emulator. You'll actually still have to use IpStyle URLs, but you might find some of the features of the emulator helpful, so I thought I'd mention it.
Let me know if you have any more questions!

Hi @rickle-msft thanks for the tips unfortunately I'm not able to get this to work. This is what I have at the moment:
I've configured a BlobClient like this:
BlobClientBuilder()
  .endpoint(workaroundAzureSdkForJava15827(httpUploadUrl)) // This strips the port as a workaround described in this issue
  .httpClient(okHttpAsyncHttpClientBuilder.build())
  .buildClient();

This lets me configure the BlobClient with OkHttp and thus use MockWebserver. I'm able to successfully send a request to MockWebserver and have it respond like this:
mockWebServer.enqueue(new MockResponse().setResponseCode(201)
  .setHeaders(Headers.of(
                "ETag", "0x8CB171BA9E94B0B",
                "Last-Modified", "1970-01-01T00:00:00",
                "Content-MD5", "sQqNsWTgdUEFt6mb5y4/5Q==",
                "x-ms-version-id", "1970-01-01T00:00:00"
            ))
  );

These are values taken from https://docs.microsoft.com/en-gb/rest/api/storageservices/put-blob#response.
This seems to make its way to BlockBlobAsyncClient::uploadWithResponse where it creates a SimpleResponse. After which it just hangs. After 60 seconds my test fails with java.util.concurrent.ExecutionException: reactor.core.Exceptions$ReactiveException: java.net.SocketTimeoutException: timeout
For some context the program I'm building using Java8 CompletableFutures so that I can compose various asynchronous tasks. I've set up the Blob upload to look like this:
 private CompletableFuture<UploadRequest> uploadSymbolsChunked(@Nonnull UploadRequest request, @Nonnull String symbolUploadUrl, @Nonnull File file) {
final CompletableFuture<UploadRequest> future = new CompletableFuture<>();

        try {
            factory.createBlobUploadService(symbolUploadUrl).uploadFromFile(file.getPath(), true); // I never move past this point.
            future.complete(request);
        } catch (UncheckedIOException e) {
            future.completeExceptionally(exception);
        }
   return future;
}

Many thanks for the assistance and if you can suggest anything here that would be much appreciated.

@mezpahlan Can you verify that the mockWebServer actually received the outgoing request and sent the response? That will tell us if there's still an issue in getting the request out the door or if something about the mocked response needs to change to allow the response to flow back upstream

This is log from MockWebserver:

Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: --> PUT http://localhost:60583/perry/casey/xiola
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: x-ms-version: 2019-12-12
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: User-Agent: azsdk-java-azure-storage-blob/12.8.0 (1.8.0_252; Linux; 5.4.0-52-generic)
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: Content-Length: 41
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: Date: Tue, 10 Nov 2020 17:07:23 GMT
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: x-ms-client-request-id: 6a4ad2f3-1103-4b90-ba2d-543d5fd8945a
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/octet-stream
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: x-ms-blob-type: BlockBlob
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: --> END PUT
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: <-- 201 OK http://localhost:60583/perry/casey/xiola (3ms)
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: ETag: 0x8CB171BA9E94B0B
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: Last-Modified: 1970-01-01T00:00:00
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: Content-MD5: sQqNsWTgdUEFt6mb5y4/5Q==
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: x-ms-version-id: 1970-01-01T00:00:00
Nov 10, 2020 5:07:23 PM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP

In my tests I configure a JUnit test rule that starts and stops MockWebserver automatically for each test method. Once the rule is running this is how I configure my tests to send to the running instance
@Rule
public MockWebServer mockWebServer = new MockWebServer();

...

@Test
public void testSomethingUsingMockWebserver() {
  // Given
  String urlForTesting = mockWebServer.url("perry/casey/xiola").toString();

  // When
  Result result = foo.doSomethingThatEventuallySendsARequestTo(urlForTesting);

  // Then
  assertThat(result).isAwesome();
}

Ok. So it seems the server is successfully receiving and sending a request/response. So the first problem of properly constructing the URL is resolved. That means something is getting lost as the response returns to the client. Would you be able to crack open the blob dependency and set a breakpoint in common.policy.scrubEtagPolicy#scrubEtagHeader? This is the first piece of code that should get run on receiving a response, so if we can confirm this code is getting hit, then the client is indeed receiving the response.
Alternatively, if you're more comfortable enabling logging in your tests, that could work as well

I can confirm that ScrubEtagPolicy::scrubETagHeader is successfully being hit. For the test above there is effectively no scrubbing being performed because there are no quote marks (") to remove - 0x8CB171BA9E94B0B. But that bit of the code is indeed being executed.

Ok great. So we can confirm the client is actually getting the response. I somewhat suspect that there's probably an issue with the conversion to CompletableFutures. Are you able to run the test using the sync client or blocking directly on Reactor's Mono object rather than using Futures just to try diagnosing that?

Yes, I think you're probably right about the conversion to CompletableFuture. I'm happy to give that a go but now you're stretching my knowledge.

Is there an OkHttpSyncHttpClientBuilder somewhere? I only see the Async one with the azure-core-http-okhttp artefact. If not how do I create one?
Same question with how to block on Reactor's Mono object. Apologies I'm more used to working with RxJava than Reactor and whilst some bits look similar a lot of it does not.

Many thanks for helping again :+1:

No worries! My knowledge on Futures is similarly limited :).

There's only an async builder for the http clients because the sync one uses that anyway. So you really just need to be building a sync blob client and using those apis directly rather than passing them to a completable future.
Blocking should just be a .block call.

Another option here is there's a toFuture method on Mono that I think you can use more safely to convert. That might also be worth a shot

I tried a few of those options but no dice :cry: .
I then started to look closer at the code in OkHttpAsyncHttpClient::getBody and noticed that the stream was never being closed. On a whim I started to google issues to do with empty bodies being returned from MockWebserver and came across this issue: https://github.com/square/okhttp/issues/6281#issuecomment-698667784. Whilst not totally the same I decided to try that workaround so now my MockWebserver response looks like this:
mockWebServer.enqueue(new MockResponse().setResponseCode(201)
  .setHeaders(Headers.of(
    "ETag", "0x8CB171BA9E94B0B",
    "Last-Modified", "1970-01-01T00:00:00",
    "Content-MD5", "sQqNsWTgdUEFt6mb5y4/5Q==",
    "x-ms-version-id", "1970-01-01T00:00:00"
  ))
  .setChunkedBody("", 1) // This line here is the workaround.

When I did that I started to notice this error from the Jackson deserialisation of the headers.
Cannot construct instance of com.azure.core.util.DateTimeRfc1123, problem: Text '1970-01-01T00:00:00' could not be parsed at index 4. So I changed this to Thu, 01 Jan 1970 00:00:00 GMT and reran the code.
Better.... but still we want more!
I then started to get null pointer exceptions in the mapper for BlockBlobAsyncClient::uploadWithResponse(BlockBlobSimpleUploadOptions, Context). What this turned out to be was that I forgot to add the header x-ms-request-server-encrypted. What was interesting about this is that the exception said null cannot be unboxed (or something similar). It was a failure in trying to create a BlockBlobItem:
BlockBlobItem item = new BlockBlobItem(hd.getETag(), hd.getLastModified(), hd.getContentMD5(),
                    hd.isServerEncrypted(), hd.getEncryptionKeySha256(), hd.getEncryptionScope(),
                    hd.getVersionId());

Specifically hd.isServerEncrypted(). Now I know that my usecase is a very hacky one and I totally understand that we've veered wildly off course from the original issue here but I was expecting this to return false if the header was not present. Maybe that's something to look at...... maybe.
Anyways, my MockWebserver response now looks like this:
mockWebServer.enqueue(new MockResponse().setResponseCode(201)
  .setHeaders(Headers.of(
    "ETag", "0x8CB171BA9E94B0B",
    "Last-Modified", "Thu, 01 Jan 1970 00:00:00 GMT",
    "Content-MD5", "sQqNsWTgdUEFt6mb5y4/5Q==",
    "x-ms-request-server-encrypted", "false",
    "x-ms-version-id", "Thu, 01 Jan 1970 00:00:00 GMT"
  ))
  .setChunkedBody("", 1)

And success! It works :smiley_cat: . For completeness, in the end I didn't have to change the way I create the BlobClient - I still use the OkHttpAsyncClientBuilder. I didn't need to modify any of the other CompletableFuture code (although I would love to get rid of it!!).
Many thanks again @rickle-msft, you've been fabulous!

@mezpahlan Thank you so much for the thorough account of your debugging steps! I am sure they will be helpful for others trying to mock out the service! You certainly found some interesting details of this interaction, and I'd say your debugging was what was fabulous as you did pretty much everything yourself :)
Happy to see you're unblocked and let us know if we can be of further support!


Refresh issues when using multiple VSCode windows

Hi. When you open up multiple VSCode windows (I work on many projects at a time), only one of the windows get the new database added to the list when you create the connection. Same applies to refresh. Refresh works in only one of the windows.
The remaining windows keeps showing up the old databases.
PS: I use Project manager extension (https://github.com/alefragnani/vscode-project-manager) to open up projects in multiple windows if that is in any way relevant.
Platform: Ubuntu 19.10
Running VSCode latest build: 1.41.1

Oops! Closing this. Didn't realize that all settings were saved to settings.json file at each and every workspace. Works now. Sorry for the confusion.


List Plugin refactor for tracking for #1291

blocked by: #1291
This PR refactors #1291. atTracking and byTracking is now uses trackingBase. trackingbase is higher order function, which returns a plugin for tracking a field on create or update operation based on config.
Config Options for trackingBase({...}):
fieldConfig: [Required] Field config, similar to field as in list schema
createField: [Required*] Name of the field for create hook
createFieldFunc: [Required*] function to return value for create operation, this function accepts parameters same as resolveInput hook
updateField: [Required*] Name of the field for update hook
updateFieldFunc: [Required*] function to return value for update operation, this function accepts parameters same as resolveInput hook

only one of createField, createFieldFunc or updateField, updateFieldFunc combination is required, both combination can be provided.

Example
this is now possible:
const { uptime } = require('os');
const { trackingBase } = require('@keystone-alpha/list-plugins');
const sysUptimeTracker = trackingBase({ // track system uptime when item is created or updated
  fieldConfig: {
    type: Text,
  },
  createdField: 'uptimeWhenCreated',
  createFieldFunc: () => uptime(),
  updatedField: 'uptimeWhenUpdated',
  updateFieldFunc: () => uptime(),
});
keystone.createList('ListWithPlugin', {
  fields: {
    // ... fields
  },
  plugins: [sysUptimeTracker],
});

@gautamsi I don't understand, why do we want uptime?

nope, it is just an example to showcase power of trackingBase it could be anything for the devs to use. like: hostname at the time of creation and updation if there are multiple containers connecting to same DB. depends on the need of anyone.

@gautamsi I discussed this with you in back channels and have thought about it some more and I'm going to close this issue. I'll add my reasoning here incase anyone else comes across it:
Plugins are the place to provide more "opinionated" starting points for applications. I.e. "We assume you want our standard tracking features and have a User list".
These kind of assumptions are not made in the core infrastructure and therefore we try to make this more configurable and flexible. List plugins are one of the mechanisms we provide for this.
For these reasons, I don't think the kind of configurability suggested here is what we want in 'core' plugins and we definitely think tracking should be one of those.
My preference would be to have many simple plugins over one complex plugin that does it all. I'd strongly recommend that to other plugin developers too but of course, in contrib land that's up to them. :)


NuProj for VS 2017 fails to build in AssignTargetFramework task

Taking over from the comment in #266.
I've installed the just published extension, the vsix manifest information is as follows:

id: NuProj.15
version: <IP_ADDRESS>53

Unfortunately, is not working. At least not for me in 3 different computers (including one where Nuproj for VS2015 used to work perfectly fine). Even if nuproj files open correctly, when building they fail with:
MSB4018	The "AssignTargetFramework" task failed unexpectedly.
System.IO.FileNotFoundException: Could not load file or assembly 'NuGet.Frameworks, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. El sistema no puede encontrar el archivo especificado.
File name: 'NuGet.Frameworks, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
   at NuProj.Tasks.AssignTargetFramework.ConvertToPackageFile(ITaskItem output)
   at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at NuProj.Tasks.AssignTargetFramework.Execute()
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext()

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

And the source points to
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\NuProj\NuProj.targets	376

As per the environment configuration:

2 machines with Windows 10.1 Pro and VS 2017 Professional
1 machine with Windows 10.1 Home and VS 2017 Community

Tomorrow I'll try to take a look with more detail with the fusion log. So far, I'm reporting this so you are aware and can take a look to.

Ok. with the help of the Fusion log I was able to get it working. However I believe is a big and dangerous workaround. What I did was taking some Nuget dlls from their published packages and store them in the nuproj installation path. The packages I had to take those from are:

NuGet.Frameworks
NuGet.Packaging.Core
NuGet.Packaging.Core.Types
NuGet.Packaging
NuGet.Versioning

All of them with version 4.0.0
From the Fusion Log I can see that probed paths were:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Nuproj

Hope this helps you to find a more proper solution. Cheers

I had to uninstall Nuproj for VS2017 extension and it worked fine with CLI.

In order to make the extension work for 2017 (at least for the project that I had) follow these steps:

download source from https://github.com/nuproj/nuproj
open src/NuProj.sln with Visual Studio 2017
remove projects NuProj.ProjectSystem.12, NuProj.ProjectSystem.14 and NuProj.Setup
build the project
right-click on NuProj.ProjectSystem.15 and add a new folder called NuProj
right-click on the NuProj folder and add (use Add as Link!! the little arrow on the Add button in the dialog) existing items from the bin/Debug folder of the NuProj.Tasks project:
NuGet.Frameworks.dll
NuGet.Packaging.dll
NuGet.Packaging.Core.dll
NuGet.Packaging.Core.Types.dll
NuGet.Versioning.dll
select the five files and change:
Install Root to MSBuild
Include in VSIX to True
open Solution items -> version.json and change version from what it is (ex: 0.20-beta) to something larger (0.26) - optional, but it will stop saying you need to update and if you have your local nuget server it will tell you to update the one you have
rebuild (not just build) solution
The resulting .vsix file (src\NuProj.ProjectSystem.15\bin\Debug\NuProj.ProjectSystem.15.vsix) is the extension that allows using .nuproj in VS2017

The packages are somewhat obsolete, but you can right click the solution and managing the nuget packages and selecting them all for update. In this case, if you upgrade NuGet, you will not need  NuGet.Packaging.Core.Types anymore.
This solution gets you the working vsix, but disables completely the batch file system of building and installing. It works only from within VS2017.

Same problem for me.
I have both VS2015 Community AND VS2017 Community installed.
The solution suggested by @dmalanij worked for me: I copied the assemblies only in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Nuproj folder.

WTF why hasn't this been updated in the latest release already?

Ok. with the help of the Fusion log I was able to get it working. However I believe is a big and dangerous workaround. What I did was taking some Nuget dlls from their published packages and store them in the nuproj installation path. The packages I had to take those from are:

NuGet.Frameworks
NuGet.Packaging.Core
NuGet.Packaging.Core.Types
NuGet.Packaging
NuGet.Versioning

All of them with version 4.0.0
From the Fusion Log I can see that probed paths were:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Nuproj

Hope this helps you to find a more proper solution. Cheers

DOES ANYONE HAVE THE CORRECT FILES THAT THEY CAN SHARE?  Here is a detailed step list

Download the nuget packages. Just to avoid you the search in the repository here you have the links:

https://www.nuget.org/packages/NuGet.Packaging/4.0.0
https://www.nuget.org/packages/NuGet.Packaging.Core/4.0.0
https://www.nuget.org/packages/NuGet.Packaging.Core.Types/4.0.0
https://www.nuget.org/packages/NuGet.Frameworks/4.0.0
https://www.nuget.org/packages/NuGet.Versioning/4.0.0

Unpack them
Take the dlls from lib\net45
Place those files in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Nuproj or the equivalent in your Visual Studio install path.

Now, I'm not sure why I'm doing this... you've been really rude in both your posts. If you expect some help from the community you definetely should review your manners. People involved in this kind of open source projects (no matter whether is the developers, contributors or simply users sharing ideas) are doing this by vocation, so the least you could do is show some respect for them. And if you find that the product is not good enough you're totally free to get some other alternative or even build yours.
Cheers

@dmalanij good comment

@dmalanij Thanks for your assistance. Unfortunately, that didn't resolve the issue completely.
If you checked "include source files in pdb", you'll get the error:
The "ReadPdbSourceFiles" task failed unexpectedly
Seems if you run: regsvr32.exe "%VSINSTALLDIR%\Common7\IDE\msdia120.dll It will resolve it.

Unfortunately, I wasn't here when the decision was made to use this.
I apologize for my indiscretion.


Editing the _meta.json not changing the navigation bar

Hello!
I change the _meta but it is not reflected on the top navbar navigation.

Having the same issue


added proof-of-work solutions

@pokhrelanmol  Thanks for Helping the Ethrerum Community

@pokhrelanmol  I will shift your code to the BlockChain Cryptography folder because it is the folder for the 1st week's content. Okay ?

Okay, no problem


Android: Orientation Change Event not firing if Orientation is locked

Hi,
my app orientation is locked on Portrait,
on iOS I can add and receive , orientation change events even when orientation is locked.
But on Android, orientation change events are not firing if orientation is locked.
Can we fire orientation events event if the orientation is locked?

+1

@aschwizzy710 @facuescobar currently the package doesn't support this. If orientation is locked, it doesn't fire on Android. It's a known caveat and PRs are highly encouraged.

So I do use an orientation lock, but it is on a specific component and I unlock the orientation in componentWillUnmount() on that component which is locked.
Is that the cause for the orientation being permanently locked on Android? @andrerfneves

Try unlockAllOrientations() as well as lockLandscape() in componentWillMount of the next view. That way it will unlock and re-lock it correctly. Maybe the willUnmount call is not affecting it.

Well, I am not trying to lock it to landscape. I just want it to rotate to landscape when the phone orientation is changed and change back freely when it is changed back to portrait. I will try unlockAllOrientations in the componentWillMount of the next view.

Oh, if all you want is the ability to free up the view to move portrait/landscape, then just make sure your native Android code allows for that, and run a unlockAllOrientations() in componentWillMount of the view. As a side, in order to control my application, I explicitly lockPortrait() in my entry-point View, and only unlock them for views that need landscape access as well.

Excellent - it worked! I appreciate your feedback.

@aschwizzy710 glad it worked. Closing this for now.

Hey @andrerfneves - how do you go about calling lockPortrait()  in your entry point view (to control the app as you mentioned) if you need that view to be landscape responsive as well?

In that case just unlockAllOrientations at your entry point and it should behave accordingly.

Thanks!


Runtime estimation

When I'm training a model using keras, I always calculate the runtime by hand (Using the estimated time per epoch)
It would be great if when I use mlnotify the estimated runtime will appear too (next to the time since the start of training)
You can do that using the ETA provided by keras times the number of epochs.

@yoni12345678
Hey, it sounds great, however I'm not sure it fits all scenarios since not all ml libraries have this feature and the manual API certainly can't get it.
Do you think we should implement a best-effort estimation?

In my experience with TensorFlow, the time estimation is pretty accurate, so I think best effort estimation should give accurate results.

Would love to have this feature. Another use case example:
Let's say we need to run a long for loop for some purpose. Other than just knowing it is completed, it is also good to know the expected ETA. The 'tqdm' library does provide the second functionality of displaying the ETA but does not notice as we have in MLNotify. So a merger of these functionalities will be great.
Yeah, there is no ML happening here, but will be useful for a general Python programmer too... If possible it can be included in the manual API.
One API usage example would be
a_long_list = [...]
mlnotify.start(total=len(a_long_list))

for i in a_long_list:
    # do something time consuming
     mlnotify.update(1)   # update by 1 after every iteration

mlnotify.end()

If there is already some library supporting such a feature, please do let me know.


Documentation includes mention of deleted token salting

The documentation includes a setting for SALT_LENGTH even though salting was removed long ago.
Would be great to remove this reference as it suggests that salting of tokens is still used!
I searched the head of the develop branch and could not find a mention of this setting except in the changelog and migrations so I'm not sure why it's showing up in the docs!
See https://jazzband.github.io/django-rest-knox/settings/.

This most likely comes from the fact that the gh-pages branch is outdated ... https://github.com/jazzband/django-rest-knox/blob/gh-pages/settings/index.html.
I will mention this in https://github.com/jazzband/django-rest-knox/issues/304

@johnraz @dgerlanc We now have a Ci job that automatically deploys the latest documentation to GH-pages.
https://github.com/jazzband/django-rest-knox/pull/343

@giovannicimolin wonderful doc is up to date now! Closing this one


[Bug]: zmq.error.ZMQError: Can't assign requested address error trying NetGear+WebGear example

Brief Description
I have a Raspberry Pi400 (using latest 64bit Bullseye release) with a UVC USB3 camera attached. On the same local network I have a Macbook Pro that I want to use to connect to the RPi and display the camera output. Speaking to @abhiTronix on chat, he recommended the following code: https://abhitronix.github.io/vidgear/latest/help/netgear_ex/#using-netgear-with-webgear
However, the client code crashes when trying to set up ZMQ connection:
zmq.error.ZMQError: Can't assign requested address
RuntimeError: [NetGear:ERROR] :: Receive Mode failed to bind address: tcp://<IP_ADDRESS>:5454 and pattern: 1! Kindly recheck all parameters.

I've tested the ZMQ hello world test code and it works fine on the same machines. I also tested uvicorn to run a simple server and it works. I have reinstalled zmq (brew reinstall zmq) on the client machine, and using venv have reinstalled all pip packages from scratch. Python is V3.10.2 on the client and V3.9.2 on the server.
The code (slightly modified to fix a couple of errors I got from Python (no self, cannot return a value in async) and error logs are found at this gist:
https://gist.github.com/iandol/ebf722f75251bb9f5bc57ba967fd12ba
Acknowledgment

[X] I have searched the issues for my issue and found nothing related or helpful.
[X] I have read the Documentation.
[X] I have read the Issue Guidelines.

Environment

VidGear version: 0.25* Branch:
Python version: 3.10.2
PiP version: 22.0.3
Operating System and version: macOS 12.2.1

Expected Behavior
See the video from RPi in the Macbook browser.
Actual Behavior
client.py crashes
Steps to reproduce
(Write your steps here:)

Clean 64bit bullseye OS, create a new venv on the RPi, install vidgear[core] and deps to run Netgear and CamGear. Add server.py taken from example code. Use IP taken from RPi.
Create venv on macOS, install vidgear[asyncio] and any other Netgear and Webgear deps.
Add client.py on macOS set up to correct IP of RPi.
Run server.py on RPi and cleint.py on macOS.

@iandol what's the IP address on both mac and Rpi? Follow following instructions and paste the results here:

Open a Terminal app on each machine.
Type and run /sbin/ifconfig this will list your IP address as well your MAC address.
You can use cmd+a to select all, cmd+c to copy and it can be pasted to whatever you want.

My RPi is assigned <IP_ADDRESS> and my macbook is <IP_ADDRESS>. I can ping <IP_ADDRESS> from the macbook OK, and as I said I tried to run the zmq hello world server/client, and I've also tried lazy pirate from the zmq docs using the same <IP_ADDRESS>:5454 and it works fine; code here: https://gist.github.com/iandol/a0cd594d790230435d4f3d343830da9f
RPi:
 ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether dc:a6:32:fd:fb:d1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet <IP_ADDRESS>  netmask <IP_ADDRESS>
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 50634  bytes 2534509 (2.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 50634  bytes 2534509 (2.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet <IP_ADDRESS>  netmask <IP_ADDRESS>  broadcast <IP_ADDRESS>
        inet6 fe80::6929:d8dd:27f8:3367  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:fd:fb:d5  txqueuelen 1000  (Ethernet)
        RX packets 85726  bytes 27307320 (26.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 67060  bytes 37502674 (35.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

macbook:
gears on  main [!?⇡] via 🐍 v3.10.2 (vidgear) took 5s
❯ ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
	inet <IP_ADDRESS> netmask 0xff000000
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether ac:bc:32:8c:08:39
	inet6 fe80::1cd9:3c9c:20ff:6cc2%en0 prefixlen 64 secured scopeid 0x4
	inet <IP_ADDRESS> netmask 0xffffff00 broadcast <IP_ADDRESS>
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:17:03:a6:48:00
	media: autoselect <full-duplex>
	status: inactive
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:17:03:a6:48:01
	media: autoselect <full-duplex>
	status: inactive
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=63<RXCSUM,TXCSUM,TSO4,TSO6>
	ether 82:17:03:a6:48:00
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x0
	member: en1 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 5 priority 0 path cost 0
	member: en2 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 6 priority 0 path cost 0
	nd6 options=201<PERFORMNUD,DAD>
	media: <unknown type>
	status: inactive
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
	options=400<CHANNEL_IO>
	ether 0e:bc:32:8c:08:39
	media: autoselect
	status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
	options=400<CHANNEL_IO>
	ether 3e:47:26:cc:db:ba
	inet6 fe80::3c47:26ff:fecc:dbba%awdl0 prefixlen 64 scopeid 0x9
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
llw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether 3e:47:26:cc:db:ba
	inet6 fe80::3c47:26ff:fecc:dbba%llw0 prefixlen 64 scopeid 0xa
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
feth6702: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 2800
	ether 66:65:74:68:1a:2e
	peer: feth1702
	media: autoselect
	status: active
feth1702: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 5000 mtu 2800
	ether de:6c:d8:46:89:a2
	inet <IP_ADDRESS> netmask 0xffff0000 broadcast <IP_ADDRESS>
	inet6 fe80::dc6c:d8ff:fe46:89a2%feth1702 prefixlen 64 scopeid 0xc
	inet6 fda0:9acf:233:27ef:de99:9383:ff75:8b6d prefixlen 88
	peer: feth6702
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::1261:2fb:2ba2:d54f%utun0 prefixlen 64 scopeid 0xd
	nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
	inet6 fe80::6ff3:ef9:a481:9c4f%utun1 prefixlen 64 scopeid 0xe
	nd6 options=201<PERFORMNUD,DAD>
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1000
	inet6 fe80::ce81:b1c:bd2c:69e%utun2 prefixlen 64 scopeid 0xf
	nd6 options=201<PERFORMNUD,DAD>
utun3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::9383:904d:c951:598e%utun3 prefixlen 64 scopeid 0x10
	nd6 options=201<PERFORMNUD,DAD>
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::3cd1:8cff:700d:52ef%utun4 prefixlen 64 scopeid 0x12
	nd6 options=201<PERFORMNUD,DAD>

On the RPi and macbook I also checked that nothing was holding onto port 5454, here is the result for the RPi:
❯ sudo lsof -i TCP -P
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
cupsd      623 root    6u  IPv6  13676      0t0  TCP localhost:631 (LISTEN)
cupsd      623 root    7u  IPv4  13677      0t0  TCP localhost:631 (LISTEN)
vncserver  645 root   11u  IPv6  14636      0t0  TCP *:5900 (LISTEN)
vncserver  645 root   12u  IPv4  14637      0t0  TCP *:5900 (LISTEN)
vncserver  645 root   16u  IPv4  15003      0t0  TCP localhost:43434->localhost:39405 (ESTABLISHED)
vncserver  645 root   17u  IPv4  86119      0t0  TCP <IP_ADDRESS>:5900-><IP_ADDRESS>:49564 (ESTABLISHED)
sshd       662 root    3u  IPv4  14515      0t0  TCP *:22 (LISTEN)
sshd       662 root    4u  IPv6  14517      0t0  TCP *:22 (LISTEN)
vncserver 1009   pi    8u  IPv4  15004      0t0  TCP localhost:39405->localhost:43434 (ESTABLISHED)
chromium- 1734   pi   31u  IPv4  20776      0t0  TCP <IP_ADDRESS>:59132->stackoverflow.com:443 (ESTABLISHED)

Add client.py on macOS set up to correct IP of RPi.
My RPi is assigned <IP_ADDRESS> and my macbook is <IP_ADDRESS>

@iandol This pretty obvious that you're doing it wrong. It mentioned everywhere in NetGear docs(see any example) as well in example links I provided you for reference, that you need to "Replace the IP addresses in the code with Client's IP address, i.e. <IP_ADDRESS> in your case.". Kindly read docs properly before raising an issue here.

Yikes, you are completely correct of course. I'm very sorry. My brain made a strong assumption in a client + server system it was the server address that must be used...
As a slight emphasis for others who may make such an misguided assumption, perhaps the first comment on the example page could be slightly modified:

Note down the local IP-address of the client system; it is important to understand that the IP address of the client must be used on client and server. Replace this IP in the following code.

There were 3 problems I encountered with the client code. There is a missing import for Netgear, a wrong self, and apparently return cannot return any value inside an async:

@iandol I think all the instructions are already self-explanatory, no need to emphasis on (client) keyword in example. However, You can surely make a new FAQ for NetGear https://abhitronix.github.io/vidgear/latest/help/netgear_faqs/ explaining the error and what one must do to avoid it. A PR would be great, but read the guidelines carefully: https://abhitronix.github.io/vidgear/latest/contribution/PR/


Dont' Merge ORM FTAs Test for new elinks

Jira link (if applicable)
AM-2917 & RDCC-7148
Change description
ORM Judicial FTAs to test V2 elinks.
For testing purpose only!
Will delete this PR after testing the RD elinks changes.
Checklist

[Y] Does this PR introduce a breaking change

This test PR is no longer required.


sumneko_lua is deprecated in nvim-lspconfig, but still used in navigator.lua

Hey,
Took me a while to look into the errors I see about the now deprecated sumneko_lua (renamed to lua_ls) and see that the errors are actually happening because I have navigator.lua enabled
I think the reason is that navigator.lua is including sumneko_lua.lua
Really appreciate if the issue is fixed 🙂

thanks, it was fixed

Thank you for the quick fix @ray-x


Added: The Facility Group Id in SystemMessage entity

Added: The Facility Group Id in SystemMessage entity

The changes regarding SMUS Feeds to exclude the ORSI_FAC facility group type orders are not necessary. As discussed with @gurveen.bagga, we've already incorporated the FacilityGroupMember list in the Brokered Order Order Items JSON. Therefore, we can address this scenario in the WH Brokered Order Items Feed Spec. Let's close this MR.


Exception during save data to firestore from cloud function

[READ] Step 1: Are you in the right place?

For issues or feature requests related to the code in this repository
file a Github issue.

If this is a feature request make sure the issue title starts with "FR:".

For general technical questions, post a question on StackOverflow
with the firebase tag.
For general Firebase discussion, use the firebase-talk
google group.
For help troubleshooting your application that does not fall under one
of the above categories, reach out to the personalized
Firebase support channel.

[REQUIRED] Step 2: Describe your environment

Operating System version: Cloud functions
Firebase SDK version: 5.5.1
Library version: 5.5.1
Firebase Product: cloud functions, firestore (auth, database, storage, etc)

[REQUIRED] Step 3: Describe the problem
Steps to reproduce:
What happened? How can we make the problem occur?
This could be a description, log/console output, etc.
From time to time i see an exceptions into firestore cloud functions log

Error: Stream removed
at ClientReadableStream._emitStatusIfDone (/user_code/node_modules/firebase-admin/node_modules/grpc/src/client.js:255:19)
at ClientReadableStream._receiveStatus (/user_code/node_modules/firebase-admin/node_modules/grpc/src/client.js:233:8)
at /user_code/node_modules/firebase-admin/node_modules/grpc/src/client.js:705:12

Error: Stream removed
at /user_code/node_modules/firebase-admin/node_modules/grpc/src/client.js:554:15
code: 2,
metadata: Metadata { _internal_repr: {} },
note: 'Exception occurred in retry method that was not classified as transient'

Firestore and Functions teams have been investigating this. See https://groups.google.com/forum/#!msg/firebase-talk/oza-JZMtXlo/DpFGz9BHCQAJ for the ongoing discussion. Stay tuned for an update.