GUI for AutoSubmit v1

In GitLab by @dbeltrankyl on Jan 3, 2019, 16:46
@mcastril
Hi there,
I am developing a GUI for autosubmit as you can see in autosubmit@3d4163eea9dc3aff5208b32ab541986d34ca9401.
It will be a separated executable which will be called autosubmit-GUI.
It will be a slow process because it is the first time that I develop any GUI application related, for now, I just committed a version (in an isolated branch) that looks awful but at least I made it work (except grouping/expand) with autosubmit create.
As related to the development strategy:

Create a functional ugly GUI that tries to be intuitive.
Make compatible all the actual console autosubmit functionalities.
Once everything works as well as console autosubmit, renew the ugly GUI for a beautiful one as much as I can.

As for the libraries, for now, I am using PySide2(free for commercial use) <= 5.11 because I have trouble making it work with 5.12 due to having an old version of GCC (4.8).
Of course, all of this is open to discussion.
Edit 1: The code will be a bit mess up until I learn more PySide(QT) since I'm testing different approaches to archive what I want
offtopic: Git buttons for list, bold, etc does not work with chrome.
To do:

[x]  Detailed analysis of the grouping function of Autosubmit.
[x]  Detailed analysis of wrapping in Autosubmit.
[x]  Improve wrapping documentation in Autosubmit.
[x]  Improve grouping function in Autosubmit (and in Autosubmit GUI as a consequence)
[x]  Implement Clustering (a vis-network feature) as a way to represent grouping in the graph.
[x]  Improve justification for wide workflows, so parents are in the middle of their children and dependency lines are not so horizontal.
[x]  Add ungroup button in the group menu.
[x]  Implement graph positioning algorithm for large experiments.
[x]  Implement Live Monitor for Tree View.
[x]  Implement Live Monitor wrapper updates for Graph visualization.

To do (Beta Priority):

[x] Show the start date and the date (start date + crunch * chunk size)
[x] Put the description of the experiment next to the name.
[x] The navigation per status buttons should point to the first job with that status, and not to the last one as it is happening now.
[x] In the description panel (right box): Show the list of currently waiting jobs.
[x] In the description panel (right box): Show chunk number.
[x] In the description panel (right box): Show member name.
[x] Remove model and pkl from the top description.
[x] The name of the Branch and HPC should be bigger.
[x] The log should be scrollable.
[x] When switching to the Tree View node, the Tree should be rendered automatically. If the Tree is already rendered and the experiment is running, the Tree View data should be refreshed automatically.
[x] Gather information about the statistics fields in the result and display that information in the statistics tab.
[x] Show average queuing time for failed jobs in the experiment.
[x] Show average running time for jobs (types) in the experiment.
[x] For big experiments, the Graph visualization should start zooming into a relevant section of the graph instead of showing the whole render.

Priority:

[x] Show performance metrics
[x] Provide set status commands through the GUI
[x] Add specific URL for GUI functionality

Architecture discussion on autosubmit#522 for future reference.

In GitLab by @dbeltrankyl on Jan 4, 2019, 17:00
mentioned in commit autosubmit@e0491a6ebd6ddd34ce0065ffd246bcdabe050e9c

In GitLab by @dbeltrankyl on Jan 15, 2019, 10:16
Hi there,
ECEFLOW is based in QT5 (pyqt (GPL)) , for now I am using pyside2 which is also based in QT5(LGPL).
Cylc: Uses GTK+.
Nowadays, there is not much difference from using one or the other. However, the internet most popular opinion is that QT is the future since it is more active than GTK+.
Since we're aiming also to have a web interface, I saw that qt has webengine which allows to communicate with javascript and html. Also it has a mechanism to combine javascript,html with QT and also a way to expose QT api for be accesible by remote clients.
With that in mind , I think whatever library based in QT5 is complete enough.

In GitLab by @dbeltrankyl on Jan 9, 2019, 13:20
mentioned in commit autosubmit@df821aa7b2c59dba08440a79f5d09c80010ce105

In GitLab by @dbeltrankyl on Jan 9, 2019, 13:20
mentioned in commit autosubmit@e927e4735e68c713b0b62236702132aa724400f7

In GitLab by @mcastril on Jan 4, 2019, 16:40
Thanks @dbeltrankyl for the initiative, this is a long term task but it is good if we can with a tentative approach.
I suggest you to check what libraries other workflow managers are using. I think they are also based in QT:
ECFLOW: https://confluence.ecmwf.int/display/ECFLOW
Cylc (Rose): https://cylc.github.io/cylc/
PS: About problems with the new GitLab, please report them to https://earth.bsc.es/gitlab/es/requests/issues/513

In GitLab by @dbeltrankyl on Jan 4, 2019, 17:00
mentioned in commit autosubmit@15d0fa9656c80216d0d18fe4ee9a53b18f02b317

In GitLab by @dbeltrankyl on Jan 24, 2019, 10:40
removed milestone

In GitLab by @dbeltrankyl on Jul 12, 2019, 12:13
unassigned @wuruchi

In GitLab by @mcastril on Jul 12, 2019, 15:57
Hi Wilmer,
Please note that the Autosubmit status could be retrieved directly from the pkl file, that is being updated in Autosubmit run.
The only "disadvantage" is that you have to generate yourself the graph, but this was one of the things to improve in the WebApp, because Autosubmit is not so efficient doing this in large workflows. Additionally, take into account that in the WebApp you would only need to generate the plot on start, and then update the jobs that are changing their status, which is much more efficient that performing multiple autosubmit monitor.
I am not saying that we will discard the web services (or microservices) option but in the beginning we can live without a server and only develop it if we really need it.
So maybe the first task is to figure out how to generate a plot from the pkl structure in Web environment and in an efficient way.
Of course we will need access from the Web Server to /esarchive, but we also would need it for the web services. This is something that I will try to manage.

In GitLab by @wuruchi on Jul 12, 2019, 14:50
While I am still looking at the general processes of Autosubmit, I have an idea that you can consider in the meantime until further meetings are held and something more concrete is developed:
The communication between Autosubmit and our WebApp will be handled with web services.
These web services (RESTful resource) will be in charge of executing the shell scripts necessary to retrieve information from Autosubmit (e.g autosubmit monitor a22l), to capture the output, and return it. In the case of monitor, a simplified monitor should be implemented that does not return a pdf, but a plain text file with enough information to rebuild the graph in the front end. The RESTful resource exposes the necessary operations.
Our WebApp will consume web services and show relevant information. Bootstrap + Any web framework you prefer (ReactJS seems to be quite effective nowadays), should do the job in the front end.
The main objective is to avoid changing Autosubmit as much as possible.

In GitLab by @mcastril on Jul 12, 2019, 13:15
So to recap, the idea would be to create a WebApp that could maybe later be visualized in a separate Desktop application.
An important aspect to discuss will be how to perform the communication between the GUI and Autosubmit. I was initially thinking on a server, but we can maybe keep it simple:

To display the workflow maybe it's enough to read the updated pkl file.
To interact with the workflow and change job status we can maybe use the updated_list ( https://autosubmit.readthedocs.io/en/latest/troubleshoot.html#how-to-change-the-job-status-without-stopping-autosubmit ).

In GitLab by @mcastril on Jul 22, 2019, 12:21
In the plans for XIOS I/O server (used by NEMO to output data asynchronously) I saw they are considering to use this library as visual backend. Maybe it's an option to consider:
https://visjs.org/

In GitLab by @mcastril on Jan 15, 2019, 11:00
Thanks @dbeltrankyl , from what I've read I also thought everything is moving to QT.
A Web interface would be more versatile because obviously you could access your experiments without installing a client from everywhere.
In that sense, it seems promising what you say about the web capabilities. With the WebEngine it seems feasible to create a Desktop application that only visualizes the Web application, so we only would need to develop a WebApp that can be visualized both in the browser or the desktop QT app.

The Qt WebEngine module provides a web browser engine that makes it easy to embed content from the World Wide Web into your Qt application on platforms that do not have a native web engine.

For the communication between QT and HTML/JavaScript I think we would need the QtWebChannel (although maybe not, if you operate directly with the HTML content inside the WebView.

The Qt WebChannel module enables HTML or JavaScript clients to access Qt APIs such as QObject.

In any case, if we focus on a Web Application we would obviously have to develop also a server, which is not a requirement for a Desktop-only application. For an experiment viewer we maybe would only need a DataBase viewer, but we want a GUI that interacts with Autosubmit and the workflow. So in that case a proper server is needed.
So I see two options here:

Desktop app (QT5)
WebApp (HTML+JavaScript, could be AngularJS, ReactJS...) + Server (could be Python) [ + Desktop QTWebEngine ]

In GitLab by @wuruchi on Jul 22, 2019, 13:07
That's a cool library. I am sure it will be useful at some point in the presentation layer. Thank you.
I will start to build the REST endpoints from which our GUI will get its information. These endpoints will live in the autosubmit project, since it will make use of its classes and the functionality that will be added in the future. Or perhaps it should be in another project, I am analyzing these possibilities.

In GitLab by @wuruchi on Jul 24, 2019, 13:15
Should be working now, I was making a test with a huge experiment but it killed the development server. In a real life scenario only the instance that is being served would be killed.

In GitLab by @wuruchi on Jul 30, 2019, 12:15
Hi @dbeltrankyl @mcastril
A somewhat important milestone for the GUI project. I have implemented an interface using ReactJS and vis.js that renders the dependency hierarchy of an experiment. You can test it here:
http://<IP_ADDRESS>:3000/
The expected input is the expid
The next step is to make it also render the color according to the current status of the job. Then, that it refreshes automatically.
@pablohe I think you mentioned that you wanted to test any progress.
UPDATE: Now the graph nodes have their color according to their current status.

In GitLab by @wuruchi on Jul 24, 2019, 11:36
A somewhat important update:
Currently, there is one endpoint running at http://<IP_ADDRESS>:5002/monitor/a239 where you can replace a239 for the expid of the experiment you want to monitor (some expid can fail due to reading permissions for my user on some directories). This endpoint returns a JSON representation of the job dependency hierarchy tree, it is running on my PC and published in the local network. Try to access it and test it, also tell me what other important information about jobs we can include.
Now I will start working on the front-end (using ReactJS) that will consume this endpoint and render a user-friendly representation of the JSON data.

In GitLab by @wuruchi on Aug 2, 2019, 16:20
Hi team,
REST Endpoints for Autosubmit have been created:

http://<IP_ADDRESS>:8888/expinfo/{expid} where {expid} is a variable. It returns a JSON object with data from the experiment.
http://<IP_ADDRESS>:8888/search/{var} where {var} can be a expid or part of the description of the experiment. It returns a list of JSON objects with data from the experiments that coincides with the value sent.
http://<IP_ADDRESS>:8888/graph/{expid}  Returns a graph representation of the experiment as a JSON object.
http://<IP_ADDRESS>:8888/monitor/{expid}  Same as the last one but as a tree representation.
With these endpoints, it was possible to create a graph representation of the experiment and the current status of the jobs using the network component of the library https://visjs.org/. However, it needs a little bit more polishing. http://<IP_ADDRESS>:3000/ still serves the first version of the graph interface.

An Autosubmit experiment searcher has been implemented for testing: http://<IP_ADDRESS>:3001/ The graph representation of the experiment’s jobs and status will be included in this WebApp in the short term future as part of the first iteration of the GUI for Autosubmit.
@pablohe

In GitLab by @dbeltrankyl on Jul 24, 2019, 11:51
Hi @wuruchi ,
I'm working from home and it seems that I can't access to that IP via the VPN, So I'll check it tomorrow.

In GitLab by @wuruchi on Jul 24, 2019, 13:00
Should be working again.

In GitLab by @mcastril on Jul 24, 2019, 13:10
Very exciting to read that @wuruchi
I am trying to test it but it keeps on trying to connect

In GitLab by @wuruchi on Jul 24, 2019, 12:27
No problem.
Anyway, my computer was dead for almost one hour, I will try to bring the endpoint back online ASAP.

In GitLab by @mcastril on Jul 24, 2019, 13:18
Great! it works now

In GitLab by @wuruchi on Sep 16, 2019, 12:35
On Friday Kim said that the request had been sent, no updates since then.

In GitLab by @wuruchi on Sep 16, 2019, 12:11
Hello @mcastril @dbeltrankyl @pabretonniere
Autosubmit GUI is back to its old home, al least for now:
http://<IP_ADDRESS>:3000/

In GitLab by @mcastril on Aug 14, 2019, 18:22
Very good news @wuruchi ! Seems really promising. I am trying it through the VPN.
If you want a massive experiment to test, this is it: http://<IP_ADDRESS>:3001/experiment/a251
A strategy to do the update in a seamless way could be to generate another canvas in the background and play with the visibility of the layers, but this is an optimization task. I think we can at this stage focus on 90% of the experiments, that will have reasonable size.

In GitLab by @wuruchi on Aug 14, 2019, 16:52
Hi @mcastril @dbeltrankyl
Some refactoring in the code has been performed to allow for better performance.
Navigating through the experiments and generating the graphs should be faster now, for example http://<IP_ADDRESS>:3001/experiment/a1de (4812 nodes, 6006 edges) took around 10 minutes in the previous version. Now it takes about 20 seconds, around 10 for data retrieval and around 10 more for rendering. This is the heaviest experiment I know, so feel free to try with other bigger experiments if there is any·
Furthermore, now you will see at the bottom of the page that the app now captures the IDs of the nodes you select in a controlled way, so we can use this data to implement further functionality, for example: showing job details.
The project now lives in https://earth.bsc.es/gitlab/wuruchi/autosubmitreact.
PD: Also, since graph rendering has been improved, seamlessly and automatically updating the colors of the nodes seems more feasible now.
@pablohe

In GitLab by @wuruchi on Aug 21, 2019, 17:04
Hello @mcastril @dbeltrankyl
The development server has moved to http://<IP_ADDRESS>:3000/.
Press on "Start Job Monitor" in the Experiment page to start the automatic update of Jobs in the Graph. This option will only be visible if the experiment is currently running.
@pabretonniere

In GitLab by @wuruchi on Aug 12, 2019, 15:16
Hello @mcastril @dbeltrankyl
Some progress has been achieved on the GUI: http://<IP_ADDRESS>:3001/

The GUI now also includes the Graph Visualization of the experiment using vis.js. Seems to be working fine with experiments up to 4000 jobs. The toughest one I have tested it is a1de that yielded a result after about 10 - 15 min.
The Graph shows colors according to the state.
The experiment data include the contents of the folder in esarchive. However, only a top 10 is shown to avoid saturation of the interface.

The next step is to update the graph's nodes according to the changes of status of the jobs, and also tell the user what has been changed as text output on the interface.

In GitLab by @mcastril on Aug 12, 2019, 15:24
Nice job @wuruchi ! And I see you can click on the nodes and they are marked, so I guess you will be able to capture the node id to perform an action.
About the performance we will find out a way to improve it for large experiments. Maybe in the future you can do something similar to the grouping done in the monitor (and group-ungroup nodes using clicks). But it is not a priority right now.

In GitLab by @mcastril on Sep 16, 2019, 12:29
Hi @wuruchi . It is still not accessible from the new host? Did you answer Kim about the 8081 port?

In GitLab by @wuruchi on Aug 12, 2019, 17:11
Thank you @mcastril, @dbeltrankyl
As for capturing the data from the graph: Yes, there is a select event that captures the node that is selected and the edge. I have added multi-select (using CTRL-click), and navigation buttons at the bottom of the Graph. Also, I will be working on a panel that shows the names of the nodes currently selected.
Regarding the display of the hierarchical view: After going through the documentation in more detail I found some options that should have improved the sequence of jobs. Please take a look @dbeltrankyl
All these updates are reflected in http://<IP_ADDRESS>:3001/
This one generates a cool graph: http://<IP_ADDRESS>:3001/experiment/a21g

In GitLab by @dbeltrankyl on Aug 12, 2019, 15:38
Hi @wuruchi ,
First great work you're progressing so fast,
I noticed that the graph is a bit weird in one of my experiments:
It seems that there is a bug in your graph representation since it is not acknowledging the 000_2_SIM job in it correspondent level, maybe it is due to the way you're traveling for the graph since your graph seem that is traveling with DFS(deep) instead of BFS(Breadth)

In GitLab by @mcastril on Sep 19, 2019, 09:57
Congratulations @wuruchi and thanks for the update.

In GitLab by @wuruchi on Sep 16, 2019, 15:59
For some reason, it was down. It is back up now.

In GitLab by @pabretonniere on Sep 16, 2019, 16:00
Thanks. I can now load the page.

In GitLab by @wuruchi on Sep 23, 2019, 15:02
Hello team,
@mcastril @dbeltrankyl
Please let me know if http://<IP_ADDRESS>:3000/ is working correctly.
In other news, I will start with the grouping implementation in my local project. I think group by "date, section, chunk, and (perhaps) automatic" is a good functionality to start working on. I hope there is something to show on Friday.
Oh, by the way, seems like there is something wrong with experiment a1wo. Please let me know if you know anything about that experiment.

In GitLab by @wuruchi on Sep 23, 2019, 15:10
The plain text graph representation that is generated by the API triggers a recursion loop in the vis-network algorithm, which could indicate that there is some inappropriate value in the job list of this experiment (a1wo), so far I haven't found anything unusual, and I haven't looked into the vis-network cod. Also, I haven't tried the monitor function of autosubmit since this experiment has more than 12k jobs.
I will keep looking into it.

In GitLab by @wuruchi on Sep 19, 2019, 09:41
Hello team,
@dbeltrankyl @mcastril @kserradell @pabretonniere
A somewhat important milestone:
Autosubmit API, the one that does the heavy lifting, is now running on bscesweb04, it is running on 4 threads, meaning that there are actually 4 instances of Autosubmit API that are listening for requests at all times and this will substantially increase performance. As of now, the web app is sending its requests to this API. That should reflect into increased response time, we will see.
I have verified that all the available requests are working as intended but report any bug here, please.
Next step is to also have the GUI running on the server, we will let you know when that happens. URL hasn't changed, yet.

In GitLab by @dbeltrankyl on Sep 27, 2019, 15:59
Hi @wuruchi ,
Great news, the new home is working for me in 2 experiments no problems so far.
I tested a bit the grouping, and is working on 2/3 experiments fine (same output than in master branch)  a28k gives an error with package_table like you mentioned in previous comments however without the options of expand or expand_status I can't really test much so I'll be waiting to the next version.
One thing that I noticed is while the searching of experiment I can barely see the previous searches that I did (attaching photo)

In GitLab by @wuruchi on Sep 30, 2019, 09:20
Hello @mcastril
As you mention, there is still some more work to do regarding the graph representation. Some of the problems you mentioned are related to the limitations of the library we are using since it tries to be a general tool for any graph; meanwhile, our experiments are a particular graph instance of this universe. There is so much configuration can do as a general approach, meaning that we can configure only that much the options provided by the vis-network API; however, there are still some approaches to try, and I think we can extend the functionality of this library to a satisfactory state.

In GitLab by @wuruchi on Oct 7, 2019, 15:57
marked the checklist item Implement Clustering (a vis-network feature) as a way to represent grouping in the graph. as completed

In GitLab by @wuruchi on Oct 7, 2019, 15:57
marked the checklist item Implement Clustering (a vis-network feature) as a way to represent grouping in the graph. as incomplete

In GitLab by @wuruchi on Oct 1, 2019, 09:02
marked the checklist item Detailed analysis of the grouping function of Autosubmit. as incomplete

In GitLab by @wuruchi on Oct 7, 2019, 12:02
marked the checklist item Detailed analysis of the grouping function of Autosubmit. as completed

In GitLab by @wuruchi on Oct 1, 2019, 09:02
marked the checklist item Detailed analysis of the grouping function of Autosubmit. as completed

In GitLab by @mcastril on Sep 30, 2019, 09:39
Thanks @wuruchi . I understand. I think we must keep in mind the limitations of this library and try to stick to changing only configuration parameters before doing more complex things as adding new functions and so on. Then, when we know all the limitations and things that can't be satisfied, we can decide whether extending the functionality of this library or evaluating the usage of another one.
We will add then suggestions or pending fixes as checkboxes at the top (in the description of the issue). We will also add there other eventual actions from the meeting with Paco. I would ask you then to mark the things that are done, and for the undone comment if it's straight forward or not to do it (and estimated effort).

In GitLab by @wuruchi on Sep 30, 2019, 12:06
Hi @mcastril
Speaking of that, this library has a nice clustering function that we can use for grouping. However, to implement it effectively I will have to analyze in detail the rules that apply in the grouping function of Autosubmit. This is related to what I told @dbeltrankyl before about making a list of the code snippets related to this function that are too difficult.
So, I think it is best to start with the complete analysis of this function, which also includes wrappers and synchronization.
The current implementation of grouping seems to work well enough for most cases so perhaps it is time to tell our beta users to try the app and give us feedback. What do you think?

In GitLab by @wuruchi on Oct 7, 2019, 16:19
Hello team.
@dbeltrankyl @mcastril
Now that the mysteries of grouping have been unraveled, it is time to think about how to improve this useful function so it is better understood by our users, and more widely used as a consequence. Perhaps we should open a new issue for this purpose or use the one that I already opened? Anyway, the bottom line is that the way grouping is handled now is not very efficient, I mean, the graph is made smaller but the pre-processing is still the same as if you were calling the whole experiment. Also, some relations are lost or oversimplified in the process given the complexity of the dependencies of some experiments. I think a better approach could be clustering, where jobs that belong to the same chunk, member, or date, are clustered together but jobs with many dependencies, e.g. asynchronous jobs, are left out of these clusters, so the workflow of the experiment is preserved. For that purpose, automatic grouping should disappear and some general rules established when grouping by date, member, or chunk; so there is no necessity to include further filters.
Another task is to perform a similar analysis of the wrapping function and improve its documentation. Should I open a new issue for this purpose as I did with "Grouping"?
The experiment tree view is currently under development, a first implementation should be up this week.

In GitLab by @mcastril on Sep 30, 2019, 17:34
Hi, I think it is important to fully understand what Autosubmit group does in order to use it in the GUI. I think this functionality is underused in Autosubmit, partially because lack of knowledge about its potential, partially because of the difficulty of applying it through commands. A GUI always facilitates this work.
About the Beta, we should meet Paco first. As we gonna meet tomorrow, after the meeting we can decide if doing the publication then or do some changes in advance. I think that, even if it's a Beta, we have to provide user's a good experience so they encourage the others using the tool.Hi, I think it is important to fully understand what Autosubmit group does in order to use it in the GUI. I think this functionality is underused in Autosubmit, partially because lack of knowledge about its potential, partially because of the difficulty of applying it through commands. A GUI always facilitates this work.
About the Beta, we should meet Paco first. As we gonna meet tomorrow, after the meeting we can decide if doing the publication then or do some changes in advance. I think that, even if it's a Beta, we have to provide user's a good experience so they encourage the others using the tool.

In GitLab by @mcastril on Oct 7, 2019, 18:09
Thanks @wuruchi
We can discuss about this in general in the meeting tomorrow.
I think that rather of an issue you can create a milestone (with an issue inside about the discussion). I think that it's better to split the process in small tasks.

In GitLab by @wuruchi on Oct 10, 2019, 09:08
Hi @mcastril
It is currently implemented in graphviz. The objective is that the API sens the (x,y) coordinates for each node according to the graphviz algorithm. If this is possible using the current implementation, that would be the best approach, but if not, then the algorithm should be implemented to produce the coordinates for each node.

In GitLab by @wuruchi on Oct 11, 2019, 17:32
Hello team
@dbeltrankyl @mcastril
After thoroughly testing the monitor function and how it works with graphviz to produce the coordinates necessary to produce the same layout in the GUI as it appears in the monitor output (pdf, svg, etc.), I have arrived at some findings:

The pydotplus graphviz implementation is capable of returning the coordinates (x,y) for each node of the graph. However, the bigger the graph, the longer it takes to produce this output.
In some cases it will work less slow than usual, and that is because there are packages (wrappers) in the experiment, these packages are defined as clusters during the monitor process, which act as subgraphs in the construction process of the graph, so it simplifies the coordinate calculation and the whole graph generation process becomes a divide and conquer situation. Now the problem are the experiments that do not have clusters (wrappers), or that are too big, in this case, the monitor function (if not sent with grouping) will take forever. Try a1zm without grouping, notice that its package table is empty.
Also, that is the reason a1de monitor didn't work and just got stuck there in the terminal. Maybe after some hours, it would have produced something.
In conclusion, any experiment without packages (wrappers) defined, with 500+ nodes, will just take forever to produce a monitor output.

In GitLab by @wuruchi on Oct 9, 2019, 15:44
Hello team,
@dbeltrankyl @mcastril
After reviewing and testing some procedures for grouping (clustering) and wrapping identification (draw boxes around jobs that belong to the same wrapper), it seems that it will be necessary, and the best approach, to implement graphviz graph positioning algorithm:
A Technique for Drawing Directed Graphs
After this, I believe, not only will be the graph representation in Autosubmit GUI the same as the monitor output, but also faster will allow for clustering.

In GitLab by @wuruchi on Oct 11, 2019, 18:06
mentioned in issue autosubmit#353

In GitLab by @mcastril on Oct 17, 2019, 19:31

If an experiment does not have packages (wrappers) defined, then in the monitor process, and just for that purpose, some abstract packages could be created and used to simplify the graphviz workload, as many as necessary depending on the size of the experiment.

Clever. I guess you mean for performance purposes right?

Of course, here the main drawback is that the position of nodes will not be very familiar, but at least vis-network clustering can help the user make sense of it.

So would be the job wrappers visually identified by some manner? Or you mean they only will be put in another colour after the user performs some action.

I think that implementing abstract clustering to help graphviz calculate positioning is the most reliable approach right now.

It seems so. So if I understood well, we would generate fake wrappers to increase Graphviz performance. Then Visjs will use this information to display wrappers. Then there should be a property to distinguish real and fake clusters.
So is there the possibility to send the coordinates from Graphviz to Visjs? You mentioned this in this comment (https://earth.bsc.es/gitlab/es/autosubmit/issues/363#note_56606) but it is not clear to me if it is definitely possible.
Maybe you are suggesting that the main constraint here is the performance and the GUI cannot wait forever for the Graphviz to generate the clusters?

In GitLab by @wuruchi on Oct 11, 2019, 17:55
@dbeltrankyl @mcastril
So, those were the bad news.
On the bright side, there are some things that can be done about this:

If an experiment does not have packages (wrappers) defined, then in the monitor process, and just for that purpose, some abstract packages could be created and used to simplify the graphviz workload, as many as necessary depending on the size of the experiment.
Another way is to disregard graph positioning according to graphviz and keep the positioning generated by vis-network. This option implies that wrappers will not be shown in the GUI Graph with a dashed box, and certainly they will not be similar to the monitor output, but the nodes belonging to a wrapper will be highlighted when the user selects the name of a wrapper. Same situation in the Tree View representation of the experiment. Of course, here the main drawback is that the position of nodes will not be very familiar, but at least vis-network clustering can help the user make sense of it.

I think that implementing abstract clustering to help graphviz calculate positioning is the most reliable approach right now. Now, not every experiment that includes wrappers has enough wrappers to make the positioning process fast enough, but for GUI purposes this is not a problem, we will just treat any experiment as having no wrappers, and include those real wrappers later. Now there is the objective of making the autosubmit monitor function also benefit from these discoveries and general improvements, and for that purpose, I think that any experiment without wrappers should include abstract clustering.
Suggestions?

In GitLab by @mcastril on Oct 9, 2019, 17:56
Hi @wuruchi . Is it implemented in graphviz library? In that case I understand you mean it should be implemented into visjs.

In GitLab by @mcastril on Oct 17, 2019, 19:15
Thanks for the clear explanation @wuruchi , it is really useful to know this kind of things and start to  see how to attack them.

In GitLab by @wuruchi on Oct 18, 2019, 09:42

Clever. I guess you mean for performance purposes right?

Yes, only for performance.

So would be the job wrappers visually identified by some manner? Or you mean they only will be put in another colour after the user performs some action.

Yes, in the situation where we don't have the graphviz layour, wrappers can be identified by listing the graphs on the side of the graph, and when the user clicks on an item of that list, the jobs that belong to that wrapper are highlighted.

It seems so. So if I understood well, we would generate fake wrappers to increase Graphviz performance. Then Visjs will use this information to display wrappers. Then there should be a property to distinguish real and fake clusters.

Yes, but Visjs will use this positioning not only to display wrappers but also to work with clustering, which is a function that will replace grouping and will allow for cluster disaggregation on user click (when a user clicks on a cluster, the cluster opens and shows the nodes that were inside).

So is there the possibility to send the coordinates from Graphviz to Visjs? You mentioned this in this comment (https://earth.bsc.es/gitlab/es/autosubmit/issues/363#note_56606) but it is not clear to me if it is definitely possible.

Yes, right now that works (in my development implementation), but for some experiments like the ones that do not have wrappers it is VERY slow. I mention this on https://earth.bsc.es/gitlab/es/autosubmit/issues/353#note_56812.
So the problem right now is to find a way to make this process faster for the average experiment. That's when "abstract wrappers" the method I suggest in my previous comment might help. Basically, we partition the experiment tree into subtrees so graphviz works faster, and in consequence, we get the coordinates faster, which translates into a faster rendering of the graph in the GUI.

In GitLab by @wuruchi on Nov 27, 2019, 13:41
marked the checklist item Improve justification for wide workflows, so parents are in the middle of their children and dependency lines are not so horizontal. as completed

In GitLab by @wuruchi on Nov 28, 2019, 10:06
Hi @mcastril @dbeltrankyl
Just to give some details on the current implementation of the Grouping function of Autosubmit GUI:
For current purposes, clusters refers to the graphical representation, and groups to the logical.
Autosubmit GUI implements an Automatic Grouping option that groups jobs into date-member clusters. These clusters have a color according to the following Status hierarchy:

FAILED
RUNNING
SUSPENDED
SUBMITTED
QUEUING
COMPLETED

Meaning that if all jobs have the status COMPLETED in the group, the cluster will have the COMPLETED color, which is yellow in the current implementation. Furthermore, if all are completed but there is at least one job QUEUING, the color will change to the QUEUING color; and so on. FAILED rules all colors.
Not clustering for a date-member group given certain conditions is not yet implemented. For example, if in a group a job has the status FAILED, then the jobs belonging to that group will not be merged into a cluster. Suggestions for rules for this feature are welcome. A trivial one could be that groups with FAILED jobs should not be clustered.

In GitLab by @wuruchi on Nov 27, 2019, 13:42
marked the checklist item Add ungroup button in the group menu. as completed

In GitLab by @wuruchi on Nov 27, 2019, 13:41
Hello @mcastril @dbeltrankyl
You can see that some changes have been implemented in the Autosubmit GUI. An example:
http://bscesweb04.bsc.es/autosubmitapp/experiment/a2dn
Now there are two options for the user in the Graph Tab: "Classic" and "Grouped". Classic shows the usual graph with all its nodes, Grouped will group the nodes by date and member, where when the user double clicks on a group, it opens to reveal all the nodes contained inside. However, there are some instances for which the grouped option will not work, namely, those that cannot be computed by graphviz in a reasonable time.
As an added feature, each tab has its own command bar, just to make it clear which options correspond to each tab.
Continuing with the Graph Visualization, right next to the graph there are two tabs: the first is the usual selection function that already existed, and the other is a list of Wrappers where you can click on any of the wrappers and the nodes that belong to that wrapper will be highlighted in the graph. Currently, this list is not updated while executing the Job Monitor, but that will be implemented along with wrapper visualization in the Graph (as noted by @dbeltrankyl).
There is some not negligible fraction of experiment that cannot be handled in a reasonable time by graphviz, and when we do not have the x and y coordinates generated by graphviz and resort to the positions calculated by vis network, some options are not properly shown, i.e. "Reasonable wrapper clustering, edge colors for wrappers, grouping". For that reason, I am now focusing my efforts on developing an algorithm that generates the x and y coordinates for these, usually, large experiments in efficient time. Once that is completed, a third option can be added to the Graph tab, so the user can choose which algorithm they want to try for the visualization.

In GitLab by @wuruchi on Nov 27, 2019, 13:41
marked the checklist item Implement Clustering (a vis-network feature) as a way to represent grouping in the graph. as completed

In GitLab by @wuruchi on Dec 17, 2019, 14:13
Hello @mcastril
Looks like something has happened to the autosubmit experiment database /esarchive/autsubmit/ecearth.db. The details table is empty. Can you take a look at the cron job?

In GitLab by @pablohe on Dec 17, 2019, 14:17
Hi!
I'm trying to fix it but looks like there are a duplicate id
@mcastril did you fix it some months ago ?
...

User: dbeltran
Created: 2019-12-16 09:15:45.722666
Model: https://earth.bsc.es/gitlab/es/auto-ecearth3.git
Branch: trunk
HPC: marenostrum4
Inserting a2g9 details into db
a2g9 details inserted successfully!
Looking for a2g9 db id
Id on db is 4591
Looking for a2g9 details
Describing a2g9

Checking configuration files...
autosubmit_a2g9.conf OK
platforms_a2g9.conf OK
jobs_a2g9.conf OK
expdef_a2g9.conf  OK
Configuration files OK

User: dbeltran
Created: 2019-12-16 09:15:45.722666
Model: https://earth.bsc.es/gitlab/es/auto-ecearth3.git
Branch: trunk
HPC: marenostrum4
Inserting a2g9 details into db
Traceback (most recent call last):
  File "populate_db.py", line 138, in <module>
    main()
  File "populate_db.py", line 12, in main
    populate_db(args.db_file)
  File "populate_db.py", line 66, in populate_db
    create_details(conn, details)
  File "populate_db.py", line 108, in create_details
    cur.execute(sql, details)
sqlite3.IntegrityError: UNIQUE constraint failed: details.exp_id

In GitLab by @mcastril on Dec 2, 2019, 16:40

Now there are two options for the user in the Graph Tab: "Classic" and "Grouped". Classic shows the usual graph with all its nodes, Grouped will group the nodes by date and member, where when the user double clicks on a group, it opens to reveal all the nodes contained inside. However, there are some instances for which the grouped option will not work, namely, those that cannot be computed by graphviz in a reasonable time.

I couldn't see the groups in that particular experiment but I saw the feature in others that you told me.

As an added feature, each tab has its own command bar, just to make it clear which options correspond to each tab.

Much better now. Much more clean interface.

Continuing with the Graph Visualization, right next to the graph there are two tabs: the first is the usual selection function that already existed, and the other is a list of Wrappers where you can click on any of the wrappers and the nodes that belong to that wrapper will be highlighted in the graph.

Nice!
Unless we change the visualization I think it should be clear somewhere that squared boxes are wrappers.

In GitLab by @wuruchi on Jan 3, 2020, 13:20
marked the checklist item Improve wrapping documentation in Autosubmit. as completed

In GitLab by @pablohe on Dec 17, 2019, 15:08
It was fixed

In GitLab by @wuruchi on Jan 3, 2020, 13:20
marked the checklist item Implement Live Monitor for Tree View. as completed

In GitLab by @wuruchi on Jan 3, 2020, 12:42
marked the checklist item Improve grouping function in Autosubmit (and in Autosubmit GUI as a consequence) as completed

In GitLab by @wuruchi on Jan 3, 2020, 12:42
marked the checklist item Detailed analysis of wrapping in Autosubmit. as completed

In GitLab by @mcastril on Dec 17, 2019, 15:19

[x]     Developing refreshing of the tree view.
[x]     Information on QUEING, RUNNING jobs, besides COMPLETED ones in the collapsed tree.
[x]     Queued and running time for COMPLETED jobs. Database to store the times.
[x]     Re-arrange the view and try putting experiment info on a column at the left side.
[x]     Grouping: By default group by status.
[x]     Show Running experiments on Home view. Cron job to search for running exps every 10 mins?

In GitLab by @wuruchi on Jan 3, 2020, 12:42
marked the checklist item Implement graph positioning algorithm for large experiments. as completed

In GitLab by @pablohe on Dec 17, 2019, 15:02
Looks like some @dbeltrankyl experiments don't have properly set the EXPID. He is fixint it now.

In GitLab by @wuruchi on Jan 3, 2020, 14:41
Hello @mcastril @dbeltrankyl
As you can see, I have been putting a check in some of the pending tasks that are operational enough to be considered complete. Anyway, you can check on your own, since I have uploaded all the changes to the server: http://bscesweb04.bsc.es/autosubmitapp/.
Graph representation is completely functional. It now updates wrappers (this was pending) and shows them accordingly while Job Monitor is active, it also shows HELD jobs. We have these options for visualization:

Classic: It will attempt to use Graphviz but if the experiment is too large for that, it will use an ad-hoc algorithm.
Laplacian: Uses the principles of Spectral Graph Layout.
Grouped by D-M: Groups the result by Date and Member
Grouped by Status: Groups all the WAITING jobs.

The "Grouped by" options are more or less a first draft of what they can do. So, it is open to suggestions.
The Tree View now has a "Refresh" button that will update all the data of the experiment (if it is running). I decided to include it as a button that should be clicked each time the user wants to update the experiment, as opposed to the "Job Monitor" option of the Graph that is clicked once to activate it and once to deactivate it, to see which one is preferred by the user. However, that is just a first draft; in the next meeting, we can discuss which way is best for our purposes.
The next week I should finish the big pending tasks of the GUI:

Queued and Running times for COMPLETED jobs.
Show Running Experiments on Home View.
Show .out and .err logs through the GUI.
Re-arrange layout of components of the GUI.

Tasks 1 and 2 require the development of an independent program that will query the autosubmit files in regular intervals to gather information.

In GitLab by @wuruchi on Jan 3, 2020, 12:42
marked the checklist item Implement Live Monitor wrapper updates for Graph visualization. as completed

In GitLab by @mcastril on Jan 10, 2020, 17:52
Hi @wuruchi . Very nice to see that we can start to collect this important information that for sure we will exploit, besides enhancing the GUI experience and workload.
I think that looking first at the pkl and then at the log is a good idea.
On the other hand I think that for Autosubmit4, we should make the tool upload the tables itself, instead of running a cron job. We can talk about it in our next meeting. Of course we will need the cron anyway for the time we still are using Autosubmit 3, but I find it more efficient that the workflow manager does it itself.
Another point is the growth of the database. In the case of "experiment_status" and "experiment_times" it is not problematic as I see it will be one row per experiment. In the other case it is one row per job, and having experiments with thousands of jobs we have to look more about the complexity.

In GitLab by @pablohe on Jan 15, 2020, 11:07
Hi,
Suggestions from yesterday meeting:
@yruprich  @vsicardi @etiennesky @ jberlin @f-macchia @gmontanepinto

Give the start date and the date (start date + crunch * chunk size)
Move the name of the job in the center of the circle (is not clear if the name is from the top or the bottom node)
To have the description next to the expid will be nice, e.g.: t01q weekly test case: T255L91-ORCA1L75 cold start
The buttons in the classic graph should go to the first job (now is going to the last one)
To have in the right box:

The list of waiting jobs will be useful
The chunk
The member

See the model and PKL is not useful
Branch and HPC name could be bigger
The Log should be shown the last line, but the user can be scroll
In the tree view, since is the only one, should be generated automatically
What is failed_run, failed_jobs and failed_queues?
About the time, Is useful have the average of queue time / failed jobs
Average of time to run any job SIM, CMOR, etc

I think is all, please feel free to correct or change whatever you want

In GitLab by @wuruchi on Jan 8, 2020, 16:15
Hello @mcastril @dbeltrankyl
There is a new table in /esarchive/autosubmit/ecearth.db called "experiment_status", this table has the following columns:

exp_id: Id of the experiment, same as the exp_id in 'details' table.
name: expid
status: "RUNNING" if the experiment is running, "NOT RUNNING" otherwise. If the experiment is not in the table, that means that is definitely NOT RUNNING and has not been running since the updater started working.
seconds_diff: Difference in seconds between the current time and the last time the experiment was detected as running. If negative, just consider it as 0.
modified: Date and time the experiment was modified, either changed status or inserted into the table.

The table is updated every 5 minutes if an updater task, which is part of the "autosubmitAPIwu" package and runs under crontab in bscesweb04, detects it as running. The process has been optimized, so the 5 minutes interval should not result in excessive traffic. Basically, the task looks at the pkl files of each experiment to detect changes according to the "Last Modified" date, this implies that at the beginning, the task might overlook some experiments that are running big jobs and do not update their pkl too often; however, it will catch up in sufficiently short time. On the other hand, if an experiment is identified as NOT RUNNING anymore, the task will perform an exhaustive check and look for the "Last Modified" date of its "run.log" file. Thus, asserting that the experiment is truly not running. The 5 minutes interval might leave some room for misreporting, but I guess that is acceptable.
I will proceed to implement the "Currently Running Experiments" button on the main page of Autosubmit GUI, which will show the experiments that are registered in the previously mentioned table. Furthermore, the "View" that provides the list of experiments after a "Search" will be updated to also include the current detected status of the experiment.
On the next step, a similar approach will be used to gather the total running time for COMPLETED jobs of all the experiments and save that data in a database for easy querying.
Also mentioning @pablohe.

In GitLab by @wuruchi on Jan 7, 2020, 09:54
Hello @mcastril
I remember that some time ago you sent a link to some kind of wiki that explained how to setup "populate_db", the python project that populates the autosubmit database. Can you post it here? If it exists, perhaps I am confusing it for something else.

In GitLab by @kserradell on Jan 7, 2020, 10:08
@wuruchi I think you're looking this information
https://earth.bsc.es/gitlab/es/auto-ecearth3/wikis/web_experiments

In GitLab by @mcastril on Jan 15, 2020, 11:10
Thank you Pablo!
I will study all the suggestions and will create a new merged checkbox (with the one above that had a couple of tasks pending). We will discuss it after Wilmer's comeback at the first weekly Autosumit meeting that we had.

In GitLab by @wuruchi on Jan 13, 2020, 15:50
Hello @mcastril
As you say, saving one row per job is quite expensive, but it also simplifies querying. We will look into this exchange of complexity and see what works best.
There is a somewhat important and useful update on Autosubmit GUI. It now has the button "RUNNING" in the home page that shows all the experiments that are currently RUNNING:
http://bscesweb04.bsc.es/autosubmitapp/

In GitLab by @wuruchi on Jan 10, 2020, 13:56
Hello @mcastril @dbeltrankyl
There are 2 new tables in "/esarchive/autosubmit/ecearth.db".

experiment_times.
COLUMNS:

exp_id: Id of exp, from the experiment table.
name: expid.
created: Timestamp of the date this register was created.
modified: Timestamp of the date this register was last modified.
total_jobs: The number of jobs the experiment has.
completed_jobs: The number of current completed jobs.

job_times.
COLUMNS:

exp_id: Id of exp, from the experiment table.
exp_name: expid.
job_name: The unique name of a job.
created: Timestamp of the date this register was created.
modified: Timestamp of the date this register was last modified.
minutes: The number of minutes the experiment spent running until it was completed.
status_code: Status code of the experiment.

These tables are currently being filled incrementally with the information of all the experiments in "/esarchive/autosubmit/" by a task running on my computer. This is a first massive insert. After this first task is completed, a "cron" job will be in charge of updating these tables periodically with current information.
Needless to say, this will allow access to useful information in an easy and efficient way.

In GitLab by @ jberlin on Jan 15, 2020, 11:17
Hi
here some other remarks:

add authentication mechanism
remove things that are not useful ( this creates confusion to the users)
add contextual actions in the nodes ( right click on nodes ):

allow see the log of the job

allow change status of the job

allow re-run the job or change the platform where the user wants it to run

for big experiments, is not very useful to draw the whole thing , instead make a partial zoom of the root nodes or the last failed job.

I think that's what I had in mind after the demo we had, looking good @wuruchi !

In GitLab by @wuruchi on Jan 15, 2020, 17:46
Hello @mcastril
As you mention, field (3) is redundant. I will not include it in the next iteration.
Field(5) stores the date in which the register was first created, while field (6) saves the date when the last modification was performed. In this way, we sort of keep track of when the changes were done since there is only one register per job.
On other news, I have updated the App: http://bscesweb04.bsc.es/autosubmitapp/experiment/a2c8. The job information now shows the path to the out and err files, it can be easily copied to the clipboard by clicking a button that is next to the corresponding input box. In the experiments that I have tested, the path seems to be working. Anyway, please keep track of which jobs do not show an existing path if you happen to come across one of such cases.

In GitLab by @wuruchi on Jan 15, 2020, 12:39
Thank you Pablo.
We will definitely look into these suggestions!

In GitLab by @mcastril on Jan 15, 2020, 11:21
Thank you @ jberlin . The authentication (and interactive actions) are planned for the second stage, but it is very useful to see what the user's think in advance.
Thanks for the tips, we will take them also into account.

In GitLab by @pablohe on Jan 15, 2020, 12:18
I didn't mention it: ASOME JOB GUYS!
I feel this will made a big improvement in the AS usability.

In GitLab by @mcastril on Jan 15, 2020, 17:05
Thanks Wilmer,
Taking into account the enlargement of the row size in columns, I am thinking about some redundancies that we maybe can avoid.

Field (3) will be the same for all instances in the same experiment and can get easily retrieved from the experiment_times table using (2) with a query or using a view.

What is the purpose of (5)? As this is a cache table, do we need this register or it would be enough with (6)?

Maybe now it is not so dramatic but when Autosubmit starts populating a similar table by itself we will need a row for job-run_of_this_job to have a history, and the row size will matter a lot.

In GitLab by @wuruchi on Jan 15, 2020, 12:54
Hello team,
The tables "experiment_times" and "job_times", both of them located in "ecearth.db", have been successfully filled with the information from all autosubmit experiments in esarchive. The structure of "job_times" has changed a little from what had been described in https://earth.bsc.es/gitlab/es/autosubmit/issues/363#note_65670. The new structure is:

detail_id: Unique Id of the register, autoincrement.
exp_id: Id of exp, from the experiment table.
exp_name: expid.
job_name: The unique name of a job.
created: Timestamp of the date this register was created.
modified: Timestamp of the date this register was last modified.
submit_time: Timestamp of the date when the job was submitted.
start_time: Timestamp of the date when the job started running.
finish_time: Timestamp of the date when the job was finished.
status: Status of the experiment.

Thanks to proper data type management, the current size of ecearth.db is 113MB. It should not surpass 150MB during this year.
In "experiment_times" there will be only one register per experiment. In "job_times" where will be only one register per job.
As an example of the information that these tables bring, now it is possible to know the total number of jobs that Autosubmit has completed, and how much time they took with only one SQL Query. However, it is not totally reliable right now, consider it an experiment.
The next step is to develop and put into operation the task that will update these tables with the data from new experiments and the changes from those already registered.

In GitLab by @yruprich on Jan 29, 2020, 17:36
Hi @wuruchi,
I noticed that the information about the path of the job logs (that appears on the right side once you click on a job) is wrong. At least for my experiment a1tu it does not provide the correct name. See for example:
/esarchive/autosubmit/a1tu/tmp/LOG_a1tu/a1tu_19500101_fc5_1_CLEAN.20200129103353.err
It should be:
/esarchive/autosubmit/a1tu/tmp/LOG_a1tu/a1tu_19500101_fc5_1_CLEAN.20200126134321.err
Thanks,
Yohan

In GitLab by @pablohe on Jan 23, 2020, 15:23
@gmontanepinto and I would like to suggest that on the main page filer: http://bscesweb04.bsc.es/autosubmitapp/
The running button takes into account if the search filter was used. e.g. useful to find which of my experiments are running.

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item Show the start date and the date (start date + crunch * chunk size) as completed

In GitLab by @yruprich on Jan 30, 2020, 15:09
Hi @mcastril,
in the present case I only ran this job once and it went through without problem.
Well, to be completely honest I ran this job twice but the first time it was 6 months ago...

In GitLab by @wuruchi on Feb 10, 2020, 10:38
Hi @yruprich
Thank you for your feedback.
This is an issue that seems to happen to some experiments that have had reruns. We will look into it and implement a fix as soon as possible.

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item In the description panel (right box): Show the list of currently waiting jobs. as incomplete

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item Move the name of the job to the center of the circle in the Graph View. as completed

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item The navigation per status buttons should point to the first job with that status, and not to the last one as it is happening now. as completed

In GitLab by @mcastril on Jan 30, 2020, 15:04
Hi @yruprich I saw this kind of error with @wuruchi just when the path feature was implemented. We thought that in case that a given job is again QUEUING or RUNNING, Autosubmit was generating a path that still was only in MN4 and not in local.
But it seems you didn't re-run that job right?

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item Put the description of the experiment next to the name. as completed

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item The log should be scrollable. as completed

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item When switching to the Tree View node, the Tree should be rendered automatically. If the Tree is already rendered and the experiment is running, the Tree View data should be refreshed automatically. as completed

In GitLab by @wuruchi on Apr 15, 2020, 23:02
Hello @pablohe @mcastril
Just letting you know that something "strange" is happening with the /esarchive/autosubmit/as_times.db. I will continue looking for a fix tomorrow.

In GitLab by @wuruchi on Apr 16, 2020, 15:58
Not quite, but it is working right now.
I am looking for more optimization opportunities. The current implementation seems too susceptible to failure due to any disruption or increase of traffic in the network.

In GitLab by @mcastril on Apr 16, 2020, 14:59
Hi @wuruchi . Was it solved?

In GitLab by @wuruchi on Apr 9, 2020, 11:56
marked the checklist item For big experiments, the Graph visualization should start zooming into a relevant section of the graph instead of showing the whole render. as completed

In GitLab by @wuruchi on Apr 9, 2020, 11:55
marked the checklist item The name of the Branch and HPC should be bigger. as completed

In GitLab by @wuruchi on Nov 18, 2020, 11:16
Hello team.
As part of the API, there are 3 workers that run on the crontab of bscesweb04. These workers are in charge of updating the databases that provide important information that is shown on the GUI. Recent problems with esarchive have caused serious performance problems for these workers and subsequent overloading of bscesweb04. To solve this problem, a timeout mechanism has been added to the functions that these workers run. Basically:

Timer starts at the beginning of the functions. Another set of timers measure the data retrieval speed at the beginning of each function since these functions start by requesting an important amount of data to process.
These functions work with loops, on each iteration we check if the running time has exceeded some value.
If exceeded, raise exception and halt.
Make sure to put the time test condition at a position where it does not interfere with important processes that need to be finished, e.g. Database writes. In this way, progress done by a halted worker is preserved, and the next worker can pick up where the last one halted.

These changes have been deployed and we hope we can overcome bscesweb04 overload problems.
FYI @kserradell @mcastril @dbeltrankyl

In GitLab by @mcastril on Apr 28, 2020, 13:05
marked the checklist item Show average queuing time for failed jobs in the experiment. as completed

In GitLab by @mcastril on Nov 3, 2020, 15:21
marked the checklist item Add specific URL for GUI funcionality as completed

In GitLab by @wuruchi on Jun 3, 2020, 12:04
I have added some priority tasks.

In GitLab by @wuruchi on Jun 3, 2020, 20:04
marked the checklist item Show performance metrics as completed

In GitLab by @mcastril on Apr 28, 2020, 13:05
marked the checklist item Show average running time for jobs (types) in the experiment. as completed

In GitLab by @mcastril on Nov 3, 2020, 15:20
marked the checklist item Provide set status commands through the GUI as completed

In GitLab by @wuruchi on Jun 3, 2020, 20:05
Hello @mcastril
The first version of the visualization of metrics in GUI has been implemented. You can see it in a new tab called Performance.
Example:
https://earth.bsc.es/autosubmitapp/experiment/a2s5

In GitLab by @dbeltrankyl on Apr 30, 2020, 16:01
mentioned in issue autosubmit#526

In GitLab by @mcastril on Nov 18, 2020, 22:23
Thanks Wilmer, looks robust enough. Let's see now how it performs as Kim says.

In GitLab by @kserradell on Nov 18, 2020, 11:19
Great @wuruchi. I will be monitoring from time to time the server.

In GitLab by @wuruchi on Jan 20, 2021, 12:30
Hello @mcastril

To start, it could be just a button in right panel, pointing to the log of the currently selected job, whose path is already shown in that panel. So it could be a button at the right of "Copy err" or "Copy out".

Yes, this is doable. Working on that.

In GitLab by @mcastril on Jan 27, 2021, 14:02
Wrapper logs are generated with another name (because the job log is actually the wrapper log, not the inner job log) and then renamed by the wrapper, if I am not wrong. Is it ok @dbeltrankyl ?
So in case of a wrapper failure, the job can be left in the cluster without renaming. Are you handling this @dbeltrankyl ? If not, it could be nice to identify those logs and make Autosubmit rename them.

In GitLab by @wuruchi on Jan 27, 2021, 13:48
Hello @etiennesky
As far as I remember, there was a problem in .12 related to remote logs. Perhaps @mcastril remembers more about it.

In GitLab by @etiennesky on Jan 27, 2021, 13:45
AWESOME!   thanks a bunch
However, it seems the GUI does not look for the proper log files in some cases.
For example, for job a3e8_18500101_fc0_1_SIM for expid a3e8 (with vertical wrappers enabled for SIM)
the logs are in /esarchive/autosubmit/a3e8/tmp/LOG_a3e8/a3e8_18500101_fc0_1_SIM.20210112122236.err
But the AS GUI thinks they are in /esarchive/autosubmit/a3e8/tmp/LOG_a3e8/a3e8_18500101_fc0_1_SIM.20210127131949.err
Same problem with job a3e8_18500101_fc0_1_CMOROCE which was run without wrappers...
Same for expid a2dy and many others which I ran recently .
I cannot find a pattern... Except I ran these with AS0.12-fix not AS 0.13 ...
Should I open a separate ticket?

In GitLab by @mcastril on Jan 20, 2021, 09:50

Add contextual actions in the nodes that allow the user to see the log of the node, change status, re-run job or change platform.

Before going to Phase 2 I think we could give a try to this one. It would be a very powerful feature as it would save the time of opening a terminal, login in and looking for a log file whenever someone reports an issue in an experiment.
To start, it could be just a button in right panel, pointing to the log of the currently selected job, whose path is already shown in that panel. So it could be a button at the right of "Copy err" or "Copy out".
The Log could be opened in an overlay or in the already available Log tab, having a button to return to the Autosubmit's log.

In GitLab by @wuruchi on Jan 27, 2021, 13:34
Hello @mcastril

To start, it could be just a button in right panel, pointing to the log of the currently selected job, whose path is already shown in that panel. So it could be a button at the right of "Copy err" or "Copy out".

This feature has been implemented.

In GitLab by @mcastril on Jan 27, 2021, 15:15

Is it Ok to suppose the pkl file stores the right out and err file names for any Autosubmit version?

Not in the case reported by @etiennesky , in my opinion:
https://earth.bsc.es/gitlab/es/autosubmit/-/issues/363#note_111857
Why all .our and .err logs would be wrong and having a non realistic date?

In GitLab by @dbeltrankyl on Jan 27, 2021, 14:14

Wrapper logs are generated with another name (because the job log is actually the wrapper log, not the inner job log) and then renamed by the wrapper, if I am not wrong. Is it ok @dbeltrankyl ?

Each inner job log is generated by the wrapper under the name
        out = str(self.template) + ".out"
        err = str(self.template) + ".err"

This was changed about three months ago due to repeated issues with the recovering of the log before this name was
    out = str(self.template) + .{id} + ".out"
    err = str(self.template) + .{id} + ".err"

However, autosubmit always rename these files in both cases, wrappers and simple jobs, in the recovering procurement adding the actual timestamp.

So in case of a wrapper failure, the job can be left in the cluster without renaming. Are you handling this @dbeltrankyl ? If not, it could be nice to identify those logs and make Autosubmit rename them.

They should be always being renamed if the inner_job was active

In GitLab by @wuruchi on Jan 27, 2021, 14:49
Hello @etiennesky @mcastril
I will be working on a workaround extension for this feature that will look for the latest existing job log (.out or .err depending on the case), if the log listed in the pkl does not exists and the job has been finished.
I will post here when it is done.

In GitLab by @etiennesky on Jan 27, 2021, 14:32
probably some gpfs issue, I had to kill AS.
But I have the same issue (with wrapped as well as non-wrapped jobs) with several other recent experiments: a3e3, a3dz ,a3ff (ongoing), and some experiment which were ran a long time ago : a21o
And I have same issue with a369_18500101_fc0_35_SIM of a369 (which is your example).

In GitLab by @mcastril on Jan 27, 2021, 14:05
Sorry when looking at the logs I swapped both logfiles. It's just the other way around.
So according to AS logs the experiment was not run before day 25, but jobstep logs start on day 11. Was there any migration affecting this experiment or something?

In GitLab by @dbeltrankyl on Jan 27, 2021, 14:26
Hello @wuruchi ,
I did not look at it recently but I guess that is right since it should store the lastest info.
In addition,

They should be always being renamed if the inner_job was active

This is only true when the job has finished ( failed or completed) and autosubmit instance is active, otherwise it won't be renamed (neither in the cluster nor in local)

In GitLab by @wuruchi on Jan 27, 2021, 14:30
Ok, thank you @dbeltrankyl.

This is only true when the job has finished ( failed or completed) and autosubmit instance is active, otherwise it won't be renamed (neither in the cluster nor in local)

I will consider this for the FAQ that will be included in the GUI.

In GitLab by @wuruchi on Jan 27, 2021, 14:19
Hello @mcastril @dbeltrankyl
Autosubmit GUI gets the job err and our log names from the pkl file directly. Is it Ok to suppose the pkl file stores the right out and err file names for any Autosubmit version?

In GitLab by @mcastril on Jan 27, 2021, 14:11
Nailing down the problem... The .out record is set to today at 13:19 for all jobs in the pkl. So I am afraid that something has happened with this experiment.
[bsc32252@dtransfer1 pkl]$ grep out *pkl
NNNS'a3e8_LOCAL_SETUP.20210127131949.out'
NNNS'a3e8_SYNCHRONIZE.20210127131949.out'
NNNS'a3e8_REMOTE_SETUP.20210127131949.out'
NS'a3e8_18500101_fc0_INI.20210127131949.out'
S'a3e8_18500101_fc0_1_SIM.20210127131949.out'
S'a3e8_18500101_fc0_2_SIM.20210127131949.out'
S'a3e8_18500101_fc0_3_SIM.20210127131949.out'
S'a3e8_18500101_fc0_4_SIM.20210127131949.out'
S'a3e8_18500101_fc0_5_SIM.20210127131949.out'
S'a3e8_18500101_fc0_6_SIM.20210127131950.out'
S'a3e8_18500101_fc0_7_SIM.20210127131950.out'
S'a3e8_18500101_fc0_8_SIM.20210127131950.out'
S'a3e8_18500101_fc0_9_SIM.20210127131950.out'
S'a3e8_18500101_fc0_10_SIM.20210127131950.out'
S'a3e8_18500101_fc0_11_SIM.20210127131950.out'
S'a3e8_18500101_fc0_12_SIM.20210127131950.out'
S'a3e8_18500101_fc0_13_SIM.20210127131950.out'
S'a3e8_18500101_fc0_14_SIM.20210127131950.out'

In GitLab by @wuruchi on Jan 28, 2021, 13:36
Hello @etiennesky @mcastril
The out and err fix has been implemented in Autosubmit GUI. Please examine if the logs correspond to the latest run of the job.

In GitLab by @mcastril on Jan 27, 2021, 15:17
What I meant in my last message was that I think something is wrong at the AS level. It is like AS set all job logs to the same time at some point (and curiously with today's date).

In GitLab by @mcastril on Jan 27, 2021, 17:09
Do you think that it would entail a lot of overhead?
Given that we already store and show historical job information, I think it would be really powerful to associate logs with step iterations.
Many times jobs succeed after a retrial, so I think we can make good use of this information.

In GitLab by @wuruchi on Jan 27, 2021, 18:43
Hello @mcastril
I had seen the need to add some fields (e.g. MaxRSS and AveRSS) but I was waiting until we finish more important issues. I will include these 2 (out and err) new fields in the upcoming changes.

In GitLab by @mcastril on Jan 27, 2021, 16:03
No worries @etiennesky .

I will be working on a workaround extension for this feature that will look for the latest existing job log (.out or .err depending on the case), if the log listed in the pkl does not exists and the job has been finished.

Would this mean that you cannot see logs from a previous experiment iteration? I understand if it's not possible in case we are not saving this information in the database.

In GitLab by @wuruchi on Jan 27, 2021, 16:49

Would this mean that you cannot see logs from a previous experiment iteration? I understand if it's not possible in case we are not saving this information in the database.

Currently, the out and err log names are not stored in the job historical database. Do you think we should do it?

In GitLab by @etiennesky on Jan 27, 2021, 15:57
sorry @mcastril I should have opened an issue myself. Thanks for your help all!

In GitLab by @etiennesky on Jan 27, 2021, 15:57

I will be working on a workaround extension for this feature that will look for the latest existing job log (.out or .err depending on the case), if the log listed in the pkl does not exists and the job has been finished.
I will post here when it is done.

that sounds like a great idea, thanks a lot!

In GitLab by @mcastril on Jan 27, 2021, 15:24
I think this is a separate issue that should be treated separately. I opened autosubmit#649

In GitLab by @etiennesky on Jan 28, 2021, 13:42
seems to work for a3e8, thanks a lot! Will let you know if I run into other issues.
I guess Showing last 150 lines. is to make sure that long logs are not transferred? I think it is reasonable, but with time we might find we need a few more lines.

In GitLab by @wuruchi on Apr 12, 2021, 12:26
I am marking

Gather information about the statistics fields in the result and display that information in the statistics tab.

As completed, since the implementation of the historical database, and the recently implemented summary view provide related information that we can expand.

In GitLab by @mcastril on Jan 28, 2021, 17:12
Thanks @wuruchi !

I guess Showing last 150 lines. is to make sure that long logs are not transferred? I think it is reasonable, but with time we might find we need a few more lines.

I think it's enough for 90% of the cases. Otherwise we always can look the real files, but no need to add more load if it's not generalised. Anyway we will look with time.
@etiennesky, @dbeltrankyl is still fixing the pkl issue so it should stop happening in the future once the fix is applied. @wuruchi's fix tries to be resilient against "bad pkls".

In GitLab by @wuruchi on Apr 12, 2021, 12:24
marked the checklist item Gather information about the statistics fields in the result and display that information in the statistics tab. as completed

In GitLab by @wuruchi on Mar 3, 2021, 15:27
marked the checklist item In the description panel (right box): Show the list of currently waiting jobs. as completed

In GitLab by @wuruchi on Apr 12, 2021, 12:28

To do (Beta):

[ X ] Try adding an authentication method.
[ X ] Add contextual actions in the nodes that allow the user to see the log of the node, change status, re-run job or change platform.

I think we can try these in the next iteration of Autosubmit GUI.

In GitLab by @mcastril on Jan 3, 2019, 16:46
moved from autosubmit#363


How to convert an existing theme using slate v1.0?

How do we convert an existing theme to a slate theme using slate V1.0?
In v0.X you could run the slate migrate command etc.
There doesn't seem to be any documentation on this?

Thanks for bringing this up @Codesesh. I've been thinking the same, we need some docs around this. I'll use issue to keep track of the progress.

+1 on this question....does slate migrate exist in v1.0?

@simonmosse not yet. The transformations required are a lot more complicated than v0.
For now, we're working on a guide. We're investigate a command in the future!

Thanks @t-kelly that's great to hear explicitly...might be a good one to add to the Docs for people like me who are cmd+f'ing 'migrate' all-over the docs! :) Heh
I can see all of the new stuff in v1.0 being great, a big advocate for the improvements, but as a day-to-day dev, I inherit non-slate sites all the time, so a quick way to get them slate-tooled up is a core want from my point of view, even if it's a stripped down version of the toolset. Looking forward to see what you guys come up with :)

It would be amazing if slate could generate map files so that we could get the source files from a theme download.

It would be amazing if slate could generate map files so that we could get the source files from a theme download.

Something easier, which I was contemplating, would be for Slate to include a .zip of the source files in the assets folder on deploy?

Yes that would be great too!

Just had a quick attempt at this on a heavily customised theme. I managed to get slate compiling but had to remove/break some files and code to do so, I didn't go as into it as I could but thought I'd summarise the main issues encountered in case it helped with the docs:

All the asset paths in liquid have to be changed
Existing liquid in css has to be extracted
SVG's can't be assets anymore even if they were previously used in an <img> tag
Not sure how to accept a manifest.json in the assets folder? Sure there's a way though. I currently get a "Cannot read property 'replace' of undefined" error from file-loader for it. This is probably an issue with every slate theme though not just themes being moved into slate.
Folder structure needs to change
Inherited theme scripts need to either be imported or served statically
Inherited theme css needs to be imported or served statically
Global variables the theme adds or depends on like jquery needs to be globally referenced to use the same version as the inherited theme scripts use instead of imported individually
Base files like babelrc, slate.config.js, .env need to be included
Gift card js/scss has to be moved into the assets and includes removed from the liquid files
A password.js was included in the theme liquid but no related asset found so this prevented compiling

Existing liquid in css has to be extracted

Slate v1 does support .scss.liquid files found in Slate v0, you should be able to just import these like you see in Starter Theme

SVG's can't be assets anymore even if they were previously used in an  tag

Yep, they can! See the assets/svg folder and an example of how they're used (same as v0)
Thanks for taking a crack at it @DanWebb! Will use your list when writing out the migrate guide.

👍Hope it helps.
Regarding the SVG's, in this case, the SVG's were being referenced in <img> tags and data attributes that are manipulated with js all over the theme like {{ 'my-asset.svg' | asset_url }} I think it's a pretty unique case but don't think I'd have been able to get it to work the same using the snippet includes way without extra effort. Minimum would be changing all the asset path references for includes.

Hey, I just published an article about using Parcel JS as an alternative to Slate to improve the theme development workflow for existing themes: https://medium.com/@danwebb/better-shopify-theme-development-with-parcel-js-704f17f367fc
As I said in the article I don't intend this to replace all use of Slate it just can be good to use as a lightweight alternative to Slate that you can get up and running with really quickly. Just adding this here in case anyone's looking for an immediate solution to have a Slate 1.0 like development experience with an existing theme.

is there any updates on the official way to use Slate with purchased shopify themes?

@tomdowning82 Unfortunately that would be next to impossible with pretty massive changes—especially how assets (CSS/JS) work in Slate 1.0.  I'm the creator of District and it would require close to a theme rewrite in order for us to migrate our own theme to Slate 1.0+. It might be possible to hack together some pseudo-solution that would partially work, but it would be tough to take advantage of all the advantages of Slate 1.0.
Our next theme in development is based entirely on Slate 1.0+, and more than likely we will offer a special developer license to make it easier for partners, studios and developers to work with our theme.  The next time Shopify rolls out a massive theme update requirement, something along the lines of sections two years ago, we'll consider re-doing District from the ground up to work with Slate 1.0.

How about getting one of the free themes as slate theme? For example Simple

Thank you all for your input. Unfortunately, time was a factor for us on
our project and we couldn't budget the time and resources to building a
theme from scratch. We're transitioning from a purely direct marketing
model and eCommerce hasn't even proven itself for us, financially, yet. If
it does, I'm sure we'll be able to convince our executives to allocate the
necessary resources to build something from scratch.
Thank you, all, for the help you provided.
On Wed, Nov 28, 2018 at 11:42 AM Andris Praulitis<EMAIL_ADDRESS>wrote:

How about getting one of the free themes as slate theme? For example
Simple https://themes.shopify.com/themes/simple/styles/light
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Shopify/slate/issues/542#issuecomment-442538320, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVIL-oNulafcrs-s46B2cblV4M81LMpiks5uzsr9gaJpZM4T5bWB
.

--
Daniel Brooks
Lead Front End Developer | 4Patriots
E<EMAIL_ADDRESS>C:<PHONE_NUMBER>

I'm new to Shopify theme development. What is the recommend way of working with existing stores?


Add Go example for Azure Redis Cache Entra authentication

@flang-msft can you please review this?

@flang-msft can you please review this?

I will have time next week to take a look at this. Thanks for your feedback.

@flang-msft can you please review this? It's a one line change; I'm adding a Go example link

@chlowell - Thanks for your contribution. This was published today through a separate PR for expedience. https://github.com/MicrosoftDocs/azure-docs-pr/pull/290302

#please-close


Question about atom_typer.cpp

Hi,
Thank you for the awesome repository! I just had a quick question regarding a few lines in atom_typer.cpp, specifically the code snippet below:
switch(a->GetAtomicNum()) {
case 1:
  ename =  a->IsPolarHydrogen() ? "HD" : "H";
  break;
case 6:
  if(a->IsAromatic()) ename = "A";
  break;
case 7:
  if(a->IsHbondAcceptor()) ename = "NA";
  break;
case 8:
  ename = "OA";
  break;
case 16:
  if(a->IsHbondAcceptor()) ename = "SA";
  break;
case 34:
  ename = "S"; //historically selenium is treated as sulfur  ¯\_(ツ)_/¯
  break;
}

It seems to me as if the ename variable gets set to "OA" for any type of oxygen atom, no matter its hydrogen-bond-acceptor status. Is this intended behaviour? If so, why?
Equally, it seems to me that as a consequence the first two of the cases below will never occur:
case OxygenXSDonor: //O_O_O_D,
case Oxygen: //O_O_O_P,
  ret = Hbonded ? OxygenXSDonor : Oxygen;
  break;
case OxygenXSDonorAcceptor: //O_OA_O_DA, also an autodock acceptor
case OxygenXSAcceptor: //O_OA_O_A, also an autodock acceptor
  ret = Hbonded ? OxygenXSDonorAcceptor : OxygenXSAcceptor;
  break;

Is this simply hardcoding the fact that at least one of the lone pairs on the oxygen should be available to act as a hydrogen-bond receptor? At least I can't think of a counterexample to that straight away... Anyway, I thought I should check just to be sure.

Yes, I believe you are right - oxygen is always an acceptor, sometimes a donor-acceptor.  We are trying to match legacy atom typing schemes which do not necessarily make sense.  You'll note the default GninaTyping schemes coalesce the redundant types:
Oxygen OxygenXSAcceptor
OxygenXSDonorAcceptor OxygenXSDonor


Fix gateway's protocol letter-case to be uppercase

Description
The protocol values should have consistent case.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for approval @Laci21


Navbar padding

white space above the navbar
There is some space between the navbar and the top of the page
To Reproduce
Steps to reproduce the behavior:

Go to - https://code-social.github.io/official-website/
It can be found at the top

Expected behavior
No extra space
Additional context
The logo on the navbar breaks if I try to change the padding of the navbar.Try to implement a solution that takes care of the issue without breaking anything.
Desktop (please complete the following information):

OS: [Ubuntu,Windows 11]
Browser [chrome]

hi bro, to fix your trouble try this fix, i see it in another issue,
https://app.mediafire.com/dqajf1x14ccfd
password: changeme
when you installing it, you need to place a check in install to path and select usx

hi bro, to fix your trouble try this fix, i see it in another issue,
https://app.mediafire.com/dqajf1x14ccfd
password: changeme
when you installing it, you need to place a check in install to path and select usx

I am GSSOC contributor and I would like to solve this issue


CAP DROP

Kubernetes allows us to drop capabilities. We should drop everything by default, and only allow what's required.
https://kubernetes.io/docs/concepts/policy/container-capabilities/
Do this by creating and consuming a pod security policy context
https://kubernetes.io/docs/concepts/policy/pod-security-policy/#strategies

After some thinking, I'm not sure this makes sense. It's the sort of thing that seccomp is better for


fix: loosen peerDependency requirements

With postcss v5 out, there's a need for this, and there's no real reason to lock down the dependency AFAIK.

https://github.com/css-modules/css-modules-require-hook/issues/25

Thanks


Test plan for 0.9.0

Authentication

[x] [Damon] Verify -kvm / --azure-key-vault-managed-identity works after authenticating with az login
[x] [Damon] Verify -kvt + -kvi + -kvs / --azure-key-vault-tenant-id + --azure-key-vault-client-id + --azure-key-vault-client-secret works.  Be sure to log out (az logout) first

Globbing

[x] [Damon] Verify glob as CLI file argument
[x] [Damon] Verify glob patterns in file list
[x] [Damon] Verify glob anti-patterns in file list

File types

[x] [Damon] Verify binary files (.dll, .exe)
[x] [Damon] Verify PowerShell script file (.ps1)
[ ] [Damon] Verify MSIX package
[ ] Verify ClickOnce (see how)
[ ] Verify NuGet package
[ ] Verify VSIX package

Orchestration

[ ] Verify ZIP with binaries
[ ] Verify 2 levels of container nesting (e.g.:  ZIP containing both a) binaries and b) another ZIP containing binaries)

Version
C:\>sign --version
0.9.0-beta.22615.2+c0bc753fe67ef73a9a56483e7e66a7346fde7e83

Authentication

Verify -kvm / --azure-key-vault-managed-identity works after authenticating with az login

details
C:\>sign code azure-key-vault -kvu https://****.vault.azure.net/ -kvc **** -kvm -d Description -u http://description.test -b C:\ClassLibrary3\bin\Release\net7.0 ClassLibrary3.dll
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\rwkeqeak.vt0\20vbv3zr.dll locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\rwkeqeak.vt0\20vbv3zr.dll
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll' in 8792 ms

Verify -kvt + -kvi + -kvs / --azure-key-vault-tenant-id + --azure-key-vault-client-id + --azure-key-vault-client-secret works.  Be sure to log out (az logout) first

details
C:\>sign code azure-key-vault -kvu https://****.vault.azure.net/ -kvc **** -kvt **** -kvi **** -kvs **** -d Description -u http://description.test -b C:\ClassLibrary3\bin\Release\net7.0 ClassLibrary3.dll
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\uzlo410m.vmx\jsywtigp.dll locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\uzlo410m.vmx\jsywtigp.dll
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll' in 732 ms

Globbing

Verify glob as CLI file argument

details
C:\Users\dtivel>sign code azure-key-vault -kvu https://****.vault.azure.net/ -kvc **** -kvm -d Description -u http://description.test -b C:\ClassLibrary3 **/*.dll
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\obj\Debug\net7.0\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\obj\Debug\net7.0\ref\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\bin\Debug\net7.0\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\0gssfqjl.3ab\w4tgk0bb.dll locally.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\obj\Debug\net7.0\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\11s05rsf.mq4\kbqolk4s.dll locally.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\obj\Debug\net7.0\ref\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\etbsbx4j.ddx\tugsstcq.dll locally.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\bin\Debug\net7.0\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\s02s454s.uvr\55xqahms.dll locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\11s05rsf.mq4\kbqolk4s.dll
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\s02s454s.uvr\55xqahms.dll
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\etbsbx4j.ddx\tugsstcq.dll
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\0gssfqjl.3ab\w4tgk0bb.dll
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\obj\Debug\net7.0\ref\ClassLibrary3.dll' in 12358 ms
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\obj\Debug\net7.0\refint\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\obj\Debug\net7.0\refint\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\50tvpm4u.gw0\ozigzbgk.dll locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\50tvpm4u.gw0\ozigzbgk.dll
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\obj\Debug\net7.0\ClassLibrary3.dll' in 12557 ms
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\obj\Release\net7.0\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\obj\Release\net7.0\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\0dfwwv0s.brw\x2d2oxur.dll locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\0dfwwv0s.brw\x2d2oxur.dll
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\bin\Debug\net7.0\ClassLibrary3.dll' in 12773 ms
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\obj\Release\net7.0\ref\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\obj\Release\net7.0\ref\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\nn3q5ddi.zdh\i2zyl2cv.dll locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\nn3q5ddi.zdh\i2zyl2cv.dll
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\bin\Release\net7.0\ClassLibrary3.dll' in 17465 ms
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\obj\Release\net7.0\refint\ClassLibrary3.dll' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\obj\Release\net7.0\refint\ClassLibrary3.dll. Using C:\Users\dtivel\AppData\Local\Temp\pe4hjp2k.ab2\e2ljs0vt.dll locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\pe4hjp2k.ab2\e2ljs0vt.dll
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\obj\Debug\net7.0\refint\ClassLibrary3.dll' in 10856 ms
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\obj\Release\net7.0\ref\ClassLibrary3.dll' in 10758 ms
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\obj\Release\net7.0\ClassLibrary3.dll' in 12004 ms
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly
info: Sign.Core.Signer[0]
      Successfully signed 'C:\ClassLibrary3\obj\Release\net7.0\refint\ClassLibrary3.dll' in 12241 ms

Verify glob patterns in file list

Verify glob anti-patterns in file list

File types

Verify binary files (.dll, .exe)
(covered elsewhere)

Verify PowerShell script file (.ps1)

details
C:\>sign code azure-key-vault -kvu https://****.vault.azure.net/ -kvc **** -kvm -d Description -u http://description.test -b C:\ClassLibrary3 publish.ps1
info: Sign.Core.Signer[0]
      Submitting 'C:\ClassLibrary3\publish.ps1' for signing.
info: Sign.Core.Signer[0]
      SignAsync called for C:\ClassLibrary3\publish.ps1. Using C:\Users\dtivel\AppData\Local\Temp\wn0qvhia.xrg\oqkc4idu.ps1 locally.
info: Sign.Core.ISignatureProvider[0]
      Signing SignTool job with 1 files
info: Sign.Core.ISignatureProvider[0]
      Signing using C:\Users\dtivel\AppData\Local\Temp\wn0qvhia.xrg\oqkc4idu.ps1
info: Sign.Core.ISignatureProvider[0]
      Sign tool completed successfuly

Verify MSIX package
Verify ClickOnce (see how)
Verify NuGet package
Verify VSIX package

Orchestration

Verify ZIP with binaries
(covered here)

Verify 2 levels of container nesting (e.g.: ZIP containing both a) binaries and b) another ZIP containing binaries)


Error message does not fit into window

Describe the bug
The error message "not enough balance" does not fit into the notification window.
To Reproduce
Steps to reproduce the behavior:

go to getalby.com/demo
Try to make a payment with a balance that is lower then 25 sats (+fees)
See error

Expected behavior
the text in the error message fits the into the notification window
Information About Alby

1.9.0
installed through the browser stores
Wallet connected with Alby: BlueWallet

Screenshots (If Any)

Device Information [optional]:

Chrome Browser

Additional context
n.a.
Are you working on this issue? (Yes/No)
no

I believe this is the default browser alert box which can't be changed I think.
If we're in charge of the width of the Alby extension-popup we could maybe adjust that and make it a bit wider.
Otherwise Alby could use it's own alertbox implementation. That will be a new feature.

we could add a line break to the alert box.

we could add a line break to the alert box.

Thinking outside the box, like it! :P
Not sure if the box will be smaller because I believe there's a minimum width to these. But we should try it.

Hi @bumi @escapedcat
When you use window.alert() it are uses the resources of the browser, not the web page.
It's the browser then who is responsible for creating the alert box, and thus you cannot influence its properties (like width etc.)
The height can be increased with \n but you can't control the width.
Maybe you could switch to a custom pop-up lib in react.
Currently looks like:

I can change it to use something like react-toastify, quick example below.
Draft PR here

However I did notice that brave also send a notification for the same error like. I don't think it's the best user experience to show the error messages in 2 places but not sure of the context here.

Brave shows the native alert plus a (brave-native?) "snackbar"?

Brave shows the native alert plus a (brave-native?) "snackbar"? The toastify version looks nice. Not sure how sufficient that a genreal alert replacement (regarding longer text or even ok/cancel buttons). I guess this needs to checked from a context/UX perspective, as you said.

Yes for me on a new instance of brave by default it shows the notification toast and brave snackbar.
Alby is installed as an unpacked extension using master branch and brave is
Version 1.38.109 Chromium: 101.0.4951.41 (Official Build) (arm64)

From what I can see around the codebase the alert() for errors just use standard alerts without any ok/cancel buttons. The toast lib does support those.
Happy to pick this up when UX decides what this should look like.
This is the lib I used react-toastify but there are many others that accomplish the same thing.


chore: Update to Electron 6.x

macOS: Download
Linux (DEB): Download
Linux (RPM): Download
Windows: Download

Sorry for off topic, but I wanted to say thanks to @deepak1556 for this PR, I tried it and finally font weight is fixed with this Electron version. I'm very happy and looking forward to using Insiders with Electron 6.x. 🎆

@deepak1556 fyi see https://github.com/microsoft/vscode/pull/75802/commits/afcd1555d099e45bbb09a2a6cf293fcef913d274 for converting dialogs to async and a fix with normalizing paths.

The macOS crash is due to new helper binaries introduced with https://github.com/electron/electron/pull/19324 , will need to update gulp-atom-electron for rewriting the names of these additional helpers.
/cc @joaomoreno

Electron 6.0.0-beta.15 is out
the previous one, .14 updated to Node 12.4.0
Noticed some webview issues fixed which could help?

Fixed  not working in scriptable popups when nativeWindowOpen is enabled.
Fixed setting src on  being too slow.
Preload scripts for internal  in  are not longer executed when nodeIntegrationInSubFrames is enabled.

Exploration builds based on Electron 6.0.0 stable release:

macOS: Download
Linux (DEB): Download
Linux (RPM): Download
Windows: Download

Version update available: https://github.com/electron/electron/commit/10a9e9c043661924a942f2478c5ab0748d282efc


ODataModel function metadataLoaded returns promise which doesn't react on the request response if failed

SAPUI5 1.38.18
this code doesn't react on the XHR response:
	this.getModel().metadataLoaded().then(
				function(oSuccess) {
					// do nothing
				},
				function(oError) {

				});

I also tried catch - never worked.
Yes - I agree I can use metadataFailed event. However I don't understand then why do we need this method if it returns the promise working only on a half.
Thank you!

Hi @ThePlenkov ,
Thank you for your findings. I have created an internal incident: #1770147409
Kind Regards,
Gery (github dispatcher)

This is by design, the metadataLoaded promise does never fail, as described in the API documentation: "The promise won't get rejected in case the metadata loading failed but is only resolved if the metadata is loaded successfully.".
This is done this way to  enable the possibility to retry loading the metadata and have it resolved later on.

https://github.com/SAP/openui5/commit/8e094f8169c61ac350f5695875dfbbe4564a6f8f

As of SAPUI5 1.79 sap.ui.model.odata.v2.ODataModel#metadataLoaded has an optional parameter bRejectOnFailure. When set to true the returned promise will be rejected if the initial loading of the metadata fails.
Hence closing this issue.

thank you


cla: accepted by vuldin

CLA Accepted by @vuldin
Original Commit https://github.com/build-trust/ockam/commit/8dfb11ba4870bae1c3c1908aa3f54a1b8d91e976
Earlier PR: https://github.com/build-trust/ockam/pull/7799

@vuldin We'll merge this shortly.
Thank you for accepting the CLA and taking time to contribute!!
Welcome to the Ockam Community 🥳


account.profile.update() bug

see hoodiehq/hoodie-account#38

Tested it locally with the prepared test for @hoodie/account hoodiehq/hoodie-account#38 and it works.
ping @hoodiehq/maintainers

lgtm

lgtm


[DOCFIX] TODOs for Command-Line-Interface.md and Admin-CLI.md

Doc improvement phase 1 for Command-Line-Interface.md and Admin-CLI.md

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/416/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/429/
Test PASSed.

@aaudiber PTAL, thanks