[SIP-35] Proposal for Improving Superset’s Python Code Organization

[SIP] Proposal for Improving Superset’s Python Code Organization
Motivation
As I was in the weeds fixing a bunch of Pylint failures, Max and I started going back and forth on this PR: https://github.com/apache/incubator-superset/pull/8777, which we ultimately closed. The root cause of that was a lack of shared understanding on the best code structure for Superset going forward. Talking with others at Preset, I realized that the issue was larger than just a new contributor not understanding project practices. Without a shared understanding, we are lacking a cohesive approach towards refactoring Superset - we need a technical North Star for project structure.
Preset’s developers met and identified a number of pain points in the existing code base. Here they are, with a bit of color to make the meaning clear:

Lack of separation of concerns

There are a number of modules that have grown organically over time to truly massive size. Often these modules contain utility functions, classes of multiple types, and a broad array of other functionality. These modules are very hard to work with.
Business logic is frequently intermixed with display logic, making code reuse more difficult.
The model layer contains a large amount of business logic which has nothing to do with data storage.

Dependency Confusion

Circular dependency issues crop up regularly with the existing module design
Many modules have implicit dependencies on other modules which requires a specific load order

Lack of Composability

There’s a substantial amount of app-specific code in shared modules
Views are currently loaded in all contexts, which makes light-weight command line usage (for example) impossible

Deep Class Inheritance

In some instances, particularly where there is pre-existing inheritance in Flask-AppBuilder, the inheritance chain can be over 10 classes deep. This can make understanding some classes very challenging.

Proposed Change
In order to address these concerns, we’d like to propose a few guiding principles for Python code structure going forward:

Prefer small, functionally-homogeneous modules with only classes and pure functions in module scope
Separate business logic from display logic

Keep the view layer lightweight and delegate to business logic objects
To support this work, we’d like to recommend a Command pattern - more on that below

Prefer thin ORM objects

To support this work, we’d like to recommend introducing Data Access Objects or DAOs to Superset to hold complicated query logic. More on that below.

Keep __init__.py files empty

This will help cut down on implicit dependencies

Prefer composition over inheritance

Collaborator objects can be simpler to understand than a deeply-nested class hierarchy.

Organize modules function and back-end objects rather than products

Prefer:

Dashboard
Chart
Query
Database
Models
api

Avoid:

SQL Lab
Explore

New patterns to introduce:
Command:
There are multiple patterns named Command, and the one we reference here is most similar to an Alembic migration. Commands perform a single cohesive business operation, have a very small public API, and either entirely succeed or entirely fail. In practice, this requires that database transaction behavior be controlled at the Command level. When possible, commands should be modeled such that they perform a single database transaction. In the case of failure, they raise an exception or return an error code, and it is the responsibility of the caller to translate that into a correct response for a user.
Example command:
# This is obviously not a full finished command. The point
# is to illustrate the structure of a command as
# encapsulating a business operation, and illustrate
# a likely public API.

import DashboardDAO from daos.dashboard

class MissingDashboardError(CommandError):
  pass

class UpdateFailedError(CommandError):
  pass

class InvalidDashboardError(CommandError):
  pass

class UpdateDashboardCommand < Command:
  def __init__(current_user, dashboard_id, dashboard_properties):
    self.actor = current_user
    self.dashboard_id = dashboard_id
    self.dashboard_properties = dashboard_properties

  def run():
    self.dashboard = DashboardDAO.find_dashboard(actor, dashboard_id)
    if self.dashboard IS NONE:
      raise MissingDashboardError("Dashboard could not be found")

    if !self._is_dashboard_valid():
      raise InvalidDashboardError("Dashboard parameters are invalid.")

    update_result = DashboardDAO.update(dashboard, dashboard_properties)

    if update_result != True:
      raise UpdateFailedError("Dashboard could not be updated.")

  def _is_dashboard_valid():
    # Check validity of dashboard data before save

DAO (Data Access Object):
A DAO in the context of Superset would be an object that would manage the lifecycle of SQLAlchemy models. Custom queries would be implemented in DAOs and then called from Command objects. In comparison to Command objects, DAOs have relatively broad public interfaces. DAOs should not depend on other DAOs to avoid circular dependencies. If results are needed cross-DAO, that should be orchestrated in the Command. Here’s a sample simplified DAO for illustrative purposes:
# This is not a fully finished DAO. The intention
# is to illustrate the public interface of a
# Data Access Object, and how it might interface
# with a SQLAlchemy model.

from superset import db
import DataAccessObject from daos.base
import Dashboard from models.dashboard

class DashboardDAO(DataAccessObject):

  @staticmethod
  def find_for_update(actor, dashboard_id):
    dashboard = db.session.query(Dashboard).filter(id=dashboard_id).one_or_none()
    if dashboard
      if actor in dashboard.owners:
        return dashboard
      else
        return None
    else
      return None

  @staticmethod
  def update(dashboard, dashboard_params):
    for key, value in dashboard_params.iteritems():
      setattr(dashboard, key, value)
    try:
      db.session.merge(dashboard)
      db.session.commit()
    except SQLAlchemyError as e:
      logger.error('Failed to update dashboard: ?', e.message)
      db.session.rollback()
      return False

    return True

Proposed example package structure that follows the above principles:
superset/dashboards/api.py # Contains FAB and Flask endpoints related to the public API
superset/dashboards/schemas.py # Contains marshmallow schemas used by both the api and view endpoints
superset/dashboards/boot.py # Contains initialization and configuration code for API endpoints, invoked by `create_app()`
superset/dashboards/views.py # Contains FAB and Flask endpoints related to page-refresh views.
superset/dashboards/dao.py
superset/dashboards/commands/add_user.py
superset/dashboards/commands/create.py
superset/dashboards/helpers.py
superset/charts/api.py
superset/charts/daos.py
superset/charts/boot.py
superset/charts/schemas.py
superset/charts/commands/add_to_dashboard.py
superset/charts/commands/create.py
superset/charts/helpers.py
superset/models/chart.py # Models are thin SQLAlchemy objects - they do not implement custom queries or filters. That functionality will be found in the DAO, which manipulates the underlying SQLAlchemy model.
superset/models/dashboard.py

In this design, all systems related to a specific back-end resource have been grouped under a top-level folder. __init__.py files should be left empty to enable only pulling in the portions of the system necessary for a specific entrypoint (Celery shouldn’t need api.py or views.py for instance)
New or Changed Public Interfaces
Over time, the internals of Superset will evolve towards the new structure. Public HTTP interfaces will not be likely to change as a result of the above proposal, but code will move and alter to conform. This will impact organizations that apply their own customizations to Superset.
New dependencies
None
Migration Plan and Compatibility
Introduce refactors to existing code at a manageable pace to allow organizations relying on Superset internals time to adapt.
Rejected Alternatives
Preset discussed Service Objects as an alternative to both Commands and DAOs. We felt that Commands provided easier entrypoints for the ports of our application (API endpoints, views, command line invocations, Celery tasks) than Service Objects, and that introducing DAOs as well helped further break down concerns.
We also considered structuring top-level folders by function (api, models, etc.) but found this resulted in drastically more Python modules overall without substantially simplifying the question of where code should live.
Individuals consulted in creating this SIP
@mistercrunch @craig-rueda @dpgaspar @robdiciuccio @suddjian @nytai

Looks good overall, however I would caution grouping things like models and/or daos under their domain as you have done above in favor of simply grouping them by function -- i.e. a package for daos, and one for models under a parent data package. Each child "data" package can, and should be further split based on domain. These lower-level bits invariably end up being reused and should be a little more centrally located.

@craig-rueda I've centralized the models in the recommendation, but have left the DAOs distributed - I don't think the DAOs will be too heavily reused outside of their domain. If the future proves me wrong, it can be refactored.

🏷 security

🗑 🏷 security
showing someone how to use the bot labels ...

🗑🏷 security

I agree with many issues/concerns raised in this SIP though I’m not overly familiar with commands and DAOs. Are there examples of other apps (preferably Flask) which use these constructs?
Note at Airbnb we have a number of Flask apps which follow best practices, i.e., are modular, have empty __init__.py files, use Flask blueprints, etc. which are fairly easy to understand and structured in a way that result in no circular dependencies.
I’m all for refactoring the Python codebase (leaning more on blueprints), though I’m unsure whether need to restructure the logic at this time to leverage commands and DAOs.

The root cause of that was a lack of shared understanding on the best code structure for Superset going forward

What software architecture will you choose?
What is your opinion of the Clean architecture

There are a number of modules that have grown organically over time to truly massive size. Often these modules contain utility functions, classes of multiple types, and a broad array of other functionality. These modules are very hard to work with.

How do you plan to analyze this problem? Will you create a single mother ticket and a number of child tickets for each issue?
How will you refactor all the tests in the current code base?

Business logic is frequently intermixed with display logic, making code reuse more difficult.

Do you have specific examples of this problem?
Isn't there a business reason for everything you display? how easy is it to separate the two?
do you have a code reviewer? if so, who let these problems get into the codebase in the first place?

Circular dependency issues crop up regularly with the existing module design

Would Inversion of control or dependency injection help with this? If not, what will help with it?

Many modules have implicit dependencies on other modules which requires a specific load order

This does not seem like much of a problem to me. Except that PyCharm (and other tools) run a module-reorder as part of their automatic code cleanup and this might prohibit something from loading.

There’s a substantial amount of app-specific code in shared modules

I do not understand the problem: Are you saying there is code duplication? If the purpose of a module is to do something for the app, then it stands to reason it would have app-specific code, wouldnt it? I'm confused.

In some instances, particularly where there is pre-existing inheritance in Flask-AppBuilder, the inheritance chain can be over 10 classes deep. This can make understanding some classes very challenging.

It can be challenging, but IMHO the code is well-written. I had to trace throgh 7 levels of OO inheritance here:
https://github.com/apache/incubator-superset/issues/8695#issuecomment-559968158
I would be curious to see how you would improve things and whether the problems is rooted in Flask itself and FAB and Superhave no choice but to bow to their Creator.

Organize modules function and back-end objects rather than products

this phrase does not express a complete thought. I think there is some sort of grammatical error here. I have no idea what you are suggesting.

Avoid: SQL Lab, Explore

One of these is a noun/product just like the things you prefer. The other is a verb.... What is unacceptable about SQL Lab?

class DashboardDAO(DataAccessObject):  def find_for_update(actor, dashboard_id):

Why is this preferred over a find_for_update in the class that actor belongs to? ditto for
class DashboardDAO(DataAccessObject):  def update(dashboard, dashboard_params): - this looks like a method that belongs to the Dashboard class.

The empty __init__.py files alone would provide enormous value IMO. If this SIP as a whole is approved I would suggested implementing breaking it into incremental, independently-valuable chunks. (whether it should be a single large SIP or not is a different question, perhaps... is the monolithic SIP coupling controversial suggestions with uncontroversial ones?)

First, thanks for your comments! I appreciate the community taking the issue seriously.
@DiggidyDave I agree that empty __init__.py files will help substantially. @dpgaspar has a PR open right now that's linked to this issue that demonstrates the patterns we're suggesting. IMHO the result is very readable and easy to work with. We hope for this SIP to be a North Star that we gradually work towards. I like your suggestion of implementing valuable bits in chunks. I'd recommend being pragmatic and attacking small enough pieces that they are comprehensible. One of the areas I'm most excited to tackle is slimming down the model layer using DAOs and commands - I think models are far too thick at the moment, but I would treat this work alone as multiple coherent changes rather than one large PR due to the potential complexity involved.
@john-bodley Superset currently has examples of both the Command pattern and a pattern similar to DAOs. Alembic's individual migration classes are an example of a command and FAB's shared filters are very similar to a DAO, in that they encapsulate shared query logic outside of the model. A DAO allows us to extract query logic from model classes to a clear destination. As we refactor, we need a target to move towards. Blueprints separate functionality well at the API layer, but will not provide a clean abstraction at the model or business logic layer without a few other patterns to lean on. I agree that fully refactoring the code is a large unit of work and expect this to be a target we move towards gradually where it makes sense. Currently the model layer is very thick and we have a lot of business logic in endpoint code. To me this is a difficult problem to solve without a shared concept of where this code should go moving forward and a pattern to leverage.

@metaperl

who let these problems get into the codebase in the first place?

Disorganization is a result of many people independently contributing to a project, focusing on shipping functional code quickly. This has worked well for Superset, but has resulted in a fair amount of tech debt. This SIP is about putting some structure in place that can help with future efforts. Blaming committers for these issues is not helpful or appropriate.

I have no idea what you are suggesting.

The idea is that backend code should be focused on the entities being acted on, rather than the specific part of Superset that is interacting with that entity. For example, SQL Lab and Explore ("products") interact with Datasources, Queries, Users ("objects"). This is proposing that the backend should be focused on providing APIs that are object-centric (POST /datasources, GET /queries, rather than APIs that are specific to any one product within Superset.

IMHO the code is well-written. I had to trace throgh 7 levels of OO inheritance

I would argue that you should not have to delve through 7 levels of inheritance to understand what a piece of code is doing. Simpler is better.

Isn't there a business reason for everything you display? how easy is it to separate the two?

You may want to take a look at principles of Clean Architecture for some information on this.

Blaming committers for these issues is not helpful or appropriate.

@suddjian - It certainly is not helpful or appropriate. And if you could explain why you included that comment, I would appreciate it.

@metaperl
Reviewers should help ensure that code is organized, and I think that's the idea you were going for with that bullet point. This will be especially true once the Superset community arrives at a consistent plan for code organization. Reviewers can point to the plan and request that code fit the established patterns.
But wording that idea as "who let these problems get into the codebase in the first place" reads like pointing fingers at the folks who wrote/reviewed the existing code, which is why I responded to it.

Chime in my two cents as a passerby.  I really like that different modules have their independent folders and we are introducing conventions and increasing consistency. I think an intuitive code structure and consistent paradigms help a lot for future contributors to understand the code.
Sharing the approach I took in one of my older projects (a general purpose CMS).  It's slightly different than what is proposed, but very similar in concept.
All my python files are organized like this:
├── __init__.py
├── app.py
├── core/   # core data models, user accounts, base data entities etc
├── ext/      # 3rd-party/non-essential extensions, s3 uploads, etc
├── lib/       # db mixins, cache decorators, utilities
├── modules/   # modules as organized in the main menu (URL route)
│   ├── __init__.py  # import and compose all modules
│   ├── module_1
│   │   ├── __init__.py
│   │   ├── admin.py   # admin views, a separate Blueprint for editing objects
│   │   ├── model.py   # operation on data entities, objects for data access
│   │   └── view.py     # User facing Blueprint for "module_1", handlers parsing parameters
│   ├── module_2
│   │   ├── __init__.py
│   │   ├── admin.py
│   │   ├── model.py
│   │   └── view.py

I find the separation of core/shared functionalities especially useful because it makes the whole project easier to navigate.
Loading modules  and booting the app is as simple as this. It seems we can avoid boot.py this way.
Hope this helps.

I really this we need to untangle the app from the various components. This is the partially to blame for the spaghetti code, circular dependency issues we’re currently facing.
All these modules, components, etc. need to be app agnostic and should be implemented as blueprints. If there are config or similar aspects these modules need leverage one can use the current_app proxy. The loading and registering (binding) of blueprints should occur within app.py.

Adding my two cents here:
Regarding blueprints: I'm inclined on delegating the actual registering to each module manager. We can still say that they occur within app.py but the specifics are encapsulated on the module manager itself.
This way:

We would reduce the centralized complexity of app.py
Handle module specifics at the module level

For example:
- superset/
    - datasets/
        - manager.py (implements `register_views`)
        - commands/
          - CRUD (create.py, update.py, delete.py ...)
          - Custom (export.py, ...)
        - api.py
        - schemas.py
        - dao.py
    - dashboards/
      ...
    - charts/
    - databases/
    - commands/
       - base.py
         ....
    - api/
        - base.py
....

So app.py would call each module's manager.register_views() with additional pre and post hooks. This kind of pattern can open up the app for external extensions, since additional modules can be independently installed (pip) and loaded in order using config, for example:
ADDONS = [
    middleware1.Middleware1Manager,
    middleware2.Middleware2Manager,
    ...
]

Just a thought, could db_engine_specs follow a similar pattern?

The blueprints are handled by FAB's `BaseApi` or it's child `ModelRestApi` that creates a blueprint with all routes registered under the same API resource namespace, `api/v1/dataset/*`, `api/v1/databases` and OpenAPI spec tag/grouping.

@john-bodley I didn't address your comment directly regarding blueprints. Let me do so now:
I concur 100% with the idea of untangling the app references from the rest of the code. The problem we currently face with regards to blueprints is that:

Blueprints cannot currently be nested (see this issue on Flask)
Flask-AppBuilder (FAB) currently leverages blueprints internally.

Were we to move entirely to blueprints being defined in Superset, that would have substantial implications for the extent that we can leverage FAB in endpoints without either reworking FAB internals or adding the ability to nest blueprints to Flask. As we've been building out /api/v1, we've leveraged Flask's base classes, especially for GET endpoints.
How would you recommend reconciling this problem?
I think that @dpgaspar's suggestion for adding a register_views function to the manager.py files is a possible resolution - this would provide an API similar to blueprint functionality that could be leveraged in app.py. I'd be interested in your opinion on how close this would get us to resolving this issues you're bringing up.

@willbarrett maybe there's a disconnect with my understanding of FAB. Does FAB prevent us from defining all the API endpoints as blueprints and housing these under a root /api directory?

@john-bodley My understanding here is a little fuzzy too, so I'll let @dpgaspar correct me if I'm wrong. FAB creates blueprints under the hood whenever we use it to define endpoints. ModelView and ModelViewApi classes create collections of endpoints under a blueprint, which is then registered on the application when we call appbuilder.add_view... or a similar function. If we wanted to group those items together into an API blueprint, it would not be possible since that would result in nested blueprints.
@dpgaspar how far away from the truth am I :)?

@willbarrett that's a good explanation.
@john-bodley
The proposed FAB model is that a blueprint is related with a BaseApi class and the blueprint's url_prefix is defined on the route_base class attribute. By default BaseApi and ModelRestApi will compose /api/<version>/<resource_name>/ for the url_prefix of the blueprint.
It's possible to overlay url_prefix has long has the endpoints don't collide.
Yet, I'm curious about the need to create a big blueprint that holds all API endpoints? Note that we aggregate all API classes under a base API class, where we can impose specific behavior/config.

I welcome this effort, and the amount of circular imports that typing has exposed really shows that something needs to be done. To get started, I would almost recommend grepping for packages with TYPE_CHECKING to see the most common circular dependencies, as those are most likely the most spaghettied pieces of code that need to be untangled. If it is possible to start by refactoring those into their new respective locations, we would be well positioned to continue refactoring the rest of the code as proposed.


feat(editors/publisher): add read only gse-control-element-editor

Adding read only GSEControl element editor with same UI as for Report. Would love to hear your review.

@danyill I will do so with the SampledValue read-only setting. Thx for hint.


YEAR_RANGE is quite short by default

https://github.com/in0rdr/diary/blob/3c2a58397740cb58bfbc4cae2ab0371bcf32a212/diary.h#L21
I find it confusing that the diary just stops at a year old - I always wonder what has happened to my old entries. Could it perhaps instead be set by default to 100 years? I don't anticipate needing more.
PS: I've apparently been using this program on-and-off for over four years now ... so thanks : )

I like your suggestion, but noticed that with 100 year range the program is too slow at startup. Can we fix it to 10y for the next release?
I hope that I can find time to implement a proper config (ini) file or so, to make this configurable..

I like your suggestion, but noticed that with 100 year range the program is too slow at startup. Can we fix it to 10y for the next release?
I hope that I can find time to implement a proper config (ini) file or so, to make this configurable..

@bovine3dom first draft for the config file is in https://github.com/in0rdr/diary/pull/61

@bovine3dom first draft for the config file is in https://github.com/in0rdr/diary/pull/61

The year range is now fully configurable through command line options or the optional config file, see release 0.5.


[BUG] GAME CRASH [MOBILE]

Describe the bug
Not sure how to reproduce this:
There is an issue where if you switch to any other tab, sometimes the game will crash and the UI disappears.
This will require further investigation as we need the console to see whats happening in mobile land.
To Reproduce
N/A
Expected behavior
Game should not crash when switching tabs in game
Screenshots
N/A
Browser and Version
Chrome
Smartphone (please complete the following information):
Iphone
Additional context
N/A

Added Exploration.


Allow logging in an author or administrator by ignoring "well-known" …

…portals SDK does not support
BUG-755199

@vrseraphin Please make similar change in SDK-A and SDK-WC soon so they're in sync. Thanks.


Paths

fixed some paths so that hopefully we work better in some versions of agent

Fake failure 🙃

also, super happy about the github actions import. this would have never failed otherwise


Update /templates page styles according to design

Update /templates page styles according to design. Here's the figma design.
Use mobile-first approach.

Let's release the minimalistic template for now and try whether the overall concept works :)


Feature request: run scenarios separately with Native

Hello, its me again :)
One of main advantages I can see for using the Core is an option to run each scenario (implemented as a function) separately. But using Core needs manual writing which kinda sucks.
Native allows automation. Tests runs automatically but debugging optionsare limited here (as written in the README).
Is it possible to enhnace Native implementation with an option to run scenarios separately? Eg with help of NativeFeature Parser generate scenarios as functions into a separate .swift file as XCtest functions so that a tester could run scenarios the same way as with Core.

Hey @tomesm. This is something I've run into as well. Because we create the tests at runtime they are classified as 'runtime tests' and Xcode only lists them in the test inspector after the first time you've run them. We've toyed with the idea of generating tests from feature files before e.g.:
Given the following feature file:
Feature: Basic tests
    Scenario: Basic steps
        Given A situation that I want to start at
        Then This value should be 100

run a command that will output BasicTests.swift with:
import XCTest
import XCTest_Gherkin

final class BasicTests: XCTestCase {
    func testBasicSteps() {
        Given("A situation that I want to start at")
        Then("This value should be 100")
    }
}

Is this what you mean? I think this would be a great feature to have as a command-line option that we run on demand/as part of a pre-build step. We'd be happy to look at a PR that implements this if you would like to take the initiative on it 👍

We've also tried a few other things with very little success (i.e. tags - https://github.com/net-a-porter-mobile/XCTest-Gherkin/issues/3). It would be really nice to be able to support (somehow) passing in a tag (or even scenario name), ideally on the command line so CIs can run subsets of tests as well as just while debugging. However, I just can't work out a good way to do it!

This has also been raised before here - https://github.com/net-a-porter-mobile/XCTest-Gherkin/issues/55 - are these the same issue?

@deanWombourne yes this looks like pretty much the same request.

hey @tomesm! I know it's quite a while since you opened this but I've recently submitted a PR to add in tags as was being described above :)


Adding support for unlimited dimensions

This PR adds support for unlimited dimensions.
I'm not very familiar with netncdf but as far as I can tell the underlying HDF5 data sets are automatically resized if required so that is what I implemented here. I think I caught the most common indexing patterns.
It is tested in the sense that netcdf can open the resulting files just fine and everything appears to be correct. There is also full test coverage for all changes and this PR does not change the behavior of limited dimensions.
It would be great if this could be merged with a subsequent release fairly soon as we have a number of downstream project that depend on it and we really need unlimited dimensions. Let me know if I can help with things!

Sorry for the delay, especially because I was asking for handling this quickly...
I changed it to only offer an explicit resize() method using the new API and will put the automatic resizing logic into a separate PR. Let me know if this still needs some work.

Okay - I had to dig a bit deeper than expected and it is getting more complicated. There is now a resize_dimension() method at the group level and it tries to assure that all variables with that dimension are always consistent. Please have a look at the tests which should explain the behavior.
Files generated by netCDF4 can be read by h5netcdf, and vice versa.
Note that the always consistent file is not exactly the behavior of netCDF - it does not appear to keep a file internally consistent and I'm not sure why.
Another thing that is kind of awkward: h5netcdf at init time (and for unlimited dimensions) will look at all variables with that dimension to determine the dimension's current size. I'm not sure how netcdf does this but I could not find anything in the hdf5 files it generates (see the currently last test).
I'm also not very familiar with netcdf semantics so it would be great if you can have a look.
I can work a bit more on this tomorrow if necessary.

Regarding: https://github.com/shoyer/h5netcdf/pull/33#discussion_r143879258
I'll comment here cause it will otherwise get hidden once I push to this branch and I think its important.

I'm actually not sure we need or want section. As long as resizing is done consistently across all variables (necessary to ensure a valid netCDF file), just looking at the dimension scale should be enough.

This will work if the file has been written with h5netcdf, but files written with netCDF4 don't show this behavior. And even if this is bug within netCDF4 (as you mention below) h5netcdf would somehow have to work around it to be able to work with existing netcdf files. And h5netcdf does need access to the current size of an unlimited dimension to be able to properly work with it.
The following example does not change the dimension scale when data is assigned to it. And in the h5dump output I don't see any way to get the current size of a dimension aside from looking at all datasets.
But I'll remove the current loop and will use the REFERENCE_LIST  of a dimensions to get its data sets - should be quite a bit faster especially with nested files.
import netCDF4

with netCDF4.Dataset("out.nc", "w") as f:
    f.createDimension('x', None)
    dummy1 = f.createVariable('dummy1', float, ('x'))
    dummy1[:] = [1, 2, 3, 4]

$ h5dump out.nc
HDF5 "out.nc" {
GROUP "/" {
   ATTRIBUTE "_NCProperties" {
      DATATYPE  H5T_STRING {
         STRSIZE 57;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "version=1|netcdflibversion=<IP_ADDRESS>|hdf5libversion=1.8.18"
      }
   }
   DATASET "dummy1" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SIMPLE { ( 4 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): 1, 2, 3, 4
      }
      ATTRIBUTE "DIMENSION_LIST" {
         DATATYPE  H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT }}
         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
         DATA {
         (0): (DATASET 331 /x )
         }
      }
   }
   DATASET "x" {
      DATATYPE  H5T_IEEE_F32BE
      DATASPACE  SIMPLE { ( 0 ) / ( H5S_UNLIMITED ) }
      DATA {
      }
      ATTRIBUTE "CLASS" {
         DATATYPE  H5T_STRING {
            STRSIZE 16;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "DIMENSION_SCALE"
         }
      }
      ATTRIBUTE "NAME" {
         DATATYPE  H5T_STRING {
            STRSIZE 64;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "This is a netCDF dimension but not a netCDF variable.         0"
         }
      }
      ATTRIBUTE "REFERENCE_LIST" {
         DATATYPE  H5T_COMPOUND {
            H5T_REFERENCE { H5T_STD_REF_OBJECT } "dataset";
            H5T_STD_I32LE "dimension";
         }
         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
         DATA {
         (0): {
               DATASET 628 /dummy1 ,
               0
            }
         }
      }
   }
}
}

I implemented all the review points and rebased on the latest mater. Let me know how you want the consistency when handling unlimited dimensions to be handled.

Just a quick ping in case you forgot. Sorry for being a bit pesky!
Also please let me know if you think this will still take a while - then I'll find some internal work-around for our dependent packages.

Ready for another review round.

Big thanks @krischer! I'll issue a new release shortly, probably later today.

Great :) Thanks for your help!
I still have the automatic resizing logic flying around in some branch but I'm not sure it is necessary especially as the logic for it is really fairly complex. Raise your hand if you still want it and I'll open a separate PR with the current state.

v0.5.0 is out now on pypi and conda-forge.

Thanks a lot for ! Also for the very useful package!
In case you are looking for another success story of h5netcdf: We utilize it in pyexodus to build exodus files which are meshes we use for large-scale high order finite element wave propagation simulations (see http://salvus.io).
The bottom line is that it enables us to skip two large C codes (NetCDF + Exodus) and it ends up being up to 100 times faster and more memory efficient.


Third-party Usage For Face Recognition

Thanks for your great work, we use Swin transformer for face recognition in our project, and Swin-S obtain a quite promising result of 98.17 in MegaFace rank1 protocol, which is the best accuracy in our project. More results, pretrained models and logs can be got from https://github.com/JDAI-CV/FaceX-Zoo/blob/main/training_mode/README.md

Very cool! I have added the link in README. Thanks a lot!


Crashes?

Hi,
Im trying to use the current version but when i launch the app, it must crash right away because nothing is in the system tray and nothing is running on task manager. v1.0.6 worked though. Im running windows 10.

This is caused by a corrupt settings file. To fix this:
Navigate to C:\Users\AppData\Local_uRawns\ and delete all the folders within. They should all start Reddit_Wallpaper_Changer._Uri.
These are the RWC setting files/folders. Once deleted, launch RWC and it will rebuild the new settings files.


Dashboard replace processor dialog is broken

The dialog needs to be polished as what we do to submit user application in #1435.

This is the preview. Also changed the order of "New Parallelism" and "Description", because changing any of the first three items will create a new processor, but change the other will not.

Also reordered the input boxes


Add Sorting and Filtering Options

Current Setup: The blogs are presented in a grid format, but there's no visible option for users to filter or sort blog posts.
Contribution Idea: Implement sorting (by date, popularity, category, etc.) and filtering options (by tag, category, date range) to enhance user experience
Please assign ... i would love to work on this

please label this


[Proposal] add modifyReleasePlan config option

Affected Packages
get-release-plan, config, cli
Problem
People have started using changesets in the wild which is great! Sometimes they don't agree with how we create and update versions, which is fine, and sometimes they have weird edge cases we don't want to support in the main version command.
Currently the solution is to write your own version command and run that instead of changeset version
The command basically looks like:
async function customVersionCommand () {
  let baseReleasePlan = await getReleasePlan()
  let modifiedReleasePlan = await customMutationOfReleasePlan(baseReleasePlan)
  await applyReleasePlan(modifiedReleasePlan)
}

This, however, does not work with any standard tooling we build for changesets, as they all assume they should run changeset version and that the output of that is correct.
Proposed solution
There are two options:

all tools that implement the option to run a custom script that isn't changeset version

pros: push away doing dangerous as heck stuff
cons: doesn't help with status, means implementing this in every other tool

add a config option called something like modifyReleasePlan or unsafeModifyReleaesPlan that accepts an async function, passes in a release plan, and then returns a release plan which gets applied.

pros: works with all existing tooling, easier to implement for consumers
cons: making it easier to do may mean it's easier for consumers to footgun themselves, as people should really understand what they are doing before modifying this.
Option 2 is where we are leaning, but we want to make it really clear how much we want people to take caution around doing this, as messing with what gets released and how
We are thinking of naming it:
modifyReleasePlanAndMaybeCreateBrokenReleases
Secondly, I want to add a package/function here which is validateReleasePlan - it:

checks you have a valid repo state after applying a release plan
checks your release plan makes sense (doesn't reference packages not in the monorepo is the main obvious one)

so consumers can validate their release plan before it gets applied and error.
There are some other concerns that @mitchellhamilton can probably expand on

what if we make it really easy to create your own CLI

It sort of technically is. You probably need anything to use changesets as the binary name for external rules that do stuff? Overloading this sounds dangerous though.

Two use-cases for modifying a release plan:
1) We do not want to always publish some packages
Some packages are used in a product, other packages that are built alongside it are designed for people writing plug-ins for the product. The packages targeted at plugin-writers should only be published after the product-packages have been updated and deployed within the product itself.
2) A person disagrees with how we are doing versioning
The biggest example is sometimes something is incidentally bumped as a dependency, when it is unnecessary to do so, and a person does not want to apply it, so they have a way to dial it down.


cucumber-html-formatter docker image

The HTML formatter is a Node.js application. Since it's meant to be used with any Cucumber implementation, the users may not be familiar with Node.js, or have it installed.
The same might be true for Docker, but having more ways to distribute the tool increases the number of ways it can be installed.
This PR adds scripts for packaging and publishing cucumber-html-formatter. Version 2.0.1 has been published already. We need a little more documentation to the release process how to use this. We need to document that it cannot be published from the docker container we use to release everything else - it has to be built and released from the host OS.

We need to document that it cannot be published from the docker container we use to release everything else - it has to be built and released from the host OS.

We should avoid that I think. Instead of pushing to DockerHub, let's use the pull functionality. Every time a tag and/or commit is pushed, the image will be built.
On hiptest-publisher, we have the following rules set on DockerHub:

This way:

each tag has a corresponding image built
latest points to the latest tag image
master allows using the master version (not sure it would be useful here)

So I'd deactivate the publish target in the Makefile to let DockerHub handle that (less manual steps, less risks of forgetting it :) )

@vincent-psarga I think that requires setting up a read-only mirror on GitHub where the Dockerfile is at the root.


InatImportJob js feedback

Improve Job Tracker by updating it with Stimulus, improving details.
Tasks

[ ]

Stimulus examples
app/javascript/controllers/field-slip-job_controller.js
#2520


Helm Repo is down

Hi there, thanks for maintaining this project!  It looks like the helm repo is down:
❯ helm repo add christianhuth https://christianhuth.github.io/helm-charts
Error: looks like "https://christianhuth.github.io/helm-charts" is not a valid chart repository or cannot be reached: Get "http://charts.christianhuth.de/index.yaml": dial tcp: lookup charts.christianhuth.de on <IP_ADDRESS>:53: no such host

Sorry for the inconvenience. I restructured some stuff on my end. It should work now again using the Helm Repository https://charts.christianhuth.de.


[Cherry-pick-3.1.0][Feature][Registry] Support etcd as registry  (#10981)

(cherry picked from commit 2eb8b9f7f0159b9119afe963193328d22cd746a7)

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Codecov Report

:exclamation: No coverage uploaded for pull request base (3.1.0-prepare@c4953e8). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             3.1.0-prepare   #11997   +/-   ##
================================================
  Coverage                 ?   39.59%
  Complexity               ?     4712
================================================
  Files                    ?     1016
  Lines                    ?    38125
  Branches                 ?     4259
================================================
  Hits                     ?    15096
  Misses                   ?    21413
  Partials                 ?     1616

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

cherry-pick #10981


Linting while running webpack raise error (lacking type info)

Bug Report

TSLint version<EMAIL_ADDRESS>TypeScript version<EMAIL_ADDRESS>Running TSLint via: Node.js API (using webpack)

TypeScript code being linted (Not dependent on code)
import { pageBuilder } from "@/pageBuilder";
import "../dist/components";

window.kentico = window.kentico || {};
window.kentico.pageBuilder = pageBuilder;

with tslint.json configuration:
{
  "defaultSeverity": "error",
  "extends": [
    "tslint:recommended"
  ],
  "jsRules": {},
  "rules": {
    "indent": [true, "spaces", 2],
    "interface-name": [
      false
    ],
    "no-default-export": true,
    "no-unused-variable": true,
    "object-literal-sort-keys": [true, "ignore-case", "match-declaration-order", "shorthand-first"],
    "max-line-length": [true, 180]
  },
  "rulesDirectory": []
}

I am also getting that underlined in vscode with message (but it is may separate issue):
Array has too many items. Expected {0} or fewer.

Webpack config part
module.exports = {
  entry: ['./src/main.ts', './src/assets/styles/icon-font-face.scss'],
  output: isProduction ? productionOutput : devOutput,
  module: {
    rules: [
    ........,
    {
      test: /\.tsx?$/,
      use: [
        'babel-loader',
        {
          loader: 'ts-loader',
          options: { // Dont thing is related
            appendTsSuffixTo: [/\.vue$/],
          }
        },
        'tslint-loader'  // <--- Here is the linter configured
      ],
      exclude: /node_modules/
    },

Actual behavior
While running webpack, I am getting the error:
CENSORED\Client\src\main.tsError: object-literal-sort-keys needs type info to use "match-declaration-order".
    at Rule.apply (CENSORED\Client\node_modules\tslint\lib\rules\objectLiteralSortKeysRule.js:45:19)
    at Linter.applyRule (CENSORED\Client\node_modules\tslint\lib\linter.js:197:29)
    at CENSORED\Client\node_modules\tslint\lib\linter.js:139:85
    at Object.flatMap (CENSORED\Client\node_modules\tslint\lib\utils.js:151:29)
    at Linter.getAllFailures (CENSORED\Client\node_modules\tslint\lib\linter.js:139:32)
    at Linter.lint (CENSORED\Client\node_modules\tslint\lib\linter.js:99:33)
    at lint (CENSORED\Client\node_modules\tslint-loader\index.js:70:10)
    at Object.module.exports (CENSORED\Client\node_modules\tslint-loader\index.js:140:3)
    at LOADER_EXECUTION (CENSORED\Client\node_modules\loader-runner\lib\LoaderRunner.js:119:14)
    at runSyncOrAsync (CENSORED\Client\node_modules\loader-runner\lib\LoaderRunner.js:120:4)
    at iterateNormalLoaders (CENSORED\Client\node_modules\loader-runner\lib\LoaderRunner.js:229:2)
    at Array.<anonymous> (CENSORED\Client\node_modules\loader-runner\lib\LoaderRunner.js:202:4)
    at Storage.finished (CENSORED\Client\node_modules\enhanced-resolve\lib\CachedInputFileSystem.js:40:15)
    at CENSORED\Client\node_modules\enhanced-resolve\lib\CachedInputFileSystem.js:77:9
    at CENSORED\Client\node_modules\graceful-fs\graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:511:3)

Expected behavior
no error while linting

I'm not using Webpack, but I am getting the same "Array has too many items. Expected {0} or fewer." message on my tslint.json file in vscode. I am also using the exact same rule:

"object-literal-sort-keys": [true, "ignore-case", "match-declaration-order", "shorthand-first"]

For me, this issue goes away if I delete any of the three string options. (It also goes away if I delete the boolean option, but then I get the expected error Incorrect type. Expected "boolean".

I as well!  Just to chime in..

The error tells you that you need to configure tslint to use type information. Seems like a tslint-loader configuration issue.

How does that error mean that @adidahiya ??  And maybe a more specific hint as to what configuration?  Your link isn't very helpful and is fairly generic.  I am having no problems with TSLINT at all EXCEPT when I tried to add a 4th parameter to the very specific object-literal-sort-keys .. then it says it's expecting only 3.  That seems like a bug in the tslint logic that isn't allowing 4 parameters..  And I think my tslint is configured to use Type Information -- at least I have a project file and configuration file -- and have had for a while.
Cheers

Client\src\main.tsError: object-literal-sort-keys needs type info to use "match-declaration-order".

@sjmcdowall The type information link from adidahiya goes over how type information works with some TSLint rules. object-literal-sort-keys is an example of these and is an optionally typed rule, meaning it can have extra behavior added when provided with type information (a tsconfig.json). The match-declaration-order option requires type information to be able to tell what that declaration order is.
The issue here is that, in order to have the TSLint running in Webpack via tslint-loader use that option, you need to provide type information tslint-loader's configuration. See https://github.com/wbuchwalter/tslint-loader#loader-options.

Thanks @JoshuaKGoldberg -- I will follow the link etc. and see if that fixes the issue!


Outdated Predefined Modern Group Names compared to IDs

I noticed that the "Predefined modern group" names associated with the group IDs have changed within the SharePoint web part picker when adding a new web part. I was able to deduce which new names go with which old IDs personally for the most part, but this section needs an update to continue to be helpful to developers.
Thanks in advance and thank you for all the hard work on these pages and content. These are very helpful and used by so many!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3043faa1-f278-c483-17f4-e1af81d9b097
Version Independent ID: df4b13b3-4c12-0592-7f29-f54075807c3b
Content: Simplify adding web parts with preconfigured entries
Content Source: docs/spfx/web-parts/guidance/simplify-adding-web-parts-with-preconfigured-entries.md
Product: sharepoint
Technology: sharepoint-framework
GitHub Login: @VesaJuvonen
Microsoft Alias: vesaj

While you're correct, it's not entirely clear which of the new names map to the old ones in SPO. But furthermore, I suspect the SP on-prem deployments will continue to use the old name, so the docs should match.
I just bugged this in #7797... once we see the schema I reference get updated, I'll update this doc.


Include build stack in BindingResolutionException message.

Time and time again I'll run into issues where some dependency of a dependency cannot be met and I am given very little to work with other than the actual name of the service/class/interface that was requested. This is nice if it is obvious, but more often than not it is an instance of a very generic interface for which there are countless services trying to have that interface injected.
This PR includes the current build stack in the message if there is a build stack available. This will hopefully provide more clues as to what was needing the missing dependency in order to track it down more quickly.
Here is an example of this PR in action:

Well then. I clicked submit too soon and it looks like this is now targeting 5.2 instead of 5.1. I'm happy to close this PR and send a new one against 5.1 as that was my intent. Let me know if this PR is good and how best to move forward to get this into 5.1 as well.
My bad. :(

This isn't really a bug fix, so 5.2 is fine.

Looks good thanks :)


How do I generate the Sourcecode?

Hello to Everyone who reads this. Maybe even @vigna.
I want to help with this project. Make improvements etc.
I have no issue with working with drv files and do all these edits even if there is no compiler help.
What I have an issue with is test the code because there is no real instructions/tutorial on how to actually generate the source code.
What this issue is about is very simple. Please provide a instruction list/tutorial how to generate these things.
If there is a difference between linux and windows and mac then provide instructions how to do these.
Why am i requesting this help?
Very simple. There is some requirements to write tests & test them. Ensure code is actually working.
Which has to be done A either in the head or B not at all.
This is prone to cause errors or mistakes in the code that may be not found.
The single requirement:
Make them easy and simple to understand so that someone who has never worked with sh scripts & make-files before can understand them. Because right now you need actual experience to know what to do to do simple exporting of the source code.
Also the Wikisection is great for such things.
Thanks for reading.
Speiger

Non of these things exist @incaseoftrouble
only executable files are gencsource.sh, find-deps.sh and test.sh (which throws errors without end)
and these files do not even work.
(I am working with windows10)

Uhm no. I just cloned the repo to double check (windows 10 with cygwin) and sources are generated just fine with make sources (you may need to run dos2unix if your git checks out CRLF). Makefile is not executable since the make command them. Yes you need a linux environment (like cygwin or WSL, not sure if the latter works though) but thats a five minute install

From what I was told (and it could be wrong or I missunderstood it) that this could be used on Win10 but I would have to start up a linux environment run the make source and then close the environment.
Then check if everything was working (well i could keep it open).
Is that correct?
Also. It would be nice if there would be a instruction list of what to do, because not everyone who can program is knowing linux well enough to make this connection to understand what to do.
(I had to ask a couple friends before i knew that linux was required even if only some assisted software)

I don't get the point. The readme tells you that you need to "make source". So you make source, thats the whole instruction list.
If you don't have make, you need to install it. If you really want to work on Windows, you can install cygwin. I guess we can add a sentence on this in the README but I don't think Windows users are the target audience in any case.

Ok and how do i "make sources"? Is it a CommandLine command? Is it a Gradle Task? Also if it is a command what arguments do i need to supply?
Again never worked with this and this is like compiled machinecode to me.

Thank you for the help.

@incaseoftrouble ok I managed to get through the process and I have the make folder included (latest version that the installer was able to find)
But when executing Make sources (with a clean copy of fast util and no changes made)
this error appears
$ make sources
./gencsource.sh drv/Iterable.drv src/it/unimi/dsi/fastutil/booleans/BooleanIterable.c >src/it/unimi/dsi/fastutil/booleans/BooleanIterable.c
./gencsource.sh: Zeile 2: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 9: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 12: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 15: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 18: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 21: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 24: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 27: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 30: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 33: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 36: $'\r': Kommando nicht gefunden.
): No such file or directoryarnung: setlocale: LC_ALL: Kann die Locale nicht ändern (C
: Ungültiger Shell Optionen Name. extglob
./gencsource.sh: Zeile 39: $'\r': Kommando nicht gefunden.
./gencsource.sh: Zeile 69: Syntaxfehler beim unerwarteten Wort `$'do\r''
'/gencsource.sh: Zeile 69: `for((k=0; k<${#TYPE_CAP[*]}; k++)); do
make: *** [makefile:130: src/it/unimi/dsi/fastutil/booleans/BooleanIterable.c] Fehler 2

Basically a lot of "Command not found" errors and syntax errors.
Any idea what to do?

You checked out the sourcecode with git autocrlf (your files have windows line endings \n\r, cygwin expects unix line endings \n). Runing dos2unix gencsource.sh should fix the problems (not sure about the Locale problem though, we'll have to see)

That seems to do something.

@incaseoftrouble it generates the JavaFiles but all Java files are empty. Nothing in there.
this is the log:
$ make sources
cc -w -I. -DSMALL_TYPES=1   -DASSERTS_VALUE=false -E -C -P src/it/unimi/dsi/fastutil/booleans/BooleanIterable.c \
        | sed -e '1,/START_OF_JAVA_SOURCE/d' -e 's/^ /  /' >src/it/unimi/dsi/fastutil/booleans/BooleanIterable.java
/bin/sh: cc: Kommando nicht gefunden.

./gencsource.sh drv/Iterable.drv src/it/unimi/dsi/fastutil/bytes/ByteIterable.c >src/it/unimi/dsi/fastutil/bytes/ByteIterable.c
cc -w -I. -DSMALL_TYPES=1   -DASSERTS_VALUE=false -E -C -P src/it/unimi/dsi/fastutil/bytes/ByteIterable.c \
        | sed -e '1,/START_OF_JAVA_SOURCE/d' -e 's/^ /  /' >src/it/unimi/dsi/fastutil/bytes/ByteIterable.java
/bin/sh: cc: Kommando nicht gefunden.

./gencsource.sh drv/Iterable.drv src/it/unimi/dsi/fastutil/shorts/ShortIterable.c >src/it/unimi/dsi/fastutil/shorts/ShortIterable.c
cc -w -I. -DSMALL_TYPES=1   -DASSERTS_VALUE=false -E -C -P src/it/unimi/dsi/fastutil/shorts/ShortIterable.c \
        | sed -e '1,/START_OF_JAVA_SOURCE/d' -e 's/^ /  /' >src/it/unimi/dsi/fastutil/shorts/ShortIterable.java
/bin/sh: cc: Kommando nicht gefunden.

./gencsource.sh drv/Iterable.drv src/it/unimi/dsi/fastutil/chars/CharIterable.c >src/it/unimi/dsi/fastutil/chars/CharIterable.c
cc -w -I. -DSMALL_TYPES=1   -DASSERTS_VALUE=false -E -C -P src/it/unimi/dsi/fastutil/chars/CharIterable.c \
        | sed -e '1,/START_OF_JAVA_SOURCE/d' -e 's/^ /  /' >src/it/unimi/dsi/fastutil/chars/CharIterable.java
/bin/sh: cc: Kommando nicht gefunden.

you cygwin needs a c / c++ compiler. install build-essential package or similar

Also thanks for the help and your time.

These are "Build" Packages that I have

@incaseoftrouble ok almost everything went right.
Only Problem is that I have around 430 Java errors because the "Linked" Variants of the code weren't generated.

I think this might be due to LinkedXY & CustomXY actually being symlinks which cannot be mapped properly on windows file systems? I'm not sure though. What should work is that you manually copy the symlinks.
If you check out https://github.com/vigna/fastutil/tree/master/drv some of the files have a "symlink" mark. (Linked* and OpenCustom*). Click the linked files to see which file they are linked to. On your repo, replace the link with the linked file. For example, OpenCustomHashMap.drv is linked to OpenHashMap.drv. You should copy OpenHashMap.drv over the OpenCustomHashMap.drv file. Do this for all 6 symlinks and make sources again, this should work I think.

Oh so just a copy paste code of the drv files would be enough. That sounds a lot easier then my idea.
Thanks for the help.
I hope this was the last time i have to do this.

Thanks @incaseoftrouble that was the last of the issues.
(I will try to make a sum up tutorial for others so that this issue won't show up again)


ReSharper 2017.2 Support

Or for rider))

For the ones having this issue, the code is already updated, it's just the build failing and the binaries not uploaded to the R# gallery.  I just cloned the repo and run the build script to generate the .nupkg package, then follow the steps at https://www.jetbrains.com/help/resharper/sdk/Extensions/Deployment/LocalInstallation/InstallCustomSource.html to install a package from a custom source.

Sorry folks that it took so long. It went forgotten multiple times and I actually thought I didn't even update to the latest SDK yet. I just pushed the package to the gallery. I'm closing this, if anything doesn't feel right, please comment/re-open here.
@smad2005 about Rider support: I can't promise anything as of now. If you need it urgent, you could send a PR. As a reference, I would use the Unity plugin or HeapAllocationViewer.


Make restoreMovedElements streaming, by using the FullXmlFilter

Could you rebase on master? The console docker image can't build because it is missing in here.


Add support to release linux aarch64 wheels

Problem
On aarch64, pip install mceq build the wheels from source code and then installs it. It requires the user to have a development environment installed on their system. Also, it takes more time to build the wheels than downloading and extracting the wheels from PyPI.
Resolution
On aarch64, pip install mceq should download the wheels from PyPI.
@afedynitch, Please let me know your interest in releasing aarch64 wheels. I can help with this.

Hi!
Thanks for submitting this issue and the news that it works on aarch64. Do you mean aarch64 on Mac OS?
This is definitely very high on my priority list, just waiting for my new laptop that should arrive within the next few days/week.
If it's aarch64 on linux then your help with that would be appreciated. In this case, please follow the commands in the azure pipeline (the .ci folder) to build and share the wheel.

Closed by PR #35
Cheers.


Install issue  error Code: 0x80070005

Windows version 21H2 Build 19044.1415
having issues installing from Windows store getting  Code: 0x80070005
have tried
sfc /scannow
dism.exe /Online /Clenup-Image /RestoreHealth /StartComponentCleanup
wsreset
any help would be appreciated
thank you

These threads might be helpful: #4686, #7193

Spent four hours trouble shooting, found that it was somewhere in the User registry, also had an issue installing pengwin wsl, even though Ubuntu wsl would install. Finally removed my profile and recreated it and all is working. Thank you

Thanks for confirming!


Laravel/Framework Updated to 5.1.24 but Some Files didn't changed.

Currently I have issue running IronMQ on Laravel 5.1 and opened issue #10828. The issue has been fixed trough pull request #10848. So I updated my laravel/framework to v5.1.24. But files merged with pull request #10848 didn't changed on my vendor.
I've removed all vendor, then clear composer cache before running composer install again.
I also got Class 'IronMQ\IronMQ' not found error. I'm using "iron-io/iron_mq": "4.*" on my composer.json.

In the meanwhile v2 will still work. Seems v4 isn't even namespaced anymore. I've got no idea what the ironmq devs are doing...

Too bad.
I got endpoint error when using v2 of IronMQ.

You need to switch to the old API. There must be something in the settings to do that.

Thanks, I'll look into it

Hi, I also ran into the same issue as @justmyfreak
After doing some digging it seems like the changes from this commit: https://github.com/laravel/framework/commit/ad8ce4ca211f9f55ae006648c00281faf4c92e31
do not exist on the tagged v5.1.24 release
https://github.com/laravel/framework/blob/v5.1.24/src/Illuminate/Queue/Jobs/IronJob.php#L86
for some reason, which leads to the vendor/* files not being updated on composer update
Bad merge? Am I missing something?

It's in the process of getting reverted.

so is v5.1.25 going to include the fix?

No.

Taylor will explain in more detail soon. :)

any updates on this @GrahamCampbell ?

Any updates on this @GrahamCampbell ?

Yes. We reverted the changes. Only is supported. Also note that all ironmq support was removed from L5.2. It's been moved to the community to maintain. Feel free to go to that laravel collective repo and ask them to support this.


Revamped IBM Stencils with only five static simple shapes and new icons for IBM Cloud.

[x] I agree to follow the Code of Conduct that this project adheres to.
[x] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Is your feature request related to a problem? Please describe.
The existing IBM stencils that drawio implemented have been excellent but we are no longer submitting updates.
Describe the solution you'd like
We would like to request drawio's help to implement the revamped new shapes and icons described here with five static simple shapes using existing drawio features as described in the attached IBMShapes.pdf.  This will help to 1) ensure that users can create diagrams for our growing product offerings, and 2) grow our user base on drawio for simple and complex architecture diagrams.  The first slide of IBMShapes.pdf describes how the new shapes relate to the old shapes with the new shapes also being simple and static but in code only and using only icons from svg2xml.
The refreshed and validated icons are in the attached IBMIcons directory which includes icons for our latest product offerings including AI.  All icons are from our design center with quality processes, and we have recently added additional processes for our icons to work flawlessly with svg2xml.
We are planning on only two sidebars and request that More Shapes contain selections for "IBM Cloud" and "IBM Shapes" sidebars with content as described in the attached IBMSidebars.xlsx, and both sidebars will use the same set of icons.  The IBM Shapes sidebar is intended more for non-cloud use cases eventually to grow our user base across multiple use cases in drawio, but most important are the shapes described here for cloud so we can grow our use base on cloud in drawio before looking at other use cases.  Let us know if there is another method for providing the details of the sidebars.
Contents of the attached IBMStencils.zip :

IBMShapes.pdf – Details for the five new static simple shapes.
IBMSidebars.xlsx – Spreadsheet describing contents of IBM Cloud and IBM Shapes sidebars.
IBMColors.json – Contains the IBM Color Palette.
IBMFonts.json – Contains the IBM Plex Fonts on Google Fonts.
IBMIcons – Directory of icon svg files ready for input to svg2xml.

Our shapes will only be static simple shapes using existing drawio features.  For other features/complexities we are looking at handling on our side with diagram-as-code as appropriate.  We look forward to the drawio team's implementation of this revamped proposal and refining to meet drawio's recommendations and standards.  Let us know if you need anything else or have any questions.
Describe alternatives you've considered
No alternatives, drawio is the best tool.
Additional context
Refer to IBMShapes.pdf for screenshots and details.

Before assigned, the IBMStencils.zip has been kept updated.
When IBM Cloud and IBM Shapes sidebars are available in More Shapes the old sidebar named "IBM" can be removed from More Shapes.
The old nine templates under Cloud / IBM can be removed and we will submit new templates when IBM Cloud and IBM Shapes sidebars are available.
Let me know if you have any questions.
Thanks!

Just a couple questions:

Can you please describe in more detail what dropin images are?
Is defining icons ok via style (there is an entry in the style string that defines what icon will be used) or does it explicitly has to be dropping on a shape/container?
What is the difference between IBM icons and dropin images?
If you want to use IBM Plex Sans, you have to host is somewhere and the link must not change over time. Otherwise the app will regress to a default font.

Is the updated IBMStencils.zip attached?
Also, adding new icons is fine, but please keep the changes to existing ones to a minimum, since we have to keep all the old variants too.

For questions 1-3,
the style for the VPC Virtual Server under IBM Cloud / Compute might include the following depending on your naming:
shape=mxgraph.ibmshapes.pnode;icon=ibm-cloud--virtual-server-vpc;image=;
The IBM Icon refers to that "icon=ibm-cloud--virtual-serer-vpc;" where that icon name is in the xml you create with svg2xml.  The IBM Icon is used but if an image is dropped onto the shape then that image is used and the "image=;" is set to the image.  May be useful if another icon is needed that is not available in the sidebar, but not used very often if you don't think the "image=;" is needed.

For question 4,
the IBMFonts.json included in the IBMStencils.zip defines the fontCss from Google Fonts and customFonts names.
The fontCss will eventually need to include IBM Plex for Chinese which is not available yet.
We have been using semibold (weight 600) for shape labels by using the following along with "html=1;" specified in the style:
<b style='font-weight:600>label
I didn't mention using semibold for shape labels in the ppt/pdf in IBMStencils.zip because semibold is not available in the Format Panel which has bold (weight 700), but if shapes labels can start out with semibold that would be a consideration, connectors labels always start out with regular font.  I understand too if using semibold would be confusing with the bold in the Format Panel.

For the IBM Shapes (non-cloud) sidebar, "icon=undefined;" would be set in the style on each shape.
It would be useful if there is an existing way for users to select from a list of icon names available in ibmicons.xml (or whatever name you use for the output of svg2xml) which then changes the "icon=undefined;" to another icon name , but alright if not.  The icon name can also be specified in our diagram-as-code that I mentioned in the original description which helps create drawio files.

With the fonts the issue is where will the fonts be hosted? We don't host fonts. There needs to be a fixed, permanent, URL where the font files can referenced from.
Or, each user needs to install the fonts locally (which they'd have to do on the desktop version, anyway).

Sure, defaulting to a regular font is alright too and we mention another font (Arial if I recall) as being similar to Plex.
Plex would be desired if possible.
Users can install the Plex fonts locally.
And is Google Fonts considered a permanent location?
The IBMFonts.json was created from Google Fonts:
{
"fontCss":
"@import url('https://fonts.googleapis.com/css2?family=IBM+Plex+Sans+Arabic:wght@400;600;700&family=IBM+Plex+Sans+Devanagari:wght@400;600;700&family=IBM+Plex+Sans+Hebrew:wght@400;600;700&family=IBM+Plex+Sans+JP:wght@400;600;700&family=IBM+Plex+Sans+KR:wght@400;600;700&family=IBM+Plex+Sans+Thai:wght@400;600;700&family=IBM+Plex+Sans:ital,wght@0,400;0,600;0,700;1,400;1,600;1,700&display=swap');",
"customFonts":
["IBM Plex Sans",
"IBM Plex Sans Arabic",
"IBM Plex Sans Devanagari",
"IBM Plex Sans Hebrew",
"IBM Plex Sans JP",
"IBM Plex Sans KR",
"IBM Plex Sans Thai"]
}
This was created following this blog.  I was impressed with how Google Fonts puts everything together nicely into a single import statement.
The blog is for Confluence which may not apply generally.
We have a permanent location for Plex also which I will need to look into.
Thanks!

fontFamily of IBM Plex Sans is the default.
With customFonts in IBMFont.json the complete list in the Format Panel is
IBM Plex Sans
IBM Plex Sans Arabic
IBM Plex Sans Devanagari
IBM Plex Sans Hebrew
IBM Plex Sans JP
IBM Plex Sans KR
IBM Plex Sans Thai
And Chinese will come later.
If that is too much to implement initially, the IBM Plex Sans is primary use case to start with.
If Arabic, for example, is intended for an entire diagram, setting fontFamily to Arabic is done for each shape unless there's a way to have Arabic be used for an entire diagram.

Let me know if you have any questions about IBMColors.json and IBMFonts.json in IBMStencils.zip.
Before our mtg we were specifying the font source in the styles and manually selecting stroke and fill colors that correspond to the IBM Color Palette.
Then we moved to the drawio customization:

For IBMColors.json, using the drawio customization for the IBM Color Palette is excellent, and customizing the color schemes is perfect for us since we recommend using a light fill color from the same color family as the border color or a fill color of white, all combinations are included in our color schemes, and IBMColors.json enables users to know what colors are in the IBM Color Palette.
For IBMFonts.json, using the drawio customization to include the import url from Google Fonts is also excellent, and we can easily give you an updated import url when Chinese is added to Plex.  Google Fonts also makes it easy to create a single import url of all the Plex fonts we select.  If you include this import url we would verify all of the languages.

The IBMColors.json (won't change) and IBMFonts.json (easy change when Plex Chinese is available which is the last Plex language) are examples for your consideration, depending on your direction and how customization is intended.
Thanks!

For the IBM Cloud sidebar sheet in IBMSidebars.xlsx with IBMShapes.pdf included in IBMStencils.zip:

Categories are Groups, Actors, AI, Compute, Containers, Data, DevOps, Network, Observability, Security, Storage, and Connectors (Connectors have labels with end types in parens if any).
Zone group shape types are set to container=0 and strokeWidth=2 with dashed border.
All other group shape types are set to container=1 and strokeWidth=1 (one case of strokeWidth=2) with solid border.

For Zone groups with container=0, the excellent feature you implemented for our previous sidebar still applies where selecting shapes inside of container=0 groups may require using alt-click or option-click or change the Z order by moving shapes backward.
The container=0 versus container=1 can be easily missed with the groups, so just emphasizing the container=0 for Zone groups only.
Thanks!

Please check out our implementation and let us know if it matches your expectations.
https://ea8f1c63.drawio-dev.pages.dev/

Everything looks excellent and works perfectly!
Depending on your direction, is it alright for shape labels to default to 14px and connector labels 12px (although connector labels are added by user), and will changing strokeColor of group border also change sidebar color and icon color.
Only a few minor corrections for your consideration:

In Groups category, typo in spreadsheet if you can please change label “Generic Core” to “Generic Group”.
In Containers category, if you can please change label for OpenShift, and add Kubernetes Service icon from spreadsheet.
In Data category, if you can please change labels for MongoDB and MySQL to mixed case.
In Network category, typo in spreadsheet if you can please change Classic Load Balancer to use internal icon name “load-balancer—classic”.

Thanks!

The minor corrections and shape label sizes are done, but connector labels and color changing needs additional code on project level and there are no short or medium term plans to implement

I understand, we are continuing to exercise and really like the implementation, thanks!

If still possible, could you please check the container=0 behavior of the Zone shapes in the Groups category:

Shapes placed inside the Generic Zone shape don't move when the Generic Zone is moved which is working as expected.
However, shapes placed inside the other Zone shapes (such as Security Group) are moving with the Security Group as if container=1 is set but container=0 is set.

Thanks!

Sounds good, all of the Zone shapes correctly have container=0 set, and we'll look forward to the fix in a follow-on release so shapes placed inside the Zone shapes don't move when the Zone shape is moved, which applies to all of the Zone shapes (Access Group, Account Group, Resource Group, Security Group, Instance Group, Placement Group, and Availability Zone).  Thanks!


Need to think about an update strategy

pkg upgrade seems to overwrite our configuration.
For example, our changed rc scripts are seemingly replaced with those from the pacakges, and our carefully configured slim.conf is moved to slim.conf.pkgsave.
How does e.g, the Mac work when "over-installing" a newer version of the OS?

Was the slim.conf.pkgsave effect reproducible?

… How does e.g, the Mac work when "over-installing" a newer version of the OS?

IIRC it's quite blunt. Something like, writing all affected files to a temporary path then a move into place – typically without considering the before/after state of things (because things are so neatly separated).

The OS in a read-only image + user modifications in a read-write unionfs overlay would solve this very neatly...

https://github.com/helloSystem/hello/issues/37#issuecomment-751341802 recently I might have misunderstood what's to be expected within the constraints of 'live' media.
If the OS will be read-only: that's fine, if:

user data is saved to persistent storage on the medium; and
user data will be transferable, fairly easily, from an old medium (i.e. outdated OS) to a new medium (up-to-date OS).

Alternatively: some way to periodically download then 'slot in', to the medium, an updated OS?

What I mean is that the OS should be conained in a read-only file (in the worst case: ZFS dataset) but overlayed with read-writeable space so that any changes by the user get redirected into the read-writeable area. This way, the system appears entirely r/w to the user.
When an OS update comes along, the read-only file is switched to another one, but the user-generated content "on top" still is there.

Overlapping:

System Update/Upgrade utility · Issue #33 · helloSystem/Utilities

https://github.com/helloSystem/ISO/issues/55#issue-775400817

pkg upgrade seems to overwrite …

Not a problem with pkg(8); see https://github.com/helloSystem/hello/issues/67#issuecomment-772307831
From https://github.com/helloSystem/ISO/issues/135#issuecomment-800032910:

A system that works for, not against, you

Freedom: lets you load software without restrictions. … easy to modify applications. …

https://github.com/helloSystem/ISO/blob/56a03e4d5e3514e9b08857ccf80d84c4d9a53aec/settings/script.hello#L329
pkg-upgrade(8) is not pkg-update(8)
https://github.com/helloSystem/ISO/blob/56a03e4d5e3514e9b08857ccf80d84c4d9a53aec/settings/script.hello#L328
Why are there still these restrictions?
Why work against the natural desire to install, or update, an application?
This is the opposite of ease.

Why work against the natural desire to install, or update, an application?

Because you can end up with an untested, messed up, and potentially no longer working system.
helloSystem is (also) for non-technical users who should never have to see a terminal.
Note that it says "not yet". We still "need to think about an update strategy" and it will probably be more like Android/iOS.

end up

Are boot environments forgotten again?
https://github.com/helloSystem/Utilities/issues/33#issuecomment-800529129

non-technical users who should never have to see a terminal.

If you stop crippling pkg, these users will be blissfully unaware of the improvement.

Boot Environments just allow to go back once disaster has struck. I want to avoid the disaster in the first place.

pkg is not mentioned

Maybe not mentioned at https://hellosystem.github.io/docs/index.html but pkg is integral to FreeBSD.
From https://github.com/helloSystem/hello/issues/110#issuecomment-776911839:

… we want to stay "true FreeBSD" wherever possible. So at least for what comes as part of the Live ISO, I think the existing system wih ports and packages serves us well …

From main page https://github.com/helloSystem/hello#what:

Based on FreeBSD. Less, but better!
… Without lockdown. Without Big Brother. The user in full control.

Crippling pkg, for any length of time, does not make helloSystem better than FreeBSD.

Boot Environments just allow to go back once disaster has struck.

That's a somewhat narrow vision.
Less dramatically: boot environments allow you to progress¸ with the option of going back.
If there's a fear of boot environments, then why is Boot Environments amongst the preferences for helloSystem?

There is no fear of BEs, but

We need a way to select BEs from the bootloader, which means we need to work on the bootloader to be completely silent unless a key combo (e.g., Command-B-E for Boot Environments) is held down
In order to be able to update the system (not just some apps) using pkg, someone would need to package all of helloSystem

In order to be able to update the system (not just some apps) using pkg, someone would need to package all of helloSystem

If you use SystemImageKit and have a pure FreeBSD image with an early init script to copy over hello-specific files from a second image with geom_rowr or something, then you can use rsync to update the two.
Does this simplify things or complicate them?

So your idea is to have one image that contains an unmodified FreeBSD, and another one that contains the helloSystem modifications? Would work but would require a working union filesystem, because geom_rowr doesn't allow you to use the second image without the original first image.
Also, I think even updating an unmodified FreeBSD is too complicated. I'd just replace the whole image with a new one.

Yeah, pretty much. Even if you mount /var from a persistent partition, and symlink /etc, /Applications, and /Users to subfolders of /var, like Silverblue does, would that work?

up.bsd.lv Binary Upgrades Report
From https://up.bsd.lv/:

2021-07-26 - This Proof of Concept was extremely informative and is taking a break to build a Proof of Concept replacement for freebsd-update. Thank you everyone who helped out with it!

pkg is integral to FreeBSD.

More accurately:

the normally used pkg is a port – ports-mgmt/pkg
the pkg that's installed with FreeBSD is more like a stub – when run, it invites installation.

https://cgit.freebsd.org/src/tree/usr.sbin/pkg

https://github.com/ChimeraOS/frzr

frzr is a deployment and automatic update mechanism for operating systems. It deploys pre-built systems via read-only btrfs subvolumes, thus ensuring safe and atomic updates that never interrupt the user.
Updated system images are downloaded at boot time and deployed to an entirely separate subvolume so as not to interfere with the currently running system. Upon next boot, the newly installed system is started and the old one is deleted, completely seamlessly and invisibly.
Persistence is handled by making /home and /var separate subvolumes mounted into the read-only root. /etc is made read-write via overlayfs.

Goes in the direction I mean. But I'd like to use filesystem image files rather than btrfs subvolumes.

Yes, that might be just what we need. As for the btrfs thing, we might be able to fork it...

No btrfs for me, so much is sure ;-)

Yes, and that's why we could fork it. ZFS is welcome, right? And that naturally uses a similar but more flexible subvolume system. So, could we not fork it and add ZFS support?

Or yeah, there's the option of having ufs2 images that can be downloaded, that may be better.

Also I want my Firefox updates ;~)

As for Firefox, it shouldn't be too hard to make Application Bundles for each version.

I want my Firefox updates

What's the issue?

Oh, I posted it on a separate issue.
Basically, updating Firefox updates Inkscape, python and Thunderbird, uninstalls krita and removes Qt, which explains why krita went but hey, now the system doesn't boot because my display manager crashes cause of no qt libraries.

a separate issue

Link please. Thanks.

I'll try find it in all these repositories

Quick hint: Entering h something in Falkon will search for "something" in all helloSystem GitHub repositories.

https://github.com/search?q=org%3AhelloSystem+author%3Akettle-7+Firefox&type=Issues | https://github.com/search?q=org%3AhelloSystem+commenter%3Akettle-7+Firefox&type=issues @kettle-7 @kettle-7 I can't find it. If not Firefox, what was the keyword?

FreeBSD PkgBase
https://github.com/helloSystem/hello/issues/37#issuecomment-751341802
https://github.com/helloSystem/hello/issues/161#issuecomment-927250627
https://github.com/helloSystem/ISO/issues/166

https://github.com/helloSystem/ISO/issues/267#issuecomment-917700004

#267 (comment)

Thanks, that's somewhat lost (with unanswered questions) in the midst of an issue that may be non-related. I suggest raising a separate issue so that it can be actionable and closable.

I feel it is an issue with pkg

I feel it is an issue with pkg,

I doubt it.

not helloSystem.

Have you raised the issue elsewhere?
Installation and Maintenance of Ports or Packages | The FreeBSD Forums
https://github.com/freebsd/pkg/
https://lists.freebsd.org/subscription/freebsd-pkg

During initial installation, the installer uses bsdintall zfsboot to create and mount ZFS partitions:
zroot/ROOT/default on /mnt (zfs, local, noatime, nfsv4acls)
zroot/tmp on /mnt/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /mnt/usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /mnt/usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /mnt/usr/src (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /mnt/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /mnt/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /mnt/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /mnt/var/mail (zfs, local, nfsv4acls)
/dev/ada0p1 on /media/msdosfs (msdosfs, local, noatime)
zroot/var/tmp on /mnt/var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot on /mnt/zroot (zfs, local, noatime, nfsv4acls)

On the running system, it will end up as
% zfs list
NAME                 USED  AVAIL     REFER  MOUNTPOINT
zroot               1.40G   214G       96K  /zroot
zroot/ROOT          1.36G   214G       96K  none
zroot/ROOT/default  1.36G   214G     1.36G  /
zroot/tmp            264K   214G      264K  /tmp
zroot/usr           41.6M   214G       96K  /usr
zroot/usr/home      41.3M   214G     41.3M  /usr/home
zroot/usr/ports       96K   214G       96K  /usr/ports
zroot/usr/src         96K   214G       96K  /usr/src
zroot/var            692K   214G       96K  /var
zroot/var/audit       96K   214G       96K  /var/audit
zroot/var/crash       96K   214G       96K  /var/crash
zroot/var/log        212K   214G      212K  /var/log
zroot/var/mail        96K   214G       96K  /var/mail
zroot/var/tmp         96K   214G       96K  /var/tmp

Currently there is no way for the installer to install a newer version of the operating system "over" an existing installation, because the installer always runs bsdintall zfsboot first.
We would like to change this so that users can install a newer version of the operating system over an existing one. For this we would like to:

Snapshot the whole zroot
Then clone it to make it writable and mount the clone at mnt and its subdirectories in the same manner as zfsboot install does?`
Install the operating system over the clone
Change whatever needs to be changed so that the next boot (only the next boot) will be from the clone
Once the cloned system is fully booted all the way into the desktop, change whatever needs to be changed so that all subsequent boots will be into the snapshot. Possibly ask the user for confirmation.

What are the exact commands needed to do this from the Live ISO?
Actually, since we would be using a clone, we might not even need to boot into the Live ISO to install the new OS over the clone but could do it from the running system. At least that is how I understand it - that you can snapshot and clone the running system without bad side effects.
Is creating a "recursive snapshot" what we need here? sudo zfs snapshot -r zroot@updated.
We now can do ls /.zfs/snapshot/updated/.
Now we can make a writable clone:
sudo zfs clone zroot@updated zroot/updatedclone.
Then we get sudo ls /zroot/updatedclone but there is nothing in there.

Using Boot Environments would probably be the correct way to do this, but Boot Environments do not cover all zfs datasets, at least not by default. So none of these are covered by Boot Environments:
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)

Can this be changed?

Just using cpdup -o overwrites critical files n /etc like the user and password files. We would need to find a way to spare them out.

vermaden says:

Its pointless to have /home or /usr/ports on ZFS Boot Environment
but if you must, then just move these needed datasets
into the BE like that:
zfs rename -u zroot/usr/src zroot/ROOT/default/usr/src

How can we come up with a list of files in /etc that must not be touched by over-installing a new OS? /etc/passwd is an obvious one but which others are like it?

When running from the Live ISO, can we get the whole zpool with all of its datasets mounted under /tmp so that the installer could do its thing there without having to format the whole disk?
# zpool import -f -R /mnt zroot
cannot mount '/mnt/var/crash': failed to create mountpoint: Read-only file system
cannot mount '/mnt/zroot': failed to create mountpoint: Read-only file system
cannot mount '/mnt/zroot': failed to create mountpoint: Read-only file system
cannot mount '/mnt/usr/home': failed to create mountpoint: Read-only file system
cannot mount '/mnt/usr/home': failed to create mountpoint: Read-only file system
cannot mount '/mnt/usr/home': failed to create mountpoint: Read-only file system
cannot mount '/mnt/usr/home': failed to create mountpoint: Read-only file system
cannot mount '/mnt/var/tmp': failed to create mountpoint: Read-only file system
cannot mount '/mnt/usr/ports': failed to create mountpoint: Read-only file system
cannot mount '/mnt/usr/src': failed to create mountpoint: Read-only file system

How can we come up with a list of files in /etc that must not be touched by over-installing a new OS?

Maybe something like
etc/resolv.conf
etc/fstab
etc/passwd
etc/spwd.db
etc/pwd.db
etc/master.passwd
etc/group
etc/wpa_supplicant.conf
etc/login.conf.db
etc/localtime
etc/opiekeys
etc/hostid
etc/rc.conf.local

Are there more?

TODO: Make use of kenv zfs-bootonce.

@jsm222 helloDesktop https://wiki.freebsd.org/helloDesktop seems like a very good idea, if it can be in ports for FreeBSD https://github.com/helloSystem/hello/issues/161 and maintained.

@jsm222 helloDesktop https://wiki.freebsd.org/helloDesktop seems like a very good idea, if it can be in ports for FreeBSD helloSystem/hello#161 and maintained.

Would be a nice thing indeed. I looked a little into it again today, but got caught up with qterminal crashing, when the menu is running..  When all packages needed for helloSystem are in the ports tree freebsd-upgrade and pkg upgrade would indeed not break anything.

For the record https://github.com/lxqt/qterminal/issues/992

@jsm222 https://github.com/helloSystem/ISO/issues/51

https://github.com/helloSystem/hello/discussions/377#discussioncomment-3728908

…

pkg check --dependencies --dry-run

pkg-check(8)

Maybe we should have Update.app run this command always as its first thing? …

If you allow pkg-upgrade(8) to complete its work (for your current case, you should probably run repeatedly), then pkg check --dependencies --dry-run should find nothing missing.

Hello,
i hope i am not in the wrong post/issue, but before finding this post i decided to update the BSD system below hellosystem following those steps
https://www.freebsd.org/releases/13.2R/installation/#upgrade
now the system is updated, but i believes something wrong happened ... the video driver is not found anymore (standard i915) and quite a lot of error (e.g. runnig falkon or dock)
No more logout, no more reboot or halt from the menu (still working if i do sudo reboot / sudo halt from terminal but halt is hanging forever) and other stuff probably.
not a big issue is a test machine i am playing with so i can re-install the whole shebang
but i think i learned to not run the update of the system as i did.
which one is the right way? should i wait for the next ISO? just out of curiosity
thanks
M

Hello @m4lin. Currently we don't have an update system in place. So you need to do a clean reinstallation of helloSystem every time. We know that we need to think about an update stratefy, but so far we haven't found a good way of doing it. One way would be to package all helloSystem components as proper FreeBSD packages, but so far no one has succeeded in doing this.

We should use the Install.app in the built image to upgrade the system, which only needs to replace the system files, which can also preserve user data and test the latest developer preview version.

https://github.com/helloSystem/ISO/issues/55#issuecomment-986067729
This is what I expected too.

Point in case:
https://github.com/helloSystem/hello/issues/523
User updated all the packages using pkg, now the system is broken and there is no more Xorg.


Differences between matlab script and previous fsl shell script?

The former fsl script dvars.sh and DVARSCalc.m give different values for RDVARS although perfectly correlated. Is this a problem? Which implementation is preferred?

Hi Cameron,
tl;dr: to have a standardised DVARS measure which is agnostic to intensity normalisation and comparable across cohorts/subject, I would use pDvar from DSEvars.m. Note there are similar standardised measure for the slow components too.
I suppose the implementation for Standardised (now called Relative) DVARS uses AR1 estimation available via fslmaths original standardised DVARS while DVARSCalc.m uses a robust estimate of autocorrelation via intraclass correlation coefficient -- see madicc function inside DVARSCalc. I have never seen dvars.sh so I can't certainly comment about its function.
TBH, I prefer neither. To get a standardised DAVRS (which is comparable across cohorts/subjects and is invariant of intensity normalisation) you can use pDvar which is DVARS normalised by the whole variability (i.e. A-var) and is available via DSEvars.m. Alternatively you can also use SDVARS_* in DVARSCalc.m which is the back-projection of p-values to either a normal or Chi^2 distribution.


Create a strip frameworks script?

i.e. https://github.com/realm/realm-cocoa/blob/master/scripts/strip-frameworks.sh

Maybe we can just link to that one from the README?

I'd rather we have one that can be invoked from the Rome directory


Add additional Prometheus metrics

Feature Description
It's great that people are aware of Prometheus, but what is the usefulness of metrics added in #678 ?
Realistically these can't be used for alerting.
How about we add some actually useful metrics:

Successful/failed SSH login attempts
Successful/failed login attempts via web
Currently open TCP connections (recently our Gitea gets hammered by DDoS attacks)
HTTP response latency histogram
Errors encountered while interacting with Git repos
Failed/successful webhook calls

These were just few to mention. Some of these can be handled by haproxy that sits in front of Gitea which is the case for us of course...
Screenshots
No response

If I am reading right Gitea uses Chi framework so enabling some of those metrics is trivial simply by adding, just make sure no high entropy labels are included: https://pkg.go.dev/github.com/m8as/go-chi-metrics

Merged into #14724


getArrayObserver, 2 push triggers unexpected results

Hi
This might just be me again misunderstanding how its supposed to work.
I have this array observer on a collection in my grid element  binded to parent collection Im working on.
let arrayObserver = this.observerLocator.getArrayObserver(this.collection)
      arrayObserver.subscribe(function (changes) {
        debugger;
      }.bind(this));
      this.subscriptionsArray = arrayObserver

Then I I call function to push in 2 more: (over look the
  push(data){
    this.collection.push(data[0]);
    this.collection.push(data[1]);
}

Now the unexpected comes.
My observer gets called...
addedCount:2
index:0
removed:Array[0]

I then though, ok, great I they have added added 2 from index 0.
I can just add those to my cloned array inside the custom element.
But it gets called again, and with same data....
So is it just me thinking this is not right?
Ofc good chance Im doing something wrong... again...

Sounds like you might have called subscribe multiple times with the same array.  Here's a gist that shows it's working properly:
https://gist.run/?id=41d5fb436a1d630ce8a6

Restared gulp watch and now it works normal.
Sorry.


Project scope

This is a topic that came up briefly on Slack but I think it's worth putting here to gather some opinions.
I think it's crucial that we be strict about what features get included as there are plenty that are certainly relevant to forums, but not specific to them. An important distinction between this project and, say, Flarum, is that Flarum is a standalone forum product, whereas this is a package intended to bring forum capabilities to a Laravel/Lumen application. People will install Flarum and get a forum with all the bells and whistles, but IMO that shouldn't be the case here.
So there are some things I would call out of scope, which currently includes #63 and #81 (maybe #23 and #32 as well). I think they should either be totally omitted, or developed in parallel as separate packages, or at the very least disabled/easy to remove out of the box.
Thoughts?

Having thought about it some more over the past few days I agree. I still think we should include statistics (#23), even if never visible on the frontend. As for #32 we only really need an admin interface for managing groups imo, we can figure out another way of doing this or leave it to the end user to integrate within their own application.

Let's define in this issue what all we want to cover in MVP. That way we can list what is left to be done

I feel everything else assigned to the 1.0 milestone would be great for MVP.

that's true but we still don't know what is left

Pretty sure we've got this sorted out now.


How to add support for thread binding cpu_cores ?

see  https://github.com/lix19937/multithreaded_programming/blob/main/26_threadpool/src/thread_pool.hpp


Patches get silently overridden on package update / native compilation

Hi,
A few of my patches keep getting silently overridden, I assume because the original definition is reevaluated by native compilation.
Using advice as suggested in #60 would be one way to fix this, perhaps.

Yes, that's a known issue. There are reasons to stick to the existing way of doing things, as mentioned in https://github.com/radian-software/el-patch/issues/60#issuecomment-1126230778, but I suppose it would not be a bad idea to at least make it customizable whether the user would prefer for functions to be patched using advice rather than being overwritten.

See #63 for a possible implementation and further discussion on the pitfalls of advising.


Support/various multibuy 1.5 fixes

❓ Context

Impacted projects: live-desktop

This PR add various small improvements to the multibuy feature

added dynamic buttons in empty state accounts
cleaned the architecture a bit and stopped using old platform provider

✅ Checklist

[x] Test coverage: Did you write any tests to cover the changes introduced by this pull request?
[x] Atomic delivery: Is this pull request standalone? In order words, does it depend on nothing else?
[ ] No breaking changes: Does this pull request contain breaking changes of any kind? If so, please explain why.

📸 Demo

🧪 Test Cases covered
Compared the following existing behaviour on this feature branch with the prod app:

Buy/Sell apps load correctly ✅
Add account via Multibuy 1.5
Discover apps load and work correctly (with experimental, debug and staging) ✅
Navigate to card from navbar ✅

New behaviour verified:

Dynamic buttons in empty accounts correctly generate and go to multibuy

🚀 Expectations to reach
Please make sure you follow these Important Steps.
Pull Requests must pass the CI and be internally validated in order to be merged.

/generate-screenshots

/generate-screenshots

/generate-screenshots

@Justkant @IAmMorrow I noticed Loopipay is shown twice in the feature branch but not in a prod app with the same app settings


Multi-gpu support?

Hello,
Can have DataParallel support?

I'm really sorry, but we do not have a plan to support the DataParallel function.
Sorry for the late response.