12 UNIMPLEMENTED: unknown service events.Events

Hi,
I've been trying to fetch events data from mainnet1 forward with gRPC calls and I have been getting this error. I tried different clients, same thing.
    err: {
      "type": "Error",
      "message": "12 UNIMPLEMENTED: unknown service events.Events",
      "stack":
          Error: 12 UNIMPLEMENTED: unknown service events.Events
              at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/src/call.ts:81:24)
              at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client.ts:343:36)
              at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:430:34)
              at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:392:48)
              at /app/node_modules/@grpc/grpc-js/src/call-stream.ts:299:24
              at processTicksAndRejections (node:internal/process/task_queues:78:11)
      "code": 12,
      "details": "unknown service events.Events",
      "metadata": {
        "content-type": [
          "application/grpc"
        ]
      }
    }

syntax = "proto3";

import "google/protobuf/timestamp.proto";

package events;

service Events {
  rpc GetEventsForHeightRange(GetEventsForHeightRangeRequest) returns (GetEventsForHeightRangeResponse) {}
}

message GetEventsForHeightRangeRequest {
  string type = 1;
  uint64 start_height = 2;
  uint64 end_height = 3;
}

message GetEventsForHeightRangeResponse {
  message Result {
    bytes block_id = 1;
    uint64 block_height = 2;
    repeated Event events = 3;
    google.protobuf.Timestamp block_timestamp = 4;
  }
  repeated Result results = 1;
}

message Event {
  string type = 1;
  bytes transaction_id = 2;
  uint32 transaction_index = 3;
  uint32 event_index = 4;
  bytes payload = 5;
}

import * as grpc from "@grpc/grpc-js";
import * as protoLoader from "@grpc/proto-loader";

const protoObject = protoLoader.loadSync(
		path.join(__dirname, 'events.proto'),
		{
			keepCase: true,
			longs: String,
			enums: String,
			defaults: true,
			oneofs: true,
		},
);
const definition = grpc.loadPackageDefinition(protoObject);
const client = new definition.events.Events(
     "access-001.mainnet1.nodes.onflow.org:9000",
     grpc.credentials.createInsecure(),
);
client.getEventsForHeightRange(
	{
		type: eventType,
		startHeight: startBlock,
		endHeight: endBlock,
	},
	(err, data) => {
		console.log(err);
                console.log(data);
	},
);

When I tried with mainnet7 forward, the message was a bit different "message": "12 UNIMPLEMENTED: ", with date and server in metadata obj "date": ["Fri, 10 Dec 2021 15:29:42 GMT" ], "server": ["envoy"]

Fixed, it was the package name and service name that needed to be correct.


Add labels as metadata to features

What this PR does / why we need it:
-> Extension for #463
Update (@suwik):

Addressed previous review comments
Added labels on the feature set level
Removed the python SDK changes related to labels implemented inside field.py (30d63fa0fa)
Made a small unit test refactoring on the way.

Which issue(s) this PR fixes:
Fixes #463
Does this PR introduce a user-facing change?:
Feature spec and Feature set spec will both have a new field called labels

Hello @woop I cut the branch from the #499 and latest of #508 and implemented the SDK for the same
When this request gets merge, we can close #499 and #508 altogether

/ok-to-test

Implementation Detail
-> Cut Branch from #499
-> Cut Branch from #508
-> Implement the new python SDK for labels

Demo List Features
from feast import Client,Entity, ValueType, FeatureSet, Feature
from pprint import pprint

client = Client(core_url="localhost:6565", serving_url="localhost6566")
client.set_project("test")

client_features = client.list_features()
pprint(vars(client_features['city']))

Output
{'_dtype': <ValueType.STRING: 2>,
 '_labels': {'key_test': 'value_test'},
 '_name': 'city'}

Demo List Feature Set
Requirements
Add this function so it can prety print the feast/feature.py
    def __repr__(self):
        return "<Test name:%s dtype:%s labels:%s>" % (self._name, self.dtype, self.labels)

Create the python function to list feature set and print the features
from feast import Client,Entity, ValueType, FeatureSet, Feature
client = Client(core_url="localhost:6565", serving_url="localhost6566")
client.set_project('juan')
test = client.list_feature_sets(project="juan",name="driver",version="1")
for featureset in test:
    print(featureset.features)

Result:
[<Test name:city dtype:ValueType.STRING labels:{'key_test': 'value_test'}>]

Example Set Labels
from google.protobuf.duration_pb2 import Duration
from feast import Client,Entity, ValueType, FeatureSet, Feature

client = Client(core_url="localhost:6565", serving_url="localhost6566")
client.set_project('juan')

feature1=Feature(name='total_transaction4',dtype=ValueType.INT64, labels=dict())
feature1.set_label("test","tost")
feature1.set_label("tast","tust")
feature1.set_label("bost","bust")

customer_fs = FeatureSet("customer_transactions",
                         entities=[Entity(name='customer_id4', dtype=ValueType.INT64, labels=dict())],
                         features=[feature1],
                         max_age=Duration(seconds=432000))
client.apply(customer_fs)

Result
Feature set updated/created: "customer_transactions:"

DB
feature_set_id          | juan/customer_transactions:5
bool_domain             |
domain                  |
float_domain            |
group_presence          |
image_domain            |
int_domain              |
labels                  | {"tast":"tust","test":"tost","bost":"bust"}
mid_domain              |
name                    | total_transaction4
natural_language_domain |
presence                |
project                 | juan
shape                   |
string_domain           |
struct_domain           |
time_domain             |
time_of_day_domain      |
type                    | INT64
url_domain              |
value_count             |
version                 | 5

/retest

@imjuanleonard we absolutely need tests in order to sign this off. Can you please add them in the same PR?

https://github.com/gojek/feast/pull/536#issuecomment-599852464
The test has been added

I was a bit misled by the PR title and trying to follow the history from #499—this PR is not only Python SDK, it's complete implementation of feature labels.
Conflicts need to be fixed now (mostly generated files that have been removed from Git)—if in the process you could rebase and squash some of the false-start commits like list_features functionality that was removed, and typos, formatting, and comment updates, that would be fantastic. Not absolutely essential for you to do it, but being a significant PR, I'd love to be able to review it commit-by-commit.

/retest

https://github.com/gojek/feast/pull/536#issuecomment-602178208
Hi @ches , have rebased and cleanup the commit as you suggested, thanks for the input.
Ah by the way, for the fixed typo it is actually when I am reading the code, saw some mistype. So I was thinking to make the change on this branch together, But I guess to avoid misleading, better to create it on another branch to do the typo fixing :)

/retest

Demo List Features
from feast import Client,Entity, ValueType, FeatureSet, Feature
from pprint import pprint

client = Client(core_url="localhost:6565", serving_url="localhost6566")
client.set_project("test")

client_features = client.list_features()
pprint(vars(client_features['city']))

Output
{'_dtype': <ValueType.STRING: 2>,
 '_labels': {'key_test': 'value_test'},
 '_name': 'city'}

Is there a reason why it is printing keys with underscores? Aren't those private fields?

def __repr__(self):

    return "<Test name:%s dtype:%s labels:%s>" % (self._name, self.dtype, self.labels)

I don't see this in the code.

You are introducing a user facing change, so please update your release note.
I believe we should prevent empty label keys. You need a test for that.
We need to test that removal of labels works. (both Core API and Python)
We need to test that comparison of feature sets work (when they have labels). This can just be an extension of an existing test if there is one. (both Core API and Python)
The labeling addition that you have made to the notebook feels very out of place. Please remove it for now.
Open comments need to be resolved.

You are introducing a user facing change, so please update your release note. ✅
I believe we should prevent empty label keys. You need a test for that. ✅
We need to test that removal of labels works. (both Core API and Python) ✅ Its working on my previous basic.ipynb, users will need to get their feature set first, and kind of update the feature using the apply method
We need to test that comparison of feature sets work (when they have labels). This can just be an extension of an existing test if there is one. (both Core API and Python)
-> yes the existing test already working with this one ✅

The labeling addition that you have made to the notebook feels very out of place. Please remove it for now. ✅  Removed
Open comments need to be resolved.

 Its working on my previous basic.ipynb, users will need to get their feature set first, and kind of update the feature using the apply method
This requires a test.

yes the existing test already working with this one

Can you point me to where this happens?

https://github.com/gojek/feast/pull/536#issuecomment-605896686
Added on the test_client.py
it is on the test_feature_set_apply_success
on line 450 , there is a remove labels part
and when applied
online 476 there are only one feature

Hello @ches, if you don't mind, can you help to review this PR?
Thanks

Hi @imjuanleonard, sorry for the delay, I'll take another look now.
From last time I reviewed and the PR description, this is for feature level only and not feature set level yet, right? Totally reasonable to add feature set level in another PR, but I think we should not say this one closes #463 then, since it says:

Unless there are any objections, we will implement labels as a map<string, string> at the feature level for 0.5. We will also add basic getters/setters at the feature set level.

We have a use case for both, so I do hope to track getting it in 0.5. Might be able to lend some help if no one gets to it first.

Hi @ches thanks for the heads up, ah yeah just checked it was required for basic setter getter on the feature_set level as well, might be missed this when I check the thread last time
Will extend the labels for feature_set level by this weekend and let you know

/test test-end-to-end-batch

Looks good to me. Will you let me know when its ready to be merged?

Thanks for being so patient here @ches. I guess the conversation around Field, Entity, and Feature has temporarily been answered, although I personally don't think we have reached our destination yet. I do feel like the door is left more open with our current approach.

/test test-end-to-end

/test test-end-to-end

/lgtm


Stream Analyzer not fetching realtime stream

when using the stream analyzer with generic cameras using  a rtsp feed the captured frames are all the same and usually from 30 minutes before the request is made.
Version: 1.3.1
action: llmvision.stream_analyzer
metadata: {}
data:
remember: false
duration: 2
max_frames: 8
include_filename: true
target_width: 1280
max_tokens: 100
temperature: 0.4
expose_images: true
provider: ""
message: >-
""
image_entity:
- camera.down_drive
- camera.up_drive
response_variable: response
I've had a little success with a workaround of recording a small clip in the automation before starting the analyzer but kinda ruins the timings on certain cameras.

In Developer Tools > States can you confirm that the attribute entity_picture is present for your cameras? When you append the value of entity_picture to your home assistant url, you should see the current frame.

When I do that I get a frame from what's seems like the camera last activity in HA so hours ago in this case when I refresh the current frame then appears I tried on another generic camera and same thing happened but needed 2 refreshes to get current frame

What camera are you using? Is the generic camera generated from an rtsp stream?

They are swann cctv camaras using the rtsp streams from the nvr to create the generic camera in HA

Could be the NVR not serving a realtime stream. There is not much that can be done on the integration side as the issue for this seems to be upstream. Feel free to reopen this if you have any questions!


Add a macro to serve files from OUT_DIR

Hey,
I really like this crate but I'm facing a problem: I have some static files that get minified and packed using vite / rollup which itself gets called in a build.rs buildscript. One useful environment variable in such a build.rs script is OUT_DIR where I can store the build artifacts of npm build.
I want to include those build artifacts with memory_serve. However, you're just turning the input TokenStream in your proc Macro into a String so macros like env!("OUT_DIR")  won't get resolved. In fact, as far as I know resolving TokenStreams is unstable at the moment.
Maybe we can add another macro load_assets_from_out_dir! or something similar.

Thank you @BeneSim, and sorry for my late response.
That seems reasonable, but I would prefer not introducing a new macro. Would it be an option for you if I added a MEMORY_SERVE_ROOT environment variable that would have priority over CARGO_MANIFEST_DIR when resolving a relative path? See #5

I have released version 0.4.4 - hopefully this solves your issue. If not - please reopen this issue.


Proposal: Optimize tests

Submitting this MR as an example. What are your thoughts on:

leveraging groups to run tests in parallel
Making tests repeatable (where it makes sense)
Making tests explicitly dependent (where it makes sense)
Adding ct:logs

In the included example, I would prefer to combine the create and delete into one test that first creates and then deletes the same topic instead of passing state between two tests. That would allow for true isolation and parallelism. I did it this way here as an example of what might be the better option for some of the other test modules.
I'm curious if you would welcome me doing some test refactoring to try and make them more explicit, isolated, repeatable, and potentially faster overall execution time.

some of the test cases (integration test with real Kafka) may use the same topic-partitions, e.g. test-topic to send / receive messages.
otherwise running in parallel should work ok.

I'm curious if you would welcome me doing some test refactoring to try and make them more explicit, isolated, repeatable, and potentially faster overall execution time.

Sure, this is very much welcome.


Added the form in index.html (medium task)

Hey 👋 please check my PR, If need to change please inform . Thanks
solved issue #1058
I have added the required form creation code in index.html , I have also checked Code is responsive . Please Review it.

Merged

This is an automated message from Fork, Commit, Merge [BOT].

Thank you for your contribution! Your pull request has been merged. The files have been reset for the next contributor.
What's next?
If you're looking for more ways to contribute, I invite you to check out my other projects. Just click here to find more. These projects contain real issues that you can help resolve. You can also check out the Influences section in the README to find more projects similar to this one.
Also please leave a star to this project if you feel it helped you, i would really appreciate it.
I look forward to seeing your contributions!


Add menu example

Same question. I think we covered this in our meeting this morning, but let me know if you still want comments on specific code.


Utf 8 support

CircuitPython supports UTF-8 output both while printing out from code and while in the REPL. This PR adds that support for language-circuitpython as well.

Great work, thanks!

Released and published in<EMAIL_ADDRESS>

Thanks @jb3!


[CLI] Option --full ignored when --output is JSON or XML

Option --full only works in default text mode display.
It would be nice to make it work for these modes too.

I think this could be useful for JSON as there is no schema attached to that.
However, since the XML output has been updated, it now has a schema that it should validate against.
<MediaInfo
    xmlns="https://mediaarea.net/mediainfo"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="https://mediaarea.net/mediainfo https://mediaarea.net/mediainfo/mediainfo_2_0.xsd"
    version="2.0">

So using -f would produce invalid XML for that schema.
However, you can get the full output by invoking
--Output=OLDXML
so
$ mediainfo --Output=OLDXML -f out.mkv
or preferably
$ mediainfo --Output=OLDXML -f --Language=raw out.mkv

Sorry for the reply delay.
Thanks for the OLDXML tip as it seems to be helpful in my case. I still hope that JSON will benefit from the --full option though.

Reopen as I think it is worth an answer about --full for XML and JSON (I don't know yet which one, will answer later)

Thanks @kieranjol I was suffering looking how to get more XML details! this worked like a charm!


Factory function for JSON model requires rebuild every single update

Dear developers,
Here is a very serious issue we detected in SAPUI5.
The business requirement:
A view contains chat history. Every new message should be shown immediately.
The problem:
With each new message the app works slower and slower
The code:
We catch message from websocket and update the JSON model with the code like this:

var aMessages = oMessagesModel.getProperty(sPath);
...
aMessages.push(oData);
oMessagesModel.refresh();

As this model is bound to the view it forces the update of the corresponding list.
We use factory function to insert diffrerent content in one list.
However in ManagedObject.updateAggregation method we have an insane line that is destroying all items and recreating them again.

// If a factory function is used, aggregation must be completely rebuild
if (!oBindingInfo.template) {
thisoAggregationInfo._sDestructor;
}

The program's behavior becomes completely unacceptable.
What we need is just insert a new item into the list.
Why should we rerender all the control?
Could you please find a workaround how to avoid the whole list destruction and provide an opportunity just to update it with new lines.
Thanks!
Petr Plenkov.

Which control is used in your view to show the chat history? I could imagine a "growing enabled" control like sap.m.Table could handle this better, as it does not run the code in ManagedObject that is troublesome for you. But I am not an expert for sap.m.Table...

I use list. In fact it's inherited from sap.m.List control now.
What I did to solve the issue - I deactivated items deletion while update. Also I remembered items for later rendering after update is finished.
sap.ui.define([
                               "sap/m/List"
                ],
                function(Control) {
                               "use strict";

                               return Control.extend("fiori.plugin.chat.control.MessageList", {

                                               renderer: "sap.m.ListRenderer",

                                               updateItems: function() {

                                                               var fnDestroyItems = this.destroyItems;

                                                               this.destroyItems = function() {};

                                                               var oResult = Control.prototype.updateItems.apply(this, arguments);

                                                               this.destroyItems = fnDestroyItems;

                                                               return oResult;

                                               },

                                               addItem: function(oItem) {

                                                               var oThis = Control.prototype.addItem.apply(this, arguments);

                                                               if (!oThis.itemsToRender) {
                                                                              oThis.itemsToRender = [];
                                                               }

                                                               oThis.itemsToRender.push(oItem);

                                                               return oThis;

                                               }

                               });
                }
);

Hello ThePlenkov,
what you describe above is the current default behavior of the list and table controls in the sap.m namepsace. They rerender all items when you add or remove an item in the "items" aggregation.
To overcome the issue you experience, you can turn on the "growing" property and set a meaningful "growingThreshold" (in case you don't want growing set it to a very high number) and then only the altered items in the aggregation will be rerendered. The growing mode enables a different update behavior of the content.
Hope that helps,
Michael

Dear Michael,
This issue was not about the list behavoir in common.
It was about using factory function.
When it's implemented the items are not just being rerendered, they are being newly created every single update.
That was the issue.
Treshhold property won't help to avoid this case as it's described in ManagedObject therefore applied to all inherited objects.'
Could you somehow provide an option to deactivate manually this rebuild?
Thanks!
Petr.

I ran into the same problem. I am using a factory function to create a list of items in a customer-provided Accordion control. Now I wanted to implement endless scrolling, by adding new items to the end of the list (using aggregation binding). However, as soon as I new items are added, the whole control re-renders, which is pretty useless for endless scrolling.
@ThePlenkov Did you ever find a solution to this?

@derwaldgeist hi, actually growing list feature did help a lot.


[Feature] Running network speed test at the specified time

Description of the feature
Currently it is possible to control when network speed test is executed only using DASHDOT_SPEED_TEST_INTERVAL parameter. Unfortunately there is no easy way to set speed test schedule that it is executed only during selected hours.
Additional context
No response

Hi there, sorry for the late reply. I could potentially implement some kind of cron syntax for the interval variables, so that it only updates then. Would that work for you?

Yes, definitely!


add_collision_exception_with

add_collision_exception_with doesn't seem to work with KinematicBody2D. Does exactly nothing.

I confirm this problem.

Should be fixed since 28c4afeb5733f9ca9725ab2a5f4066af8e02b2a6.


EC_KEY_can_sign() must check eckey for determining can sign or not.

must output can not sign
This bug_report_EC_KEY_can_sign.c output below two lines.
$ ./bug_report_EC_KEY_can_sign
eckey can sign
only_verify_eckey can sign

But only_verify_eckey must out put
can not sign

Therefore I give bug_report_EC_KEY_can_sign.patch.
my environment
$ openssl version -v
OpenSSL 1.1.1s  1 Nov 2022

Sorry, ...
I forget to attach a bug_report_EC_KEY_can_sign.zip.
bug_report_EC_KEY_can_sign.zip

The EC_KEY_can_sign() function is undocumented and is deprecated since 3.0. Since it is undocumented it is unclear exactly what it should do. However our 3.x migration guide does mention it:
https://www.openssl.org/docs/man3.1/man7/migration_guide.html

Applications should use EVP_PKEY_can_sign(3) instead.

So, app developers are encourage to use EVP_PKEY_can_sign() as a direct replacement. That function is documented as follows:

EVP_PKEY_can_sign() checks if the functionality for the key type of pkey supports signing. No other check is done, such as whether pkey contains a private key.

i.e. the function tells you whether the key type (not this specific key) supports signing. It explicitly does not check for the presence of a private key.
So, IMO, EC_KEY_can_sign() is supposed to do the equivalent thing and is operating as intended, i.e. this is not a bug. It checks whether the curve supports signing, but does not tell you whether this specific key does.

Closing since this is marked as resolved.


proposal: reflect: add Named() bool to Type interface

Right now, the only way to tell if a reflect.Type is a named type is to do t.Name() != "". This is quite common:
$ git grep -n '\.Name() [!=]= ""'
src/encoding/gob/type.go:844:   if rt.Name() == "" {
src/encoding/gob/type.go:866:   if rt.Name() != "" {
src/encoding/json/encode.go:1148:                               if ft.Name() == "" && ft.Kind() == reflect.Ptr {
src/encoding/xml/marshal.go:638:        } else if typ.Name() != "" {
src/encoding/xml/read.go:188:   if t.Name() != "" {
src/reflect/type.go:1575:       if T.Name() != "" && V.Name() != "" || T.Kind() != V.Kind() {
src/reflect/value.go:2323:      if dst.Kind() == Ptr && dst.Name() == "" &&
src/reflect/value.go:2324:              src.Kind() == Ptr && src.Name() == "" &&

However, the function implementation is quite complex:
func (t *rtype) Name() string {
        if t.tflag&tflagNamed == 0 {
                return ""
        }
        s := t.String()
        i := len(s) - 1
        for i >= 0 {
                if s[i] == '.' {
                        break
                }
                i--
        }
        return s[i+1:]
}

I don't think it would ever be possible for the compiler to know that we just care about the first branch, since it's possible that t.String would return a string like foo., in which case s[i+1:] at the end would also be the empty string.
I propose that we add a method like Named() bool to the interface, implemented like:
func (t *rtype) Named() bool {
        return t.tflag&tflagNamed != 0
}

A call to this function would be inlineable and not require string handling, so it would be massively cheaper than the original t.Name() != "". As proof, here's what happens if we implement it and use it in JSON decoding:
diff --git a/src/encoding/json/decode.go b/src/encoding/json/decode.go
index 2e734fb39e..97d44d4639 100644
--- a/src/encoding/json/decode.go
+++ b/src/encoding/json/decode.go
@@ -451,7 +451,7 @@ func indirect(v reflect.Value, decodingNull bool) (Unmarshaler, encoding.TextUnm
 	// If v is a named type and is addressable,
 	// start with its address, so that if the type has pointer methods,
 	// we find them.
-	if v.Kind() != reflect.Ptr && v.Type().Name() != "" && v.CanAddr() {
+	if v.Kind() != reflect.Ptr && v.Type().Named() && v.CanAddr() {
 		haveAddr = true
 		v = v.Addr()
 	}

name           old time/op    new time/op    delta
CodeDecoder-4    27.3ms ± 1%    26.5ms ± 1%  -2.80%  (p=0.002 n=6+6)

name           old speed      new speed      delta
CodeDecoder-4  71.1MB/s ± 1%  73.1MB/s ± 1%  +2.89%  (p=0.002 n=6+6)

One can imagine that the gob and xml packages would also see many-percent speedups, on top of the reflect package itself getting faster.
I realise that this is a case of adding extra API just for the sake of performance. However, I fail to see another way to cheaply see if a reflect.Type is named, which is a common operation that should be trivial. If anyone has any ideas on how to make Name() != "" just as fast somehow, please do speak up.
On the plus side, there aren't any other flag bits that would be interesting to export, so this shouldn't lead to adding a countless number of boolean methods.
cc @josharian @randall77 @ianlancetaylor @Quasilyte

Here's a possible alternative - somehow help the compiler realise that we just care about t.tflag&tflagNamed in the Name implementation. For example:
func (t *rtype) Name() string {
        if t.tflag&tflagNamed == 0 {
                return ""
        }
        s := t.String()
        i := len(s) - 1
        for i >= 0 {
                if s[i] == '.' {
                        break
                }
                i--
        }
        name := s[i+1:]
        if name == "" {
                panic("impossible") // or perhaps return "BADNAME"
        }
        return name
}

One counter-point to the JSON case is that we could instead do the reflect work ahead of time for structs; see #16212.
Still, this seems like a common reflection operation that should be way cheaper. And reflection is used in many more places than just encoding/json.

Is adding Named a backward-compatible change?

@as yes, because Type has unexported methods.

While I understand that adding Named() would be much simpler from an implementation perspective, it sets a bad precedent (as correctly noted in the first comment) and doesn't solve the same problem elsewhere. It would be better to actually have some (even restricted) form of partial inlining, and let the compiler do the right thing.

@CAFxX do you have implementation ideas of this partial inlining?
Ones come to my mind either require more memory consumption per inlining candidate and/or more computations to analyze the call site, which would probably result in wasted cycles for 90% of places. This can be optimized from the toolchain speed point of view, but changing the inliner is complicated since it's hard to determine whether inlining something will make things better or not (at least on architectures like x86).
I've tried several changes to make Go inliner more "dynamic" and make decisions with some caller context. It made some benchmarks faster, some slower. This might be the reason why we still have current cost model, for example. It's trivial to make it worse, but hard to make it universally better (it may actually require profile-guided optimization, since guessing statically is non-trivial).

@CAFxX of course it would be best if this was fast with no changes to the API - but the hard problem is figuring out how that would work. Make the compiler smarter? Special-case the reflect package in the compiler? Somehow rewrite the method to do less work?
I haven't come up with a way to do it, and I'm not even sure it's possible, hence this proposal.

<3% in one case does not seem worth the extra API here.
As others noted, the code could cache the result if needed.
Or we could arrange for it to be faster if needed.
Neither of those requires new API.
New API is much more costly than 3%.

As others noted, the code could cache the result if needed.

Sorry, I don't know what you're referring to.

Or we could arrange for it to be faster if needed.

I opened this proposal precisely because I didn't see an easy way to do this :) I could make the reflect struct heavier to then cache the name string, but I don't know if that's a tradeoff we want to make.

@rsc sorry to comment on a frozen issue, but I'm still confused about the arguments to close this proposal. Like I said in the comment above, I don't see a way to make reflect.rtype.String significantly faster, or to cache its result in encoding/json. If anyone has a specific idea, I'll gladly use that instead of adding new API.
I completely forgot to follow up on this and the thread got locked. Unlocking for now.

Additional context: @mvdan asked if it was alright to unlock and follow up. I said yes.

If we think it's worth it, and if it's faster, we could use a sync.Map to cache (*rtype).String in the reflect package.  Or in the encoding/json package.
The current rtype format is optimized for space.  We could change that.
We could implement partial inlining as suggested above.
The encoding/json code should check CanAddr first, although I don't know how often that is false.  CanAddr is simple.
There are quite a few areas we could explore rather than adding new API.  New API seems like a lot, and the performance improvement doesn't seem like all that much in the larger scheme of things.

Thanks, @ianlancetaylor. Next time I'm in the mood for some encoding/json performance tuning I'll try your ideas and report back. I'm not sure why I hadn't noticed your reply until now, I must have read the notification and forgotten about it.


[1.18] Port of matching: disable hcm integration by default (#16387)

Port of https://github.com/envoyproxy/envoy/pull/16387 to 1.18 release branch

Ping @snowp


Prevent null pointer exceptions

What is the purpose of the change
Prevent null pointer exceptions
Brief changelog
XXXXX
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[ ] Run mvn clean install -DskipTests & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #2944 into master will decrease coverage by 0.04%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #2944      +/-   ##
==========================================
- Coverage   63.92%   63.88%   -0.05%
==========================================
  Files         584      584
  Lines       26005    26005
  Branches     4543     4543
==========================================
- Hits        16624    16613      -11
- Misses       7195     7206      +11
  Partials     2186     2186

Impacted Files
Coverage Δ

...ng/telnet/support/command/StatusTelnetHandler.java
0% <0%> (ø)
:arrow_up:

...ache/dubbo/remoting/transport/AbstractChannel.java
75% <0%> (-12.5%)
:arrow_down:

.../remoting/transport/netty4/NettyClientHandler.java
75% <0%> (-11.12%)
:arrow_down:

...che/dubbo/remoting/transport/mina/MinaChannel.java
43.42% <0%> (-10.53%)
:arrow_down:

...ng/exchange/support/header/HeartbeatTimerTask.java
73.68% <0%> (-5.27%)
:arrow_down:

...c/main/java/org/apache/dubbo/rpc/RpcException.java
85.71% <0%> (-3.58%)
:arrow_down:

...rg/apache/dubbo/common/timer/HashedWheelTimer.java
60.62% <0%> (-1.4%)
:arrow_down:

...pache/dubbo/registry/support/AbstractRegistry.java
81.95% <0%> (+0.75%)
:arrow_up:

...he/dubbo/registry/multicast/MulticastRegistry.java
65.51% <0%> (+1.72%)
:arrow_up:

...he/dubbo/remoting/transport/netty/NettyServer.java
71.42% <0%> (+3.57%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d4b6cb...28618af. Read the comment docs.


Item.Image doesn't honor size prop

The wrapped Image produced by Item.Image doesn't seem to honor the size prop.
Item.Image renders as a wrapped image with ui=false.  Looking at http://semantic-ui.com/views/item.html#image seems to indicate that their wrapped image uses
 .
Indeed adding ui=true to the Image produced by Item.Image seems to resolve the size issue.
Update: there is an explicit test ensuring ui=false, so perhaps there's something I don't understand about how semantic is supposed to work.

Please open a new issue with a complete report.  We are happy to help.


Add uptime monitoring for the BuyBC Agents and Controllers

Add uptime monitoring for the BuyBC Agent and controllers in the dev, test, and prod environments.
URL endpoints to be monitored can be found here:

dev: https://console.apps.silver.devops.gov.bc.ca/k8s/ns/e79518-dev/routes?name=buybc
test: https://console.apps.silver.devops.gov.bc.ca/k8s/ns/e79518-test/routes?name=buybc
prod: https://console.apps.silver.devops.gov.bc.ca/k8s/ns/e79518-prod/routes?name=buybc

Depending on the version of aca-py the agents are running, we may need to upgrade them to be able to access the /status/ready endpoint on the agent without requiring the API key.
The endpoint on the controller you want to monitor is the /health endpoint.

@rajpalc7, For the Buy BC Controller checks, it would be better to expect the string {"success":true} to indicate the controller is working rather than explicitly matching "success":false to indicate the controller is down.  There are many conditions where the controller could be down and you would never explicitly match "success":false.  When possible you always want to look for the positive condition, that way any other response or behavior of the endpoint will be considered down.
When monitoring aca-py agents, you want to monitor the /status/ready endpoint (as indicated in the issue description) whenever possible.  This endpoint is a k8s compatible health endpoint that provides an indication of whether or not the agent is ready to accept requests.  You can use this single endpoint to monitor the agent, eliminating the need to monitor both of its client and agent endpoints.  The Buy BC agent instances are currently running an older version of aca-py that requires the use of an API key to authenticate with the status /status/ready endpoint.  So that leaves you with two options; use the API key to authenticate with the endpoint (not recommended), or upgrade the Buy BC instances to a newer version of aca-py that allows unauthenticated access to the /status/ready endpoint (recommended).

Raj to update d blocker and other details

@rajpalc7 could you please add a comment with the current status for this issue, and what is preventing it from being completed? Thank you!

@esune - This ticket is completed now and can be reviewed by wade.


put a few copy-pasteable usage examples to the README

Resolved via #30, in particular fda42e6dd0b2291868c228d4d2bb1fdd73008649


Chat room tab info wont update

The chat room tab information, viewers, and game playing and uptime don't update when a channel goes live, unless you close and reopen the tab or the whole chatterino client

Can confirm. Pajlada also acknowledged the issue.

Yep, confirmed. I'm working on this issue :slightly_smiling_face: /


Satellites track your movement around them on the map as if they were planets

The mini-map that is


Add a new table mapping

Summary of Changes
Include a summary of changes then remove this line
Tests
What tests did you add or modify and why? If no tests were added or modified, explain why. Remove this line
Documentation
What documentation did you add or modify and why? Add any relevant links then remove this line
CheckList
Make sure you have checked all steps below to ensure a timely review.

[ ] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"

In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

[ ] PR includes a summary of changes.
[ ] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain docstrings that explain what it does

[ ] PR passes make test

Add one more value in the mapping of the ES filters, the cluster value. In our case we are leveraging this filter. It should not affect teams which don't use it


Add support to dyn keyword

Hi, I'm facing the following issue, the tool for the following function generates documentation correctly:
#[utoipa::path(
    post,
    path = "/api/v1/dummy-context/{dummy_id}",
    responses(
        (status = 201, description = "Dummy was successfully created", body = DummyContextResponse),
        (status = 400, description = "The request is not valid"),
    )
)]
pub async fn post_dummy_message(some_dummy_param: web::Data<SomeDummyType>) {
    // Some code here
}

But when i change the params and add one with that have the dyn keyword, for example the following(I just change the params, the rest is exactly the same):
#[utoipa::path(
    post,
    path = "/api/v1/dummy-context/{dummy_id}",
    responses(
        (status = 201, description = "Dummy was successfully created", body = DummyContextResponse),
        (status = 400, description = "The request is not valid"),
    )
)]
pub async fn post_dummy_message(some_dummy_param: web::Data<dyn SomeDummyTypeTrait>) {
    // Some code here
}

My code related to utopia breaks with the following message:
unexpected type in component part get type path, expected one of: Path, Reference, Group
I cant avoid adding the keyword because I need traits on the function.
Is this behavior expected or can be considered as a bug? I'm missing something to make this work? Thank you!
Versions:
utoipa = { version = "^2.1.0", features = ["actix_extras", "uuid", "chrono"] }
utoipa-swagger-ui = { version = "2.0.1", features = ["actix-web"] }
actix-web = "4.2.1"
rustc 1.62.0 (a8314ef7d 2022-06-27)

Hi, right. This could be considered a bug, funny that no-one hasn't reported that before or they are not using dyn that much. I'll take a look at this when I get time and hopefully resolve this to the next release.

Thank you, let me know if there is something i can help with (if you have an idea of how to solve it i will try)


Monthly Chat Agenda November 2022 (2022-11-07 and 2022-11-21)

This is the Agenda for the Monthly CRS Chat.
The general chat is going to happen on https://owasp.slack.com in the channel #coreruleset on Monday, 2022-11-07, at 20:30 CET. That's the 1st Monday of the month. A separate issue chat is happening at the same location, same time on Monday, 2022-11-21. That's the 3rd Monday of the month. Please note that we have a CRS calendar (maintained by @fzipi).
Archived previous meetings and their decision are here.
The monthly meeting follows a new agenda structure and focuses more on project overview and less on individual PRs. See https://github.com/coreruleset/coreruleset/wiki/DevRetreat22WorkshopRoadmap
What happend in the meantime since the chat last month
Outside development

Blog about GeoIP & ASN by Christian Folini / @dune73
Portrait of CRS developer Andrea Menin / @theMiddle
Successful CRS Developer Retreat in Italy. Blog Post pending, notes at https://github.com/coreruleset/coreruleset/wiki/Dev-Retreat-2022
Next ModSec / CRS webcast by Christian Folini: Nov 14

Inside development - Sub project status reports:
Rules

Bug progress with over 30 bug bounty fixed.
We had a report on an issue that lead to all regular expressions failing to compile on Ubuntu 22.04, which, effectively, disabled the WAF. The culprit appears to have been a bad package, possibly nginx-mod-lua, from a specific apt repository.

CRS Sandbox

Not much movement
Talks about retiring the elastic monitoring for a lack of use (dune73 thinks it would take a custom sandbox dashboard to make it interesting)

CRS Bug Bounty and Security

Decision to track security findings, namely those received via email, as issues in a separate private repository
bug bounty findings all covered, but checks reveal not all tests are successful. Link to private checklist
see below for discussion.

Plugins

Extensive discussion on allowed method variable in nextcloud plugin.
WordPress Plugin: updated the exclusion rules and planning to in Labels
bookmark Meeting Agenda
cluding top used WP-Plugin. Little other movement.

Documentation and Public Relations

Release of first dev portrait (see above), 2nd in the making
Blog post about dev retreat in the making.

Project Administration and Sponsor relationships

OWASP HQ accepted monthly payment for administration
Several upcoming sponsorship announcements in the making

Tools

go-ftw fixed several bugs during retreat
crs-toolchain to replace perl based regex-assemble and more tooling (regex-assemble part now on go implementation of regex-assemble which allows more powerful configuration).
Pending question: Migration of regex-assemble to crs-toolchain

Testing incl. Seaweed and many future plans

Currently on hold

Containers

Decision during retreat to let go of the naked ModSec container and to concentrate on a CRS container.

CRS Status Page

We decided to aim for a MVP where we check 80% of our PL1 rules.
We are currently on 70% of PL1 rules for CRS v4.
More information here

Project Decisions

How to cover for bug bounty findings that have negative tests (-> false negatives!)?

When and how to migrate to CRS toolchain?

Release plan for CRS 4?

PRs that have been merged since the last meeting

#2970
#2969
#2968
#2967
#2965
#2964
#2952
#2962
#2877
#2894
#2938
#2932
#2963
#2956
#2960
#2958
#2937
#2832
#2921
#2955
#2951
#2907
#2904
#2935
#2923
#2925
#2928
#2931
#2927
#2922

We merged 30 PRs since the last monthly project chat.
Open PRs
Open PRs marked DRAFT or work in progress or needs action

#2974
#2975
#2966
#2954
#2972
#2971
#2948
#2878
#2945
#2957
#2949
#2950
#2941
#2947
#2946
#2942
#2939
#2933
#2637
#2573
#2791
#2666

2nd Meeting (Monday, 2022-11-21) - Originally the separate Issue Meeting
Tasks until CRS v4

[x] Merge as many PRs as we can (before toolchain migration)
[x] Toolchain migration
[ ] MUST Close all BB findings, recheck (stats right now: 73.33% blocked at PL1, 88% at PL2, 88% at PL3, 95% at PL4)
[ ] MUST close Shivam Bathla's bypasses
[ ] MUST list updates: still critical with 20 open issues, 13 closed
[ ] MUST Document  and phase 1 rules
[ ] MUST Split tasks in ready for RC2 vs GA
[ ] MUST CI: integrate all checks that @airween has developed
[ ] CAN WordPress plugin updates
[ ] CAN Setup CI for plugins (nice2have)
[ ] CAN Add TX:/crs_arg_.*/ to all rules that check ARGS  (-> check performance impact first)

Status of issues covered last month

Issue slot 1: #FIXME
Issue slot 2: #FIXME
Issue slot 3: #FIXME
Issue slot 4: #FIXME
Issue slot 5: #FIXME
Issue slot 6: #FIXME
Issue slot 7: #FIXME
Issue slot 8: #FIXME
Issue slot 9: #FIXME
Issue slot 10: #FIXME

Stats

Covered in chat: FIXME
Closed: FIXME
Pending: FIXME

This month's issues
There are FIXME open issues at the beginning of the issue chat.
We generally cover 10 issues per month in a separate issue meeting. Add them as you see fit.

Issue slot 1: #FIXME
Issue slot 2: #FIXME
Issue slot 3: #FIXME
Issue slot 4: #FIXME
Issue slot 5: #FIXME
Issue slot 6: #FIXME
Issue slot 7: #FIXME
Issue slot 8: #FIXME
Issue slot 9: #FIXME
Issue slot 10: #FIXME

How to get to our slack and join the meeting?
If you are not yet on the OWASP Slack, here is your invite: https://owasp.org/slack/invite .
Everybody is welcome to join our community chat.

Decisions

Bug Bounty Findings, 2nd pass: Every dev is asked to look at 5 bug bounty findings that are still on 200 despite PRs and investigate (-> Issue -> PR). See https://github.com/coreruleset/bug-bounty-tests for list. (Test ID means number of curl call of finding.)
@theseion finishes the toolchain. Once he is done, @lifeforms will give the signal for migration, also for migration of CI. The old regex-assemble will be replaced with a script that fails with notice to run toolchain instead.
We are in no position to do a CRS4 release plan. Too many things open on bug bounty findings, 30 PR and toolchain. Discussion postponed to meeting in 2 weeks.

Decisions Nov 21

deny at phase:1 is a problem for NGINX -> we need to explain to people they should stick to phase 2 blocking and they should not activate early blocking ->  documentation: extend early blocking doc with phase 1 and its caveats/problems, maybe blog post
boosting performance removing unneeded variables from rules and maybe remove unnecessary PL-skip rules in every rule file -> will be part of a larger performance campaign for 4.1 -> in the meantime: implement as plugin
@franbuehler will work on Shivam Bathla's new bypasses, should be finished in 2 weeks
@fzipi will work on list updates, but needs some help. It's a big task. He will tag people on the list update.
@theseion will focus on the BB issues, but everyone is invited to help. @theMiddleBlue will help on "curl failed issues"


Add a Testcontainers demo/example

Summary
I would like to add an example that demonstrates use of the extension with Testcontainers modules for WireMock, in particular Java, Python and Golang. As a part of it, it might even make sense to add some APIs to the SDKs to make the use of the extensions easier. To be seen
References

https://wiremock.org/docs/solutions/testcontainers/

I think it would be good to work on this after wiremock-testcontainers-java has officially supported for WireMock Version 3. However, I do plan on using it for Integration Tests!

Hi @nilwurtz. It is officially supported now, kinda. At least it is fully compatible

The testcontainers-kotlin example has now been released 🥳


hash binarydata in configmaps

Fixes #73

How is the migration path here? Do the hashes of CMs/Secrets change? That would lead wave to restart all deployments when wave is updated.
Thank you for all your contributions over the last few days, I am looking through them.

Yes it will restart all deployments referencing a configmap. I would mention this in the release.
We could avoid it but would involve either another hashmap or format strings. In my opinion a restart is fine.


KAFKA-9069: Flaky Test AdminClientIntegrationTest.testCreatePartitions

https://issues.apache.org/jira/browse/KAFKA-9069
Make getTopicMetadata in AdminClientIntegrationTest always read metadata from controller to get a consistent view.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@guozhangwang @hachikuji  Please review this patch. Thanks.

retest this please.

Merging to trunk and 2.4 (copy the changes to old classes).


[GOBBLIN-1501] Documentation + Code Cleaning for IcebergMetadataWriter

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

https://issues.apache.org/jira/browse/GOBBLIN-1501

Description
This PR mostly adds documentation, renaming and fixing typo in the gobblin-icberg module, with extra focus on IcebergMetadataWriter for:

Change the way that Transaction object is initialized as it is shared in many different places.
Change the way for accessing or initializing AppendFiles and DeleteFiles object. All these changes in TableMetadata object.
Adding documentation for places where commit does happen or doesn't happen for reasons.
Remove some of the IOException in the method signature wherever they are not required.

Tests
This PR doesn't add or delete any of the functionality so pass all unit tests should be sufficient.
Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@ZihanLi58  Can you please take a look when you got a chance? Thanks.


Repository Discussions

Hey all, I'm having a hard time trying to locate documentation for Discussions API at the repository level.  I do see it at the teams level and also I see this library provides that.  What I can't seem to find is API docs even on GitHub's site for repository level discussion.
I can query the REST endpoint here using httpie
 http https://api.github.com/repos/go-chi/chi/discussions

Is this hidden in this repo somewhere that I'm not seeing?
I appreciate the help!

As far as I can tell, the official GitHub v3 REST API only provides methods to access "Teams Discussions" and not "Repository Discussions".

You might want to contact GitHub Tech Support and ask them if they support repository discussions with the v3 API.

Ok great,  so I'm not the only one!  It's really confusing that the API is there and I can hit it. Thanks, I'll ping them.

Thanks, @incognick !
Feel free to report back to this issue any findings you may get, and we can reopen the issue to support the API if it exists.

Thanks for the quick response Glenn!


✨ python 3.12

Support for Python 3.12

:tada: This PR is included in version 0.10.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Support for Zarf in a Connected Environment

Is your feature request related to a problem? Please describe.
I'd like to use Zarf for declarative deployments, but I'm not deploying in an airgapped environment. In other words, I don't need the Gitea server, container registry or mutating webhook.
Describe the solution you'd like
I'd like to be able to create a Zarf package that doesn't depend on the Gitea server, container registry or Zarf agent. In other words, I want to use Zarf without its mutating webhook.
Describe alternatives you've considered

I know I can add the zarf-agent=ignore label to namespaces, but I'd like to not deploy the agent at all
Technically, I could just use the Gitea server and container registry, but the resulting package is >10GB and it was error-prone during creation (I had issues related to caching); this is not tenable for my non-platform dev audience to use.

Additional context
Why I want to use Zarf even though I'm not in an airgapped environment:

Instead of having app teams fork my IaC/CaC, I want an immutable artifact that can be deployed that abstracts away some of the complexity around K8s, Flux, etc.
This artifact needs to be immutable so I can ensure app teams don't change the IaC/CaC and inadvertently violate security controls
I want non-platform devs to be able to easily perform cluster upgrades

Some questions as this is a pretty fundamental shift for zarf that I'm not sure will be super easy and will break full portability/some planned enforcement things:

Was this a big bang deployment? What was the total binary size?
Can you clarify you had issues creating packages, but not deploying, or both?
What were the cache issues? When I've seen these they were often a one-off and not a continual issue once the offending entry was resolved.
How are you distributing your package? Via OCI, file transfer or just giving users a repo and having them create the package?
To clarify the issue isn't the webhook or other init package assets as much as a very large package having trouble being built, correct?
If these components were removed would you still deploy everything through zarf?

Technically you can do this now, though we consider it somewhat of an anti-pattern today. Some methods (all these assume you don't specify images or repos in later zarf packages):

use a custom init package without the extra stuff
remove or disable the agent in a secondary package (to avoid creating your own init)
the webhook ignore label works on namespaces too, so you can create a namespace label for the namespaces you don't want zarf to care about

Was this a big bang deployment? What was the total binary size?

it's Big Bang + some 3rd party apps on EKS; I don't recall the total binary size

Can you clarify you had issues creating packages, but not deploying, or both?

I had issues with SBOM generation during package creation, but this was resolved by clearing the Zarf cache. Understood the SBOM generation is not mandatory

What were the cache issues? When I've seen these they were often a one-off and not a continual issue once the offending entry was resolved.

Not sure exactly, but clearing the cache fixed the issue

How are you distributing your package? Via OCI, file transfer or just giving users a repo and having them create the package?

Haven't nailed this down, but I'd like to attach the package to Github Releases

To clarify the issue isn't the webhook or other init package assets as much as a very large package having trouble being built, correct?

Negative. My org simply doesn't want to use the container registry or Gitea server because we are already connected to external ones

If these components were removed would you still deploy everything through zarf?

I want to use Zarf to create a declarative upgrade process

@UncleGedd and I have met and discussed this issue now and have a lightweight way ahead in PR #1105 that I do not believe breaks the core design of Zarf, but allows a lighter transition into Zarf's declarative style.


ProcessedDate not set in outbox for actor messages nor in queuedInternalCommands

Current Behavior

When a request has been processed and an actor message has been sent, the message does not get a processed date set in the Outbox, nor does the command get a processed date set in the queued internal commands table.
Expected behavior

A processed date is expected to be set when a command/message has been processed.
How to reproduce

Send in a succesful request.

When this is completed remember to close #641 and #637

ActorMessageDispatcher has not yet been implemented. That´s why ProccesedDates for ActorMessages are never set.

Works for internal commands.
This unfortunately does not fix various end dates not being set (eg. #641) as we suspected it might.


Improve _sqnorm

Would you add multi-gpu tests (both with and without cpu)?

@beam2d I fixed the code.

LGTM!


Fix rtd build error

Perhaps it's because the dependency resolution didn't work properly...
https://readthedocs.org/projects/kakaowork-py/builds/14243283/
Running Sphinx v4.1.1

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/kakaowork-py/envs/latest/lib/python3.7/site-packages/sphinx/config.py", line 327, in eval_config_file
    exec(code, namespace)
  File "/home/docs/checkouts/readthedocs.org/user_builds/kakaowork-py/checkouts/latest/docs/conf.py", line 18, in <module>
    from kakaowork import __version__  # noqa: E402
  File "/home/docs/checkouts/readthedocs.org/user_builds/kakaowork-py/checkouts/latest/kakaowork/__init__.py", line 8, in <module>
    from kakaowork.client import (Kakaowork, AsyncKakaowork)
  File "/home/docs/checkouts/readthedocs.org/user_builds/kakaowork-py/checkouts/latest/kakaowork/client.py", line 5, in <module>
    import aiosonic
  File "/home/docs/checkouts/readthedocs.org/user_builds/kakaowork-py/envs/latest/lib/python3.7/site-packages/aiosonic/__init__.py", line 31, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

Fixed dependencies resolution regex of aiosonic. Just wait for the next release without any changes


381-fix

Adding bedpe as a file format, fixing validation rules for stripe window and loop window

@adamjtaylor the regex I used for loop window is list::-?\d+ . Let me know if that and everything else works!

Humm... actions change with a weird git error about your data-models-docs branch
Run r-lib/actions/pr-push@v2
[@octokit/rest] `const Octokit = require("@octokit/rest")` is deprecated. Use `const { Octokit } = require("@octokit/rest")` instead
[@octokit/rest] `const Octokit = require("@octokit/rest")` is deprecated. Use `const { Octokit } = require("@octokit/rest")` instead
Collecting information about PR #395...
git push ***github.com/aditigopalan/data-models-docs.git -q HEAD:381-hic
remote: Permission to aditigopalan/data-models-docs.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/aditigopalan/data-models-docs.git/': The requested URL returned error: 403
Error: The process 'git' failed with exit code 128

It also looks like this version reverts some changes from before our recent data model release.


Django 3.1.3 support

ImportError: cannot import name 'FieldDoesNotExist' from 'django.db.models.fields'
From release notes:
The compatibility import of django.core.exceptions.FieldDoesNotExist in django.db.models.fields is removed.

ImportError: cannot import name 'FieldDoesNotExist' from 'django.db.models.fields' (C:\Users\Raphael\Desktop\gb_tools\GleebeesTools\sst2\venv\lib\site-packages\django\db\models\fields\__init__.py)
I also tried with a django-3.1.2 installation.

Hi, I've prepared a fix for this error in https://github.com/pivotal-energy-solutions/django-datatable-view/pull/246.


Clicking documentation in the navigation shifts the whole navigation over

Regular page:

Support page:

It seems the Support page also misses the link to "Enterprise Edition"

The docs have not yet been adoped to the new style. Please open an issue in https://github.com/owncloud/documentation/issues/

I'll take care of it, let's just track it here.

fixed.. By danimo and the last little pieces here https://github.com/owncloud/documentation/pull/1941

Thanks for the quick turnaround on this!


ValueError: Could not interpret value coord for parameter x

Hi
I'm trying to use methylartist composite for plotting methylation over several SVAs.
I think the software finishes profiling the coordinates, but then it rans into
Traceback (most recent call last):
  File "/sw/pkg/methylartist/1.2.4/bin/methylartist", line 4855, in <module>
    main(args)
  File "/sw/pkg/methylartist/1.2.4/bin/methylartist", line 4570, in main
    args.func(args)
  File "/sw/pkg/methylartist/1.2.4/bin/methylartist", line 4358, in composite
    ax0 = sns.lineplot(x='coord', y='meth', data=meanplot_table, ci='sd', lw=2, hue='sample', palette=sample_color)
  File "/sw/easybuild/software/Seaborn/0.11.2-foss-2021b/lib/python3.9/site-packages/seaborn/_decorators.py", line 46, in inner_f
    return f(**kwargs)
  File "/sw/easybuild/software/Seaborn/0.11.2-foss-2021b/lib/python3.9/site-packages/seaborn/relational.py", line 692, in lineplot
    p = _LinePlotter(
  File "/sw/easybuild/software/Seaborn/0.11.2-foss-2021b/lib/python3.9/site-packages/seaborn/relational.py", line 367, in __init__
    super().__init__(data=data, variables=variables)
  File "/sw/easybuild/software/Seaborn/0.11.2-foss-2021b/lib/python3.9/site-packages/seaborn/_core.py", line 605, in __init__
    self.assign_variables(data, variables)
  File "/sw/easybuild/software/Seaborn/0.11.2-foss-2021b/lib/python3.9/site-packages/seaborn/_core.py", line 668, in assign_variables
    plot_data, variables = self._assign_variables_longform(
  File "/sw/easybuild/software/Seaborn/0.11.2-foss-2021b/lib/python3.9/site-packages/seaborn/_core.py", line 903, in _assign_variables_longform
    raise ValueError(err)
ValueError: Could not interpret value `coord` for parameter `x`

I thought maybe it was because of low methylation calls in the coordinates, so I tried with a subset of coordinates that I've used with methylartist locus (finishing succesfully), but it fails with the same error.
My command is:
methylartist composite -d methylation_calls_files.txt -s SVA_F.bed -r hg38.fa -p 20 --svg -t SVA_F.cons
I'm using version 1.2.4
I tried using --excl_ambig and to change -l to a more loose threshold, but I got the same error.
The contents of methylation_calls_files.txt are:
op_043_001.sorted.bam op_043_001_methylation_calls.db
op_043_002.sorted.bam op_043_002_methylation_calls.db

Any advice would be greatly appreciated!
Best,

Hi, Sorry to hear it's crashing on you. Could you include the full output including anything before the Traceback, and the first few lines of SVA_F.bed?

Hi, thank you for the quick response! Of course! Here are the first few lines of SVA_F.bed:
chr1    108488347       108490674       SVA_F_dup43     -
chr1    108675095       108676716       SVA_F_dup44     +
chr1    112897625       112899310       SVA_F_dup49     -
chr1    113068817       113070035       SVA_F_dup50     +
chr1    113069789       113071152       SVA_F_dup51     +
chr1    117428797       117431187       SVA_F_dup52     -
chr1    150727945       150729963       SVA_F_dup55     -
chr1    150780850       150782545       SVA_F_dup56     +
chr1    151650596       151652242       SVA_F_dup57     -
chr1    155700508       155701698       SVA_F_dup62     +
chr1    156556909       156557922       SVA_F_dup63     +
chr1    174348481       174351069       SVA_F_dup70     -
chr1    182508683       182510657       SVA_F_dup72     -
chr1    21023240        21024912        SVA_F_dup4      +
chr1    213151489       213153280       SVA_F_dup85     -

And here is the full output of composite:
850897.composite_SVAs.txt

Hi,
Thank you! This fixed the error :-)
I have some follow up questions that I think are related to what you mention about the SVAs being of different length (I attach a screenshot and annotated it by question number):

The line plot from the upper panel seems to start at around position ~80 and finishes at position ~780. The bed file I input only contained entries of >1kbp and my consensus sequence is ~1.5kbp, so I'm confused on to why the methylation % is only calculated from position 80 to 780. Is it because the rest of the positions didn't have that many methylation calls? I just want to make sure it is not the sample legend that is on the way and for some reason stops the plot.
Sorry if this is a very noob question, but I see that some of the line plots in the lower panel start as late as position ~440 - is this the result of aligning the sequences from the bed file on to the consensus? in other words, did their start region aligned around position 440 of the consensus?

Best,

Q2 probably has to do with the individual TEs' alignment to the consensus, but I have no idea what's going on regarding Q1 on the consensus plot. Will see if I can reproduce with SVA annotations on my end.

Ok! Thank you. Just so you know, this plot in particular was with all SVAs of >1kbp.

I see similar results for SVA, it was due to their heterogeneity and an automatic cutoff that determines which sites are included in the mean methylation plot. Because each individual element (in this case SVA) might include different sites (in this case, CG dinucleotides), each position in the reference SVA aligned to the individual SVA in is assessed to see how many SVAs have a CG at that position, call this the "coverage" at that CG. The median coverage is taken across all CGs and used as a cutoff for determining which to include in the plot. Since SVAs are very heterogeneous this doesn't work well so I've added a parameter --meanplot_cutoff in f6f7fee, let me know if it works for you, looks like this for me (default vs --meanplot_cutoff 5):
default:

--meanplot_cutoff 5:

Thank you so much! This worked perfectly.
I was wondering - are there matrices I could output to traceback a specific element from the plot? E.g. If I want to know which elements are not methylated towards the end? Or in other words, a matrix that would have the methylation score per position in the consensus? (per element?)

I've added a --output_table option to composite in commit 737a5c0. The output currently looks something like:
chrom   start   end     strand  sample  coord   meth
chr16   24108350        24110141        -       054     33      0.6178728620349763
chr2    202534282       202536111       -       054     33      0.90873449102059
chr20   32723118        32724962        +       054     33      0.9506836249230268
... etc ...

where coord refers to a position in the reference element (.fasta passed to -t option). The positions included in the table are determined the same way the positions included in the mean methylation plot (default to median coverage or --meanplot_cutoff. Let me know if this works for you.

This is so great. Thank you so much!! I'll close this issue now :-)

I'm sorry to reopen this. I thought of a related question...
I tried loading this table to R and plot it as x=coord, y=meth, group =  (chrom+start+end) as a sanity check (it should output the same plot as composite if I understood right).
But I get a much fuller plot:

Here is the code I'm using in R
sva_methyl <- fread("methylation_calls.SVA.meanplot_cutoff_5.composite.table.tsv", data.table = F)
sva_methyl$coords_genome <- paste(sva_methyl$chrom, paste(sva_methyl$start, sva_methyl$end, sep="-"), sep=":")
ggplot(sva_methyl, aes(x=coord, y=meth, group = coords_genome)) + geom_line(colour = "steelblue", alpha = 0.5) + theme_classic()

Do you filter this somehow before creating the plots from composite? How can I recreate the plot from the table I got?
Thank you so much again!!

If you haven't tried it, try setting --maxsegs to a value greater than or equal to the number of SVAs in the table.

I've tried that, but it still looks quite sparse in comparison.
I'm running
methylartist composite -d methylation_calls.txt -s hg38_SVAs_plus1kb.bed -r hg38.fa --svg -t SVA.fa --lenfrac 0.5 --meanplot_cutoff 5 --maxsegs 2400 --output_table
Where hg38_SVAs_plus1kb.bed has 2326 entries.

Ignore the above comment, I've modified the arguments to composite in commit 13522b7 so that this makes some sense now I think. There's now a --minelts and --maxelts which default to 1 and 200, respectively. To plot everything you can just give --maxelts a very large value e.g. 5000 e.g.:
methylartist composite -d 054.data.txt -s te/SVA.1kbp.bed -r Homo_sapiens_assembly38.fasta -t SVA_F.fa -p 64 --output_table --lenfrac 0.5 --maxelts 5000 -a 0.1


Boolean session parameters are not handled properly

Provider Version
0.25.22
Terraform Version
1.0.3
Describe the bug
snowflake_task supports setting session parameters. For example, one can specify:
session_parameters = {
      "ODBC_QUERY_RESULT_FORMAT" = "ARROW",
}

Some Snowflake session parameters require a boolean value. However, the provider always coerces the value to a string. Consider this example:
session_parameters = {
      "ERROR_ON_NONDETERMINISTIC_MERGE" = false,  # This parameter is of type BOOLEAN in Snowflake.
}

The generated SQL is:
ALTER TASK "my_db"."my_schema"."my_task" SET ERROR_ON_NONDETERMINISTIC_MERGE = "false"

Note the quotes around false.
Expected behavior
The correct SQL should be:
ALTER TASK "my_db"."my_schema"."my_task" SET ERROR_ON_NONDETERMINISTIC_MERGE = false

Hey @aryann. Thanks for creating the issue.
The task resource has been dramatically reworked since v.0.25.22. Can you confirm if the problem persists after upgrading to the newest provider version?

We are closing this issue as part of a cleanup described in announcement. If you believe that the issue is still valid in v0.89.0, please open a new ticket.


[Feature] Multiplayer realtime editing

https://github.com/yjs/yjs

We had a similar feature which got depricated, Gonna close this issue for now.

We had a similar feature which got depricated, Gonna close this issue for now.


Running CARLA from Docker failed

Hello community!
After we installed,

Docker and
nvidia-docker-2

we followed the instructions https://carla.readthedocs.io/en/latest/build_docker/.
In brief, we pulled the CARLA image 0.9.9
docker pull carlasim/carla:0.9.9

and ran CARLA under docker.
docker run -p 2000-2002:2000-2002 --runtime=nvidia --gpus all carlasim/carla:0.9.9

Unfortunately after running the command above we got the following error message:
sh: 1: xdg-user-dir: not found
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default

Has anyone an idea to solve this?
Thanks for your contributions!

@Leon-Adler it didn't fail. It just has an annoying problem with ALSA lib, but there is no segfault. Did you try to connect to the server?

Hi @Leon-Adler !
As @germanros1987 said, these errors are not affecting the server. You should be able to run your client without problems.

I try to run Carla using docker on a remote server. I use the command docker run -it -p 2002-2004:2002-2004 -e NVIDIA_VISIBLE_DEVICES=1 carlasim/carla:0.9.6 but get the following error:
4.22.1-0+++UE4+Release-4.22 517 0
Disabling core dumps.
sh: 1: xdg-user-dir: not found
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=131119
Malloc Size=48848 LargeMemoryPoolOffset=179984
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
CarlaUE4.sh: line 5:    13 Segmentation fault      (core dumped) "$UE4_PROJECT_ROOT/CarlaUE4/Binaries/Linux/CarlaUE4-Linux-Shipping" CarlaUE4 $@

I want to run headless carla. There is one Carla running on another user account on this PC outside of docker. Does it affect?
I also get the following error by running docker run -p 2000-2002:2000-2002 --runtime=nvidia --gpus all carlasim/carla:0.9.6:
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.

The tutorial on the website is for v8.2. Isn't it old?

sudo docker run --gpus 0 --env SDL_VIDEODRIVER=offscreen carlasim/carla:0.9.10 /bin/bash CarlaUE4.sh
it works for me
good luck


Some timeout refactoring

Changes
Add more details about how the timeout handling works 🕒
While investigating #2905, I struggled to understand how the timeout
handling works, especially with TimeoutSet having very little comments,
so I've added some. I didn't add anything for backoffs yet because I'm
hoping we can separate that into a separate structure since it has a
very specific purpose that doesn't generalize to all timeouts.
Also changed the name "finished" to consistently use "done" so the
reader doesn't have to wonder about the difference between "finished"
and "done" (there isn't one)
Move timeout handler into its own package 📦
I'd like to move the "backoff" logic into its own file, separate from
the other timeout logic, so it's clear which parts apply to what (i.e.
the timeout handler is being used for 2 purposes: timing out Runs which
take too long, and backing off when pod creation is failing - this is
totally fine but it's hard to understand when reading the code)
As a first step, I've moved the timeout handler into a separate package,
so we can have a file and tests dedicated to the backoff logic separate
from the other handling.
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[x] Includes tests (if functionality changed/added)
[n/a] Includes docs (if user facing)
[x] Commit messages follow commit message best practices
[x] Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.
Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes
If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.
Release Notes
NONE

@bobcatfish I learnt some timeout handler with this PR 😜  and excited to see more changes ...
one minor NIT which can be addressed with next set of changes
/lgtm


Eagerly publish JSON data files for every variable

What
Publish a JSON data file for every variable in MySQL, regardless of whether it is used in a chart.
Why? / Why now?

As part of the Data Pages project, we want to be able to make grapher charts with no dedicated grapher page

Considerations

To build grapher charts without dedicated grapher pages, we will need access to both variable data and variable metadata

Here's what I've done so far https://github.com/owid/etl/pull/778, but I guess it's gonna change after more shaping.


Set generation height (ocean level)

Currently terrain gets generated based on the load marker radius, but the only way to generate a flat surface terrain is to limit Y generation level at the max radius.
This results in terrain which is generated really high and you have to fly high up to the terrain to interact with it. This is getting increasingly annoying.
To fix the issue we should add support for setting the "ocean level" or surface generation height.
In other words terrain generation should not be influenced by the load marker. Instead the load marker should load terrain when terrain is encountered in the world.
Ultimately the goal is of course that the load marker follows the player and we load and unload terrain based on a bunch of rules.
But as far as Weltschmerz is concerned we should have support for setting the surface generation level which is not dependent on the location of the load marker. Instead the load marker and camera should be placed at the surface generation level.

Support for this is planned. It is one of the base enhancment for generating this https://github.com/starandserpent/Weltschmerz/issues/2 in terra

Terrain hight is configurable


tsconfig.json wiki should indicate which version of tsc supports tsconfig.json

tsconfig.json is a great new feature, but older versions of tsc do not support it. The documentation at https://github.com/Microsoft/TypeScript/wiki/tsconfig.json should indicate which versions of tsc supports tsconfig.json.

Thanks, added a note that it's supported starting with 1.5.


FormData() with 'Content-Type': 'multipart/form-data'?

Trying to send an image across the network. Can't figure out what isn't working.
Headers:
{ 'Content-Type': 'multipart/form-data' }

Content is a FormData Blob object:
https://developer.mozilla.org/en-US/docs/Web/Guide/Using_FormData_Objects

using the request API with
{url: '...', method: 'put', data: blob, headers: {}}

Data seems to get swallowed up and not sent by axios. Body reported by chrome as {}.
This was working with the 'rest' package before I switched over.
Any idea what might be going wrong?

Sounds like a bug. I will have to investigate.

This should be fixed with 0.5.4


Improve diagnosability of LogBuildStarted test

This test seems a bit flaky; it's failed a few times in CI builds for no
discernable reason (and passed on retry of the same source).
Unfortunately, the failure was just that an Assert.True failed, which
didn't provide much actionable information (not even which of the two
asserts it was).
In the hopes of making the next failure clearer, this change:

Splits the two asserts into two tests so we can see which is failing
Moves to Assert.Equals, to get xunit's stringification of the objects
on failure (so we can hopefully see a difference).

LGTM


Forgot Password URL always adds "/"

Description
I had my Internal URL/External URL as https://jellyfin.myServer.xyz/ and the forgot password URL redirected to https://jellyfin.myServer.xyz//web/index.html#!/forgotpassword.html
Problem is the "//" here. URL won't lead anywhere.
Not that big of a deal and probably a quick fix, but just signaling it.
Version
1.9.2
Steps to Reproduce

Go to Settings > Jellyfin > Scroll down to "Jellyfin Settings"
Put URL such as "https://jellyfin.myServer.xyz/" in External/Internal URL
Save changes
Try to hit the password reset on the login page. It won't lead anywhere.

Screenshots
No response
Logs
No response
Platform
desktop
Device
PC
Operating System
Windows 10
Browser
Firefox
Additional Context
No response
Code of Conduct

[X] I agree to follow Jellyseerr's Code of Conduct

You shouldn't add a trailing slash when you write the forget password url.


[GithubCodeFolding] Not working anymore

It seems Github changed something and because if that the code folding script doesn't work anymore.

Hi @malnvenshorn!
Thanks for letting me know! It looks like a few other userscripts were broken in their update. I'll have it fixed soon.


Allow number of settings per H2 session to be configurable

(cherry picked from commit d91a8ca17bb17b08ce7f040b868e3c286a7bb0a6)
Conflicts:
doc/admin-guide/files/records.config.en.rst
mgmt/RecordsConfig.cc
proxy/http2/HTTP2.cc
proxy/http2/HTTP2.h
proxy/http2/Http2ConnectionState.cc
proxy/http2/Http2ConnectionState.h

Can ignore the autest failure, because it runs on F30 now, and 7.1.x will not support gcc9 I imagine.


Build automation system for Java server

We need to use a build automation tool, so that we can run the java server code across multiple platforms. This can be done really fast inside any Java based IDE. We may use Gradle.

Added the gradle wrappers for the Java server: https://github.com/ipapapa/IoT-MicroLocation/pull/33


Add the build of the source image

Should be as simple as adding:
  - name: build-source-image
    value: 'true'

To the pipelines in .tekton.

Already done


Reenable OpenApi_Add_NSwagTypeScript tests

I disabled the tests as part of the 3.1 TFM update. https://github.com/aspnet/AspNetCore/blob/release/3.1/src/Tools/Microsoft.dotnet-openapi/test/OpenApiAddFileTests.cs#L137.
The problem is that the KnownFrameworkReferences right now only contains AspNetCore.App 3.0 since they have not ingested our 3.1 build. If we want to enable this test immediately we need to hack the itemgroup to mock a AspNetCore.App 3.1 reference as is done in the Template tests: https://github.com/aspnet/AspNetCore/blob/release/3.1/src/ProjectTemplates/test/Infrastructure/TemplateTests.props.in#L19-L25. However, I think at this point, we can wait and re-enable this test when a new SDK is available.

Thanks @JunTaoLuo.
@ryanbrandenburg can you please re-enable these tests when we get an updated SDK dependency.


Recommendation 52: Relevant elements of the Ecosystem should support the delivery of higher-level and analysis-ready data products in well-documented and broadly used protocols and formats...

Recommendation 52
Relevant elements of the Ecosystem should support the delivery of
higher-level and analysis-ready data products in well-documented and broadly used
protocols and formats, even where those formats might not be appropriate for primary
data. This should include broadening support across the Ecosystem for a wider variety of
data and information formats, such as engineering data; data models; sound and imaging
data; and physical collections attached to planetary missions.

PDS Management Response:

The PDS does host calibrated, analysis ready products for many instrument datasets.  These products are indicated by their processing level.
PDART opportunities may further improve the calibration and the availability of ARD.


Forked snapshot

Fixes #N/A
Description
How was this change tested?
Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
To ensure accuracy in future PRs, please see these guidelines.
A quick fix for this PR: rebase it; your next report should be accurate.

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 82.489%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4918

Relevant Lines:
5962

💛  - Coveralls


Project Structure for Neochat & Architecture

Frontend- react
Backend - go - mux
Folder Structure:
Root folders & Files:
client
server
Dockerfile
package.json
Makefile
config.yml // json file
.env file

Makefile:
install:
cd client/ && npm install && npm run build
We can setup a yaml file to get the domain name, if env is dev run in localhost
if env==prod
install caddy and point domain to client and server app

proxy all /api request to go binary which is running inside systemctl

server
- go.sum
- go.mod
- main.go

Dockerfile:
FROM node-alphone and go-alphine and caddy
CMD cd client && npm install
CMD cd client && build
// caddy commands
Expose 4200 // node port
Expose 3000 // go port


Add WAMR API bindings in Python

Before adding the new bindings:

Moved wasm-c-api in a subfolder in the package.
Adapted the tests to be able to run in this new structure.

New:

Added the WAMR API in another folder in the same level as wasm-c-api.
Created an OOP proposal.
Added an example using this proposal.

Can you @lum1n0us take a look at this? I'd love to hear your feedback since you were the one who did the wasm-c-api adaptation.
Planning
My plan is to extend as much as possible the OOP so that we can use features of WAMR such as debugging. In parallel, completly hide the ctypes API from the user.

@wenyongh, what do we do with the char - in the Python code?

@wenyongh, what do we do with the char - in the Python code?

I have no idea, maybe @lum1n0us has more details.

There's a plan to extend the API from the pov of the OOP to support more functionalities and add more tests. I'm currently working on supporting malloc/free and native symbols. Should I add them in this PR?

@wenyongh, what do we do with the char - in the Python code?

I have no idea, maybe @lum1n0us has more details.

@tonibofarull I am afraid I am not following. What do you mean


CORS preflight error on http & https when calling createStorageAccount

When attempting to create a storage account on web frontend its rejecting due to blocked content-type header field on the preflight check. Should just be a matter of adding it to AllowHeaders on nginx/endpoint config.
Calling code:
this.drive.createStorageAccount("Example", "1MB");

Endpoint: [POST] https://shadow-storage.genesysgo.net/storage-account
Status code: (Failing on preflight so no status) - CORS Error
Error: Access to fetch at 'https://shadow-storage.genesysgo.net/storage-account' from origin 'https://jovial-gingersnap-4394fd.netlify.app' has been blocked by CORS policy: Request header field content-type is not allowed by Access-Control-Allow-Headers in preflight response.
Request (Images come from an earlier test on localhost but same result on the https endpoint):

Thanks for reporting, we're resolving this now. Will post an update here once that is fully rolled out.

This has been resolved @cloakd. Let us know if you face anything else.


structure.sql created if schema_format = :sql is incorrect if postgres schemas are used.

The structure.sql dump file created by the postgres database dump tool does not leave the search_path paramater in a predictable state if the database contains different schemas (other than public).  Accordingly the statements to insert the migration rows appended to the structure.sql might not work if the search_path doesn't include the public schema.
This produces the following error

ERROR:  relation "schema_migrations" does not exist

The following monkey patch fixes this problem for postgres
module SequelRails
  class Migrations
    class << self
      alias_method :dump_schema_information_orig, :dump_schema_information

      def dump_schema_information(opts = {})
        if opts.fetch :sql
          "SET search_path = public, pg_catalog;\n"
        else
          ''
        end + dump_schema_information_orig(opts)
      end
    end
  end
end

Thanks for reporting the issue, please test if it works for you so I can release a new version.


Draining nodes without interrupting busy runners

Is your feature request related to a problem? Please describe.
We'd like to be able to kubectl drain a node but allow running jobs to run until completion. This is so that we can trigger low urgency tasks such as Kubernetes updates or node maintenance. We'd like this to be (semi-)automated, since generally our node updates are automated.
Describe the solution you'd like
A means of draining nodes, without interrupting running jobs.
Describe alternatives you've considered
kubectl drain simply evicts the pods, unsurprisingly 😢.
kubectl cordon is plausible, since runners won't be rescheduled once they're consumed (we're running in ephemeral mode). But this could take a very long time, since a job has to run on the runner first.
I was planning on making a separate CLI tool that would follow roughly this logic:
cordon node
for len(runners) > 0 {
    for runner := range runners {
        if runner is not busy according to GH API {
            evict runner
            remove from runners
        }
    }
}

It's also possible that a mutating admissions webhook can be placed on whatever API kubectl drain uses, which is probably a better solution so can just use kubectl drain rather than a separate utility.
Additional context
Would be happy to work on this myself and contribute back (since I'll have to work on it anyway :smile:). Just wanted to check first if there was something I was missing or if anyone had a better solution.

https://github.com/actions-runner-controller/actions-runner-controller/issues/470#issuecomment-841428853 I guess it's worth mentioning that in theory, quite soon jobs won't fail if there isn't a runner matching the specified labels (this will allow us to scale from 0 without a fake runner). Once that is here you could in theory just setup a scheduled override and wait for the runners to naturally hit 0? Doesn't help you now obviously :D.

Hmm that does sound interesting. I think though would still need to kubectl cordon the node, but at least we'd have fewer replicas sitting around waiting to be consumed.
Perhaps then we can even accomplish this without full horizontal autoscaler support:
cordon node
scale down workers to some low number, thus terminating excess runners
scale back up
wait for runners on that node to be zero
update node

A bit hacky though... would definitely have to run inside a maintenance window. I'd still like to see some safe drain support, because in general we'd like to run updates outside of maintenance windows (as our previous CI system did) or be able to take a node out of service at any time.

We plan to ship most of this by the end of the summer and we plan to also ship it in GHES after we test it on dotcom.

We're on GHES, so see you in a year or two 🤣.

I think though would still need to kubectl cordon the node, but at least we'd have fewer replicas sitting around waiting to be consumed.

yeh you'd need to stop new pods being scheduled and nodes being scaled out but you can do that via regular kubectl commands, no code to be maintained :D

You could almost certainly do something with the scheduling feature and tolerations + taints to do a green / blue style rollover of runners allowing for a seamless upgrade without the need for a window

Definitely feasible... only thing I'm missing is a way to gracefully terminate a runner (i.e. don't terminate if running a workflow).
I suppose the root of the issue is that if the runner is given SIGTERM/SIGINT, it doesn't wait for the current running job to finish. If the runner did have that guardrail, then a NoExecute toleration can be set against set some taint, with a reasonable grace period on the runner pod spec--and all would be well.
Having typed this out just now, it occurs to me that it's probably better to address the pretty serious shortcoming in https://github.com/actions/runner itself rather than add to your maintenance burden :).
On my end I'll probably still need a quick-and-dirty CLI to do the strategy I outlined in the OP, but I'll look into seeing what it takes to patch actions/runner instead. I wonder even if there's some way we can probe whether the actions process is busy then set a preStop hook (querying API, reading process memory/fds, etc.).
Thanks for talking this through with me!

I wonder even if there's some way we can probe whether the actions process is busy from the runner end

@inahga Hey! Yeah I thought the same thing and read actions/runner code a bit. My conclusion at the time (a few months ago) was that there's no hook or any interface for that.
So, the only potential solution would be to programmatically unregister the runner at the very end of the workflow, if that doesn't break the running workflow itself.
Even if that's possible, all the changes you need would reside in your own actions workflow definition, rather than actions-runner-controller. That's why I had not put further effort on this matter, because I thought I'd better focus on actions-runner-controller itself, assuming that's what people expect me to do.

Thanks for the heads up!
I was leaning on querying GH API from the preStop hook, using an in-cluster API proxy to protect the admin:enterprise token.
So the preStop hook would execute a script something like:
until [ curl http://cluster-local.api.gateway/status/$(hostname) returns 200 ]; do
  sleep 5
done

I can't think of a reason for this not to work... but I have said that many times before when working with kubernetes :rofl:

Sorry for reviving this issue, but we need this (drain nodes without cancelling jobs) at my organization too. We would probably be okay with using the ps aux | grep hack described here until they provide a better interface, or maybe even checking for existence of _diag/Worker* files. But to do this we would need to be able to set container lifecycles and probably terminationGracePeriodSeconds too for runner/docker containers. Is this something that could be implemented?

@Natanande Hey! Thanks for the ideas.
terminationGracePeriodSeconds already exist in Runner spec so you can set whatever duration in it and the runner pod will inherit it(this should apply to the pod so you won't need runner/docker container distinction for this).
Container lifecycles aren't configurable at the moment for Runner and RunnerDeployment. But the experimental RunnerSet (#629) which I suppose is being the standard for deploying runners with actions-runner-controller allows you to configure container lifecycles today.
For Runner and RunnerDeployment, I'd appreciate it if you could submit a PR to add it to the RunnerSpec!
Out of curiosity, how would you use container lifecycles for this use-case? Are you willing to defer dockerd container until the runner stops with it?

@Natanande I've done a similar extension of the API in https://github.com/actions-runner-controller/actions-runner-controller/pull/580, if you need help getting started. I just haven't done this myself because I am on vacation, so no business-related OSS contributions for me :wink:. I can do after I return if you are not comfortable.
ps aux | grep sounds like a wonderful simplification over polling the API. I probably should have thought of it :laughing:. Thanks for the tip, would love to test and see how it works out.

But the experimental RunnerSet (#629) which I suppose is being the standard for deploying runners with actions-runner-controller allows you to configure container lifecycles today.

Not to derail the thread, but @mumoshu is there any plans for the RunnerSet to be extended to support ephemeral runners? I am secretly very excited for RunnerSet for its stronger API, but can't use it in production without ephemeral support. Would be happy to test/contribute if needed (after vacation :smile:).

RunnerSet to be extended to support ephemeral runners?

@inahga I thought I have not yet tested it but theoretically, it should just work 😄 Would you mind giving it a shot?
Also, ephemeral runners are fundamentally unreliable while being stopped. We need https://github.com/actions-runner-controller/actions-runner-controller/issues/470#issuecomment-841428853 to improve it. You might have already read that but I thought it worth being mentioned.

@inahga I thought I have not yet tested it but theoretically, it should just work smile Would you mind giving it a shot?

For sure!

Also, ephemeral runners are fundamentally unreliable while being stopped. We need #470 (comment) to improve it. You might have already read that but I thought it worth being mentioned.

Yep we have the risks documented. Given the size of our org, the number and complexity of the builds, and the fact that our previous CI had good support for ephemeral, we really do need ephemeral.
The autoscaling issues we're not worried about since we're not using it. The way our cluster is set up there is no cost saving benefit to autoscaling--so we just run at capacity and ensure the number of runners reflect the peak workload.
As for the issue with --once https://github.com/actions-runner-controller/actions-runner-controller/issues/466, and the lack of a real ephemeral flag, we haven't actually encountered these issues yet. Our build count is well into the thousands by now and myself hundreds, all of varying complexity, and it seems to just work. I suspect maybe some other changeset in actions/runner accidentally quietly fixed it? I have not dug into it though.


Add support for custom set commands on accounts

Fixes #116

The custom set commands are run immediately after account object
creation.

Will leave open for a couple of days before merging and publishing. @offsky, this implements the feature you were requesting as a more general solution. Set commands are permitted to be account specific and hence this pushes you to use the account object to customize for your scenario. Please see the README and additional unit test I added for clarification.

@sonicparke the second part of this PR contains the feature you wanted in #71. Please have a review and provide any feedback you might have. Thanks.

This looks good. It will resolve my issue completely.

Merging as no response on original commit in 2 days and sign-off received on the second feature addition by @offsky.