Simple Non-Image Based Example

Hey @mats-claassen thank you for building this amazing library! This is exactly what I was looking for!!!
I noticed you have a lot of examples with Image based ML and am struggling to get started with a simple non-image example. I have a model built in tensorflow that is 1 layer using Sigmoid that takes in 20 features and outputs 1 label, Binary Classification. I am struggling to get my model to run successfully on Bender. Here are the errors I am seeing.
2017-06-29 17:48:17.286368-0700 bender_test[409:138318] libMobileGestalt MobileGestaltSupport.m:153: pid 409 (bender_test) does not have sandbox access for frZQaeyWLUvLjeuEK43hmg and IS NOT appropriately entitled
2017-06-29 17:48:17.286403-0700 bender_test[409:138318] libMobileGestalt MobileGestalt.c:550: no access to InverseDeviceID (see <rdar://problem/11744455>)
2017-06-29 17:48:17.391311-0700 bender_test[409:138318] /BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalImage/MetalImage-61.5/MetalPerformanceShaders/Filters/MPSKernel.mm, line 370: error 'Source 0x101118830 texture type must be MTLTextureType2D
'
/BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalImage/MetalImage-61.5/MetalPerformanceShaders/Filters/MPSKernel.mm:370: failed assertion `Source 0x101118830 texture type must be MTLTextureType2D'

Do you have an example I can look at where the input MPSImage is composed of a 20 feature tensor?

not what you were looking for - but distantly related albeit this is for server side stuff.
swift + tensorflow + custom models.
https://github.com/PerfectExamples/Perfect-TensorFlow-Demo-Vision

@johndpope do you know if it provides iOS support and if the library size doesn't increase the apps binary size to much? Im capped at increasing our apps size by 5mb.

@dernster any thoughts on how to run a simple non-image predict on Bender from a TensorFlow file?

hi @bvallelunga  - no ios support. the native c++ tensorflow stuff out of the box does have support for ios - as of the other week they now have a development cocoapod of 500mb. not sure how this impacts final build size. I've found it a challenge to do the most basic loading of custom models that are not image based though.
It's likely if there's demand that the swift wrapper that @RockfordWei created for desktop tensorflow could be adopted to fit ios down the track. though i suspect that there's no immediate plans. there's a complicated stripping and scaling down that happens when you build for ios, not all ops are supported.

@johndpope that interesting, though 500mb is like crazy large. I am also having trouble getting a very basic model to work on bender. I am getting these errors:
"\n\n\nNodes in my graph (pre optimization):"
"inputs/x-input"
"model/W"
"model/MatMul"
"model/b"
"model/add"
"model/y_pred"
"inference/Greater/y"
"inference/Greater"
"inference/inference"
"\n\n\nNodes in my graph (after optimization):"
"inputs/x-input"
"model/W"
"model/b"
"model/add"
"inference/Greater/y"
"inference/Greater"
"inference/inference"
"Bender:: Unsupported layer found: Placeholder"
"Bender:: Unsupported layer found: Const"
"Bender:: Unsupported layer found: Const"
"Bender:: Unsupported layer found: Const"
"Bender:: Unsupported layer found: Greater"
"Bender:: Unsupported layer found: Cast"

This is the TensorFlow python script I used to generate the model:

# This script is used to train the model. It repeats indefinitely and saves the
# model every so often to a checkpoint.
#
# Press Ctrl+C when you feel that training has gone on long enough (since this is
# only a simple model it takes less than a minute to train, but a training a deep l
# earning model could take days).

import os
import numpy as np
import tensorflow as tf

checkpoint_dir = "/tmp/voice/"
print_every = 1000
save_every = 10000
num_inputs = 20
num_classes = 1

# Load the training data.
X_train = np.load("X_train.npy")
y_train = np.load("y_train.npy")

print("Training set size:", X_train.shape)

# Below we'll define the computational graph using TensorFlow. The different parts
# of the model are grouped into different "scopes", making it easier to understand
# what each part is doing.

# Hyperparameters let you configure the model and how it is trained. They're
# called "hyper" parameters because unlike the regular parameters they are not
# learned by the model -- you have to set them to appropriate values yourself.
#
# The learning_rate tells the optimizer how big of a steps it should take.
# Regularization is used to prevent overfitting on the training set.
with tf.name_scope("hyperparameters"):
    regularization = tf.placeholder(tf.float32, name="regularization")
    learning_rate = tf.placeholder(tf.float32, name="learning-rate")

# This is where we feed the training data (and later the test data) into the model.
# In this dataset there are 20 features, so x is a matrix with 20 columns. Its number
# of rows is None because it depends on how many examples at a time we put into this
# matrix. This is a binary classifier so for every training example, y gives a single
# output: 1 = male, 0 = female.
with tf.name_scope("inputs"):
    x = tf.placeholder(tf.float32, [None, num_inputs], name="x-input")
    y = tf.placeholder(tf.float32, [None, num_classes], name="y-input")

# The parameters that we'll learn consist of W, a weight matrix, and b, a vector
# of bias values. (Actually, b is just a single value since the classifier has only
# one output. For a classifier that can recognize multiple classes, b would have as
# many elements as there are classes.)
with tf.name_scope("model"):
    W = tf.Variable(tf.zeros([num_inputs, num_classes]), name="W")
    b = tf.Variable(tf.zeros([num_classes]), name="b")

    # The output is the probability the speaker is male. If this is greater than
    # 0.5, we consider the speaker to be male, otherwise female.
    y_pred = tf.sigmoid(tf.matmul(x, W) + b, name="y_pred")

# This is a logistic classifier, so the loss function is the logistic loss.
with tf.name_scope("loss-function"):
    loss = tf.losses.log_loss(labels=y, predictions=y_pred)

    # Add L2 regularization to the loss.
    loss += regularization * tf.nn.l2_loss(W)

# Use the ADAM optimizer to minimize the loss.
with tf.name_scope("train"):
    optimizer = tf.train.AdamOptimizer(learning_rate)
    train_op = optimizer.minimize(loss)

# For doing inference on new data for which we don't have labels.
with tf.name_scope("inference"):
    inference = tf.to_float(y_pred > 0.5, name="inference")

# The accuracy operation computes the % correct on a dataset with known labels.
with tf.name_scope("score"):
    correct_prediction = tf.equal(inference, y)
    accuracy = tf.reduce_mean(tf.to_float(correct_prediction), name="accuracy")

init = tf.global_variables_initializer()

# For writing training checkpoints and reading them back in.
saver = tf.train.Saver()
tf.gfile.MakeDirs(checkpoint_dir)

with tf.Session() as sess:
    # Write the graph definition to a file. We'll load this in the test.py script.
    tf.train.write_graph(sess.graph_def, checkpoint_dir, "graph.pb", False)

    # Reset W and b to zero.
    sess.run(init)

    # Sanity check: the initial loss should be 0.693146, which is -ln(0.5).
    loss_value = sess.run(loss, feed_dict={x: X_train, y: y_train, regularization: 0})
    print("Initial loss:", loss_value)

    # Loop forever:
    step = 0
    while True:
        # We randomly shuffle the examples every time we train.
        perm = np.arange(len(X_train))
        np.random.shuffle(perm)
        X_train = X_train[perm]
        y_train = y_train[perm]

        # Run the optimizer over the entire training set at once. For larger datasets
        # you would train in batches of 100-1000 examples instead of the entire thing.
        feed = {x: X_train, y: y_train, learning_rate: 1e-2, regularization: 1e-5}
        sess.run(train_op, feed_dict=feed)

        # Print the loss once every so many steps. Because of the regularization,
        # at some point the loss won't become smaller anymore. At that point, it's
        # safe to press Ctrl+C to stop the training.
        if step % print_every == 0:
            train_accuracy, loss_value = sess.run([accuracy, loss], feed_dict=feed)
            print("step: %4d, loss: %.4f, training accuracy: %.4f" % \
                    (step, loss_value, train_accuracy))

        step += 1

        # Save the model. You should only press Ctrl+C after you see this message.
        if step % save_every == 0:
            checkpoint_file = os.path.join(checkpoint_dir, "model")
            saver.save(sess, checkpoint_file)
            print("*** SAVED MODEL ***")

@dernster or @mats-claassen any thoughts?

@bvallelunga Thanks. You can take a look at this test. Here we use an internal Texture class that can be created from arrays, then it's converted to a MTLTexture and finally to a MPSImage. In your particular case I see that your model has a "Greater" layer and we don't have support for it at this moment. We are open to receive pull request though :)

@dernster Thank you! Yea I just noticed you don't support the Greater layer. Do you plan on adding support for it soon?
Also does code look good for building a MPSImage?
extension MPSImage {
   convenience init(input: [Float], device: MTLDevice) {
        // The input data must be placed into the MTLTexture from the MPSImage.
        // First we convert it to 16-bit floating point, then copy these floats
        // into the texture slices. (Note: this will fail if the number of channels
        // is not a multiple of 4 -- in that case you should add padding bytes to
        // the example array.)
        var input = input
        let input16 = float32to16(&input, count: input.count)
        let inputImgDesc = MPSImageDescriptor(channelFormat: .float16, width: 1, height: 1, featureChannels: input.count)
        self.init(device: device, imageDescriptor: inputImgDesc)
        input16.withUnsafeBufferPointer { ptr in
            for i in 0..<self.texture.arrayLength {
                let region = MTLRegion(origin: MTLOriginMake(0, 0, 0), size: MTLSizeMake(1, 1, 1))
                self.texture.replace(region: region, mipmapLevel: 0, slice: i, withBytes: ptr.baseAddress!.advanced(by: i*4), bytesPerRow: MemoryLayout<Float16>.stride * 4, bytesPerImage: 0)
            }
        }
    }
}

@bvallelunga there's no plan to add it soon. But maybe you can try to implement it yourself, by taking a look at some other layer!

@bvallelunga Not sure but you can see our Texture class. It already has that functionality and without the multiple of 4 restriction. Maybe we would need to add it to Bender, since right now it's at the Example project.

@dernster would you be able to give me an example of using the functionality without the multiple of 4 restriction?

Thanks, @johndpope for the comment. For TensorFlow iOS solution, however, I think CoreML maybe a better approach - you can import TensorFlow or even Keras models to CoreML. That's one of the most important reasons that I am not going to migrate Python core code into Swift right now without our sponsors' permission. Sorry about that.

Thanks, @johndpope for the comment. For TensorFlow iOS solution, however, I think CoreML maybe a better approach - you can import TensorFlow or even Keras models to CoreML. That's one of the most important reasons that I am not going to migrate Python core code into Swift right now without our sponsors' permission. Sorry about that.

Take into account that Core ML doesn't support TensorFlow without Keras, it only supports versions 1.0.X and 1.1.x and it doesn't support adding custom layeers. Also, it works with iOS 11+, which hasn't been released.

If anyone can be the sponsor I think I can translate the whole TensorFlow
python library into Swift
On Fri, Jun 30, 2017 at 17:53 Santiago Castro<EMAIL_ADDRESS>wrote:

Thanks, @johndpope https://github.com/johndpope for the comment. For
TensorFlow iOS solution, however, I think CoreML maybe a better approach -
you can import TensorFlow or even Keras models to CoreML. That's one of the
most important reasons that I am not going to migrate Python core code into
Swift right now without our sponsors' permission. Sorry about that.
Take into account that Core ML doesn't support TensorFlow without Keras,
it only supports versions 1.0.X and 1.1.x and it doesn't support adding
custom layeers. Also, it works with iOS 11+, which hasn't been released.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xmartlabs/Bender/issues/36#issuecomment-312380985,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVIpO_TisRZQhyE2LlQRGf3fPnQNwoArks5sJW5PgaJpZM4OKCfL
.

@RockfordWei would the the completed framework support iOS and be lightweight (under 2mb)?

On Fri, Jun 30, 2017 at 17:57 Brian Vallelunga<EMAIL_ADDRESS>wrote:

@RockfordWei https://github.com/rockfordwei would the the completed
framework support iOS and be lightweight (under 2mb)?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xmartlabs/Bender/issues/36#issuecomment-312381615,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVIpOxE3oH6liZySaaosqXOvwRYQlI18ks5sJW9LgaJpZM4OKCfL
.

No guarantee, I didn't poke it much

@RockfordWei what kind of sponsorship are you looking for?

On Fri, Jun 30, 2017 at 18:08 Brian Vallelunga<EMAIL_ADDRESS>wrote:

@RockfordWei https://github.com/rockfordwei what kind of sponsorship
are you looking for?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xmartlabs/Bender/issues/36#issuecomment-312383434,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVIpO0EfBNeoZvXgXazbKT41XU_s-E2nks5sJXHXgaJpZM4OKCfL
.

See perfect.org

@RockfordWei I have checked out the site, is there a specific page you would like me to look at?

On Fri, Jun 30, 2017 at 18:13 Rocky WEI<EMAIL_ADDRESS>wrote:

On Fri, Jun 30, 2017 at 18:08 Brian Vallelunga<EMAIL_ADDRESS>wrote:

@RockfordWei https://github.com/rockfordwei what kind of sponsorship
are you looking for?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xmartlabs/Bender/issues/36#issuecomment-312383434,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVIpO0EfBNeoZvXgXazbKT41XU_s-E2nks5sJXHXgaJpZM4OKCfL
.

See perfect.org

We are attracting the third round of investment, actually

Ahhh, no thank you. I am just a single developer working on a side project.

@bvallelunga Sorry for the delay, here's an example:
Create your texture:
let texture = Texture(
            data: [
                [1.0, 2.0, 3.0, 4.0, 5.0, 6], [9.0, 10.0, 11.0, 12.0, 13.0, 14],
                [17.0, 18, 19, 20, 21, 22], [25, 26, 27, 28, 29, 30],
                [33, 34, 35, 36, 37, 38], [41, 42, 43, 44, 45, 46]
            ],
            size: LayerSize(f: 6, w: 2, h: 3)
)

Here, each array corresponds to z-values.
Then, you need to create a metal texture:
let metalTexture = texture.metalTexture(with: device)

And finally a MPSImage instance:
let mpsImage = MPSImage(texture: metalTexture, featureChannels: texture.depth)

Take into account that the class Texture isn't included in Bender, for now it's only on the Examples. Maybe you can just copy that class.


Add ability to generate own empty model and iterate to new empty model after save (also change or force title)

As part of a quicker workflow we should be able to directly jump to the form and have it generate a new empty model on the fly, rather than transitioning through an empty model in the graph editor.
As well, once a model is saved, a button should be made available to create/jump to a new empty form of the same type as previously run.
Note that in the Noctua stack, the model title is special in that it is required before a model can be flushed/stored. Currently, the SAE either takes the "parent"'s name or generates its own from the first entries. For the SAE to operate as a "full" client, it needs to be able to have users be able to initialize and manipulate the title.
(This was available in the previous version of the form.)

Implemented as start a new model function


Wishlist?

Anyone wants to speculate about good to add features?
My list is mostly contribs to add:

websocket endpoint benchmarking tool
basic demultiplexing example wrapper

improved documentation, tutorials, cookbooks, etc
easy installation from standard package managers (apt-get, brew, chocolatey, etc)

I'd also like to see if there's features we can remove or cleanup before hitting 1.0.0 stable (it becomes much harder after that).

Ah.. good point on 1.0

we probably need to define what LEVEL means in --loglevel=LEVEL and clarify things
make sure IPv6 works
I personally really want to do --reverselookup=false as default
think if we need custom error pages for 403/404 and other ones
since devconsole, dir, staticdir, cgidir, and command are all forms of routing.... maybe we should have universal "--route plugin:src:dest" or similar sort of option... that would also make ability to mix devconsole and staticdir

Since we rolled out changes that were done in Server/Handler split we need to re-check how websocketd handles programs that ignore SIGTERM and/or take some time to terminate.

I'd prefer easy installation from standard package managers over any additional features.

Another vote for package manager integration.

Removal/non-removal of CGI also seems to be a candidate to decide before 1.0 (see #138)

Sorry to be late in here!

Broadcast support. Haven't seen any mentions of it.

I'd like to have the possibility to use separate chain certificates (intermediate certificate) and server certificates. So there would be another option like --sslkey and --sslcert, for example --sslchain

@Aginor404 why? They are concatenated eventually, I do not see benefit of splitting those.

I don't see any benefit either, in fact I don't like it, but some of the certificate authorities do split them nowadays, which is why, for example, the Apache guys have the option to just put them into the directory like they are and include both in the config file.
It looks like this:
SSLCertificateFile /etc/ssl/crt/public.crt               //Locate Certificate File
SSLCertificateKeyFile /etc/ssl/crt/private.key               //Locate Private Key File
SSLCertificateChainFile /etc/ssl/crt/intermediate.crt          //Locate the Intermediate File
Maybe I suck at concatenating, but I had the problem that after just putting them into one file together made them not work (not be recognized as valid).
I ended up installing websocketd on another server that does not have an intermediate certificate.

Basic auth of some sort mentioned in #172

I'd like to have the possibility to send/receive multiple-line messages for integration Asterisk server API (AMI - Asterisk management interface) with my SuiteCRM.

Hi,
I've just started using websocketd and it's so nice.
I like the developer console too.
I would love to use websocketd as a websocket client sending messages interactively in command line.
It can do that in the browser using --devconsole and change the address to the websocket server.
I wish I could do that in command line too.
I had a look through the documentation but I haven't figured out if this feature is already there (and if so so how to use it)
Is that something websocketd already supports ? If not, would that be something worth adding ?
Thank you,
George

You kind of can ask your websocket for stuff using curl:
curl -s -H "Connection: Upgrade" \
           -H "Upgrade: websocket" \
           -H "Host: localhost:8080" \
           -H "Sec-WebSocket-Version: 13" \
           -H "Sec-WebSocket-Key: ok" \
          http://localhost:8080 \
  | hexdump -C

But I doubt it's what you want...  I used nodejs wscat before but it stopped installing cleanly for me now, not sure what's there... Anyone has other tools they use?

Thank for the quick answer.
Here's what I'm doing using the html interface:
open ws://<IP_ADDRESS>:7890/
send {"type":"list_connected_devices"}
onmessage {"type":"list_connected_devices","devices":[]}

I imagine a websocket client for nodejs/python/java/etc. will do the trick.
I wish I could do the same using command line instead of the web interface with websocketd.

Special very fast mode without start and stop application for each request. I do not tell about FastCGI, I tell about mode like FastCGI, I tell about mode without start and stop application for each request. I think this is very important for this project. If you do not make it then this make others and they will be faster than your project.

I would love to see the ability to subscribe to stderr on a second websocket.

@meisterluk I like this idea. Not sure if it should be on different port or same, just with query string in URL... Say url?stderr=true&stdout=false  would give you only stderr socket...

Anyone else wants this?
Problem is that subscribing to stderr and stdout on two different sockets requires coordination :( So when you subscribe to output you should specifically request "same stderr" and this is only possible via having some kind of ID for out/err pair.
Solution I suggest above with choices "out or err or out+err" is somewhat reasonable to keep things down to single socket but it might be not the best for people who want this feature for something practical.

UDP connection to server program, for binary data
Correct me if I'm wrong, but I don't see a mechanism for websocketd to do packetized binary transfers to and from a the data-serving program. Since a simple datagram server is easy to write in most languages, and in Linux can be done using a local socket, UDP endpoint support in websocketd would simplify implementation of binary Websockets.

My personal wish is a rewrite of this in pure C

Can add support to send both stdout and stderr messages of the process to the websocket client? Personally I use the following code for the purpose.
func (pe *ProcessEndpoint) StartReading() {
	//go pe.log_stderr()
	if pe.bin {
		go pe.process_binout()
	} else {
		//go pe.process_txtout()
		go pe.process_txt(pe.process.stdout, pe.process.stderr)
	}
}

func (pe *ProcessEndpoint) process_txt( readers ...io.Reader ) {
	var wg sync.WaitGroup
	for _, r := range readers {
		wg.Add(1)
		reader := r  // this line to capture variables, without which, will always read the last reader
		go func(){
			defer wg.Done()
			buf := bufio.NewReader(reader)
			for {
				buf, err := buf.ReadBytes('\n')
				if err != nil {
					if err != io.EOF {
						pe.log.Error("process", "Unexpected error while reading STDOUT from process: %s", err)
					} else {
						pe.log.Debug("process", "Process STDOUT/STDERR closed")
					}
					break
				}
				pe.output <- trimEOL(buf)
			}
		}()
	}

	wg.Wait()

	close(pe.output)
}

is anyway can  A(client) send message to B(client)?


chore: add all resources individually to tarball

Helm doesn't automatically perform CRD upgrades. One thing we can do to help this is publish them separately so admins can upgrade these manually apart from helm.
This PR helps the situation by also publishing all our resources individually and including them in the helm tarball, so admins can easily update the CRDs with something like:
helm template open-feature-operator-v0.2.28.tgz -s templates/apiextensions.k8s.io_v1_customresourcedefinition_featureflagconfigurations.core.openfeature.dev.yaml | kubectl apply -f -
Besides this added benefit, there's no change. You can still install the operator with helm install/upgrade as before.

@toddbaert can you please address the merge conflicts? Once this is merged, please create a separate PR to update the docs.

@toddbaert can you please address the merge conflicts? Once this is merged, please create a separate PR to update the docs.

I'd like to wait for a release to update the docs, or it won't be possible/accurate. I'll update the docs then.


Implement zip file streaming

When searching large zip files (e.g. multi-GB logs or something), it's desirable not to decompress the whole thing in memory at one time.  Instead, a psuedo-FILE object can be created with fopencookie(3) (glibc) or funopen(3) (BSD) that only needs to seek forwards and decompress small chunks at a time, and that FILE* can be passed to the ordinary streaming search routine.
Here's an example of what fopencookie/funopen-wrapped zip streaming looks like: https://github.com/cemeyer/Zlib-FILE  (that implementation is permissively licensed, feel free to just drop it in to the_silver_searcher).
Aside from the FILE-wrapper code, the actual patch should be fairly small.  Something like this:
--- a/src/search.c
+++ b/src/search.c
@@ -315,14 +315,10 @@ void search_file(const char *file_full_path) {
     if (opts.search_zip_files) {
         ag_compression_type zip_type = is_zipped(buf, f_len);
         if (zip_type != AG_NO_COMPRESSION) {
-            int _buf_len = (int)f_len;
-            char *_buf = decompress(zip_type, buf, f_len, file_full_path, &_buf_len);
-            if (_buf == NULL || _buf_len == 0) {
-                log_err("Cannot decompress zipped file %s", file_full_path);
-                goto cleanup;
-            }
-            search_buf(_buf, _buf_len, file_full_path);
-            free(_buf);
+           log_debug("%s is a zip file. stream searching", file_full_path);
+           fp = decompress_file(zip_type, ...);
+           search_stream(fp, file_full_path);
+           fclose(fp);
             goto cleanup;
         }
     }

See #1106.


rm 2020 application link

Pushed some suggested changes.
I think "(COVID-19 dependent)" without instructions on where to find more information raises more questions than it answers, so I removed it from the head and added more text to a banner below.

I'd prefer to pull the dates from the banner if we aren't going to have the COVID announcement above the fold. If you want to do that I'm fine to merge after that. I think it should go out today.

done. If you're happy, I can squash/merge.

applied the suggested changes, squashed and pushed


Update lifecycle.md

Hello @TheBigBirn ... Actually, this is correct ... but "IF" "ISN'T" or "FALSE" "DON'T" ... Geebers McScreebers! :smile: lol ... this sentence is a bit of 🧠 mind bender 🧠. I'd like to work on it and improve what it's trying to say. Basically ...

After the first render :point_right: don't perform additional operations.
If ShouldRender is false :point_right: don't perform additional operations.

I've opened https://github.com/dotnet/AspNetCore.Docs/issues/21139 to work on it. Thanks for calling this 👹 beastly language 👹 out.

Hello @TheBigBirn ... Actually, this is correct ... but "IF" "ISN'T" or "FALSE" "DON'T" ... Geebers McScreebers! :smile: lol ... this sentence is a bit of 🧠 mind bender 🧠. I'd like to work on it and improve what it's trying to say. Basically ...

After the first render :point_right: don't perform additional operations.
If ShouldRender is false :point_right: don't perform additional operations.

I've opened https://github.com/dotnet/AspNetCore.Docs/issues/21139 to work on it. Thanks for calling this 👹 beastly language 👹 out.


UPDATE animals table

FEATURES

Add 7 new values to the table to data.sql

Add new column 'species' and update the values to 'pokemon' and 'digimon' can see it in schema.sql

Transaction queries using UPDATE, DELETE can see in queries.sql

Create Aggregates, can see it in queries.sql

SCREENSHOTS
Transactions

Inside a transaction update the animals table by setting the species column to unspecified. Verify that change was made. Then roll back the change and verify that the species columns went back to the state before the transaction.

Second transaction where change the species to digimon and pokemon and commit the changes

Delete all column and rollback the change.

Last transaction where I commit to update the table as requested in the project req.

Group by queries

How many animals are there?
How many animals have never tried to escape?
What is the average weight of animals?

Who escapes the most, neutered or not neutered animals?
What is the minimum and maximum weight of each type of animal?
What is the average number of escape attempts per animal type of those born between 1990 and 2000?

Note: I use the Pugglin PostgreSQL from Chris Kolkman for this project and that is the interface you see in the screenshots.

Add LICENSE files to fix broken link in MIT License and change 3 queries show in the screenshot below.


feat: Use Database availability as indicator for /healthz response

As the original PR #2718 got reverted, here the next try to implement a /healthz that uses database availability as idicator for the response.
The difference in this PR is, that the response for the "happy case" is not changed and thus backwards compatible (like the original PR initially also tried before the more detailed status messages were introduced).

@Tallicia @swalchemist Could you please review as you were already involved in the initial PR?

I will take look.


Deploy preview is always broken

It always seems broken to me. So it's not really helpful, we should fix it, to be able to test better.

Can you please check if it's better without the https://docs.netlify.com/site-deploys/deploy-previews/#preview-changes-without-the-netlify-drawer ?


Remove deprecated icon links from backend

This PR just removes deprecated icons used in backend to match current Solidus backend style

This is no more needed when we'll merge #69, closing.
cc @adammathys


Fix ctu extdef mapping file with space problem

@Szelethus Could you please commandeer this patch? I guess we'd need some tests for the input files and some unit tests for a common function that is used in both places.

In general, wouldn't it be more expressive (albeit perhaps a little bit slower?) to use a regular expression for the capturing of the individual bits of the input lines, instead of manual juggling of lengths and offsets?

Related Clang patch https://reviews.llvm.org/D102669


adding boregon-trail into OSS

Adding in Boregon Trail game into OSS.  Please let me know if there are any questions.  Thanks in advance!

/validate 860f628d4df3a2d45b5b72b38295c1f38f9c392a


Web server not working

Hello, sorry for the inconvenience, but the PADLOC web server suddenly stopped loading. Is there any way that can be solved?
The message on the page is as follows:
This site can’t be reached
padloc.otago.ac.nz took too long to respond.
Thank you :)

No worries, it just started working again! thank you :) have a lovely weekend :)


Allow customizing default unhandled exception handler

I would like to be able to override default error screen behavior - probably by implementing IBlazorErrorHandler (not sure why is that in Microsoft.MobileBlazorBindings.WebView project now), and providing it somewhere.

In Hybrid Blazor? You can by binding an IBlazorErrorHandler implementation to the ErrorHandler parameter on the BlazorWebView component:
https://github.com/xamarin/MobileBlazorBindings/blob/master/src/Microsoft.MobileBlazorBindings/Elements/BlazorWebView.cs#L30

Nope, I'm mostly interested in same for native components.

And that totally makes sense! I'll tag @Eilon to look at this. It seems trivial to move IBlazorErrorHandler to Microsoft.MobileBlazorBindings and find a way to expose it to the outside world (Either through the DI container or on a registration method during composition.

We can't move IBlazorErrorHandler to Microsoft.MobileBlazorBindings, because that will introduce circular dependency.
While we can easily move it to Microsoft.MobileBlazorBindings.Core instead, I would prefer to move all hybrid related staff to Microsoft.MobileBlazorBindings.WebView, and remove that dependency from Microsoft.MobileBlazorBindings (so that I wouldn't have to import all Web related staff if i need native components only - currently it clutters IntelliSense a lot.

Hmm definitely some interesting ideas here, will have to think about the layering. Much of the layering here is "for convenience" so that 99.99% of projects can reference 1 NuGet package and it pulls in everything you need. Of course, it's always going to be more than you need because no one needs everything. At some future point we'll definitely reconsider the overall layering so that projects can be leaner. But app trimming is probably the best solution to this anyway because that will do more than we ever could solve with layering.
But for error handling I'm sure we can come up with something good.
Can you share some info on what you would do in the error handler? That will help ensure the design we come up with is suitable.

App trimming will help with app size, but not with cluttered IntelliSense.
I'd like to handle some custom ApiExceptions in different way (probably showing some alert with more or less friendly message), plus I'd like to avoid showing exception details in non-development builds.

So you want to be able to "handle" certain errors, and not just show a different (nicer) error message? Right now once the error gets to the error handler the app is considered "dead" but I think we can easily change that.

@jspuij Thanks for the steer on how to set the custom error handler. I'd expected it to be work automatically based on a custom service registration with the DI container. When I registered a custom IBlazorErrorHandler implementation into DI and it wasn't used, I was slightly confused!
I've updated Main.razor in my solution to accept an injected variable of type IBlazorErrorHandler, and pass that on to the BlazorWebView via its ErrorHandler parameter.
I wonder if the code generated by the Hybrid App template should contain this by default, so that people don't need to go searching for why their custom implementation is not being used as they might have expected.
I can see why the DI-based approach was not used in the code as it stands, since the BlazorWebView class currently has a static constructor to initialise it just one time. However, maybe that is something that could be changed to make it more obvious how to customise the behaviour using standard DI registration practices? At the moment, there are a couple of steps needed to be able to output the details of an exception to ILogger, which I'd expected to happen automatically (but doesn't, as far as I can tell.)

@TheCakeMonster we could try to make this work. For now it's just an area where we put in a quick feature to make some really-bad scenarios be less-bad, but it's still not great! We'll have to think about this one a bit to come up with a reasonable solution.


Displayed parameter string for strcmp not fully displayed until null byte

Version and Platform (required):

Binary Ninja Version: 3.3.3996 Personal (Build ID e34a955e)
OS: MacOS Ventura
OS Version: 13.2.1
CPU Architecture: M1

Bug Description:
I am reversing a binary that is linked to libc.
Type: ELF
Platform: linux-x86_64
Architecture: x86_64

Libraries:
  libc.so.6

I got the the interesting part where a password comparision happens. On first sight, I thought the password is the one that is displayed within strcmp. But this is not the full password.

However, going deeper (assembly) gave me the same information.

After all, I looked at the memory location and was suprised that the string end is not awaited until a null byte.

Steps To Reproduce:
I don't know if it is up to the binary, but the binary is currently part of an ongoing CTF which I am not allowed to share.
Expected Behavior:
The string should be displayed until there is a null byte.

The annotation not showing up in disassembly is correct and expected behavior. However, this does highlight another issue which I'm going to rename this issue to address. Specifically, even if you change the type of the argument to be a "char *" the "guess a string" annotator that's used in disassembly is used in HLIL which is incorrect. So the behavior you're observing here is mostly correct (except for the bug described in #4083), however the work-around I was going to suggest (setting the type of the argument -- indeed, depending on the type libraries for this binary the type might already be set) SHOULD result in correct rendering at the HLIL and that is indeed a bug.

The reason we don't display the full string is because we don't support line wrapping yet. I'm closing this ticket in favor of #2366


How can I get the vocab distribution at each step in the decoder

Hello dear folks,
How can I get the vocab distribution at each step in the decoder.
After I successfully trained the model,
I would like to access the probability distribution over the vocabulary at each step, in the decoder. Say given the context of a target word, how can I get the probs for the target?
Ideally, given a sentence w_1 w_2 ... w_i ...w_n, is there any possibility to get the probability P(w_i = w_j | context)  for all w_j I have in the vocabulary, given that context consist of both the prefix and the postfix of w_i. If postfix is not possible, then only from the prefix.
I think it might be in fconv.py in the Decoder class, the incremental_inference function. But I can't figure out exactly how.
Thank you so much ahead,
Victor

You can't get postfix with a translation model, translations are left to right, so the model is trained to see prefixes only. For language models the story is different, but again, you seem to be talking about translation.
You can get log probs for each word in the vocabulary by inspecting lprobs variable here:
https://github.com/pytorch/fairseq/blob/master/fairseq/sequence_generator.py#L299
Be careful about batching and beam size though, it might be easier to put a breakpoint and inspect the content of this tensor before changing it.

@shirakad  do you solve your problem, I have the same problem as well. Thanks!


Problem with ping360_sonar install on Xavier

CMake Error at /opt/ros/melodic/share/cv_bridge/cmake/cv_bridgeConfig.cmake:113 (message):
Project 'cv_bridge' specifies '/usr/include/opencv' as an include dir,
which is not found.  It does neither exist as an absolute directory nor in
'${{prefix}}//usr/include/opencv'.  Check the issue tracker
'https://github.com/ros-perception/vision_opencv/issues' and consider
creating a ticket if the problem has not been reported yet.
Call Stack (most recent call first):
/opt/ros/melodic/share/catkin/cmake/catkinConfig.cmake:76 (find_package)
CMakeLists.txt:10 (find_package)

Closing this so we can keep everything to issue #206


Changing the URL to access dashboard

Hey all,
I have a lab server at http://lab.martynkeigher.com
On this Ubuntu 12.04 server I have both Apache installed (using port 80) and dashing (using port 3030).
I copied the 'sampletv' dashboard and create a new erb file called dashing.erb and it is viewable at: http://lab.martynkeigher.com:3030/dashing
So, in an effort to remove the ':3030' from the URL, I have added the following to the Apache/sites-available/default file: http://pastebin.com/AfkiTmVi
If you need to see the full contents of the default file then please go here: http://pastebin.com/k4WViFur
So now, if you go to http://lab.martynkeigher.com/dashing it works, kinda...but not fully!
Can you please go to the above URL and let me know why and what I need to change?
Thank you so much! :)
Martyn.

There is a great wiki page on using dashing on parked domains. The page is at https://github.com/Shopify/dashing/wiki/How-To:-Run-dashing-on-a-sub-path

hi @bean5 i tried this but it is not working for me
i got a return 404

@MartynKeigher did u solve ur problem?


chore: test release

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

:tada: This PR is included in version 1.9.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Select blocks the render of complete page

Current Behavior
@natew , I upgraded tamagui from 1.7.8 -> 1.9.4 in my project and now Select component blocks the render of the whole page before it loads. Is this expected?
I'm using expo-router + tamagui. The screen is "not" navigated until Select component finishes loading/rendering.
Suggestion: I'll suggest upgrading the starter to use expo-router which provides fantastic file based navigation, just like Nextjs. It will also make the starter template a lot easier for people to understand.
Expected Behavior
The Select component should load it's value faster or not block the render of whole page.
Tamagui Version
1.9.4

Reproduction
-

System Info
No response

this should be fixed


create env file and remove not used import

Hi everyone !
This short PR is made in order to remove the hardcoded version of the url in the pizero/runner.py script. It is better to put the IP and PORT in a separate environment file under the model of python.env.dist. Also I removed an import which was not used in this script, namely PIL (if needed after for an improvement of the script, it could be added at this time).
Consequently I updated the README.md and the previous env file in pi_utils so that all variables have the same name.
Also I have a more general question: why was this folder pizero created ? I believe that the runner script could be put inside pi_utils since this directory already has pizero's requirements + the monitor script + the environment file corresponding to the IP and PORT of the local webserver which will run on the main rpi. I don't see why we would need to create separate directories for these two scripts, in my opinion they go hand in hand. What was the motivation ?
As always, feedback is welcomed 🤗

Yes let's wait for @MateoLostanlen's input ! :)

@chloeskt agree to move the pizero runner and delete the folder 😉

Alright ! In that case, this evening I shall modify this in this current PR then and ask for your reviews again. Or do you prefer that I open a new PR ?

1 PR = 1 action :)

I can open the new PR if you want

I can open the new PR if you want

I already did everything, I just want to check that all requirements, the flow, etc is working on rpi


Small improvements

Check List:

[x] Tests pass (./test.sh <username> shows no differences between expected and actual outputs)
[x] All formatting changes by the build are committed
[x] Your launch script is named calculate_average_<username>.sh (make sure to match casing of your GH user name) and is executable
[x] Output matches that of calculate_average_baseline.sh
[x] For new entries, or after substantial changes: When implementing custom hash structures, please point to where you deal with hash collisions (line number)

Execution time: ~30 ms better
Execution time of reference implementation: a lot

Comes in a bit slower actually:
Benchmark 1: timeout -v 300 ./calculate_average_artsiomkorzun.sh 2>&1
  Time (mean ± σ):      2.060 s ±  0.010 s    [User: 0.002 s, System: 0.004 s]
  Range (min … max):    2.034 s …  2.066 s    10 runs

Summary
  artsiomkorzun: trimmed mean 2.0625630242949997, raw times 2.0336724294199997,2.06647070542,2.06597330042,2.0661385814199997,2.05878286342,2.06480267942,2.0643172384199997,2.0615065864199997,2.0622090584199997,2.05677388642

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:02.062 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_artsiomkorzun.java)| 21.0.2-graal | [Artsiom Korzun](https://github.com/artsiomkorzun) | GraalVM native binary, uses Unsafe |

@gunnarmorling reverted some changes. Please try once more.

Almost the same now as the current entry:
Benchmark 1: timeout -v 300 ./calculate_average_artsiomkorzun.sh 2>&1
  Time (mean ± σ):      2.022 s ±  0.015 s    [User: 0.001 s, System: 0.004 s]
  Range (min … max):    1.996 s …  2.036 s    5 runs

Summary
  artsiomkorzun: trimmed mean 2.0264423235666666, raw times 1.9956557839,2.0358313169000004,2.0279398379,2.0246280749000003,2.0267590579

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:02.026 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_artsiomkorzun.java)| 21.0.2-graal | [Artsiom Korzun](https://github.com/artsiomkorzun) | GraalVM native binary, uses Unsafe |


feature: API access via the portal when using external SSO

Is it possible to have the portal issue API keys for logged in users? Or is any way to facilitate access to APIs behind the portal to cli users?

@silvajp2 , please elaborate on the specific use case and authentication flow. Please describe the process in its entirety.

hey @greenpau , sorry for not being clear!I don´t have a full flow yet, just a general question / something I would like to have / exploring.
As a starting point, having a /token endpoint that returns the token that the user can then copy paste. Would this be possible?

As a starting point, having a /token endpoint that returns the token that the user can then copy paste. Would this be possible?

@silvajp2 , I am not great at hypotheticals 😄 However, the JWT is the token you could use to get "/whoami" endpoint programmatically. See here: https://github.com/greenpau/caddy-auth-portal/issues/159#issuecomment-892076005
generally, are we talking about?

authentication
data access

Not at the moment.
i am redesigning local auth in such a way that SSO account (“connected account”) become linked to a local account. So it will likely happen in the next couple months.

That's awesome!
This will also allow MFA for SSO then?

@silvajp2 , the portal is now issuing API keys.
Please reopen in github.com/greenpau/caddy-security if this is still relevant for you. Please consider sponsoring the work on the project! 👍


try to avoid calling actions from update_decls callbacks

in lang/talon/talon.py we call actions.user.create_spoken_forms_from_list from an "update_decls" callback. https://github.com/knausj85/knausj_talon/blob/master/lang/talon/talon.py#L59
This has a complicated interaction that causes an error similar to that in https://github.com/talonvoice/talon/issues/429 to occur. we can temporarily solve this by using Any instead of T, but that's just papering over the underlying problem.
Had a long discussion with aegis on slack about this, starting at https://talonvoice.slack.com/archives/CJ2EE3LEB/p1637107224077200. Resolution seems to be: calling actions from an update_decls callback is a bad idea, we should find a way to avoid it. aegis is considering banning calling actions from these callbacks.
Ways to avoid it:

move the create spoken forms logic and the talon.py list updating logic into the same file
import create_spoken_forms into talon.py, deliberately breaking the no-cross-imports rule
something else?

aegis suggests that eventually create_spoken_forms might go away (I assume he means "become a talon feature", but not sure), so it might be okay if the solution here isn't the prettiest for now.

I thought of a possible solution on my side, which is instead of banning actions in update_decls, I use stale decls for those actions (the actions that were present before the reload). Unsure if viable but it would mean no changes needed in knausj_talon if it worked.


e.message error fix

I hope this all works like this, because I have not tested it localy.

Original Message:
https://github.com/DasKraken/CR-dl/blob/master/src/m3u-download.js#L76
on exactly this line here, the Downloader will quit because nodeJS has an In-time compile error. the variable "e" is not found in this block.

should be fixed now in 4.0.0


Crashes when activated

I just installed this plugin and as soon as I click to enable the plugin Obsidian freezes and Obsidian has to be force closed (Windows). I've tried a few times now and got the same result. This is the error that occurs in console:

Same is happening to me

Same thing here now

It crashes for me too. I hope this can be fixed, it seems like a very useful plugin

Still crashes when enabling.

this plugin is causing the entire ObsidianMD ui to disappear and be a blank white window with no connection to the inspector even.
copypaste:
16 Uncaught (in promise) TypeError: Cannot read plugin:msg-handler: 10818
properties of undefined (reading 'split')
at extractEMLAttachments (plugin:msg-handler: 10818:43)
at getMsgContent (plugin:msg-handler: 10725:20)
at async MsgHandlerPlugin.handleFileCreate (plugin:msg-handler:11550:28)
• Uncaught (in promise) TypeError: Cannot read plugin:msg-handler: 10818
properties of undefined (reading 'split')
at extracteMLAttachments (plugin:msg-handler: 10818:43)
at getMsgContent (plugin:msg-handler: 10725:20)

when i disable this plugin it stops doing that, but i also lose the ability to read .eml files. i dual-index them in DEVONthink too so i can still access them that way but it exceeds my capabilities to troubleshoot this much.


ci(release): use PAT

Use PAT instead of default token in hopes of not having branch protection issues in git release step

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


feat:updating go.mod dependencies for alpha4

This PR updates notation cli with the main branch of notation-core-go and notation-go.
After all PRs merged, I will update go.mod to use main branch.
Test manually on virtual machine.
Now I hard code envelope type to be jws. I will add more envelope types after cose merged.
Signed-off-by: zaihaoyin<EMAIL_ADDRESS>

Codecov Report

Merging #357 (fd30969) into main (06b7b5b) will decrease coverage by 0.07%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #357      +/-   ##
==========================================
- Coverage   30.62%   30.54%   -0.08%
==========================================
  Files          25       25
  Lines        1610     1614       +4
==========================================
  Hits          493      493
- Misses       1104     1108       +4
  Partials       13       13

Impacted Files
Coverage Δ

cmd/notation/verify.go
18.62% <0.00%> (-0.38%)
:arrow_down:

internal/cmd/signer.go
0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Ran into a small upgrade issue (3.1 -> 3.2)

wanted to bring it to attention in case someone else runs into this when trying to start the service:
● arangodb3.service - ArangoDB database server
   Loaded: loaded (/lib/systemd/system/arangodb3.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2017-09-04 21:51:37 CDT; 5s ago
  Process: 4519 ExecStartPre=/usr/bin/chmod 700 /var/log/arangodb3 (code=exited, status=203/EXEC)
  Process: 4518 ExecStartPre=/usr/bin/chown -R arangodb:arangodb /var/log/arangodb3 (code=exited, status=0/SUCCESS)
  Process: 4517 ExecStartPre=/usr/bin/install -g arangodb -o arangodb -d /var/run/arangodb3 (code=exited, status=0/SUCCESS)
  Process: 4516 ExecStartPre=/usr/bin/install -g arangodb -o arangodb -d /var/tmp/arangodb3 (code=exited, status=0/SUCCESS)

Running
ln /bin/chmod /usr/bin/chmod
ln /bin/chown /usr/bin/chown

solved it for me.

Hi,
This is a duplicate of #3197 which will be fixed with the next release.
I will close this issue - thanks for reporting.


 removed

Coverage increased (+0.03%) to 38.948% when pulling c17e2a3c029a85fa8ef180449ec41c9d961184a4 on clean_document into 1ddd55a4ad88f7f1f143f8fb41e339b56d2c3c59 on master.

Coverage increased (+0.02%) to 38.939% when pulling 959119ae501ff042413cc8cd9aa3c42c6e658685 on clean_document into 618cc391c0eb732b9aae92487db70519240ca893 on master.


Repacking mod doesnt apply changes made

I unpacked a mod I wanted to balance (DBZMOD), changed a few things, and repacked it, then replaced the mod in tmodloader's mod folder, but the mod hasn't been changed at all.
TMLP version: <IP_ADDRESS>
TML version: <IP_ADDRESS>
Terraria version: <IP_ADDRESS>

I ran into this problem as well.
I unpack the same mod, make my changes, repack it and replace the mod in the Mods folder for Tmodloader.
When I load into the game none of the changes are taken, in fact I've gone out of my way to break things or use completely ridiculous values to see if something happens but nothing does. The tmod files are just ignored, it's like nothing was done.
I've made new worlds and characters and this persists, none of my changes take effect.
Am I missing a step somewhere? I haven't decompiled or recompiled anything, I simply opened the .cs files and changed values a bit in Visual Studio and repacked because DBZ things are very OP in multiplayer.

I ran into this problem as well. I unpack the same mod, make my changes, repack it and replace the mod in the Mods folder for Tmodloader.
When I load into the game none of the changes are taken, in fact I've gone out of my way to break things or use completely ridiculous values to see if something happens but nothing does. The tmod files are just ignored, it's like nothing was done. I've made new worlds and characters and this persists, none of my changes take effect.
Am I missing a step somewhere? I haven't decompiled or recompiled anything, I simply opened the .cs files and changed values a bit in Visual Studio and repacked because DBZ things are very OP in multiplayer.

That's worrying, I may have gotten a path mixed up. I'll have to look into this later.
I don't believe it's an issue on your end.

Good luck!
I'd figured since that other person never responded it was fixed or on my end so I spent all day figuring it out and got nowhere.
Hopefully it's an easy fix.

Repacking is currently no longer planned, build your mods in-game.

Oh yeah FYI this has been rectified, packing your mods should work fine.


Better support for 20+ CPU cores

So far we've focused mostly on 4 cores with profiling, but moving forward we should make sure that Mojo::Server::Prefork (and Hypnotoad) scale well much further. Aside from optimizations like high/low watermarks (for when to start/stop accepting connections), we could also experiment with different configurations, and perhaps adjust the defaults and/or documentation.
Since most members of the core team do not have access to such servers, we will need volunteers to help us.
perl examples/fast.pl prefork -w 20 -m production -l http://*:8080

For load testing we recommend this example application, and wrk.
wrk -c 100 -d 30s http://<IP_ADDRESS>:8080/

For starters we need a baseline, a series of load tests with an incrementally increasing number of workers workers (steps of 2?).

Okay -- I have run this on the server where I use Mojolicious anyway; this is Mojolicious 6.03 (I have been admittedly lazy with updates) on a HP DL-380 Gen8 server with 64 (or 66?) GB RAM and 2 x 6 cores with hyperthreading, so I let it run with up to 24 workers in steps of 2.
If really necessary, I can install and use a newer Mojolicious version, but I am afraid this means work and I am a bit hesitant to fix what ain't broken (for me).
multicore-benchmark.sh.txt
typescript.txt

Ah, and if you think my test driver script does the right thing, anyone else is of course welcome to use it, too.

Here are two more runs that don't show that don't show that extreme outlier at 20 workers like the first one, but connection problems for 16 and 18 workers instead, and a fourth one that has no connection problems, but an outlier at 20 workers again. And at 24. Dunno what to make of that.
For fun, I extracted the values for requests/s into a table; maybe that is of any use. (And pesky Github doesn't accept file names without extension, and no *.sh or *.csv files. Darn.)
typescript2.txt
typescript3.txt
typescript4.txt
mojo-workers.csv.txt

I really hope those results are not valid, otherwise we would have a lot of work to do.

Tangentially: perhaps there is a large establishment using Mojo that can donate CPUtime on a larger box for further testing.
E.g. the way ServerCentral provided @briandfoy 80 cores to look for stuff

More results with -M 2 and -M 5, respectively.
multicore-benchmark.sh.txt
typescript-M2.txt
typescript-M5.txt
typescript-M5b.txt
typescript-M2.txt:======== running with 2 workers
typescript-M2.txt:Requests/sec:   3297.08
typescript-M2.txt:======== running with 4 workers
typescript-M2.txt:Requests/sec:   6848.40
typescript-M2.txt:======== running with 6 workers
typescript-M2.txt:Requests/sec:  10258.88
typescript-M2.txt:======== running with 8 workers
typescript-M2.txt:Requests/sec:   7107.24
typescript-M2.txt:======== running with 10 workers
typescript-M2.txt:Requests/sec:   3633.44
typescript-M2.txt:======== running with 12 workers
typescript-M2.txt:Requests/sec:   4227.07
typescript-M2.txt:======== running with 14 workers
typescript-M2.txt:Requests/sec:  16374.49
typescript-M2.txt:======== running with 16 workers
typescript-M2.txt:======== running with 18 workers
typescript-M2.txt:Requests/sec:   3754.41
typescript-M2.txt:======== running with 20 workers
typescript-M2.txt:Requests/sec:   2653.64
typescript-M2.txt:======== running with 22 workers
typescript-M2.txt:Requests/sec:  11632.83
typescript-M2.txt:======== running with 24 workers
typescript-M2.txt:Requests/sec:  10521.71
typescript-M5.txt:======== running with 2 workers
typescript-M5.txt:Requests/sec:   3199.20
typescript-M5.txt:======== running with 4 workers
typescript-M5.txt:Requests/sec:   6507.45
typescript-M5.txt:======== running with 6 workers
typescript-M5.txt:Requests/sec:  10442.67
typescript-M5.txt:======== running with 8 workers
typescript-M5.txt:Requests/sec:   5065.49
typescript-M5.txt:======== running with 10 workers
typescript-M5.txt:Requests/sec:   3978.57
typescript-M5.txt:======== running with 12 workers
typescript-M5.txt:Requests/sec:   8653.46
typescript-M5.txt:======== running with 14 workers
typescript-M5.txt:Requests/sec:   9886.64
typescript-M5.txt:======== running with 16 workers
typescript-M5.txt:Requests/sec:   3199.90
typescript-M5.txt:======== running with 18 workers
typescript-M5.txt:Requests/sec:   2734.76
typescript-M5.txt:======== running with 20 workers
typescript-M5.txt:Requests/sec:  18723.67
typescript-M5.txt:======== running with 22 workers
typescript-M5.txt:Requests/sec:   2021.80
typescript-M5.txt:======== running with 24 workers
typescript-M5.txt:Requests/sec:   1614.69
typescript-M5b.txt:======== running with 2 workers
typescript-M5b.txt:Requests/sec:   3265.98
typescript-M5b.txt:======== running with 4 workers
typescript-M5b.txt:Requests/sec:   6592.93
typescript-M5b.txt:======== running with 6 workers
typescript-M5b.txt:Requests/sec:  10347.86
typescript-M5b.txt:======== running with 8 workers
typescript-M5b.txt:Requests/sec:   6635.31
typescript-M5b.txt:======== running with 10 workers
typescript-M5b.txt:Requests/sec:   4025.01
typescript-M5b.txt:======== running with 12 workers
typescript-M5b.txt:Requests/sec:   4639.24
typescript-M5b.txt:======== running with 14 workers
typescript-M5b.txt:Requests/sec:  15606.98
typescript-M5b.txt:======== running with 16 workers
typescript-M5b.txt:======== running with 18 workers
typescript-M5b.txt:Requests/sec:   4145.14
typescript-M5b.txt:======== running with 20 workers
typescript-M5b.txt:Requests/sec:   3347.29
typescript-M5b.txt:======== running with 22 workers
typescript-M5b.txt:Requests/sec:  12196.62
typescript-M5b.txt:======== running with 24 workers
typescript-M5b.txt:Requests/sec:   9203.04

Ok, if it is not the multi-accept, we have to find out what else could cause the load to be so unevenly distributed.

Some more runs with Mojolicious-6.51.
typescript-run1.txt
typescript-run2.txt
typescript-run3.txt
typescript-run4.txt
typescript-run5.txt
typescript-run6.txt
typescript-run7.txt
7runs.csv.txt

Thanks, i think we have enough results for now, as long as they don't show a new pattern.

It appears that for these kinds of load tests we have to limit the number of connections each worker is allowed to handle concurrently.
perl examples/fast.pl prefork -w 8 -c 16 -a 0 -r 1000 -m production -l http://*:8080

So, for 8 CPU cores and 128 concurrent connections, we would end up with 8 workers that each handle 16 connections. Additionally, it seems like a good idea to disable automatic restarting of workers, and increase the number of keep-alive requests per connection.
wrk -c 128 -d 30s http://<IP_ADDRESS>:8080/

We have a server that is currently solely used for tests and development tasks; this allows me to run these benchmark tests on an otherwise idle system but I have to wait for a time when it is not used for other in house tasks.
So generally I should be able to repeat the benchmark tests when asked for but I can´t promise to be able to do that immediately on request since the system may be in use at the time.
The system itself is running SLES 12 SP 1 on a 2 socket / 6 cores per socket / 2 threads per core (Hyperthreading) architecture (Xeon X5675 at 3.07GHz) with 24 GB of RAM.
I installed Perl 5.22.1 via plenv and added only Mojolicious. I´ve adapted the benchmark script by jyrgenn to use the parameters in kraih´s latest post:
multicore-benchmark.sh.txt
I did not test variations of the number of connections; only the number of workers was changed.
I´ve run the script with both v6.51 and v6.52:
mojo_651.txt
mojo_652.txt
I have to admit that I currently don´t know much about performance tuning so you´ll have to tell me what params to use for additional tests if you need any. We don´t have any Mojolicious apps under heavy load so far, but we´re working on a larger app (which will make heavy use of the non-blocking features) and I hope to gain some insight this way...

A few test runs with 6.52 and the latest parameters above showed that everything was totally fine, all processes used 100 % (or what they could get) CPU from beginning to end, totally stable, every time. Great, @kraih, I guess you nailed it!
Details to follow, must hurry to a meeting now.

Thanks everyone!

The curve for @jyrgenn 's latest data looks pretty good.

The curve for @heikojansen's data looks even better. (also 12 CPU cores with hyper-threading)

In case someone wants to run more benchmarks in the future, here's an updated script.
#!/bin/sh

PORT=8080
ADDR=<IP_ADDRESS>
WRK=/usr/local/bin/wrk
FAST=/Users/sri/repo/mojo/examples/fast.pl
PID=/Users/sri/prefork.pid
CONNECTIONS=128
MAXWORKERS=128

WORKERS=2
while [ $WORKERS -lt $MAXWORKERS ]; do
    echo ''
    let CLIENTS=(CONNECTIONS/WORKERS)+1
    echo ======== running with $WORKERS workers and $CLIENTS connections per worker
    perl $FAST prefork -w $WORKERS -c $CLIENTS -a 0 -r 100000 -m production -l http://$ADDR:$PORT -P $PID &
    sleep 2
    $WRK -c $CONNECTIONS -d 30s http://$ADDR:$PORT/
    sleep 2
    while [ -f $PID ]; do
        kill -s QUIT $(cat $PID)
        sleep 1
    done
    echo ========
    let WORKERS+=2
done

Ok, here are the results for above benchmark,
https://gist.github.com/mpapec/bd9bae3dd7a59c8ee749984d7173e9f6

@mpapec The curve looks pretty good, until there's not enough connections anymore to scale further.

Included EV in case it shows improvements over vanilla ioloop. Btw, what is your opinion on HyperThreading in scaling context (are there real benefits from quasi cpu cores)?
CONNECTIONS=512
https://gist.github.com/mpapec/3a7bff7513d12278413319e631df09ab
CONNECTIONS=512 && use EV
https://gist.github.com/mpapec/7cbfec10a6482269b575a8999300b5d8
CONNECTIONS=9000 && use EV
https://gist.github.com/mpapec/3bf1ab859506948c7d61139265123d2c

@mpapec A Hyper-Thread seems to be worth about 20% or a real CPU core.

Opteron 64 cores,
https://gist.github.com/mpapec/2f242879debd6015f02829d41c4c9600
(paired cores share FPU and L2)
http://www.theregister.co.uk/2015/11/06/amd_sued_cores/


Where is the dataset and what kind of dataset am I supposed to use inorder to train?

sorry couldn‘t find them in the repository...

If you scroll down to the bottom of the top-level README.md page, you'll see a table with hyperlinks. You should be able to click on the link, download the dataset, and unzip them inside your data/ directory. It should all just work.
For example, if you want to run the experiment for Mutagenicity

Go to https://ls11-www.cs.tu-dortmund.de/staff/morris/graphkerneldatasets (as listed in the table)
Search for Mutagenicity on that page, which should allow you to download the data files from https://ls11-www.cs.tu-dortmund.de/people/morris/graphkerneldatasets/Mutagenicity.zip
Unzip it inside your data/ directory
Run $ PYTHONPATH=. python train.py --bmname=Mutagenicity


return more struct info

Describe your changes
Issue ticket number and link
Closes #xxxx (Replace xxxx with the GitHub issue number)
Checklist before requesting a review

[ ] I have performed a self-review of my code

Check the correct box. Does this PR contain:

[ ] Breaking changes
[ ] New functionality
[ ] Major changes
[ ] Minor changes
[ ] Bug fixes

[ ] Proposed changes are described in the CHANGELOG.md

[ ] CI Tests succeed and look good!

Maybe it would be nice if the output stored in struct is more or less in line with the format in api/file_*.yaml files, that is:
type: file
example: "resources_test/common/pancreas/raw.h5ad"
info:
  label: "Raw dataset"
  summary: An unprocessed dataset as output by a dataset loader.
  description: |
    This dataset contains raw counts and metadata as output by a dataset loader.

    The format of this file is derived from the [CELLxGENE schema v4.0.0](https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/4.0.0/schema.md).
  slots:
    layers:
      - type: integer
        name: counts
        description: Raw counts
        required: true
    obs:
      - type: string
        name: dataset_id
        description: Identifier for the dataset from which the cell data is derived, useful for tracking and referencing purposes.
        required: false

      - type: string
        name: assay
        description: Type of assay used to generate the cell data, indicating the methodology or technique employed.
        required: false

      - type: string
        name: assay_ontology_term_id
        description: Experimental Factor Ontology (`EFO:`) term identifier for the assay, providing a standardized reference to the assay type.
        required: false

      - type: string
        name: cell_type
        description: Classification of the cell type based on its characteristics and function within the tissue or organism.
        required: false

      - type: string
        name: cell_type_ontology_term_id
        description: Cell Ontology (`CL:`) term identifier for the cell type, offering a standardized reference to the specific cell classification.
        required: false

      - type: string
        name: development_stage
        description: Stage of development of the organism or tissue from which the cell is derived, indicating its maturity or developmental phase.
        required: false

      - type: string
        name: development_stage_ontology_term_id
        description: |
          Ontology term identifier for the developmental stage, providing a standardized reference to the organism's developmental phase.

          If the organism is human (`organism_ontology_term_id == 'NCBITaxon:9606'`), then the Human Developmental Stages (`HsapDv:`) ontology is used.
          If the organism is mouse (`organism_ontology_term_id == 'NCBITaxon:10090'`), then the Mouse Developmental Stages (`MmusDv:`) ontology is used.
          Otherwise, the Uberon (`UBERON:`) ontology is used.
        required: false

      - type: string
        name: disease
        description: Information on any disease or pathological condition associated with the cell or donor.
        required: false

      - type: string
        name: disease_ontology_term_id
        description: |
          Ontology term identifier for the disease, enabling standardized disease classification and referencing.

          Must be a term from the Mondo Disease Ontology (`MONDO:`) ontology term, or `PATO:0000461` from the Phenotype And Trait Ontology (`PATO:`).
        required: false

      - type: string
        name: donor_id
        description: Identifier for the donor from whom the cell sample is obtained.
        required: false

      - type: boolean
        name: is_primary_data
        description: Indicates whether the data is primary (directly obtained from experiments) or has been computationally derived from other primary data.
        required: false

      - type: string
        name: organism
        description: Organism from which the cell sample is obtained.
        required: false

      - type: string
        name: organism_ontology_term_id
        description: |
          Ontology term identifier for the organism, providing a standardized reference for the organism.

          Must be a term from the NCBI Taxonomy Ontology (`NCBITaxon:`) which is a child of `NCBITaxon:33208`.
        required: false

      - type: string
        name: self_reported_ethnicity
        description: Ethnicity of the donor as self-reported, relevant for studies considering genetic diversity and population-specific traits.
        required: false

      - type: string
        name: self_reported_ethnicity_ontology_term_id
        description: |
          Ontology term identifier for the self-reported ethnicity, providing a standardized reference for ethnic classifications.

          If the organism is human (`organism_ontology_term_id == 'NCBITaxon:9606'`), then the Human Ancestry Ontology (`HANCESTRO:`) is used.
        required: false

      - type: string
        name: sex
        description: Biological sex of the donor or source organism, crucial for studies involving sex-specific traits or conditions.
        required: false

      - type: string
        name: sex_ontology_term_id
        description: Ontology term identifier for the biological sex, ensuring standardized classification of sex. Only `PATO:0000383`, `PATO:0000384` and `PATO:0001340` are allowed.
        required: false

      - type: string
        name: suspension_type
        description: Type of suspension or medium in which the cells were stored or processed, important for understanding cell handling and conditions.
        required: false

      - type: string
        name: tissue
        description: Specific tissue from which the cells were derived, key for context and specificity in cell studies.
        required: false

      - type: string
        name: tissue_ontology_term_id
        description: |
          Ontology term identifier for the tissue, providing a standardized reference for the tissue type.

          For organoid or tissue samples, the Uber-anatomy ontology (`UBERON:`) is used. The term ids must be a child term of `UBERON:0001062` (anatomical entity).
          For cell cultures, the Cell Ontology (`CL:`) is used. The term ids cannot be `CL:0000255`, `CL:0000257` or `CL:0000548`.
        required: false

      - type: string
        name: tissue_general
        description: General category or classification of the tissue, useful for broader grouping and comparison of cell data.
        required: false

      - type: string
        name: tissue_general_ontology_term_id
        description: |
          Ontology term identifier for the general tissue category, aiding in standardizing and grouping tissue types.

          For organoid or tissue samples, the Uber-anatomy ontology (`UBERON:`) is used. The term ids must be a child term of `UBERON:0001062` (anatomical entity).
          For cell cultures, the Cell Ontology (`CL:`) is used. The term ids cannot be `CL:0000255`, `CL:0000257` or `CL:0000548`.
        required: false

      - type: string
        name: batch
        description: A batch identifier. This label is very context-dependent and may be a combination of the tissue, assay, donor, etc.
        required: false

      - type: integer
        name: soma_joinid
        description: If the dataset was retrieved from CELLxGENE census, this is a unique identifier for the cell.
        required: false
    var:
      - type: string
        name: feature_id
        description: Unique identifier for the feature, usually a ENSEMBL gene id.
        required: false

      - type: string
        name: feature_name
        description: A human-readable name for the feature, usually a gene symbol.
        required: false

      - type: integer
        name: soma_joinid
        description: If the dataset was retrieved from CELLxGENE census, this is a unique identifier for the feature.
        required: false
    uns:
      - type: string
        name: dataset_id
        description: A unique identifier for the dataset. This is different from the `obs.dataset_id` field, which is the identifier for the dataset from which the cell data is derived.
        required: true
      - name: dataset_name
        type: string
        description: A human-readable name for the dataset.
        required: true
      - type: string
        name: dataset_url
        description: Link to the original source of the dataset.
        required: false
      - name: dataset_reference
        type: string
        description: Bibtex reference of the paper in which the dataset was published.
        required: false
        multiple: true
      - name: dataset_summary
        type: string
        description: Short description of the dataset.
        required: true
      - name: dataset_description
        type: string
        description: Long description of the dataset.
        required: true
      - name: dataset_organism
        type: string
        description: The organism of the sample in the dataset.
        required: false
        multiple: true

→ This would require making a distinction between type and dtype in the file_*.yaml where there currently is none.
→ Would require reformatting the struct object returned by this component


feat: Support for Telegram

Is your feature request related to a problem? Please describe.
I don't use Slack or Gotify.
Describe the solution you'd like
I'd like to be able to specify an arbitrary set of Telegram bot API credentials, as well as an arbitrary chat ID. Messages should arrive at that chat.
Describe alternatives you've considered
I tried looking for a Tekegram plugin for Gotify, but was unsuccessful.

Perhaps consider using a library like https://github.com/nikoksr/notify ?

Might want to look at apprise.  Pretty large library of supported options.

apprise is a Python module, so can't use that sadly.
I don't really see the use of that notify library. You still have to implement the services yourself, you just then give them to that notify and send with that. So you'd still need to add support for every service yourself.
I'll work on adding Telegram support later on 👍

Shoutrrr is another alternative written in go.  It does not have as many services, but might also be worth a look.

Using a library like notify lets you avoid all the implementation details and use a simple interface in your own code. The alternative is learning the minutiae of every single service's request formats and the like.

Shoutrrr is another alternative written in go. It does not have as many services, but might also be worth a look.

This looks more preferable to notify (imo). Thinking I'll replace slack: and gotify: with notify: (that uses Shoutrrr not Notify). Then make that read yaml like this
service:
  release-argus/argus:
    ....
    notify:
      argus_notifications:
        type: slack
        token: xyz
        channel: something
        params:
          bot_name: argua
          color: #ff8000
          icon: github
          title: New release

It would be nice if we could specify multiple different notification targets, but that’s just a nice to have.

It would be nice if we could specify multiple different notification targets, but that’s just a nice to have.

What do you mean by 'different notification targets'? You can (and will still) be able to setup multiple of each type of notification to go to different places.
I have Shoutrrr mostly working, am just working on automated conversions to the new format for existing slacks (to slack/mattermost) and gotify. Am also trying to figure out how the Shoutrrr code works so that I can PR icon support for Mattermost

@djmattyg007 Look into mailrise.  It is an SMTP gateway that uses apprise under the covers.  Use Shoutrr to send email to mailrise and use mailrise to then forward.  It allows you to setup different email addresses.  Each can have one or more destinations.  I use this for all of my apps.  Each app has its own channel in Discord and Gotifty.  Some I send emails.  You then have one config for all of your apps.  If you change email addresses, you have one place to go to change all of your apps.  Or if you want to change from Slack to Discord, one place to go.  No longer have to remember every place you have to go make changes.  Also opens up a lot more options of places to send.

After too many commits, I'd say that this is about ready in https://github.com/release-argus/Argus/pull/63. Have implemented conversions of slack/gotify to not make this a breaking change, but I bet there will still be a config that breaks! :/. Gonna try some more testing of random configs tomorrow and ensure they all work.

Currently, switching to Shoutrrr is gonna lose us Gotify extras (unless me/someone else adds them to Shoutrrr), and until the next Shoutrrr release, we'll lose support for sending icons with our Mattermost webhooks

Right I've done a bunch more testing and finally merged the PR. Tried to get conversions to handle places where you may have had a gotify and a slack named the same, but in the end I gave up as I thought a few edge cases would fail that way and I just want to get this out. It'll detect where slack is for slack by 'hooks.slack.com' being in the url and assume it's for mattermost otherwise. Then it will suffix all existing slack/gotify names with the type it detects, so they'll either end in '_gotify', '_mattermost' or '_slack', but I believe they should all continue working.
I'd appreciate some of you testing this and reporting back to me (with the log) if ur config fails with this new image.
The Docker :master image should be out in the next 10 mins or so

@JosephKav So Gotify notification would now look something like this using the shourtrr format for url's?
notify:
  GOTIFY_NOTIFICATION:
    url: gotify://ip_address:port
    token: <gotify_token>
    title: Argus Notification
    message: '{{ service_id }} - {{ version }} released.'
    extras:
      client_display: text/plain
      client_notification: '{{ web_url }}'
    priority: 5
    delay: 0s
    max_tries: 3

And each service would look like:
    notify:
      GOTIFY_NOTIFICATION: {}

@JosephKav So Gotify notification would now look something like this using the shourtrr format for url's?

notify:
  GOTIFY_NOTIFICATION:
    type: gotify
    options:
      message: '{{ service_id }} - {{ version }} released.'
      delay: 0s
      max_tries: 3
    url_fields:
      host: ip_address
      port: 443
      token: <gotify_token>
    params:
      title: Argus Notification
      priority: 5

Forgot to link the updated docs, sorry! https://deploy-preview-13--release-argus.netlify.app/docs/config/notify/#gotify
(I'll merge them to release-argus.io once I do a release)
We've lost the extras ability with Gotify under Shoutrrr sadly

I updated config.yml and built and deployed a new local image.  No errors in the logs.  Will let you know what happens once an app is updated.

I updated config.yml and built and deployed a new local image. No errors in the logs. Will let you know what happens once an app is updated.

Ideally, you should have been able to use the existing config. Did you try that? You can test the notifiers with -test.notify NAME

I did not, but easy enough for me to roll back the config.yml.  Argus starts fine with no errors.  I see if added _gotify to my notification name.  When I test it, I do get an error.  It is trying to send an https request to an http server.

What url were you using?

I'm guessing (and hoping) one without http or https as I've made it default the port to 443 if it can't find those or a host:

just osu.gotify:8132.

Ah, so you'll need params.disabletls under this notify

@samcro1967 What are you using as host in that one? Odd that it's got two ports

<IP_ADDRESS>:8132  Looks like you are automatically wrapping https: and 443 around whatever is entered into the Host field.  Maybe the DisableTLS: yes is not working.  I would think that should at least change https to http.  You may want to add another url_field for port.    Would be cleaner I think to let folks know they have to specify both.  Then you will know the port for sure and are not assuming it is on either 80 or 443
???

<IP_ADDRESS>:8132 Looks like you are automatically wrapping https: and 443 around whatever is entered into the Host field. Maybe the DisableTLS: yes is not working. I would think that should at least change https to http. You may want to add another url_field for port.  Would be cleaner I think to let folks know they have to specify both. Then you will know the port for sure and are not assuming it is on either 80 or 443 ???

There is one for port, it just defaults to 443. Looks like its not getting the params correctly, may be only in the testing call. Trying to fix currently

Totally missed the port in the docs.  I updated and it is now trying to get to "https://osu.gotify:8132.

Totally missed the port in the docs. I updated and it is now trying to get to "https://osu.gotify:8132.

Yeah, I've pushed a fix for that. The master tag should be updated in <10 mins. The params from defaults/notify weren't being passed down to the service notify

FYI, you can track the build of :master (happens every commit to the master branch) here

Test notification worked with the latest version and the new syntax. Do you want me to roll the config back to the old syntax and test that as well?

That'd be nice, but only if you woulnd't mind and it's not too much hassle 😄

The old config gave me the https to an http message.
docker exec -it argus bash -c "cd /etc/argus && argus -test.notify GOTIFY_NOTIFICATION_gotify"
INFO: Testing (GOTIFY_NOTIFICATION_gotify),
ERROR: GOTIFY_NOTIFICATION_gotify, failed to send notification to Gotify: Post "https://osu.gotify:8132/message?token=AdMn4Ycrgo8SUKu": http: server gave HTTP response to HTTPS client
ERROR: GOTIFY_NOTIFICATION_gotify, failed to send notification to Gotify: Post "https://osu.gotify:8132/message?token=AdMn4Ycrgo8SUKu": http: server gave HTTP response to HTTPS client
ERROR: GOTIFY_NOTIFICATION_gotify, failed to send notification to Gotify: Post "https://osu.gotify:8132/message?token=AdMn4Ycrgo8SUKu": http: server gave HTTP response to HTTPS client
ERROR: GOTIFY_NOTIFICATION_gotify, failed 3 times to send a gotify message to gotify://osu.gotify:8132/AdMn4Ycrgo8SUKu
ERROR: Testing (GOTIFY_NOTIFICATION_gotify), Message failed to send with "GOTIFY_NOTIFICATION_gotify" config
failed to send notification to Gotify: Post "https://osu.gotify:8132/message?token=AdMn4Ycrgo8SUKu": http: server gave HTTP response to HTTPS client x 3

Here is the old config:
gotify:
  GOTIFY_NOTIFICATION:
    url: http://osu.gotify:8132
    token: <redacted>
    title: Argus Notification
    message: '{{ service_id }} - {{ version }} released.'
    extras:
      client_display: text/plain
      client_notification: '{{ web_url }}'
    priority: 5
    delay: 0s
    max_tries: 3

@samcro1967
		url = strings.TrimPrefix(url, "http://")
		if strings.HasPrefix(url, "http://") {
			converted.SetParam("disabletls", "yes")
		}

🤦
I strip the http:// before I check for it and disabletls!

Something else I should try?

Something else I should try?

Not that I can think of? I'm saying it's all working as far as I can see w.r.t Gotify and HTTP. My command log was just to show that it did pick it up as http and disabletls, so tried http://. It's meant to fail as I pointed it at a non-existant server as I disabled http on my one after the testing earlier worked

So with the old config I specified http://osu.gotify:8132 as the URL.  The notification was sent to https://osu.gotify:8132.  It doesn't look like the strip is working to me with the old config, but maybe I am misunderstanding what you are saying.

So with the old config I specified http://osu.gotify:8132 as the URL. The notification was sent to https://osu.gotify:8132. It doesn't look like the strip is working to me with the old config, but maybe I am misunderstanding what you are saying.

Did you make sure to repull :master? I just tried on docker with your config and it worked. Changed my Gotify back to HTTP to confirm
❯ sudo docker exec -it c9265400f10d bash -c "cd /etc/argus && argus -test.notify GOTIFY_NOTIFICATION_gotify"
INFO: Testing (GOTIFY_NOTIFICATION_gotify),
DEBUG: GOTIFY_NOTIFICATION_gotify, Sending "TEST - NAME_OF_SERVICE - MAJOR.MINOR.PATCH released." to "gotify://gotify.example.io:444/<SECRET>" with params=map["disabletls":"yes" "priority":"5" "title":"Argus Notification"]
INFO: Testing (GOTIFY_NOTIFICATION_gotify), Message sent successfully with "GOTIFY_NOTIFICATION_gotify" config

Just rebuilt the container and retestd.  It worked.  I wonder if I pulled it too soon after you made the last commit.  Not sure if Github uses a CDN and maybe I had not replicated yet from where I pulled it.  I will test discord tomorrow.  :-)

Tested Discord and works as expected.  Also tried email, but am getting an error which I am guessing it in my config, but I am not seeing it.  I see MAILRISE_NOTIFICATION when I do a -config.check.
config.xml global block
notify:
  MAILRISE_NOTIFICATION:
    type: smtp
    url_fields:
      Host: osu.mailrise
      Port: 8025
    params:
      FromName: Argus
      FromAddress<EMAIL_ADDRESS>      ToAddress<EMAIL_ADDRESS>      Auth: None
      Subject: '{{ service_id }} Release'
      UseHTML: yes
      UseStartTLS: no

logs
docker exec -it argus bash -c 'cd /etc/argus && argus -test.notify MAILRISE_NOTIFICATION'
INFO: Testing (MAILRISE_NOTIFICATION),
ERROR: Testing (MAILRISE_NOTIFICATION), notify:
  MAILRISE_NOTIFICATION:
    ^ unknown service ""

Tested Discord and works as expected. Also tried email, but am getting an error which I am guessing it in my config, but I am not seeing it. I see MAILRISE_NOTIFICATION when I do a -config.check.

Ah, that's an inconsistency between the code and my docs. The code is expecting it as "email". I'll do a commit now to fix that (code will read and use smtp)

I built a new image and am now seeing this when I perform a test:
docker exec -it argus bash -c 'cd /etc/argus && argus -test.notify MAILRISE_NOTIFICATION'
INFO: Testing (MAILRISE_NOTIFICATION),
ERROR: Testing (MAILRISE_NOTIFICATION), notify:
  MAILRISE_NOTIFICATION:
    ^ toaddress is not a valid config key [auth encryption from fromaddress fromname starttls subject title to toaddresses usehtml]

Ahh, Shoutrrr wants me to give ToAddresses not ToAddress! Oops again! Have amended the latest commit and fixed this. Thanks for testing!

Got beyond that error, but now getting this one:
ERROR: Testing (MAILRISE_NOTIFICATION), Message failed to send with "MAILRISE_NOTIFICATION" config
error applying params to send config: usestarttls is not a valid config key [auth encryption from fromaddress fromname starttls subject title to toaddresses usehtml] x 3

error applying params to send config: usestarttls is not a valid config key [auth encryption from fromaddress fromname starttls subject title to toaddresses usehtml] x 3

as you may have seen from the Shoutrrr PR i mentioned you in, this is an inconsistency with the Shoutrrr docs that I've proposed a fix for. I'll keep Argus to using usestarttls and just convert it to starttls internally for now. (will still save it as usestarttls)

I built a new image this morning and tested again.  I am seeing the toaddresses issue again after the usestarttls commit.
 docker exec -it argus bash -c 'cd /etc/argus && argus -test.notify MAILRISE_NOTIFICATION'
INFO: Testing (MAILRISE_NOTIFICATION),
ERROR: Testing (MAILRISE_NOTIFICATION), notify:
  MAILRISE_NOTIFICATION:
    url_fields:
      toaddresses: <required> e.g<EMAIL_ADDRESS>

You need to make it toaddresses not toaddress in your params. I'll fix the PR for the new docs with this

Yup, it is working now.  I can confirm Discord, Gotify, and email are all working with the implementation of Shoutrr.  Great work!


[CT-3138] [Bug] Error: Invalid value for '--warn-error-options'

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
When installing dbt-core into an environment where an older version of click is installed, dbt-core will throw a vague error Error: Invalid value for '--warn-error-options': Cannot load YAML from type <class 'dbt.helper_types.WarnErrorOptions'>. It seems dbt-core is pinning click<9 which allows it to use older versions of click that it seems no longer compatible with. I think dbt-core should be pinning a minimum version requirement of click that plays well with the library.
Expected Behavior
No error.
Steps To Reproduce
Install click==8.0.0
Install the latest dbt-core
Attempt to use the dbt cli w/ a command dbt ls or something
Relevant log output
Error: Invalid value for '--warn-error-options': Cannot load YAML from type <class 'dbt.helper_types.WarnErrorOptions'>

Environment
- OS:
- Python: 3.9.1
- dbt-core: 1.6.3

pip freeze output
agate==1.7.1
asn1crypto==1.5.1
attrs==23.1.0
Babel==2.12.1
certifi==2023.7.22
cffi==1.15.1
charset-normalizer==3.2.0
click==8.0.0
colorama==0.4.6
cryptography==41.0.4
dbt-core==1.6.3
dbt-extractor==0.4.1
dbt-semantic-interfaces==0.2.0
filelock==3.12.4
hologram==0.0.16
idna==3.4
importlib-metadata==6.8.0
isodate==0.6.1
jaraco.classes==3.3.0
Jinja2==3.1.2
jsonschema==3.2.0
keyring==24.2.0
leather==0.3.4
Logbook==1.5.3
MarkupSafe==2.1.3
mashumaro==3.8.1
minimal-snowplow-tracker==0.0.2
more-itertools==8.14.0
msgpack==1.0.5
networkx==3.1
oscrypto==1.3.0
packaging==23.1
parsedatetime==2.6
pathspec==0.11.2
platformdirs==3.8.1
protobuf==4.24.3
pycparser==2.21
pycryptodomex==3.19.0
pydantic==1.10.12
PyJWT==2.8.0
pyOpenSSL==23.2.0
pyrsistent==0.19.3
python-dateutil==2.8.2
python-slugify==8.0.1
pytimeparse==1.1.8
pytz==2023.3.post1
PyYAML==6.0.1
requests==2.31.0
six==1.16.0
snowflake-connector-python==3.2.0
sortedcontainers==2.4.0
sqlparse==0.4.4
text-unidecode==1.3
tomlkit==0.12.1
typing_extensions==4.8.0
urllib3==1.26.16
zipp==3.17.0

Which database adapter are you using with dbt?
snowflake
Additional Context
No response

Thanks for raising this @dylan-murray !
Do you happen to know the minimum version of click that works?

The minimum version it appears to work with is click==8.0.2.

@dylan-murray opened #8684 with you as a co-author.


Bumped production RELEASE for 1.21-5

Bumped production RELEASE for 1.21-5
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/unhold


All Dev Doc pages should be responsive

Describe the bug
The Dev Doc playground isn't fully responsive. At certain window sizes the drawers overlay the main content without the ability to hide them. Theres some content backgrounds that don't fill the entire background so the main background color appears where it shouldn't and some pages scroll horizontally. There's a spot where the Menu icon appears despite the drawer being open. These are just some of my findings. Please see the screenshots below and be sure to test the site on various devices and screen widths to ensure all things are reasonably responsive.
What is the expected behavior?
Dev Doc playground should be responsive
Screenshots or links to minimum reproduction example

Environment

Anything else to add?

I made a story (BLUI-3509) to come up with the design first

This has been migrated to jira. Closing this bug as it will be a larger initiative and can be tracked in jira instead.


make functions frameless if they don't need a frame

If a function is a leaf and does not contain any spills it does not need a frame.
POC https://github.com/mraleph/sdk/commit/a70a694193917f8f43b47ebd47102346daad5736 reveals that a lot of functions in Flutter Gallery fit this definition - and dropping frame setup leads to a significant code size improvement (and potentially performance improvement as well).
Store barrier introduces a slight complication into this on ARM/ARM64 because of its outlined slow-path. This slow-path can only call leaf-runtime entry, but nevertheless calling the slow-path clobbers LR. We already support generating store barriers in intrinsics - which are also frameless and thus don't have LR saved - however this introduces explicit pushing and popping of LR which needs to be taken into account when deciding if it is worth to setup a frame (e.g. POC allows at most 1 instruction with a store barrier per frameless function). One dodgy part here is the lack of static guarantee: location summaries capture which instructions perform calls and which don't, with the exception of store barrier call. I suggest that turning POC into a real implementation requires at least adding an assertion that checks that branch-link instruction is only allows to be emitted when IL instruction is marked as clobbering LR.
part of go/dart-llvm-assessment
/cc @mkustermann for triage and assignment

While discussing this with @mkustermann I have done a bit of digging into why this had such a drastic effect on Flutter Gallery size. Turns out that it disproportionally hits localisation related getters which are generated in large numbers on Flutter Gallery because it supports a lot of different locales. We have 83906 frameless functions and 21303 framefull functions - however out of 83906 frameless functions 80938 originate from localization related generated code.
/cc @alexmarkov

We already had a bug for this in https://github.com/dart-lang/sdk/issues/40195 . Due to more context here I'll close the other issue.
@sstrickl is going to work on this soon.

@mraleph Since you made the PoC, maybe you can upstream them as well?


Hitting uncaught com.google.gson.stream.MalformedJsonException when trying to connect.

I'm trying to get librespot-java up and running on a fresh Raspberry Pi 4B. I'm using librespot-player-1.6.3.jar with the default config.
When running I am able to discover the Spotify Connect device from Spotify, but hitting a malformed JSON exception
2023-10-04 23:37:05,273 ERROR Log4JUncaughtExceptionHandler:31 - [zeroconf-client-241783884]
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 10 path $
        at com.google.gson.JsonParser.parseReader(JsonParser.java:76) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.core.ApResolver.request(ApResolver.java:88) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.core.ApResolver.fillPool(ApResolver.java:56) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.core.ApResolver.<init>(ApResolver.java:52) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.core.Session.<init>(Session.java:136) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.core.Session.<init>(Session.java:77) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.core.Session$Builder.create(Session.java:1055) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.ZeroconfServer.handleAddUser(ZeroconfServer.java:367) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.ZeroconfServer.access$200(ZeroconfServer.java:56) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.ZeroconfServer$HttpRunner.handleRequest(ZeroconfServer.java:499) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.ZeroconfServer$HttpRunner.handle(ZeroconfServer.java:574) ~[librespot-player-1.6.3.jar:1.6.3]
        at xyz.gianlu.librespot.ZeroconfServer$HttpRunner.lambda$run$1(ZeroconfServer.java:482) ~[librespot-player-1.6.3.jar:1.6.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 10 path $
        at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1659) ~[librespot-player-1.6.3.jar:1.6.3]
        at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1465) ~[librespot-player-1.6.3.jar:1.6.3]
        at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:551) ~[librespot-player-1.6.3.jar:1.6.3]
        at com.google.gson.stream.JsonReader.peek(JsonReader.java:433) ~[librespot-player-1.6.3.jar:1.6.3]
        at com.google.gson.JsonParser.parseReader(JsonParser.java:71) ~[librespot-player-1.6.3.jar:1.6.3]
        ... 14 more

This occurs regardless of the device I'm using to connect.

Alright, so I'm almost 100% sure this is a result of my pihole blocking the Spotify API URL http://apresolve.spotify.com/?type=accesspoint&type=dealer&type=spclient.
When accessing this URL from within my network I am getting "Invalid IP address provided", however when out of my network (such as 5g on my phone) I get back the right JSON.
Going to see if I can unblock this and see.

Well, I'm not sure what's happening. For some reason my network will not allow http:// and instead will only allow https:// for this apresolve URL. When I use http:// I get a response body of "Invalid IP address provided".
I am able to access plenty of http (not https) sites normally on my network, so it's not like port 80 or some other security measure is in place. And in fact, if I remove DNS entirely and use: http://<IP_ADDRESS>/?type=accesspoint&type=dealer&type=spclient I still receive this Invalid IP body. So, it's not DNS.
However, over my phone off network (5g) I'm able to get to the unsecured page.
Will keep digging. That said, when I changed the BASE_URL within ApResolver.java to use https rather than http then librespot-java is working fine. So good news is I'm unblocked, but news is I still don't know why http would not work.

Marking this CLOSED.
Turns out the issue with http/https was a webproxy setting on my network's router from long ago. This altered the HTTP header just slightly to cause Spotify to reject the request. Everything works out of the box now. Thanks!


Finish the interests in the relay server

Mostly done, needs some refactoring, but we have open issues for that.