Suggestion: store videos in the database

(I'm not sure if there's a better place for suggestions)
Has there been thought of storing the video data in the database, utilizing a Django model with title, id, url, date published, and category fields? This would just need a (weekly? monthly?) task in Heroku Scheduler to check for and add new videos. The only manual update would be categorizing each video (e.g. Meetup, Workshop, etc.).
Depending on the impact to the database, the video description could also be added as a TextField to allow Postgres full text search. There's also the possibility of tags using ManytoManyField and a Tags model. This would be more manual work for new videos, though.
Obviously, this is overkill right now. The thought behind this suggestion is that it will allow for future implementation of a video page with sort/filter/search functionality along with easy pagination using Django's Paginator.

This is a fine place for suggestions.
I think this is a pretty good idea if we can do it without adding a bunch of extra infrastructure to have a task queue. Basically we just need to parse the YouTube feed every few minutes and compare it to the database. Loading the video from the DB instead of the feed directly would definitely have some nice performance gains and avoid awkward caching. The same could probably apply to the meetup.com feed.

This would just need a (weekly? monthly?) task in Heroku Scheduler to check for and add new videos.

Just a note that the site is deployed to fly.io now and not Heroku. It used to be on Heroku but we moved it. Are there some docs that still say Heroku? If so, we should update it.

I'm not sure if the XML has a way to loop through all videos or if it's only the first page. In a past project, I used the YouTube API to add thousands of videos from a channel to a database. Here's a modified version of what that looks like: youtube_api_check.txt
(apparently .py files can't be attached here).
The downside to this is that the search endpoint does not include the video's description. So there would need to be a second call using the videos endpoint. I added this in the .txt file, but it's untested.

Basically we just need to parse the YouTube feed every few minutes and compare it to the database.

I'm not familiar with fly.io but I'm assuming they have something similar to PythonAnywhere's scheduled tasks (or maybe a cron job?). For my previous project, I had the task run every hour to check for new videos (the API's quota didn't allow for it to be more often). If this is done with XML, every 5 minutes would be fine. The loop could just be broken as soon as the first .exists() returns True (for the very unlikely occurrence of more than one new video between checks).
There's a few mentions of Heroku, I'll submit a PR for those.

I think the XML feed is for just recent videos. It isn't a full API. We'd have to use the YouTube API for that as you suggested.
However, I'm not sure we need it unless we're planning to build some alternative way to search our videos/streams on YouTube or we want to display all our videos for some reason. I'm not sure we need that. Really we just need to display recent videos which is exactly what the RSS feed does. I think we need to think about the experience we're trying to build.
I haven't done a lot with Fly.io so I'm not sure how you do scheduled tasks. We could of course run something like celerybeat, but that will probably take us above the free tier.

That's fair. My thinking was more along the lines of "just in case we need the data" which is a dangerous game to play. Long term, I was thinking of a list of all videos with pagination to keep interested users on the site instead of navigating away (to YouTube) but it really doesn't make sense to try to build a search/filter system just for that reason when YouTube will do it better.
If there's no way to automate the task without increasing the tier then it's not really worth it. The performance difference wouldn't be noticeable to most users. As long as the need for videos stays at 15 or less recent videos then the current setup is good.


Changeing Text of Circle produces alot of errors

Hey, so i was testing some stuff with your plugin(which is awsome) a strange Problem occured.
What i was trying to do: I let a circle progressbar finish to 100% and then use the callback function to start over again BUT with a different text inside the circle here is the FIDDLE. I tried changeing the text with setText() which didnt work then i tried to Change the value which strangely worked but will Chose alot of Errors if you have the Browser console open. Any idea what i am doing wrong?

I was having problems with setText(). Changing to the non-minified version fixed this https://github.com/kimmobrunfeldt/progressbar.js/blob/master/dist/progressbar.js

Thank you for the JSFiddle. Unfortunately I don't have time to investigate this :/

Not sure if it is related but I am also seeing errors when in MS Edge and I open the console I start seeing a bunch of errors SCRIPT5: Access is denied. Nothing seems to be impacted other than thousands of console errors counting up.

Closing due to inactivity, if anyone finds a fix – PR is welcome


Inexistent address after creating a nami wallet

Yesterday, June 14, I created my first Nami wallet, after which I tried to look up my address in Cardano Explorer but it comes up as non-existent. What problem could it be? Can resetting the wallet be a solution?

@rafmc98 Talk to Cardano Support on the Live support page so that your issue can be looked into.

Talk to Cardano Support on the Live support page so that your issue can be looked into.


Fix as -> AS syntax

What type of PR is this?
/kind cleanup
What this PR does / why we need it:
The CI complains that the syntax is not matching, so we change the case.
Which issue(s) this PR fixes:

None
Does this PR have test?
None

Special notes for your reviewer:
None
Does this PR introduce a user-facing change?

None

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 42.08%. Comparing base (11d77f4) to head (d47463b).
Report is 393 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2417      +/-   ##
==========================================
- Coverage   45.50%   42.08%   -3.42%
==========================================
  Files          79      109      +30
  Lines        7782    15916    +8134
==========================================
+ Hits         3541     6698    +3157
- Misses       4099     8722    +4623
- Partials      142      496     +354


scripts/netContain/Dockerfile: update OVS version

the .1 release was removed from the mirrors:  https://launchpad.net/ubuntu/+source/openvswitch
.2 is the latest release for Ubuntu 16.04 (Xenial Xerus)
Signed-off-by: Bill Robinson<EMAIL_ADDRESS>

Merging since this is blocking a release and we've proven this OVS version works elsewhere


Allows NAT role customization.

@sesas @gimballock This allows the client to subclass the HaNat class and customize its role permissions, which is needed when you take advantage of its capability for extra initialization from #11.

LGTM.  :+1:


[test:unit] create coverage badge for project README.md

Is your feature request related to a problem? Please describe.
Currently test coverage results are only visible in the CI logs.
Describe the solution you'd like. Include clear acceptance criteria.
It would be convenient to have a coverage badge on the README.
A couple of possible solutions without using 3rd party SaaS coverage tools:

https://www.npmjs.com/package/jest-coverage-badges
https://github.com/marketplace/actions/dynamic-badges

@ivelin created a short PR for this. In order to fully set this up, there's some setup steps required to be done by a maintainer

Sorry @t3chnicallyinclined for our duplicated effort on this one. Just pushed the PR and didn't assign the ticket to myself before. Good to see that we got to the same solution though. Happy to solve this together if it still makes sense :)

@t3chnicallyinclined , @midnight-madman thank you for the PR. I will take a look today and comment.

@midnight-madman @ivelin sorry I don't believe mine is working yet (i'm a noob) as I realized there some steps needed to be done from a maintainer and not sure how that process goes. @midnight-madman you can happily take this one and I will surely take a look when it's complete, thank you!! will search for something a little more noob friendly!

completed


add ASP commit list

add 2.1.1 changes mined from shiproom notes.

@leecow The sockets issue has nothing to do with the EF bugs.

Thanks Arthur, removed from the EF section.


Range parsing precedence

Currently, ranges support:
0..40
(0)..40
11..
..9
(-5..)
-10..
@2020-01-01..
@2020-01-01..@2021-01-01

There a few changes I'd propose to change.
Only literals
With current grammar, .. has a rule to only bind with literals:
take 1..@2023-02-14  # range from 1 to @2023-02-14
take 1..[]           # two arguments: range from 1 onward and an empty list
take 1..hello        # two arguments: range from 1 onward and an ident

Because literals are allowed to be enclosed in parenthesis, ranges are support nested expressions, but only if they contain literals only.
Current parsing grammar thus contains two types of parenthesis: one for normal nesting and pipeline and the other for the ranges.
This is a weird rule and I propose we eliminate it, by using normal parenthesis for ranges too.
(1)..40     # ok
(-1)..40    # ok
(0-1)..40   # not ok currenty, I propose we allow this
(5 | abs | hello [false])..40   # not ok currenty, I propose we allow this

There is not a strong case that this would greatly improve some use case, I want to change it in name of consistency.
Whitespace
In my recent push to remove effects of whitespace from PRQL, ranges proved to be the only breaking point.
take 1+1    # a single bin op
take 1+ 1   # a single bin op
take 1..1   # a single range
take 1.. 1  # two arguments: a range and an int

Whitespace is not required to bind the literal into range, but if the term is not a literal (i.e. a list) it will not be bound into range even if there is no whitespace.
I propose we remove this distinction and bind ranges the same way we bind all other binary operations.
Result
take (10 - 5 + 2 * 2)..1
take [1, 10]..[5, 15]
take 1 .. 4
take 1 ..
take ( .. 4)  # parenthesis needed

#1818 implements proposed behavior

Agree with "Only literals" point, a nice consistency win.
For "Whitespace":

I do think the ergonomics of take (..1) are worse than take ..1. I'm not sure how I'd explain it without giving the whole explanation about parsing whitespace, which suggests we're too anchored in the parser relative to the language.
But it's also not the biggest deal, and I don't want to slow you down.
Do you have a view for why Rust allows for the ..1? (I'm trying to think of other languages that I've thought of as whitespace-agnostic, this is the only case I can think of...).
Is it even possible in Chumsky to generally ignore whitespace but not in a specific place? I can look later this week; I wonder whether it's possible to define the range as a unit that can't have whitespace in the same way that -> or >= can't have whitespace (I'm presuming those can't have whitespace and we're only discussing the whitespace between parsing items?)

Other places where I had to care about whitespace when writing the pest parser:

s"foo" as distinct from a function call on s — s "foo"
1.5 as distinct from two numbers, the first without any digits after the period 1. 5
A datetime like @2021-12-31T12:00 as opposed to a date and named parameter T12 with an arg 00: @2021-12-31 T12:00
Type definitions like col<int> as distinct from the comparison col < int?

Maybe I try and add some tests for these before we merge that branch...

Is it even possible in Chumsky to generally ignore whitespace but not in a specific place? Edit: i guess this is the lexer vs parser?

Yes, this is lexer vs parser. Lexer handles all whitespace related things, and so s-strings, numbers and dates can be parsed correctly.
I used to lex whitespace into Token::Whitespace that was then consumed by parser, but it was causing many problems and made parser definitions much more verbose.
So basically, information about the whitespace is not passed from lexer to parser. I just realized that I could convert .. from being passed as other control token, to a special token that would know if it can bind to left and to right, depending on the whitespace:
Token::range { open_left: bool, open_right: bool }

Type definitions are gonna be tricky to implement too.

I managed to get the range parsing working with approach described above.
Which means that we don't have to change the behavior now, and sacrifice a little consistently for the sake of ergonomics.

As discussed, this is not a requirement for parser implementation anymore.
If anyone has strong option for improving language consistency, feel free to add to the discussion.


adjustment index page style

Coverage remained the same at 48.254% when pulling 6b0238006b99b547d86055c57d016415a6639848 on lepdou:index_page_refactor into 71da31c3b316675c9d14d4dde42bb7e6bf362b22 on ctripcorp:master.


Support the property disabled.

Introduction
This PR is aimed at fixing the issue #41.
Dear author:
I am glad to tell you that I have finished the disabled property. By the way, I have also fixed a bug of disableDragAndDrop property. In previous versions, this property would only work during the initialization, and would not work if changed by callbacks.
If you have reviewed my codes and find they are OK, I would start to modify the docs and CHANGELOG.
Thank you!
cainmagi
Example
The following codes could be used for testing the disabled property.
import uuid

import dash_uploader as du
import dash
import dash_html_components as html
import dash_core_components as dcc
from dash.dependencies import Output, Input

app = dash.Dash(__name__)

UPLOAD_FOLDER_ROOT = r"C:\tmp\Uploads"
du.configure_upload(app, UPLOAD_FOLDER_ROOT)

def get_upload_component(id):
    return du.Upload(
        id=id,
        text="Drag and Drop files here",
        text_completed="Completed: ",
        cancel_button=True,
        max_file_size=1800,  # 1800 Mb
        filetypes=["csv", "zip"],
        upload_id=uuid.uuid1(),  # Unique session id
        max_files=2,
    )

def get_app_layout():

    return html.Div(
        [
            html.H1("Demo"),
            html.Div(
                [
                    dcc.Checklist(
                        id="uploader-configs",
                        options=[
                            {"label": "Disabled", "value": 0}
                        ],
                        value=[]
                    ),
                    get_upload_component(id="dash-uploader"),
                    html.Div(id="callback-output"),
                ],
                style={  # wrapper div style
                    "textAlign": "center",
                    "width": "600px",
                    "padding": "10px",
                    "display": "inline-block",
                },
            ),
        ],
        style={
            "textAlign": "center",
        },
    )

# get_app_layout is a function
# This way we can use unique session id's as upload_id's
app.layout = get_app_layout

# 3) Create a callback
@du.callback(
    output=Output("callback-output", "children"),
    id="dash-uploader",
)
def get_a_list(filenames):
    return html.Ul([html.Li(filenames)])

@app.callback(
    Output("dash-uploader", "disabled"),
    [Input("uploader-configs", "value")]
)
def check_disabled(val_configs):
    if 0 in val_configs:  # Disabled
        return True
    else:
        return False

if __name__ == "__main__":
    app.run_server(debug=True)

What I have tested:

When checking Disabled option, the uploader would become semi-transparent. The click and the "drag & drop" would be disabled. The text message would change as The uploader is disabled.
When unchecking Disabled option, everything would work as before.
First, upload a file. After getting the complete message, check Disabled, the message would be changed to Disabled. If uncheck Disabled, it would turn back to the complete message.
During the upload of a large file, check Disabled, everything would not be changed. The disable only take effects when the file is not uploaded yet or has been uploaded.

The explanation is:

The preference order of the message is: uploading message > disabled message > complete message > default message.
The preference order of the style is: uploading style > disabled style > complete style > default style.
The preference order of the class is: uploading class > paused class > disabled class > hover class > complete class > default class.

Update report

Add properties disabled, disabledMessage, disabledStyle.
Replace the unexposed property disabledInput by disabled.
Fix the responsive bug of drag and drop region when the property disableDragAndDrop or disabled changes.
Add two arguments text_disabled and disabled to du.Upload.

Hi @cainmagi ! Glad to see this implemented. Looks neat! I added basic testing functionality for the dev branch. Could you add a test for the disabled property to this PR? It would not need to test all the possible combinations, but some of the basic functionality you were testing by hand. If you are not familiar with Selenium, I can try to help you. Or, you can just try to copy the testing functionality from the tests/test_usage.py.

Thank you! I have already run your tests. Since I have no much time for web development after each Wednesday, I may have to give you some feedbacks later (as fast as Saturday?)

Yeah sure take your time :) I am not in a hurry with this at all!

I have already finished the pytest scripts. Please check the updating report for finding more details.

I think this looks good! Could you add CHANGELOG and docs changes for the disabled property?

Have updated docs and CHANGELOG.

Thank you! I will try to start the new PR about the multi-services the next weekend.


popup disappear right after it shows up when it is being called in a UIAlerView delegate method clickedButtonAtIndex

I have a button in a uitableviewcell which will show an alertview and when pressed on OK button it will call its delegate method to show the popup
The popup shows up successfully but disappear right after it shows up.

Hi daarkfish. I've managed to replicate this bug and am working on a fix right now.

Hey daarkfish, after messing with it a little bit I've come to the conclusion that UIAlertView's take time to dismiss the actual view and along with that I believe they destroy the UIWindow as well in alertView:clickedButtonAtIndex:
I've come up with a workaround :

Basically the smallest amount of time it takes for a UIAlertView to dismiss is 0.40829 seconds. So after that time period you want to show Popup.


cmd/go/internal/lockedfile: use F_OFD_SETLK on Illumos

See the TODO here:
https://github.com/golang/go/blob/f686a2890b34996455c7d7aba9a0efba74b613f5/src/cmd/go/internal/lockedfile/internal/filelock/filelock_fcntl.go#L16-L19
The illumos tag was added in https://golang.org/cl/174457 (for #20603), so the TODO can now be done. It should be pretty straightforward.
CC @jclulow @jayconrod @bradfitz

Can we add an OS-illumos label for issues like this one?

@jclulow, done.

Fixed in CL 255258.


Can the documentation be updated to address Alphanumeric outbound caller ID

Outbound caller ID being labeled with Alphanumeric for the tenant.  Either if it's possible, or not, it's not mentioned either way.

@paulypoteat Thanks for submitting feedback on the documentation. Could you please link the issue you are referring to? Thanks!

https://docs.microsoft.com/en-us/microsoftteams/set-the-caller-id-for-a-user
I was hoping the article could mention alphanumeric Outbound caller ID (not just numeric outbound caller ID).  Either that it's possible, or even if it's not.

@paulypoteat thank you for reaching out and helping improve MS Docs. I've done some research and found the following:  From a web browser on another device or computer go to https://microsoft.com/devicelogin and then enter the alphanumeric code displayed in the Enter Code field and then click Next. (the code is not case-sensitive.)  If the Pick an account window appears and the desired account appears in the list then select that account.  If not, then select the Use another account option. If prompted to sign in then enter the username for the desired Teams user account and select Next.  Then enter the password and select Sign In.

@paulypoteat  Thank you for submitting feedback. We understand that this issue has been resolved.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.


Enhancement/op-4521_color_picker_alpha

Brief description
Add alpha support to the color picker input.
Description
Created a new component with alpha support for the color picker by extending the default input and including 4 new number inputs for "RGBA". The component receives an array of floats for each channel (normally 0-1) and returns the same when onChange is fired. The native input color picker remains the same but two new functions have been created hexToFloat and floatToHex to handle the native input only supporting hex strings.

Questions

The original "no alpha" color picker input only supports hex strings and isn't inline with InputColorAlpha.
It might be worth merging these two components together and either letting the user or developer choose if alpha channels are needed.
The same with showing the float values, should the 4 number inputs always be visible?
At the moment you can't paste colors into the input (in any format).
We may need an dropdown menu that lets you switch between formats.

Testing

Use the Demo Form.

I think in the future we could have a single configurable component - keeping it inline with other inputs will be a little bit tricky as we'd need to maintain the height of all sub-components. but at this moment, this should be enough as a proof of RGBA works in the settings.


Modeling Exercise: Visual enhancements

⚠ Should be merged after #170

This PR includes some visual improvements for modeling exercises:

Abstract class names are now italic
Elements with feedback suggestions now only have highlighted titles instead of a highlight covering them completely
Relationship highlights now perfectly match the relationship path

Before
After

@AlexanderG2207, thanks for the review! 🙏
The builds are failing because the account holder needs to accept the new Apple developer license agreement and he/she is currently on vacation 🙂


Document best practices around invoking a wasi module multiple times

Is your feature request related to a problem? Please describe.
Not a problem per se, just questions around writing production-grade code with Wazero.
Context:
I have a simple Rust program that looks like this:
use prql_compiler::compile;
use std::io::{self, Read};

fn main() {
    // example input is "from employees | select [name,age]  ";

    let mut prql = String::new();
    let stdin = io::stdin();
    let mut handle = stdin.lock();
    handle.read_to_string(&mut prql).expect("failed to read input"); // read stdin input
    let sql = compile(&prql).expect("failed to compile"); // transform the input into something else
    println!("{}", sql);
}

It's been compiled with cargo wasi build --release and produced a binary.
Validating it works as expected with wasmtime:
cat input.prql | wasmtime run target/wasm32-wasi/release/prql-wasi.wasm
SELECT
  name,
  age
FROM
  employees

I would like to provide a Go library that embeds this wasm binary.
Here's what it currently looks like, with non critical sections elided:
//go:embed prql-wasi.wasm
var prqlWasm []byte

// Engine holds a reference to the wazero runtime as well as the pre-compiled wasm module
type Engine struct {
	code wazero.CompiledModule
	r    wazero.Runtime
}

func (e *Engine) Compile(ctx context.Context, inBuf io.Reader, name string) (string, error) {
	outBuf := new(bytes.Buffer)
	errBuf := new(bytes.Buffer)
	config := wazero.NewModuleConfig().
		WithStdout(outBuf).WithStderr(errBuf).WithStdin(inBuf)

	mod, err := e.r.InstantiateModule(ctx, e.code, config.WithName(name))
	if err != nil {
		if exitErr, ok := err.(*sys.ExitError); ok && exitErr.ExitCode() != 0 {
			return "", err
		} else if !ok {
			return "", &ParseError{Input: "who knows", Err: err}
		}
	}
	mod.Close(ctx)
	errStream := errBuf.String()
	if errStream != "" {
		return "", fmt.Errorf(errStream)
	}
	return outBuf.String(), nil
}

func (e *Engine) Close(ctx context.Context) {
	e.r.Close(ctx)
}

func New(ctx context.Context) *Engine {

	r := wazero.NewRuntimeWithConfig(ctx, wazero.NewRuntimeConfig())
	wasi_snapshot_preview1.MustInstantiate(ctx, r)
	code, err := r.CompileModule(ctx, prqlWasm)
	if err != nil {
		log.Panicln(err)
	}

	return &Engine{
		r:    r,
		code: code,
	}
}

which would be used like this:
func main() {
	ctx := context.Background()
	p := prql.New(ctx)
	defer p.Close(ctx)

	content, _ := io.ReadAll(os.Stdin)
	s := string(content)
	inBuf := strings.NewReader(s)
       // simulate executing a transformation multiple times
	for i := 0; i < 500; i++ {
               name := fmt.Sprintf("run-%d", i)
		_, err := p.Compile(ctx, inBuf,  name)
                inBuf.Seek(0, 0)
                // handle err…
       }

Describe the solution you'd like
It would be greatly beneficial to have documented best practices around how to optimize running a wasi  module multiple times.
Questions that could benefit from being documented
What are the best practices around invoking a wasi module multiple times?
What are the performance implications of using wasi vs a standard wasm module?
Is it preferable to close the module each time:
mod, _ := e.r.InstantiateModule(ctx, e.code, config)
mod.Close()

or invoking it with a different name without closing it:
_, err := e.r.InstantiateModule(ctx, e.code, config.WithName(name))
// is mod.Close() necessary if we use a different name for each invocation?

Additional context
Wazero has been extremely fun to build prototypes with, and I'm curious about how to start working on production-grade code. A canonical production ready example would be a great starting point for many new projects.

I'll give a quick answer as we've had some chats about this on #wazero-dev gophers slack.
The TL;DR; is that main is not something to re-execute. In other words, wasi guests are not safe to re-execute "_start", which is done on instantiate. Same thing applies to GOOS=js and its main. Re-instantiation is the way out, and you can control the name, or use namespace to avoid conflicts when run in parallel.
In any case,  always close modules when done with them, unless you are only initing one per runtime and the runtime closes it. So, when any module is no longer needed, either due to its work being complete via init, or you no longer need its exports, close it. This advice applies to all guests, not just WASI.
If you didn't ask about wasi, and were asking about things that could be re-used (exported functions), I would suggest pooling as is done here. https://github.com/http-wasm/http-wasm-host-go/blob/main/handler/middleware.go#L151
-- more notes on wasi
We've tried it and the compilers don't expect that to run again. Many times, they will do things like "unreachable", and the behaviors of trying to execute "_start" again aren't something we can affect (as it is in how wasi-libc is used by compilers often enough.
help wanted to raise a PR in a place that makes sense weaving in this advice.

Thanks for the quick response @codefromthecrypt

help wanted to raise a PR in a place that makes sense weaving in this advice.

I'll give this a try once I figure out where it belongs.

I have create a golang wasm cgi server based on wazero, but performance is too bad, only have 100 qps
how to improve reuse wasi module performance?
Thank you for any reply!
goos: linux
goarch: amd64
pkg: github.com/shynome/go-wagi/z
cpu: AMD Ryzen 7 5700G with Radeon Graphics
BenchmarkWASI-16               1<PHONE_NUMBER> ns/op
PASS
ok      github.com/shynome/go-wagi/z    1.043s

z.go
package main

import "log"

func init() {
	log.Println("init")
}

func main() {
	log.Println("main")
}

z_test.go
package main

import (
	"context"
	_ "embed"
	"os"
	"os/exec"
	"testing"

	"github.com/shynome/err0/try"
	"github.com/tetratelabs/wazero"
	"github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1"
)

var z []byte

func TestMain(m *testing.M) {
	cmd := exec.Command("go", "build", "-o", "z.wasm", ".")
	env := os.Environ()
	env = append(env, "GOOS=wasip1", "GOARCH=wasm")
	cmd.Env = append(cmd.Env, env...)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	try.To(cmd.Run())

	z = try.To1(os.ReadFile("z.wasm"))

	m.Run()
}

func BenchmarkWASI(b *testing.B) {

	ctx := context.Background()
	rtc := wazero.NewRuntimeConfigCompiler()
	rt := wazero.NewRuntimeWithConfig(ctx, rtc)
	wasi_snapshot_preview1.MustInstantiate(ctx, rt)

	cm := try.To1(rt.CompileModule(ctx, z))

	mc := wazero.NewModuleConfig().
		WithSysNanotime().
		// WithStderr(os.Stderr).
		// WithStdin(os.Stdin).
		// WithStdout(os.Stdout).
		WithName("")

	for range b.N {
		mod := try.To1(rt.InstantiateModule(ctx, cm, mc))
		mod.Close(ctx)
	}
}

In your benchmark, you're not calling b.ResetTimer() after CompileModule, which means the<PHONE_NUMBER>ns (1 second) include compiling the module.
Back to your server, if you don't compile the module on your hot path (and you probably don't, if you get 100 qps), there's not much more to do.

Thanks for your tips. I have update benchmark, the new performance is 137 qps, it is too slow.
how do improve reuse wasi module performance?

Yes, and I don't think Go likes to have _start called again.
You should (maybe) consider TinyGo for the guest, and export functions, so you only instantiate once, and then call functions from an already instantiated module.


python3.6 Cython nametuple TypeError

I have a class that inherits from NamedTuple,I have compile it to pyd file on windows use cython,when I import the class  and create a object in another py file,It throws error TypeError: new() takes 1 positional argument but 4 were given, the class as follows:
from typing import NamedTuple

class Info(NamedTuple):
    service: str
    damage: str
    privilege: str

info = Info("http", "no", "no")

when I don't compile it to pyd,It can use normally,How to deal with it?

does any one has alternative plan? this usage is heavily used in the project.，and I don't know how to

This is a duplicate of #2552.

What you can do is to simply not use namedtuple at all. Run this code in regular Python, then print out the Info._source attribute. This is a copy of the equivalent code to define the class - just cythonise that. Get rid of the builtins imports though, Cython can do those better.


Fix notificationonclick handler

The test client.url == '/' doesn't work in every instance. I'll be happy to put something else if anyone has an idea.

I think the idea with that check is to highlight that when the notification is clicked, you need to only focus on a page that your notification would normally click to.
For example google.com could have a service worker for everything and send a notification for a new article and a notification for a G+ post. Without a check for the URL, when a notification is clicked for the article or G+ post, the click handler will find a client (http://google.com) and then just focus on that, even though the click should probably go to http://plus.google.com or http://google.com/news.
I'm happy to either leave this in as good practice or remove this all together and leave it opening a new tab everytime.

I did some more experimenting and found a few other things about this focus issue. I'm closing this for now and will send an email around when I have some time.


I want to get the page source code

I want to get the html source code of the current page

function key + f12 should open Edge Devtools if you have it installed, if you don't it would prompt you to install it.


[Bug] AvatarView: Label is visible if Source is not null

Steps to Reproduce

Create avatar view with transparent image

Expected Behavior
Label is not visible
Actual Behavior
Label is visible
Reproduction imagery

I am going to fix it


no github page

please add github page

please add github page

It's a potential feature I could realise.

It's not really a feature but more required as of now since you have no build instructions, no list of dependencies or any way of running it(without spending too much time figuring out the dependencies etc...)

Completed?? I don't see any GitHub page?


Specify main as the release branch

Semantic release expects a "master" release branch. Since we use main, we need to specify it as a release branch.
Source

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Layout changes, Storyboard support

Hello,
Have added image masking, layout manager and Storyboard support

Can't set Lookup Tags again #25 fixed

Regards,
Anton

Tags text/background changes


beacon vendor download column

There is a column when I download the vendor report that says "opportunities".  Some vendors have some solicitation names in there and others don't.  I'm curious what this column pulls.

Vendors can directly follow individual opportunities, so these are the names of opportunities that the particular vendor follows.


Release 0.1.2

Release 0.1.2 with the latest changes

https://github.com/nginxinc/telemetry-exporter/releases/tag/v0.1.2


backport: oasis-core bump

this is a backwards compatible change, so let's open it as a backport/stable change

need to reopen with different branch name


Add support to calculate npm tree only by package-lock.json file

[ ] All tests passed. If this feature is not already covered by the tests, I added new tests.
[ ] All static analysis checks passed.
[ ] This pull request is on the dev branch.
[ ] I used gofmt for formatting the code before submitting the pull request.

This PR introduce more control on the possible ways npm tree is calculated:

create the tree (which is the "ls" command) only b the package-lock file, with no mater if noe modules exists.
if package-lock exists, check if the package.json is more updated than the lock, if so, then recalculate th package-lock file.


Tensors not on the same device

When i run one of the minimal examples
# test.py
from chroma import Chroma, Protein

chroma = Chroma()
protein = Protein('1GFP')
protein = chroma.design(protein)

protein.to("1GFP-redesign.cif")

I get an error that some tensors are not on the same devices:
(chroma_env) Ξ software/chroma git:(main) ▶ python3 test.py
Using cached data from /tmp/chroma_weights/90e339502ae6b372797414167ce5a632/weights.pt
Loaded from cache
cuda
Using cached data from /tmp/chroma_weights/03a3a9af343ae74998768a2711c8b7ce/weights.pt
Loaded from cache
Traceback (most recent call last):
  File "/home/iwe34/software/chroma/test.py", line 5, in <module>
    protein = chroma.design(protein)
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/models/chroma.py", line 532, in design
    X_sample, S_sample, _ = self.design_network.sample(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/anaconda3/envs/chroma_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/data/xcs.py", line 114, in new_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/models/graph_design.py", line 826, in sample
    node_h, edge_h, edge_idx, mask_i, mask_ij = self.encode(X, C, t=t)
                                                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/data/xcs.py", line 114, in new_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/models/graph_design.py", line 536, in encode
    node_h, edge_h, edge_idx, mask_i, mask_ij = self.encoder(
                                                ^^^^^^^^^^^^^
  File "/home/iwe34/anaconda3/envs/chroma_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/data/xcs.py", line 114, in new_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/models/graph_design.py", line 1237, in forward
    node_h, edge_h, edge_idx, mask_i, mask_ij = self._checkpoint(
                                                ^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/models/graph_design.py", line 1253, in _checkpoint
    return module(*args)
           ^^^^^^^^^^^^^
  File "/home/iwe34/anaconda3/envs/chroma_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/iwe34/software/chroma/chroma/layers/structure/protein_graph.py", line 224, in forward
    node_h_l = node_h_l - self.__getattr__(f"node_means_{i}")
               ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

When I try to fix that putting everything on the GPU for this particular line, the error occur on another line (next shown below):
  File "/home/iwe34/software/chroma/chroma/layers/structure/protein_graph.py", line 1443, in forward
    h = torch.exp(-(((h.unsqueeze(-1) - rbf_centers) / self.std) ** 2))
                      ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

That looks to me that a lot of other tensors also would show the same problem and that it is not an easy way to fix, because the device type don't seemed to be share across all classes :(

Thanks for the question.
The fix is easy, you just need to specify the device in Protein.
chroma = Chroma()
protein = Protein('1GFP', device='cuda')
protein = chroma.design(protein)

protein.to("1GFP-redesign.cif")

close this due to inactivity. Feel free to reopen.


GH-37841: [Java] Dictionary decoding not using the compression factory from the ArrowReader

Rationale for this change
This PR addresses https://github.com/apache/arrow/issues/37841.
What changes are included in this PR?
Adding compression-based write and read for Dictionary data.
Are these changes tested?
Yes.
Are there any user-facing changes?
No

Closes: #37841

@danepitkin Thanks a lot for the review comments, I will address them.

@danepitkin I am not sure if this one is practical though :thinking:

@danepitkin I am not sure if this one is practical though 🤔

I'm okay with leaving it as-is if there are issues with using @BeforeEach instead of @Before.
Overall, LGTM! I think it's ready for final review from a committer. Excellent job.

@lidavidm appreciate your feedback.

@lidavidm I updated the PR. Appreciate another round of reviews.

@lidavidm I updated the PR, appreciate another round of reviews.

@vibhatha can you file a followup to add a new integration test to cover this scenario?

Issue filed here: https://github.com/apache/arrow/issues/39222

@github-actions crossbow submit java


Use of eclipse settings "org.eclipse.jdt.core.prefs"

I am currently using your plugin and previously used the plugin de.set.ecj. The Eclipse settings were used there.  Can I configure your plugin so that the Eclipse settings are used?
compileJava {
options.compilerArgs << '-properties' << '$projectDir/.settings/org.eclipse.jdt.core.prefs'
}
The background is that I need source compatibility on Java 11, but target compatibility on 17 and compliance on Java 17. Unfortunately, this cannot be mapped with the Gradle settings.
The goal is that my library can be used in Java 11 projects

Hi @dovogt,
the good news is that the compiler should pick up the properties just fine with the code you specified above. Did you try this and run into any issues?
The bad news is that, from your description, there is different problem with your approach. So let's dive into this:

The background is that I need source compatibility on Java 11, but target compatibility on 17 and compliance on Java 17. [...] The goal is that my library can be used in Java 11 projects

Source compatibility (i.e. javac's -source parameter) determines the language version the compiler will use to read your library's source code. For example, if you have text blocks in your sources, the source compatibility needs to be at least 14.
The target compatibility (-target) essentially configures the bytecode version for your compiled classes.
Java is not forward-compatible. This means, that in Java X, you cannot read bytecode from any Java version that was released after version X.[^1] Specifically, in your case, you target Java 17 bytecode which cannot be used from Java 11.
Instead, Java is backward-compatible. This means that bytecode from Java Y can be read in any Java version released after version Y. So what you want to do here, is setting -target to Java 11. This way, the compiler outputs Java 11-compatible bytecode and your library can be used from any version starting from 11.
Just one more thing to keep in mind: Using -source and -target is discouraged. While -target instructs the compiler to produce bytecode compatible with Java 11, the compiler does not restrict which APIs you are using. If you are referencing a class introduced in Java 16, your code will compile fine but still won't be able to run on 11.
The -release parameter was introduced to solve this. Using this parameter you set the version of the accepted source code, and the targeted bytecode version at once. However, you also get the additional benefit that the compiler checks that you are only using APIs available to the desired release.
So, finally, what I recommend in your case, is ditching -source and -target, and setting -release 11 instead. This can be done in Gradle too and will be respected by the compiler:
compileJava {
    options.release = 11
}

[^1]: This is not entirely correct, because the bytecode version is technically independent from the Java version, but the assumption is good enough.

I just checked and can confirm it works. However, if you are specifying sourceCompatibility and targetCompatibility in Gradle, you are overwriting the settings read from the properties file.

I have created the following configuration. If I set options.relase=11, will the configuration from the settings still be loaded? Or do I have to leave out release that it uses the settings?
	project.tasks.named("compileJava").configure {
		options.fork = true
		options.release = 11
		options.encoding = "UTF-8"
		options.warnings = false
	        options.compilerArgs << '-properties' << '.settings/org.eclipse.jdt.core.prefs' << '-proc:none' << '-log' << new File(project.buildDir, "compiler.xml").getAbsolutePath()
	}

I have created the following configuration. If I set options.relase=11, will the configuration from the settings still be loaded? Or do I have to leave out release that it uses the settings?
	project.tasks.named("compileJava").configure {
		options.fork = true
		options.release = 11
		options.encoding = "UTF-8"
		options.warnings = false
	        options.compilerArgs << '-properties' << '.settings/org.eclipse.jdt.core.prefs' << '-proc:none' << '-log' << new File(project.buildDir, "compiler.xml").getAbsolutePath()
	}

I didn't check this specifically, but -release will probably overwrite the properties just like -source and -target do. You could test this by setting -release to a different version compared to the settings and checking the bytecode of the compiled classes. However, if this configuration works for you, I recommend sticking to it.

When I set the release parameter, it seems to use this and no longer the settings from the Eclipse.  I have removed it and then it seems to work. Thanks again for your support

I had to set the options.release= 11 after all. By using Java 17 for execution, Java 17 is entered as the minimum version in the modules file for publish. As a result, an error occurs when downloading the dependency.
However, this change then leads to gradle changing the settings file in Eclipse and overwriting the compiler compliance level with 11. I have included the Eclipse plugin in build.gradle and set sourceCompatibility=17 and targetCompatibility= 11 so that it can be used for us.


Where can I download the latest Sesame5 firmware?

We are aware that in order to update the firmware of Sesame 5 using the SDK, it is necessary to download the latest firmware from somewhere and update it via BLE.
Thus, would you share download URL of the latest firmware

You'll be able to download it from SESAME Biz very soon. For now, please update using the Sesame app.

You'll be able to download it from SESAME Biz very soon. For now, please update firmware using the Sesame app, thank you.

Could you give us a rough idea of when you might be able to share the download link?

Can you give us a rough idea of when you might be able to share the download link?


SyntaxError: Invalid regular expression: /*/iu: Nothing to repeat

I get the Invalid regular expression error when I try to search query .*? with the following options:
{
  unicode: true,
  interSplit: "[^\\p{L}\\p{Emoji}\\d']+", // Notice that I add a \p{Emoji} here to supports Emoji
  intraSplit: '\\p{Ll}\\p{Lu}',
  intraBound: '\\p{L}\\d|\\d\\p{L}|\\p{Ll}\\p{Lu}',
  intraChars: "[\\p{L}\\d']",
  intraContr: "'\\p{L}{1,2}\\b",
}

Steps to reproduce:

Open https://stackblitz.com/edit/node-7bqgzj
Run node index.js in the terminal
You will see the error below:

SyntaxError: Invalid regular expression: /*/iu: Nothing to repeat
    at new RegExp (<anonymous>)
    at prepQuery (file:///home/projects/node-7bqgzj/node_modules/@leeoniya/ufuzzy/dist/uFuzzy.cjs.js:377:11)
    at Object.filter (file:///home/projects/node-7bqgzj/node_modules/@leeoniya/ufuzzy/dist/uFuzzy.cjs.js:382:17)
    at eval (file:///home/projects/node-7bqgzj/index.js:52:22)
    at Generator.next (<anonymous>)
    at _0x320919.run (https://node7bqgzj-023r.w-credentialless-staticblitz.com/blitz.41fbae16.js:352:373719)
    at _0x28eaf8._evaluate (https://node7bqgzj-023r.w-credentialless-staticblitz.com/blitz.41fbae16.js:352:379291)
    at async ModuleJob.run (https://node7bqgzj-023r.w-credentialless-staticblitz.com/blitz.41fbae16.js:181:2372)

I don't know if this is a bug in uFuzzy or an error in my options. If my options is incorrect, I would love to know what's the correct options if I want to support unicode with emoji. Thanks in advance.

interesting, apparently * is an emoji :shrug:

i guess the best we can do is run regexp escaping on the post-split tokens

My current workaround is removing all special characters from the needle before searching. It seems to work well in our cases so far.
needle = needle.replace(/\p{Punctuation}+/gu, ' ')
const indexes = uf.filter(haystack, needle)

yeah that would work. you can also try doing this instead:
interSplit: "\\p{Punctuation}+",


Mixing CUDA Fortran and SYCL CUDA

Example showing how to link together a program written in CUDA FORTRAN
that calls a routine that contains a SYCL program running on CUDA

Blocked by https://github.com/intel/llvm/pull/4004


Stable/icehouse

I've updated the repo description to say that master is tracking Kilo, not Icehouse. If you'd like to use the module to deploy Icehouse you can use the stable/icehouse branch.


Fix translation files

Coverage remained the same at 90.114% when pulling 9e2289761c669fbd9d5f19fd91c9b6576187f6ee on fix-translation-files into b73fea49b54d561ea3792073304f6fd08dae936d on master.


Fixes #2703 - document agent system requirements

coverage: 93.925% (-0.004%) from 93.929%
when pulling e34e7042d9e476666c453cfb8c7433eea0ac82f0 on cody-agent-system-requirements
into f3c7d5f1fb206136a8cb37d2a4f39cfb91203591 on main.


Rethinking wording around input/output

One point of confusion with some users is the use of the wording for the "input" and "output" command line options. Users sometimes think that these files are for humans, when really we are connoting files that the tools read & write.
In thinking about re-branding these options across the tool suite, I've come up with following list of potential renames:

in / out
from / to
read / write
load / save
load / store
load / dump
consume / produce

We should clarify that these actions are for the tool "state" or "cache". We may want to write documentation about what we mean by a cache and how each tool modifies the cache. We should also think about the naming convention of the argument. I would prefer to stay away from possible clashes with tools that we mimic, so using a only a long option might be the way to go.

Thoughts? I'm partial to "load" and "store" of a "cache", with the options being --load / --store or --load-cache and --store-cache

@gonsie I would say read/write or load/store make the most sense to me.

Part of our conversation about this brought up the fact that users are allowed to create the file name, and many users with default to a .txt extension. This can lead to confusion when a user tries to look at the data later, and ends up with a screen of jibberish.
I think we should:

put a .mfu file extension on to whatever path value the user has given us
refer to this file as "the mfu file" in our documentation and usage strings
add a page to the RTDs that describes this file.
in addition to changing the input/output option names.

thought! We could also use the terminology pipe-in and pipe-out since this is basically what users are trying to achieve. Yes a “pipe” is a specific type of file on the file system, but I don’t MPI users typically think about that.


Replace Oniguruma

Currently we use Oniguruma for expr because it has support for the GNU regex syntax.  Ideally, we should be using something like Rust regex, but AFAICT this involves either converting the input string to make it conform to the correct syntax or forking regex-syntax so it supports GNU regexes.

By design, the regex crate lacks lookaround and back-references:

Its syntax is similar to Perl-style regular expressions, but lacks a few features like look around and backreferences. In exchange, all searches execute in linear time with respect to the size of the regular expression and search text.

The fancy-regex crate supports those; I don't know how close it is to GNU syntax.

Ah, I forgot that GNU BRE has backreferences.  I'm a little wary of using fancy-regex because it has practically no documentation and is not updated very frequently.

Reopening this because I still want to replace onig. The fancy-regex crate looks pretty good now, I think. It does not necessarily need to be updated that much, because it also uses the standard regex under the hood. I'm honestly also fine with exploring whether we can use the regex crate and see whether anyone really relies on the fancy features, but that's maybe a bit risky.
Reason for my renewed interest is that onig takes a long time to build:

I have a plan for this: we could depend on the regex-syntax crate and do our own parsing, because expr requires a very specific syntax for regexes. I don't think this would be too difficult.

I've asked for advice on the regex repository: https://github.com/rust-lang/regex/discussions/1126 (no reply yet, I just posted this)


Updates to the marker to allow custom images with different anchors

Updates include allowing the setting of a google map icon which will allow more custom settings of how the marker is handled.  As well, allowing for the zIndex of the marker to be set.

Sorry for the two commits.   I submitted a prior change, but back tracked on some of that implementation to make the implantation for fluent with Google's API.


PowerToys does not save my settings. I have to manually revert each instance.

Microsoft PowerToys version
0.53.3
Running as admin

[X] Yes

Area(s) with issue?
General
Steps to reproduce
I have to revert my settings in each instance of PowerToys; specifically which app within PowerToys I've set to ON or OFF.
ex. I have set the following P.T. apps as active:  color picker, Fancy Zones, Keyboard manager & PowerToys Run.   BUT after every time I load P.T (manually after every boot   -- I use Startup Delayer)  I have to manually switch all P.T apps that I don't use, off.
PS - apologies, English is not my first language.
✔ Expected Behavior
Automatically saved settings
❌ Actual Behavior
I must manually revert settings
Other Software
Startup Delayer - by r2 Studios

Similar/Same issue.  Some features remain enabled (always on top, file explorer addons) but a restart of the application causes others (fancy zones, mouse utilities and powertoys run) to become disabled.

The same for me.

ref: #15450

Could you attach a full bug report file?
Right-click on the tray icon and click Report Bug. Then drag-and-drop the generated file into the GitHub comment to upload it. Thanks!

Having this issue as well

Bump


fix: make sure refine preserves pre-existing natural mvars

Fixes #2434. refine currently fails to include pre-existing natural metavariables from the refined term in the resulting goal list. For example, constructor creates natural metavariables, and we can close the goals with refine alone:
example : True ∧ True := by -- (kernel) declaration has metavariables '_example'
  constructor
  · refine ?left
  · refine ?right

The source of this issue is a bug in withCollectingNewGoalsFrom in elabTermWithHoles when allowNaturalHoles is false. Metavariables are separated into natural and non-natural, and then any new unassigned natural mvars (as determined by their index) cause the tactic to be aborted. If the tactic isn't aborted here, the non-natural mvars are passed through to constitute the new goal list. However, this orphans any pre-existing "old" natural mvars.
This PR fixes this by changing the process of logging and aborting upon encountering new natural mvars into an unless allowedNaturalHoles do ...-style guard. If we pass that guard, then we don't need to filter the original list of mvars at all; it's guaranteed to be free of new natural mvars, as required.

Commenting in an attempt to cause github to reopen this PR.


add link to get back to toplevel page

Hello,
I tried sigal for the first time and I like it very much. I indexed my whole picture collection (>32k images). In order to be usable for me I needed to add a link to the top-level html page in the template I use.
I propose to merge it into the main sigal.
This is my first pull request ever. That's why I choose a small change. Pls let me know if anything more needs to be considered.
thanks,
Stefan

Hi,
Thanks for the proposition, but there is already a link to the root page with the page title (the site name). I think there is no need to have another link, but maybe for the photoswipe theme this link is not obvious ? I

As discussed in #306, adding a link with a home icon would be fine. Please reopen if you want to work on this.


Add config option for stream blanker default on / off

Depending on the use case it is useful to make sure vocotocore starts with stream live instead of stream blank. This should be a config option.

+1
On Fri, Nov 9, 2018 at 4:59 AM derpeter<EMAIL_ADDRESS>wrote:

Depending on the use case it is useful to make sure vocotocore starts with
stream live instead of stream blank. This should be a config option.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/voc/voctomix/issues/234, or mute the thread
https://github.com/notifications/unsubscribe-auth/AABauWxKLx1ConKKP_m0relbatREb1SFks5utWAWgaJpZM4YWX-i
.

--
Carl K


added delete endpoint

added delete endpoint to space cloud
tested with mongo, SQL, Postgres
returns empty object if the operation is successful otherwise an error

yes, I have tested the above endpoint with the help of postman for all the databases.


Revert "Add date to addin build"

Reverts Esri/distance-direction-addin-dotnet#606
The version numbers need to be using different numbers for Pro and ArcMap

This does not need to be reverted


Convert block tags to java. Also fix compile errors related to previous block registry changes.

Merging for build error fix.


build: skip biomed ingest tests 90% of time due to ftp connectivity

The Biomed connectors have been flaky with ftp connections lately.

is the ftp connection issue particular to this source (can we host somewhere else)?

i believe so, that this is a biomed server issue. but this connector is specific to biomed (more than just ftp), so to put it somewhere else would mean we would just be testing the PDF's, not the connector per-se.

ah got it, yea fair. hm, does immediately retrying succeed? (maybe we need an option for retry attempts in the pipeline or just in our test) or are there just blanket periodic outages? (maybe we can detect that and skip the test)


Build updates

Changes needed to get Ubuntu and Windows builds working again.

Changes needed to get Ubuntu and Windows builds working again.


picture and different formats

Chrome does load jpg instead of webp.
Markup:
<picture>
    <source data-srcset="/dist/images/categories/historika_master.webp" type="image/webp">
    <source data-srcset="/dist/images/categories/historika_master.jpg" type="image/jpeg">

</picture>

Is there anything I can do to force supported browsers to load the webp format?

Can you please make a simple example using jsfiddle. Thank you.

Sorry, posted in the wrong issue tracker. Problem was regarding other library.


Figure out future DAG operators

There are some concepts we've talked about (and have in some cases touched using FunctionNodes #121) that we need to figure out how to cleanly integrate into our pipelines interfaces.
Among these are:

Block Transformers and Estimators (operating on big feature spaces split across several RDDs)
Aggregators (sampling, windowing, caching, etc.)
Joining (both in the context of block nodes, and e.g. join(identity, transformer))
Operating on grouped data (e.g. Grouped(Transformer[A, B]): Transformer[Seq[A], Seq[B]])
Hyperparameter tuning & cross validation (making sure to allow optimizations like how linear mappers can train multiple lambdas at once)

@etrain @shivaram
So I think I’ve finally come up with a valuable insight for keeping the pipeline interface linearized even when doing complex DAG operations. (Important because the linearized interface gives us nice things like being more understandable and flexible, passing training data through the whole pipeline automatically, etc.)
The key insight is instead of complex operators taking nodes as parameters, they take functions that would chain new nodes onto the current pipeline.
Some examples of what this could look like for different operators:
Join Operator:
(A => B, A => C) => (A => (B, C))
Useful for obvious reasons
nodeAB then nodeBC thenJoin {
     _ then nodeCD,
     _ then nodeCE
}

Concatenate Operator:
Seq[(A => B)] => (A => Seq[B])
Concatenate is to Join as Seq(a1, a2, a3) is to (a, b)
Block Transformers & Block Estimators would interact w/ an alternate physical concat operator than usual nodes would.
nodeAB then nodeBC thenConcat {
     _ then nodeCD,
     x => (x thenEstimator nodeCE).fit(rddOfA) then nodeED,
     _ then otherNodeCD
}

nodeAB then nodeBC thenConcat {
     (1 to 4).map(_ then randNodeCD()):_*
}

Grouped Operator:
(A => B) => (Seq[A] => Seq[B])
Used for applying single-item transformers and estimators on grouped data, e.g. in Timit training LDA on a per-frame basis even though the frames are grouped into entire utterances.
nodeAB then nodeBSeqC thenOnEach {
     (_ then nodeCD thenEstimator nodeDE).fit(rddOfA)
}

Risky point of trickiness: the chaining rules inside the grouped operator have to be subtly different from the usual. _ thenLabelEstimator nodeCDL, would have to return LabelEstimator[A, D, Seq[L]], whereas usually nodeAC thenLabelEstimator nodeCDL would return LabelEstimator[A, D, L].
Switch Operator:
(A => Int)(Array[A => B]) => (A => B)
Effectively the branching conditional of pipelines. A => Int specifies which branch to take, Array[A => B] are the potential branches.
A use case we’ve seen for this is training a separate event extractor for each event type in the event extraction pipeline.
nodeAB then nodeBC thenSwitch (c => if (c > 1) 0 else 1) {
     _ then zeroNodeCD,
     _ then oneNodeCD
}

Risky point of trickiness: if a branch contains an estimator, it will only be trained on the data that flows into that branch.

Thanks @tomerk - this is useful stuff. I don't fully grok the use case for the Concatenate operator, so let's chat about it in person. We should also discuss semantics for identifying the key for grouped and join operators.
The Switch operator seems useful, but I'm not sure what its use would be at the current level of abstraction. If it's for hyperparameter tuning, I think that belongs in a separate discussion and perhaps at a higher level of abstraction altogether. If it's for supporting more complex pipelines then it would be good to have a concrete use case to be working from.
Finally, let's not forget about Aggregation - it's possible to think of plain aggregation as a grouped operator with the null key (which could be how we implement it), but we should think about user-facing support for aggregation and what the semantics are at that level.

After talking to @etrain and @shivaram it turns out I was mistaken about the interface of the BlockLinearEstimator. So, concatenate should actually have the interface:
Seq[(T => DenseVector)] => (T => DenseVector)
This could be generalized to reduce & aggregate:
(Seq[(A => B)], (B, B) => B) => (A => B)
(Seq[(A => B)], C, (C, B) => C, (C, C) => C) => (A => C)
But I haven't seen any use cases for these yet, even in more complex pipelines

Okay @shivaram and @etrain, as per our discussion I’ve gone through MnistRandomFFT, RandomPatchCifar, VOCSIFTFisher, and the Imagenet pipeline to figure out how to better linearize our existing pipelines.
Lessons learned
The main way to linearize these pipelines is to consider how to push manual fitting & joining logic into a node.
Some cases the logic is so complicated it needs to have it’s own estimator (potentially inlined). Other patterns are generalizable enough it may be worth making nodes that take other nodes as input. e.g:
Concat(node1, node2)

OnMatrix(pcaNode)

OnSeq(ldaNode)

FisherVector(gmmNode)

Note that if you guys prefer it’s also possible to implement any of these via chaining instead of composition:
node1.concat(node2)

pcaNode.onMatrix

ldaNode.onSeq

gmmNode.toFisherVector

Note on sampling:
Sampling doesn’t quite fit into the above (because we need to maintain type information when sampling so we can fit sampling estimators into the above nodes, e.g. the current use case of ColumnSampler). We can either make estimators that we want to be able to sample take an optional sample size as a parameter, or we can make sampling a method of all estimators, node.sampling(count) and extend it while maintaining type information. Lets us do e.g. FisherVector(gmmNode.sampling())
MnistRandomFFT
val featurizer = Concat {
  Array.fill(conf.numFFTs) {
    RandomSignNode(mnistImageSize, randomSignSource) then PaddedFFT then LinearRectifier(0.0)
  }
}

val mnistClassifier = (featurizer thenLabelEstimator
    new BlockLeastSquaresEstimator(conf.blockSize, 1, conf.lambda.getOrElse(0)))
    .fit(train.data, labels) then MaxClassifier

// Do the evals
val trainEval = MulticlassClassifierEvaluator(mnistClassifier(train.data), train.labels, numClasses)
val testEval = MulticlassClassifierEvaluator(mnistClassifier(test.data), test.labels, numClasses)

The only change needed for the MnistRandomFFT is replacing the zip vectors FunctionNode w/ a Concat node that extends Transformer and takes nodes of matching types as parameters
RandomPatchCifar
// Inline an Estimator
val convolverEstimator = Estimator { (in: RDD[Image]) =>
  val patchExtractor = new Windower(conf.patchSteps, conf.patchSize)
      .andThen(ImageVectorizer.apply)
      .andThen(new Sampler(whitenerSize))

  val baseFilters = patchExtractor(in)
  val baseFilterMat = Stats.normalizeRows(MatrixUtils.rowsToMatrix(baseFilters), 10.0)
  val whitener = new ZCAWhitenerEstimator().fitSingle(baseFilterMat)

  //Normalize them.
  val sampleFilters = MatrixUtils.sampleRows(baseFilterMat, conf.numFilters)
  val unnormFilters = whitener(sampleFilters)
  val unnormSq = pow(unnormFilters, 2.0)
  val twoNorms = sqrt(sum(unnormSq(*, ::)))

  val filters = (unnormFilters(::, *) / (twoNorms + 1e-10)) * whitener.whitener.t
  new Convolver(filters, imageSize, imageSize, numChannels, Some(whitener), true)
}

// Build the pipeline
val featurizer = convolverEstimator.fit(trainImages)
    .then(SymmetricRectifier(alpha=conf.alpha))
    .then(new Pooler(conf.poolStride, conf.poolSize, identity, _.sum))
    .then(ImageVectorizer)
    .thenEstimator(new StandardScaler).fit(trainImages)

val predictionPipeline = (featurizer thenLabelEstimator
    new BlockLeastSquaresEstimator(4096, 1, conf.lambda.getOrElse(0.0)))
    .fit(trainImages, trainLabels) then MaxClassifier

// Do the model evals.
val trainEval = MulticlassClassifierEvaluator(
  predictionPipeline(trainImages), LabelExtractor(trainData), numClasses)
val testEval = MulticlassClassifierEvaluator(predictionPipeline(testImages), LabelExtractor(testData), numClasses)

The main change to the RandomPatchCifar pipeline is moving the convolver fitting logic into an Estimator (note we may want this as a not inlined estimator). It was too complicated internally to easily generalize to patterns I've seen elsewhere, but we could make it take a WhiteningEstimator as a parameter so it can use more than just the ZCAWhitener.
Note that I've removed the caching nodes here because they're using simple enough rules that a DAG planner could put them in (just cache before any estimator). Alternatively w/ a DAG planner we could also have caching nodes that only cache in the fitting stage, not the final transformer, or we could specify caching the same I specify sampling in VOC, via node.sampling()
VOCSIFTFisher
// Part 1: Scale and convert images to grayscale.
val grayscaler = MultiLabeledImageExtractor then PixelScaler then GrayScaler

// Part 2: Extract Sifts & Compute dimensionality-reduced PCA features.
val siftExtractor = grayscaler then new SIFTExtractor(scaleStep = conf.scaleStep)
val pcaFeaturizer = conf.pcaFile match {
  case Some(fname) => siftExtractor then
      OnMatrix(PCATransformer(convert(csvread(new File(fname)), Float).t))
  case None => (siftExtractor thenEstimator
      OnMatrix(PCAEstimator(conf.descDim).sampling(conf.numPcaSamples))).fit(parsedRDD)
}

// Part 3: Compute Fisher Vectors and signed-square-root normalization.
val fisherFeaturizer = conf.gmmMeanFile match {
  // Part 3a: If necessary to compute the Fisher Vectors, compute a GMM based on the dimensionality-reduced features, or load from disk.
  case Some(f) => pcaFeaturizer then
    FisherVector(
      csvread(new File(conf.gmmMeanFile.get)),
      csvread(new File(conf.gmmVarFile.get)),
      csvread(new File(conf.gmmWtsFile.get)).toDenseVector)
  case None => (pcaFeaturizer thenEstimator
      FisherVectorEstimator(GaussianMixtureModelEstimator(conf.vocabSize).sampling(conf.numGmmSamples)))
      .fit(parsedRDD)
} then FloatToDouble then
  MatrixVectorizer then
  NormalizeRows then
  SignedHellingerMapper then
  NormalizeRows

// Part 4: Fit a linear model to the data.
val model = (fisherFeaturizer thenLabelEstimator
    new BlockLeastSquaresEstimator(4096, 1, conf.lambda))
        .fit(parsedRDD, trainingLabels)

// Now featurize and apply the model to test data.
val testActuals = MultiLabelExtractor(testParsedRDD)
val predictions = model(testParsedRDD)

val map = MeanAveragePrecisionEvaluator(testActuals, predictions, VOCLoader.NUM_CLASSES)

Notice the OnMatrix & FisherVector wrapping I mentioned in the “lessons learned”, leaving it up to the DAG executor to manage the caching, and how switching between a full pipeline linearization in conf.x match { } instead of just the specific node to be chained.
ImageNetSiftLcsFV
A mix of the strategies used above should much better linearize the pipeline:

Nodes that wrap other nodes (OnMatrix, FisherVectorEstimator, etc.)
using wrapper(estimator.sampling()) instead of ColumnSampler
having conf.x match { } specify the full linearization
(Optionally but not required) leaving it up to the DAG executor to decide when to cache / unpersist cached data when fitting
moving complicated fitting logic that doesn’t fit into any of the above into it’s own estimator
using Concat to combine the Sift & LCS features

cc @thisisdhaas @sjyk who have a use case for a split operator that goes from a SparkSQL query output to a transformer

Here is my use case/what I want to be able to do.
-my side of things
The output of my loading/data transformation code is a table/dataframe with a schema. I want the user to be able to specify a "FeatureView" of this table, that is for every column specify in words the featurization semantics of data. For example, suppose I have a table Review(comment, city, type, rating), the user should be able to say:
FeatureView(Review, List( (comment, "text"), (type, "categorical"), (rating, "numerical")))
-your side of things
Ideally there would be a pipeline architecture that takes a FeatureView as input generically without having rewrite every time. Either by automatically splitting the features into branches of the pipeline or allowing transformation on an index of the list.
For each of these types of features, transformers exist, however, when they are all mixed it doesn't fit into a nice linear pipeline. I think it is fair to assume a one-to-many relationship between cols and features.

Can you enumerate the set of types you care about - are they basically,
String, Enum, Double? Are there more? I ask because I'm worried about
enforcing type-safety if possible later on in the pipeline.
If you think of the FeatureView as a TupleN[A,B,..,N], and each component
pipeline as a (A => J, B =>K, .. N=>L), then we could pretty easily come up
with something that takes a FeatureView and a TupleN( functions) and gives
back a pipeline that goes from TupleN[A,B,..,N] => TupleN[J,K,..,L], but
I'm not entirely sure whether those are obvious semantics or what the DAG
operator would be called.
On Fri, May 29, 2015 at 1:58 AM, Sanjay Krishnan<EMAIL_ADDRESS>wrote:

Here is my use case/what I want to be able to do.
-my side of things
The output of my loading/data transformation code is a table/dataframe
with a schema. I want the user to be able to specify a "FeatureView" of
this table, that is for every column specify in words the featurization
semantics of data. For example, suppose I have a table Review(comment,
city, type, rating), the user should be able to say:
FeatureView(Review, List( (comment, "text"), (type, "categorical"),
(rating, "numerical")))
-your side of things
Ideally there would be a pipeline architecture that takes a FeatureView as
input generically without having rewrite every time. Either by
automatically splitting the features into branches of the pipeline or
allowing transformation on an index of the list.
For each of these types of features, transformers exist, however, when
they are all mixed it doesn't fit into a nice linear pipeline. I think it
is fair to assume a one-to-many relationship between cols and features.
—
Reply to this email directly or view it on GitHub
https://github.com/amplab/keystone/issues/125#issuecomment-106749978.

Spoke with @shivaram and @tomerk about this - it looks like we can support the pattern you're asking for with the Concat operator and assuming the input tables can be converted to RDD[case class].
@tomerk is going to check in this operator first and then we can see how well these things are supported.

FYI here is the Spark PR open to convert DataFrames to typed RDDs https://github.com/apache/spark/pull/5713

Sounds good. We can assume that the user knows the schema of the data frame and thus can give a case class description of it.

It would be great if we could have a many-to-many relationship between inputs and outputs. For example, one common use case for us is to compute a similarity feature from two columns of the input (signature looks like (A,B => J)).
This certainly complicates the interface @etrain described above, and I don't know enough about what you're planning with RDD[case class] + Concat to understand if it will work with that interface.

So the basic idea right now is that each operator to _.concat will take in the entire row (a single object) and that we provide some simple syntax for selecting appropriate fields. Say you've got
case class ViewRecord(a: A, b: B), then the operator might look something like
preprocess thenConcat {
   CosineSimilarity(_.a, _.b),
   new EntityResolver(0.6)(_.a),
   ...
}

And the output type of this pipeline would be something like a Tuple[J,K,...] if we assume that the output of EntityResolver is of type K.
It may not end up being this exact syntax, but that's the basic idea.

Also, @sjyk - @shivaram mentioned that there may be a .toCaseClass (or something) method on DataFrames coming soon, but I can't find a reference to it, so this may not be something you need to solve.

Ah, that concat operation looks very well aligned with our use case. Looking forward to seeing it.


HttpRequestClaimValueEnricher

Hi.
What do you think about adding new enricher named HttpRequestClaimValueEnricher:
    public class HttpRequestClaimValueEnricher : ILogEventEnricher
    {
        /// <summary>
        /// The claim property name searched for value to enrich log events.
        /// </summary>
        private readonly string _claimProperty;

        /// <summary>
        /// The property name added to enriched log events.
        /// </summary>
        private readonly string _logEventProperty;

        public HttpRequestClaimValueEnricher(string claimProperty) : this(claimProperty, null)
        {
            _claimProperty = claimProperty;
        }

        public HttpRequestClaimValueEnricher(string claimProperty, string logEventProperty)
        {
            _claimProperty = claimProperty;
            _logEventProperty = logEventProperty ?? claimProperty;
        }

        public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
        {
            if (logEvent == null) throw new ArgumentNullException(nameof(logEvent));

            if (HttpContext.Current == null)
                return;

            if (HttpContextCurrent.Request == null)
                return;

            var user = HttpContext.Current.User;
            if (user == null)
                return;

            var claims = ((ClaimsIdentity) user.Identity).Claims;

            var value = claims?.FirstOrDefault(c => c.Type == _claimProperty)?.Value;
            if (string.IsNullOrWhiteSpace(value))
                return;

            var claimProperty = new LogEventProperty(_logEventProperty, new ScalarValue(value));
            logEvent.AddPropertyIfAbsent(claimProperty);
        }
    }

?

Thanks for the suggestion! This looks good. Could we add it without requiring the package to take on any additional dependencies outside the ones it already has? (Can't check right now myself, sorry :-)). If so, a PR would be very welcome. Cheers!

Yes, sure. There aren't any required external dependencies (in my opinion). I'd like to make a pull request, but can't push my changes into separate branch (403).

#38

Done! Thank you.


fix accuracy metric

Unlike MLM, when learning MNTP, accuracy must be calculated based on the preds in front of the MASK TOKEN, but the current code is based on MLM. (Calculating results and accuracy of MASK TOKEN position)
I modified preds and labels to be calculated correctly by shifting them.
The original code continues to decrease eval_accuracy, but the modified code correctly improves eval_accuracy.

Thanks, nice catch!


doc: add Wechat Bot to examples list

ChatGPT webchart bot is a WeChat assistant based on NodeJS and webchaty. This tool is more convenient for you to  use ChatGPT, you just chat with it as a good friend or a housekeeper.

Tweaked the language a bit. Thanks!


Not able to replicate results on FlyingThings++ noted in the paper

Hi,
Great work and thank you for releasing your code and documenting it.
I was trying to replicate numbers on FlyingThings++ shown in this table.

I had setup the conda environment and FlyingThings++ exactly as told in the README. I am using the model checkpoint provided in get_reference_model.sh.
I run the command python test_on_flt.py --modeltype pips and python test_on_flt.py --modeltype raft to evaluate on FlyingThings++.
Here are the results that I get.
PIPS:
1_8_16_pips_flt_17:07:18; step 002542/2542; rtime 0.00; itime 0.09, ate_vis = 10.42, ate_occ = 27.04

RAFT:
1_8_16_raft_flt_18:46:56; step 002542/2542; rtime 0.00; itime 1.03, ate_vis = 15.04, ate_occ = 39

Do you have any idea why they might be different from the results provided in the paper?

Interesting! Thanks for the heads up. At least you're doing better than the paper, instead of worse!
My first thought is that maybe the data you generated is slightly different, due to some randomness in that process. What I should probably do is put my data on google drive. If you want/need that, let me know. I'll keep thinking, in case this isn't it.

Thank you for your response. Yes, that will be great if you could provide the data that you generated. I agree that there could be some randomness in the process.

Hey @aharley , please let me know by when you will be able to share the data you generated.

Maybe more time actually. The upload time is reading 55h right now, and that's assuming no interruptions. Anyway, at the end of this we should have a very good gdrive link. I'll let you know @ayshrv!

Update: I think the problem is basically on my side, to do with last-minute updates to the dataset, after putting results into the paper. Re-running things locally I'm actually getting values very close to yours.
I will update the repo soon with new reference results in the README.
If you still want to download my exact flyingthings data you can get it here:
https://drive.google.com/drive/folders/1zzWkGGFgJPyHpVaSA19zpYlux1Mf6wGC?usp=sharing

Great, thanks for checking and providing me your version of the dataset.
Looks like you are indeed able to get better results than reported in the paper :).


PWA-207 Page headlines and product names now wrap and break correctly

Pull Request Test Coverage Report for Build 506

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 38.355%

Totals

Change from base Build 505:
0.0%

Covered Lines:
1730

Relevant Lines:
4385

💛  - Coveralls


Feature 96 creatorviewpage 01

Draft PR

Let us know when this pull request is ready for review @Ali-7s

Looks like the requested changes have been incorporated. Back button has been added, create event routes to add event page, view all events and view my events route back to home page. Home page Sign-up button now says Creator View.


Code to ignore certain lightning warnings

Right now we have
warnings.filterwarnings(
    "ignore",
    category=UserWarning,
    module="pytorch_lightning.trainer.data_loading",
    lineno=102,
)

warnings.filterwarnings(
    "ignore",
    category=UserWarning,
    module="pytorch_lightning.trainer.callback_hook",
    lineno=100,
)

warnings.filterwarnings(
    "ignore",
    category=UserWarning,
    module="torch.optim.lr_scheduler",
    lineno=129,
)

After a version bump we can revisit these and see what is still relevant.
The problem is that the lineno changes with each PL version (or torch for the lr_scheduler warning), so we can see if it is possible to do an "if" and then change it depending on what version we are at.  Then when new PL versions come out we can just add a case to the if...

import warnings

warnings.filterwarnings("ignore", ".*does not have many workers.*")

That might be better.
The things to ignore are:

[ ] https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/data_loading.py#L265-L270
[ ] https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/data_loading.py#L106-L112
[ ] https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py#L1533-L1537

@jbrightuniverse  Can this be removed now?  Did you test it on the other project?

It can be removed, it works.


docs: added emojis to normal dungeon, floor 9

For floor 9, I have added emojis in rooms up to room 4. Please note, as a mac user, there might be some of these emojis not available for you to view, which can lead to question marks like [?], so feel free to either change it or delete it. I cannot tell which emojis are not available to some users. Feel free to ask me if you need any clarification about the emoji I used.

Nice work @antonyren-carleton


libgrpc.a libssl.so.3 linking error for a Ubuntu 22.04 gRPC/protobuf application

What version of gRPC and what language are you using?
gRPC v1.62.0
C++
What operating system (Linux, Windows,...) and version?
Ubuntu 22.04.1 LTS
What runtime / compiler are you using (e.g. python version or version of gcc)
cmake version 3.22.1
gcc and g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
What did you do?
Followed the instructions at 'https://grpc.io/docs/languages/cpp/quickstart/' with updated cmake options to build gRPC dependencies from source with static linking.  Specifically the following commands were executed as root:
cd /root
export MY_INSTALL_DIR=/opt/grpc/grpc-1.62.0
mkdir -p $MY_INSTALL_DIR
export PATH="$MY_INSTALL_DIR/bin:$PATH"
sudo apt update
sudo apt install -y build-essential autoconf libtool pkg-config
git clone --recurse-submodules -b v1.62.0 --depth 1 https://github.com/grpc/grpc
cd grpc
mkdir -p cmake/build
pushd cmake/build
cmake -DgRPC_INSTALL=ON -DCMAKE_BUILD_TYPE=Release -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$MY_INSTALL_DIR \
-DgRPC_ABSL_PROVIDER=module -DgRPC_CARES_PROVIDER=module -DgRPC_PROTOBUF_PROVIDER=module \
-DgRPC_RE2_PROVIDER=module -DgRPC_SSL_PROVIDER=module -DgRPC_ZLIB_PROVIDER=module \
-Dprotobuf_BUILD_SHARED_LIBS=OFF ../..
make --jobs=$(nproc)
make install
popd
echo "/opt/grpc/grpc-1.62.0/lib" | tee /etc/ld.so.conf.d/grpc1.conf
ldconfig
which protoc
/opt/grpc/grpc-1.62.0/bin/protoc --version # output is 'libprotoc 25.1'.

The gRPC/protobuf build and install above are successful.  All good so far.
Next, I compile my application and attempt to link its libraries, but I get this linking error:
/usr/bin/ld: /opt/grpc/grpc-1.62.0/lib/libgrpc.a(ssl_transport_security.cc.o): undefined reference to symbol 'SSL_CTX_set_keylog_callback@@OPENSSL_3.0.0'
/usr/bin/ld: /lib/x86_64-linux-gnu/libssl.so.3: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

What did you expect to see?
100% successful gRPC/protobuf build and install (yes, I saw this).
100% successful build for custom Ubuntu 22.04 gRPC/protobuf application (no, I did not see this).
What did you see instead?
Unexpected linker errors in my custom Ubuntu 22 gRPC/protobuf application, specifically:
/usr/bin/ld: /opt/grpc/grpc-1.62.0/lib/libgrpc.a(ssl_transport_security.cc.o): undefined reference to symbol 'SSL_CTX_set_keylog_callback@@OPENSSL_3.0.0'
/usr/bin/ld: /lib/x86_64-linux-gnu/libssl.so.3: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

Given that I built gRPC with shared-libs off, why is libssl.so.3 needed?
Anything else we should know about your project / environment?
My custom application is being ported from Ubuntu 16 to Ubuntu 22.  No issues exist on Ubuntu 16 using gPRC v1.54.0.  After experiencing this linking issue with v1.54.0 on Ubuntu 22, I upgraded gRPC to v1.62.0 but the issue remains.  Has anyone else encountered a similar linking issue? How did you resolve it?
I see a similar issue here (https://github.com/googleapis/google-cloud-cpp/discussions/9063) but the original poster in that case was compiling gRPC using packages rather than source (modules).
If I search libssl.so.3 for its symbols, I see the undefined symbol:
# readelf -Ws /lib/x86_64-linux-gnu/libssl.so.3 | grep SSL_CTX_set_keylog_callback
583:<PHONE_NUMBER>035cc0    12 FUNC    GLOBAL DEFAULT   15 SSL_CTX_set_keylog_callback@@OPENSSL_3.0.0

Any suggestions for next steps in debugging or added visibility into the issue?

I believe the issue described above may have been related to older/out-of-date libraries that were on my development platform at alternative/additional library search paths.  Performing some clean-up on my development platform enabled me to resolve/get-past this issue.

Such build issues are always tricky/annoying to debug. Glad you got it fixed.


Ability to fix LSP errors/warnings with an LLM

A user should be able to hit a keybind when on a line with an LSP error, and then the relavent information about that error should be passed on to an llm which should try to fix it.

The use of vim.diagnostics.get() might be good for getting the locations of diagnostics


Passing values to URI

Here's a slightly different use case.
I need to create a POST that looks like this:
webhooks_request "Get data from TPP" do
  uri "<IP_ADDRESS>/vedsdk/certificates/retrieve?apikey=" + node["tpp"]["apikey"]
  #uri "<IP_ADDRESS>/"
  #uri_port 5000
  ssl_validation false
  use_ssl true
  #follow_redirect true
  post_data (
                { 'CertificateDN' => "\\VED\\Policy\\EMV Certificates\\NsCVcQg4fd", 'Format' => "Base64", 'Password' => "lexicon", 'IncludePrivateKey' => "true"}
            )
  header_data (
                  { 'content-type' => 'application/json'}
              )
  expected_response_codes [ 200, 201, 400, 401 ]
  action :post
end

So basically I need to pass in the API key as a parameter to the next request. However I don't think the uri key supports this:
TypeError
---------
no implicit conversion of nil into String

Hey there, any idea why this is not working? I think If I crack this I'll be good to go :)

Yea, np, I'll get this going for you today!

Excellent thanks.

So I made some progress here. I changed the post case action in the provider to look like this so the entire uri gets sent.
when "post"
          Chef::Log.info "Setting up action #{ action }."
          # Changed so that the full uri including query parameters are sent
          #req = Net::HTTP::Post.new(uri.path)   #let's post
          req = Net::HTTP::Post.new(uri)   #let's post

However the issue now is that the value of my default["tpp"]["apikey"]  attribute is not available in here:
webhooks_request "Get data from TPP" do
  uri "<IP_ADDRESS>/vedsdk/certificates/retrieve?apikey=#{node["tpp"]["apikey"]}"
  ssl_validation false
  use_ssl false
  post_data (
                { 'DN' => "NsCVcQg4fd", 'Format' => "Base64", 'Password' => "123", 'Key' => "true"}
            )
  header_data (
                  { 'content-type' => 'application/json', 'auth' => node["tpp"]["username"]}
              )
  expected_response_codes [ 200, 201, 400, 401 ]
  action :post
end

However the ["tpp"]["username"] attribute is.
I verified the the node.overide["tpp"]["apikey"] is working as defined here
ruby_block "Extract the TPP API Key" do
  block do
    jsonData = JSON.parse(node["webhooks"]["post_response"])
    jsonData.each do | k, v |
      if k == 'APIKey'
        node.overide["tpp"]["apikey"] = v
      end
    end
  end
  action :run
end

using this:
ruby_block "Print API Key" do
  block do
    print "\nThis should be the key: " + node["tpp"]["apikey"] + "\n"
  end
  action :run
end

Which indeed works perfectly, from within the ruby_block. So its strange why its empty from within the webhooks_request.

OK so I've got this to work now. Basically I have to do this uri lazy {"<IP_ADDRESS>/sdk/data/retrieve?apikey=#{node["tpp"]["apikey"]}"} in order to get the value of the API key from the attribute:
webhooks_request "Get data from TPP" do
  uri lazy {"<IP_ADDRESS>/sdk/data/retrieve?apikey=#{node["tpp"]["apikey"]}"}
  post_data (
                { 'data' => "NsCVcQg4fd"
            )
  header_data (
                  { 'content-type' => 'application/json', 'auth' => "#{lazy {node["server"]["apikey"]} }" }
              )
  expected_response_codes [ 200, 201, 400, 401 ]
  action :post
end

Seems weird and a bit hacky to me though but it works. I was under the impression that the all of the attributes would be available directly once they are set.

Should be able to get this in by Monday, major work being done.

Excellent that will be great.

Fixed in v0.1.3


Fixed typo in build-composer.yml that caused composer install to fail

A bug was introduced in commit e8664798e08421db9fc85edfadaf7958750a8907 that causes composer install to fail when drupalVM was added as a dependency to an existing site.

falset == portmanteau of false and not?
Don't know what I was thinking 🤪


Added aiofiles and uvicorn to setup.py, tested locally, and it's working

Codecov Report

Merging #56 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master      #56   +/-   ##
=======================================
  Coverage   93.98%   93.98%
=======================================
  Files          34       34
  Lines         582      582
=======================================
  Hits          547      547
  Misses         35       35

Impacted Files
Coverage Δ

rfhub2/version.py
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7ed712...f3a9b0a. Read the comment docs.

please don't tell me that previous builds did not work because there was no uvicorn :D

please don't tell me that previous builds did not work because there was no uvicorn :D

yeah...I will speed up with robot - acceptance tests. How about usin some bare docker image and installing package inside?

@MaciejWiczk  yes, acceptance must be run on installed package. Maybe to not require docker in CI pipeline after package is built it could be installed and run inside virtualenv? then it could be deleted after tests are executed.


Store artifact manifests in metadata

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Aug 31, 2017, 18:46
Currently when staging things we walk the extracted artifact frequently using system calls.
Since we have a system for reading back metadata that we encode at build time, we should be using that instead.
First, the Element needs to offer an API for fetching the manifest; calling that API before the element's artifact is in the local cache should cause an error. This API is implemented by storing a new manifest.yaml in the artifact metadata.
Then, we just use the new API in place of utils.list_relative_paths() for reading the artifact manifest.

In GitLab by [Gitlab user @aevri] on Oct 26, 2017, 17:42
I've started to look at implementing this in order to speed up staging.
I'm finding that for my large test case with roughly 100,000 files it spends quite a lot of time loading the manifest.yaml, which is roughly 1MB in size. This makes it even slower than before!
I find that the default 'round-trip loader' takes significantly longer than the 'safe loader':
$ time python3 -c 'from ruamel.yaml import YAML; yaml = YAML(); f = open("/path/to/manifest.yaml"); d = yaml.load(f);'

real    0m19.793s
user    0m19.700s
sys     0m0.110s

$ time python3 -c 'from ruamel.yaml import YAML; yaml = YAML(typ="safe"); f = open("/path/to/manifest.yaml"); d = yaml.load(f);'

real    0m0.787s
user    0m0.740s
sys     0m0.040s

Will it be acceptable to use something other than the round-trip loader for manifest.yaml? It doesn't feel like it would benefit from round-tripping.

In GitLab by [Gitlab user @tristanvb] on Oct 26, 2017, 17:47

Will it be acceptable to use something other than the round-trip loader for manifest.yaml? It doesn't feel like it would benefit from round-tripping.

Sure !
Only nitpick I might have is, please make it something nicely baked into the private _yaml api.

In GitLab by [Gitlab user @aevri] on Oct 27, 2017, 11:09
After more disappointing performance, I discovered it makes a big difference
how you construct your SafeLoader:
$ time python3 -c 'from ruamel.yaml import YAML; yaml = YAML(typ="safe"); f = open("/path/to/manifest.yaml"); d = yaml.load(f);'

real    0m0.787s
user    0m0.740s
sys     0m0.040s

$ time python3 -c 'from ruamel import yaml; f = open("/path/to/manifest.yaml"); d = yaml.load(f, yaml.loader.SafeLoader);'

real    0m12.256s
user    0m12.180s
sys     0m0.060s

The docs showing the 'YAML' usage also say this:

This is the new (0.15+) interface for ruamel.yaml, it is still in the
process of being fleshed out. Please pin your dependency to ruamel.yaml<0.15
for production software.

http://yaml.readthedocs.io/en/latest/basicuse.html
We don't seem to require a particular version of ruamel just now:
https://gitlab.com/BuildStream/buildstream/blob/master/setup.py#L169
I'm guessing that given this, we'd prefer not to use the 'YAML' interface until the
advice changes.
If that's the case, would the Python library json module be acceptable? It
performs very well:
$ time python3 -c 'import json; f = open("/path/to/manifest.json"); d = json.load(f);'

real    0m0.050s
user    0m0.030s
sys     0m0.010s

I can see that with everything else being yaml, you may not be keen on that
inconsistency :)

In GitLab by [Gitlab user @tristanvb] on Oct 27, 2017, 11:39

I can see that with everything else being yaml, you may not be keen on that inconsistency :)

Yeah really not; I suppose however, so long as we pin ruamel to a given specific version, it might be acceptable to pin it to a specific version that is >= 0.15 for the sake of using that API in places where it counts - this gives us a future path forward without breaking artifact format.
I think that for the huge lists you are talking about 0m0.787s vs 0m0.050s is a negligible difference.
Also, I considered just a flat file with a list of paths, but that would be quite undesirable if and when we want to augment that data with attribute information (permissions, xattrs, etc).
You might want to compare the differences in load times:
Not good, it's a list that is not very forward compat:
files:
- filename1
- filename2

Better forward compatibility:
files:
- filename: filename1
- filename: filename2

In GitLab by [Gitlab user @aevri] on Oct 27, 2017, 15:34
Unfortunately with a dict per file, the costs seem to be significant again.
It's slower than utils.list_relative_paths():
$ time python3 -c 'from ruamel.yaml import YAML; yaml = YAML(typ="safe"); f = open("/path/to/manifest.yaml"); d = yaml.load(f);'
real    0m3.053s
user    0m2.940s
sys     0m0.090s

$ python3 -c 'import timeit; path = "/path/to/files"; print(timeit.timeit("list(buildstream.utils.list_relative_paths(" + repr(path) + "))", number=1, setup="import buildstream"))'
2.478195433039218

$ time find /path/to/files > /dev/null
real    0m0.253s
user    0m0.000s
sys     0m0.230s

I had a quick go at adapting list_relative_paths() to use Python 3.5's os.scandir(), it took roughly 0.5 seconds to run instead. We're supporting minimum Python 3.4 though, so not much help.

In GitLab by [Gitlab user @aevri] on Oct 27, 2017, 15:54
Ooh, just avoiding calls to os.path.relpath() makes it much faster. I'll send an MR for that.

In GitLab by [Gitlab user @aevri] on Oct 31, 2017, 12:41
I've made an attempt at persuading you over to JSON for artifact metadata on the mailing list :)
https://mail.gnome.org/archives/buildstream-list/2017-October/msg00029.html

In GitLab by [Gitlab user @aevri] on Nov 2, 2017, 10:07
Following on from your message on the mailing list, does this mean would you accept a patch which read and wrote manifest.yaml using Python's json module? The idea being that we'd just be using it as an optimised yaml parser behind the scenes.

In GitLab by [Gitlab user @tristanvb] on Nov 2, 2017, 10:50
Ok so - while it does mean we can live with a less desirable metadata until such a time we can change to nicely formatted yaml, do we have to jump on that opportunity right away ?
Is this immensely pressing right now ?
Also what about the denormalization approach of using a flat lists also mentioned in that thread, is this an option before we resort to this ?

In GitLab by [Gitlab user @aevri] on Nov 2, 2017, 11:22

Is this immensely pressing right now ?

Nah, definitely more interested in the overall benchmarking.
I thought we could cross this one off easily now with satisfaction after the last message, maybe I just read it the way I wanted to :)
Will leave it for now and circle back when it seems the most important thing. Thanks!

In GitLab by [Gitlab user @danielsilverstone-ct] on Oct 23, 2018, 14:30
It would be interesting to know if the cas based cache and the virtual directory work done to support remote execution affect this at all, and if so if it's in a positive or negative way.

In GitLab by [Gitlab user @tristanvb] on Nov 4, 2020, 10:45
This is quite old, we now have bst artifact list-contents which achieves the purposes without requiring manifests, let's close this.