Creating an Upstash Redis Checkpoint Saver

We are using langchainjs / langgraphjs in a NextJS web app for building an AI Agent.
We want to store the state / history, but since the app runs on Vercel, we cannot use Memory Saver.
So we are trying to implement a Redis Checkpoint saver using Upstash Redis.
We have 2 questions -

Should we be using checkpoint ts or checkpoint id as the row key?

const key = `checkpoint:${thread_id}:${checkpoint_ts_epoch}`;

or
const key = `checkpoint:${thread_id}:${checkpoint_id}`;

Some of the samples suggest using checkpoint ts where as others use checkpoint_id.
We tried logging the checkpoint_id but it looked like a UUID which wouldn't allow sorting to retrieve the latest checkpoint from a list. Yes, we could override the way checkpoint_id is generated, but we would like to minimize the customization.

What's the suggested approach for serializing / deserializing the data?

We used the default serde, which may be using JsonPlusSerializer but the data dumped to redis is binary encoded.
This makes it difficult to understand and debug what's going on. Is there a way to store plain-text (JSON or Stringify-ed JSON) as a value in Redis and retrieve it back?

Hey @krusadellc - we will get some more implementations done next week and will look into Redis as well!

Should we be using checkpoint ts or checkpoint id as the row key
you should use the checkpoint id. They're implemented using UUID v6, which is highly collision resistant and maintains chronological sort compatibility. It should meet your needs out of the box.
What's the suggested approach for serializing / deserializing the data
The binary data you're seeing is the raw bytes of a stringified JSON representation of the data. Here's an example from a postgres checkpointer I wrote where the stringified JSON is put into a node buffer.
To deserialize, you'll want to use load function from @langchain/core, as I did here. Note that in this example I'm converting the raw postgres bytea data type (represented in JS as a node buffer) to a string using node's Buffer.toString().
In your checkpointer implementation you can skip the conversion to byte arrays / buffers, instead reading and writing a raw string in redis.
Still make sure you use the load function from @langchain/core correctly when deserializing though - some of the values being serialized are classes that need their prototype to be set up, unlike plain JS objects and TS types.

@Masstronaut your implementation was super helpful! It looks like a newer version of LangGraph added support for putWrites which also added a new database table.. If anyone else finds this, this is how I got this working with Sequelize:
import type { RunnableConfig } from "@langchain/core/runnables";
import { load } from "@langchain/core/load";
import {
  BaseCheckpointSaver,
  type Checkpoint,
  type CheckpointListOptions,
  type CheckpointTuple,
  type PendingWrite,
  type CheckpointMetadata,
} from "@langchain/langgraph-checkpoint";
import {
  ChatBotCheckpointModel,
  ChatBotCheckpointWritesModel,
} from "../../../../models";

/**
 * Custom LangGraph Checkpoint Saver adapted for Postgres/Sequelize
 *
 * Inspired By:
 * https://langchain-ai.github.io/langgraphjs/how-tos/persistence/
 * https://github.com/langchain-ai/langgraphjs/blob/main/libs/checkpoint-sqlite/src/index.ts
 * https://github.com/Masstronaut/langchain-endpoint/blob/6f8b22cb75bb405ede0e72bb5f308cd410411273/src/agent/postgrescheckpointer.ts
 * https://github.com/langchain-ai/langgraphjs/issues/435
 */
export class SequelizeSaver extends BaseCheckpointSaver {
  async getTuple(config: RunnableConfig): Promise<CheckpointTuple | undefined> {
    const {
      thread_id,
      checkpoint_ns = "",
      checkpoint_id,
    } = config.configurable ?? {};
    let row: ChatBotCheckpointModel | null;

    if (checkpoint_id) {
      row = await ChatBotCheckpointModel.findOne({
        where: { thread_id, checkpoint_ns, checkpoint_id },
      });
    } else {
      row = await ChatBotCheckpointModel.findOne({
        where: { thread_id, checkpoint_ns },
        order: [["checkpoint_id", "DESC"]],
      });
    }

    if (!row) {
      return undefined;
    }

    let finalConfig = config;
    if (!checkpoint_id) {
      finalConfig = {
        configurable: {
          thread_id: row.thread_id,
          checkpoint_ns,
          checkpoint_id: row.checkpoint_id,
        },
      };
    }

    if (
      !finalConfig.configurable?.thread_id ||
      !finalConfig.configurable?.checkpoint_id
    ) {
      throw new Error("Missing thread_id or checkpoint_id");
    }

    const pendingWritesRows = await ChatBotCheckpointWritesModel.findAll({
      where: {
        thread_id: finalConfig.configurable.thread_id.toString(),
        checkpoint_ns,
        checkpoint_id: finalConfig.configurable.checkpoint_id.toString(),
      },
    });

    const pendingWrites = await Promise.all(
      pendingWritesRows.map(async (row) => {
        const record = row.toJSON();
        return [record.task_id, record.channel, record.value] as [
          string,
          string,
          unknown,
        ];
      })
    );

    return {
      config: finalConfig,
      checkpoint: await load(row.checkpoint.toString()),
      metadata: await load(row.metadata.toString()),
      parentConfig: row.parent_checkpoint_id
        ? {
            configurable: {
              thread_id: row.thread_id,
              checkpoint_ns,
              checkpoint_id: row.parent_checkpoint_id,
            },
          }
        : undefined,
      pendingWrites,
    };
  }

  async *list(
    config: RunnableConfig,
    options?: CheckpointListOptions
  ): AsyncGenerator<CheckpointTuple> {
    const { limit, before } = options ?? {};

    const thread_id = config.configurable?.thread_id;
    const whereClause: WhereOptions<ChatBotCheckpointModel> = { thread_id };

    if (before) {
      whereClause.checkpoint_id = {
        [Op.lt]: before.configurable?.checkpoint_id,
      };
    }

    const rows = await ChatBotCheckpointModel.findAll({
      where: whereClause,
      order: [["checkpoint_id", "DESC"]],
      limit,
    });

    for (const row of rows) {
      yield {
        config: {
          configurable: {
            thread_id: row.thread_id,
            checkpoint_ns: row.checkpoint_ns,
            checkpoint_id: row.checkpoint_id,
          },
        },
        checkpoint: await load(row.checkpoint.toString()),
        metadata: await load(row.metadata.toString()),
        parentConfig: row.parent_checkpoint_id
          ? {
              configurable: {
                thread_id: row.thread_id,
                checkpoint_ns: row.checkpoint_ns,
                checkpoint_id: row.parent_checkpoint_id,
              },
            }
          : undefined,
      };
    }
  }

  async put(
    config: RunnableConfig,
    checkpoint: Checkpoint,
    metadata: CheckpointMetadata
  ): Promise<RunnableConfig> {
    const checkpointColumn = Buffer.from(JSON.stringify(checkpoint));
    const metadataColumn = Buffer.from(JSON.stringify(metadata));

    await ChatBotCheckpointModel.upsert(
      {
        thread_id: config.configurable?.thread_id?.toString(),
        checkpoint_ns: config.configurable?.checkpoint_ns,
        checkpoint_id: checkpoint.id,
        parent_checkpoint_id: config.configurable?.checkpoint_id,
        checkpoint: checkpointColumn,
        metadata: metadataColumn,
      },
      {
        conflictFields: ["thread_id", "checkpoint_ns", "checkpoint_id"],
      }
    );

    return {
      configurable: {
        thread_id: config.configurable?.thread_id,
        checkpoint_ns: config.configurable?.checkpoint_ns,
        checkpoint_id: checkpoint.id,
      },
    };
  }

  async putWrites(
    config: RunnableConfig,
    writes: PendingWrite[],
    taskId: string
  ): Promise<void> {
    const rows = writes.map((write, idx) => {
      const serializedWrite = Buffer.from(JSON.stringify(write[1]));
      return {
        thread_id: config.configurable?.thread_id,
        checkpoint_ns: config.configurable?.checkpoint_ns,
        checkpoint_id: config.configurable?.checkpoint_id,
        task_id: taskId,
        idx,
        channel: write[0],
        value: serializedWrite,
      };
    });

    await ChatBotCheckpointWritesModel.bulkCreate(rows, {
      updateOnDuplicate: ["value"],
    });
  }
}

Hi @brandon-pereira @Masstronaut! Do any of you have a way of explaining the Checkpointers to me? I've been pulling my hair out looking through the docs and (JS) code for the MemorySaver, the MongoDB and Postgres version, and I need to set one up for SurrealDB instead.
I'm very confused as to what is being stored where.
Is there a simple breakdown of what the data Type(s) are that is stored on the DB?
Before Checkpointers existed, I was previously storing each message in a separate collection, and a thread which contains list of messages which are linked (via relations in SurrealDB) which kept the duplication of data super slim, and I was culling all the crap that comes with LangChain types out of it.
With Checkpointers, I need to retain all that stuff, but I'm lost understanding where data transformations are occurring, what's being duplicated, where etc. , what Data Type needs to be stored, what Data Type is retrieved (or "assembled" upon retrieval).
Anyone able to help with an overview of this? I'm currently trying to mirror whats happening in the MemorySaver version (with SERDE gutted from it) but my data looks very different.
I'd love to know/see an example of: whats going IN. whats coming out (and how/if it needs transforming).
*I hope this is not too much of a noob question.

EDIT: I think I have found my saving grace - the Sqlite Saver.. I'm just following through the code and swapping parts out. I think it might be at least be good starting point 🤷🏾♂??
Any gotcha's I should be weary of? (hoping to setup the Checkpointer for Production use eventually)

Hey yes the SQLite one is a good place to start - definitely need better docs around the mechanics.
There's still a few improvements around structure from the Python library to port over but what's there should work just fine.

Hey yes the SQLite one is a good place to start - definitely need better docs around the mechanics.
There's still a few improvements around structure from the Python library to port over but what's there should work just fine.

🤞🏾I'm still making my way through it. I noticed the code on Github contains all the Types for reference so this has helped! (When clicking through the Class that is inside of the node module the Types are not available << not sure if this is a normal in the code of a package)

@jacoblee93 I've refactored my code down but the putWrites is still confusing...
Looking through the Sqlite Memory Saver, what are the Types for rows in here?
async putWrites(
		config: RunnableConfig,
		writes: PendingWrite[],
		taskId: string
	): Promise<void> {
		debugLog(debug_checkpointer, `${serviceName} putWrites▶`)

		const stmt = this.db.prepare(`
      INSERT OR REPLACE INTO writes
      (thread_id, checkpoint_ns, checkpoint_id, task_id, idx, channel, type, value)
      VALUES (?, ?, ?, ?, ?, ?, ?, ?)
    `);

		const transaction = this.db.transaction((rows) => {
			for (const row of rows) {
				stmt.run(...row);
			}
		});

		const rows = writes.map((write, idx) => {
			const [type, serializedWrite] = this.serde.dumpsTyped(write[1]);
			return [
				config.configurable?.thread_id,
				config.configurable?.checkpoint_ns,
				config.configurable?.checkpoint_id,
				taskId,
				idx,
				write[0],
				type,
				serializedWrite,
			];
		});

		transaction(rows);
	}
}

I'm trying to translate this code to and SurrealDB query

UPDATE 1: I've got something working right now with SurrealDB. However, am I correct in that with a simple Chatbot setup (Input >> Agent Node >> End), if I send 1 msg, and get 1 replay back (where 0 tools are invoked) my database stores:
3 checkpoints records
and 4 writes records

A. Have I done something wrong here?
B.  Does it need to be?
C. Is it normal?
In my previous setup I would have stored 3 records:
User msg,
AI msg (tool calls + answer merged into this),
Thread (which links to these 2 messages - no storage duplication), however I didn't have checkpointing/replay/branching / don't know if this is possible?


[Course statistics] Create a more unified version for the settings

The settings are kind of messy. Also the "Help" button could be replaced with the "Show more" button that is used on other pages.

33b7f4432921920bd7c309ad7514d2bf03ae7579


Logical OR between repeated conditions

Is there a reason at line below for performing the same condition with an OR?
https://github.com/openssl/openssl/blob/bae39163409ac3b8a1c579c2bcfbdae35370a133/ssl/statem/extensions_clnt.c#L236

This is closing the two sub-packets started a bit above this point:
https://github.com/openssl/openssl/blob/bae39163409ac3b8a1c579c2bcfbdae35370a133/ssl/statem/extensions_clnt.c#L214-L215
The first sub-packet if for the whole supported groups extension. The second is for the actual list of groups contained within that extension.


adding toJsonPretty function variant of toJson

What this PR does / why we need it:
Adding toJsonPretty, which is a copy of toJson, but instead of json.Marshal is using json.MarshalIndent to pretty-print JSON
closes #8229
Special notes for your reviewer:
Perhaps, it would be nice to add a way to specify the prefix and indent..
If applicable:

[ ] this PR contains documentation
[X] this PR contains unit tests
[ ] this PR has been tested for backwards compatibility

ugh.. found toPrettyJson function in https://github.com/Masterminds/sprig/blob/master/functions.go#L222, which seems to work just fine


how can i  solve this problem : process exited with exit code 9

My project has been launched, and it is running normally.
It can be started by normal : node www.js,
or it can be started by : pm2 www.js.
i use command: clinic doctor -- node www.js
,then print these:
node: --trace-event-categories is not allowed in NODE_OPTIONS
/root/nvm/versions/node/v8.6.0/lib/node_modules/clinic/bin.js:209
if (err) throw err
^
Error: process exited with exit code 9
at ChildProcess. (/root/nvm/versions/node/v8.6.0/lib/node_modules/clinic/node_modules/@nearform/clinic-doctor/index.js:65:13)
at Object.onceWrapper (events.js:318:30)
at emitTwo (events.js:125:13)
at ChildProcess.emit (events.js:213:7)
at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)

You should use node 9.4.0+ - in some cases this would also work on Node 8.9.4 See https://github.com/nearform/node-clinic-doctor#supported-node-versions.

Thank you, mcollina


TextEmbed incompatible with Urwid 2.5.3

Description
See issue in Urwid
https://github.com/urwid/urwid/issues/802

TextEmbed should work as good as before now. Alongside the fix, I added the ability to override the codepoints used for widget placeholders.
I just want to include a couple more updates and release at once.
Thanks.

Released v0.2.0.


Uncaught TypeError: Cannot read properties of undefined (reading 'VERSION')

WA-JS Version
3.1.1
WhatsApp Version
2.3000.1012938617
Browser Version
Chrome 124.0.6367.62
Operation System
macOS 13.3.3
Steps to reproduce

Install the Demo extension
Open URL: https://web.whatsapp.com/

Demo.zip
What is expected?
it doesn't fail and inject the code successfully
it points to this line in the code (self.Debug is undefined)
https://github.com/wppconnect-team/wa-js/blob/main/src/webpack/index.ts#L163
https://github.com/wppconnect-team/wa-js/issues/1801
What is actually happening?
fails and keeps WPP global object undefined
Any additional comments?
No response

@javix64
you are right.
Paste the wppconnect-wa.js code in the console, and it should run smoothly.
It won't work properly when injected using Tampermonkey and Chrome extensions.

Hey @code4you2021 I have found a low quality solution for this issue.
If you search in the entire project self.Debug.VERSION, it will appear only two result, change it for a higher version. For example:
compare(self.Debug.VERSION, '2.3000.0', '>=') ==> compare('2.3000.1', '2.3000.0', '>=')
Let's see what the maintainers need to say about this issue. :smile:


Conversion TODO

[ ] removal of jQuery dependency
[ ] 100% test coverage
[ ] bower availability
[ ] Travis CI build with status

Still no updates?

Sorry, unfortunately this has fallen by the wayside. I'd like to get work done on it, but it'll likely continue to be a while. If anybody wants to lend a hand, though, I'd certainly welcome it.


Active menu button is not highlighted

Checklist

[X] I have searched the existing issues for similar issues.
[X] I added a very descriptive title to this issue.
[X] I have provided sufficient information below to help reproduce this issue.

Summary
The first time the chat menu button is selected, it is not highlighted as active.
Reproducible code example
import streamlit as st
from streamlit import session_state as ss
from streamlit_navigation_bar import st_navbar
import random
import string

st.set_page_config(layout='centered')

if 'msg' not in ss:
     ss.msg = []

def generate_random_response(msg):
    letters = string.ascii_lowercase
    return f'my response to {msg} is ' + ''.join(random.choice(letters) for _ in range(6))

def main():
    page = st_navbar(["Home", "Documentation", "Chat", "About"])
    st.write(page)

    if page == 'Chat':
        # Show message history.
        for message in ss.msg:
            with st.chat_message(message["role"]):
                st.markdown(message["content"])

        # user prompt
        if prompt := st.chat_input("enter your prompt"):
            ss.msg.append({"role": "user", "content": prompt})
            with st.chat_message("user"):
                st.markdown(prompt)

            # Get response.
            with st.chat_message("assistant"):
                response = generate_random_response(prompt)
                st.markdown(response)
                ss.msg.append({"role": "assistant", "content": response})

if __name__ == '__main__':
    main()

Steps to reproduce

run the app
select chat
the chat menu button is not highlighted

Expected behavior
If chat is selected is should highlight the chat menu button as active.
Current behavior
No response
Is this a regression?

[ ] Yes, this used to work in a previous version.

Debug info

Streamlit version: 1.33
Streamlit Navigation Bar version: 3.1.2
Python version: 3.11
Operating System: windows 10
Browser: chrome

Additional information
No response

Hey @fsmosca, thanks for opening the bug report! I will take a look at this as soon as possible.

Just an update, I was able to reproduce the bug with the following code:
import streamlit as st
from streamlit_navigation_bar import st_navbar

page = st_navbar(["Home", "Chat"])
st.write(page)

if page == "Chat":
    prompt = st.chat_input("Say something")

But I'm still working on the solution.

Hi, I have the exact same issue, and I can confirm that is the chat_input element to be the source of the problem. Has someone found any kind of solution yet? Thank you.

I am having the same problem and I also think the problem is any type of chat_input on the page. I really liked this navigation bar code but its almost impossible to use it in a meaningful way because it is not reliable for any page which includes chat_input.
Thanks for the great code any ways.


Redirect to a directory

When migrating from a different site, one might have a URL such as https://example.com/articles/my-article.
Currently, if one specifies the URL to redirect from /articles/my-article, it will create an extensionless HTML file. The web server might not be able to recognise it as an HTML file, and would need to be configured to do so. This can be a pain.
My proposal is to detect whether a redirect has an extension or not. If it does not have one, then create a directory named "my-article", and generate the redirecting HTML file inside of it as index.html.

Hi! Thanks for reporting this. I agree it's an important case 😊 what about just adding the index.html into the end of your string, like https://example.com/articles/my-article/index.html? I hope I didn't misunderstand the question ☺

Hey, first off, thanks for writing this plugin; it makes migrating existing blogs much easier. I'm still evaluating Pelican, but because I'm more comfortable with Python than Ruby or Go, it's a strong contender for me.
You make a valid point. It think having the plugin generate the index.html for you is just more intuitive (and one less thing for the content creator to worry about).

I'm glad it's helpful! I think it makes sense to go along with other similar software, so I left a few comments on the PR 😄


getMuiTheme in material-ui@next (Not found)

Problem description
Hi,
in material-ui@next
import getMuiTheme from 'material-ui/styles/getMuiTheme';
Module not found: Error: Can't resolve 'material-ui/styles/getMuiTheme'
The getMuiTheme in not present in styles folder
https://github.com/callemall/material-ui/tree/next/src/styles
Link to minimally-working code that reproduces the issue
getMuiTheme(darkBaseTheme)

Versions

Material-UI: ^1.0.0-alpha.8
React: ~15.4.2
Browser: Chrome

That's right, we haven't updated the documentation. I'm gonna close that issue for #6249. We should definitely do so.

You can take a look at the source code. Replace getMuiTheme with createMuiTheme and create a palette with createPalette. Here's an example.
import MuiThemeProvider from 'material-ui/styles/MuiThemeProvider'
import createMuiTheme from 'material-ui/styles/theme'

import {grey, amber} from 'material-ui/styles/colors'

import createPalette from 'material-ui/styles/palette'

const muiTheme = createMuiTheme({
    palette: createPalette({
      primary: grey,
      accent: amber,
      error: red,
      type: 'dark'
    })
})

ReactDOM.render(
    <MuiThemeProvider theme={muiTheme}> //muitheme changed to theme
      <div />
    </MuiThemeProvider>,
    document.getElementById('root')
)

Maybe this can get you started, while you wait until the docs are updated.
If you want to learn more, take a look at the source code:
https://github.com/callemall/material-ui/tree/next/src/styles

It's seems I need to install themes? but how?  I got these errors:
Module not found: 'material-ui/styles/theme'
 @ ./src/App.js 59:13-48
Error in ./src/App.js
Module not found: 'material-ui/styles/colors'
 @ ./src/App.js 63:14-50
Error in ./src/App.js
Module not found: 'material-ui/styles/palette'
 @ ./src/App.js 65:15-52

@webmobiles the paths have changed. Update your code to the latest v1-beta release.

where i can find docs about v1-beta (i've just installed it)

@webmobiles Here: https://material-ui-1dab0.firebaseapp.com/

Still cant read the theme with @next,

@stevematdavies Change it to
import createPalette from 'material-ui/styles/createPalette'
and
import createMuiTheme from 'material-ui/styles/createMuiTheme'

FYI, colors were moved also, this is working for me in 1.0.0-beta.18:
import React, {Component} from 'react';
import PropTypes from 'prop-types';
import MuiThemeProvider from 'material-ui/styles/MuiThemeProvider'
import createPalette from 'material-ui/styles/createPalette'
import createMuiTheme from 'material-ui/styles/createMuiTheme'
import {grey, amber, red} from 'material-ui/colors'

const muiTheme = createMuiTheme({
	palette: createPalette({
		primary: grey,
		accent: amber,
		error: red,
		type: 'dark'
	})
})

@dsteinman You can simplify your code with the following:
import React, {Component} from 'react';
import {createMuiTheme} from 'material-ui/styles'
import {grey, amber, red} from 'material-ui/colors'

const muiTheme = createMuiTheme({
	palette: {
		primary: grey,
		accent: amber,
		error: red,
		type: 'dark'
	}
})

I seem to be having an issue using my own colors that are outside of material-ui/colors? Keep getting the error can't assign the string to the type of Color.
I'm using typescript and was wondering if that was intended?

@codepressd It's because a color isn't a string:
https://github.com/callemall/material-ui/blob/f0b167dc855df2960273a53f59e6d2034b68b2eb/src/index.d.ts#L19-L35

@oliviertassinari I tried that and they were undefined:
import {
	MuiThemeProvider,
	createMuiTheme
}                      from "material-ui/styles";

Both MuiThemeProvider and createMuiTheme are undefined ("material-ui": "^1.0.0-beta.19"). Strange, since I can see exports being augmented with those properties in node_modules/material-ui/styles/index.js...

@turnerhayes I have not clue. It's most likely an issue with your build environment. Feel free to post a reproduction repository if you find the root cause on Material-UI side.

@turnerhayes, the following should work. MUI beta changes the file structure. NOTE: low-level defs (e.g., in styles) seem typically to export default class constructors, whereas the top-level index exports all definitions (not the different use of braces):
import { MuiThemeProvider } from 'material-ui';
or
import MuiThemeProvider from 'material-ui/styles/MuiThemeProvider';

use
https://kradio3.github.io/react-mdc-web/components/linear_progress/
and be happy

Versions
Using  1.2.1.
Know this
The styles folder is now under the core directory. Hope this could help.
j031:~/workspace/7omp (add1-basicFrame) $ npm list | grep material
├─┬<EMAIL_ADDRESS>
Seems like there's been a change in the folder structure. You need to do:
import MuiThemeProvider from '@material-ui/core/styles/MuiThemeProvider'

NOT:
import MuiThemeProvider from 'material-ui/styles/MuiThemeProvider'

It's the same for createMuiTheme:
import { createMuiTheme } from '@material-ui/core/styles';

This is the correct path.


Added ifElseOf, whenOf, and unlessOf similar to condOf

Codecov Report

Merging #156 into master will decrease coverage by 0.57%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #156      +/-   ##
==========================================
- Coverage     100%   99.42%   -0.58%
==========================================
  Files           1        1
  Lines        1033     1047      +14
==========================================
+ Hits         1033     1041       +8
- Misses          0        6       +6

Impacted Files
Coverage Δ

dist/partial.lenses.cjs.js
99.42% <0%> (-0.58%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32f53ff...e7699d0. Read the comment docs.


Fix link to reference manual

Thanks for writing these articles!
The link to the reference manual was broken, it seems that the file location has changed.

Thank you so much :-)

Thank you so much :-)


Fails to compile in Linux with Go 1.17rc1

It is impossible to compile a project with github.com/sethvargo/go-limiter dependency using recent Go 1.17rc1.
Sample code:
package main

import (
        "context"
        _ "github.com/sethvargo/go-limiter"
        "log"
        "time"
)

func main() {
        store, err := memorystore.New(&memorystore.Config{
                // Number of tokens allowed per interval.
                Tokens: 15,

                // Interval until tokens reset.
                Interval: time.Minute,
        })
        if err != nil {
                log.Fatal(err)
        }
        ctx := context.Background()
        store.Take(ctx, key)
}

go build .
# golimitfails
github.com/sethvargo/go-limiter/memorystore.(*store).purge: relocation target runtime.walltime not defined
github.com/sethvargo/go-limiter/memorystore.newBucket: relocation target runtime.walltime not defined
github.com/sethvargo/go-limiter/memorystore.(*bucket).take: relocation target runtime.walltime not defined

go version
go version go1.17rc1 linux/amd64

I need to do more research on a suitable replacement for fasttime. This was a known risk of linking to something private, but using time.Now is significantly slower.


Correct java action heap size issue #2389.

limit the heap size to 56m, leave some space for stacks etc.

We have a defect for the heap of java actions https://github.com/apache/incubator-openwhisk/issues/2389 I don't think this fix helps in general and could make things worse relative to actions settings. What do you think?

My fault, I misunderstood the bug meaning. Yes, correct, the best solution perhaps would be a more flexible container/warmup memory strategy. For example, I can think of some user could pay a premium openwhisk service, so their actions could consume more than 512m limit size. But maybe we could set the jvm heap Xmx a little less than 512m as a temp workaround.

Going to close this I don't think it's correct. Reopen if you disagree.


Timezone -- UTC offset

Hello,
when I import photos (taken by my iPhone both in MacOS High Sierra and in Windows 10) the file are renamed using UTC datetime and not the date and time corresponding to the timezone where photos have been taken. I cannot find any comment about this behaviour in the documentation, so I suppose this is the default. Is possible to set datetime according to the timezone where photos have been taken?
Here the details:
C:\Users\aless\elodie>python elodie.py import --debug --destination="C:\sorted" "C:\unsorted\IMG_7107.JPG" C:\unsorted\IMG_7107.JPG -> C:\sorted\2018-11-Nov\Rovigo\2018-11-13_10-29-36-img_7107.jpg****** SUMMARY ******

Hi jmathai,
thank you for your reply.
I took the photo on November 13, 2018 at 11:29 am in Rovigo, Italy.
In November the time offset in Italy is UTC+1 (https://www.timeanddate.com/time/zone/italy).
I took the photo at 11:29 (Italian time) and, as you can see from my previous post, in Windows 10 image properties, "Date taken" is 11:29.
Running EXIFTOOL both on source photo and on destination photo I get:

[EXIF]            306 ModifyDate                      : 2018:11:13 11:29:36
[EXIF]          36867 DateTimeOriginal                : 2018:11:13 11:29:36
[EXIF]          36868 CreateDate                      : 2018:11:13 11:29:36
[Composite]         - SubSecCreateDate                : 2018:11:13 11:29:36.641
[Composite]         - SubSecDateTimeOriginal          : 2018:11:13 11:29:36.641

and

[EXIF]              7 GPSTimeStamp                    : 10:29:36
[Composite]         - GPSDateTime                     : 2018:11:13 10:29:36Z

In my opinion, elodie (as default) should rename the photo based on the time I took the photo, i.e. italian time (11:29) and not on UTC (10:29).
Currently I have:
Filename: 2018-11-13_10-29-36-img_7107.jpg
Winodws 10 image properties, "Date taken": 11:29.
(10-29 in Filename and 11:29 in image properties <-- Filename is not logic, I took the photo at 11:29)
Using local time I would have:
Filename: 2018-11-13_11-29-36-img_7107.jpg
Winodws 10 image properties, "Date taken": 11:29.
(11-29 in Filename and 11:29 in image properties)
So my suggestions are:
a) rename the photos using the local time as default (using for example exif tag "CreateDate" or "DateTimeOriginal");
b) have a configuration variable or command line parameter to let the user choose between local time and UTC.

Yes this is bothering me as well. File creation date is correct, while Elodie changes the date to 'the future' (I'm in CST (UTC -6))


[BUG]AWS IAM role set through IRSA is not used correctly

Describe the bug
We are using Flowise 1.8.3 on EKS (K8s on AWS). The SA is mapped to IAM role through IRSA. As a consequence, these two environment variables are added to the process:
AWS_ROLE_ARN=arn:aws:iam::(accountid):role/(rolename)
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
If we try to use the AWS ChatBedrock without explicit AWS Credentials, we get an error "Role Arn 'arn:aws:iam::(accountid):role/(rolename)' needs to be assumed with web identity, but no role assumption callback was provided."
To Reproduce
Steps to reproduce the behavior:
N/A
Expected behavior
The IRSA role should be used to access Bedrock and other AWS resources.
Screenshots
If applicable, add screenshots to help explain your problem.
Flow
If applicable, add exported flow in order to help replicating the problem.
Setup

Installation EKS v1.27.13
Flowise Version 1.8.3
OS: N/A
Browser N/A

Additional context
Add any other context about the problem here.

I think we had similar fix - https://github.com/FlowiseAI/Flowise/pull/2470 for allowing user to access Bedrock through roles.
@danieldabate, with your fix, bedrock should be able to get these env variables and use it no?

This is a limitation in langchainjs. There is an open discussion there that you might want to upvote 😀
https://github.com/langchain-ai/langchainjs/discussions/5661


Gh-2882: Improve getSchema Javadoc and remove deprecation

Related Issue

Resolve #2882

Codecov Report

:exclamation: No coverage uploaded for pull request base (v2-alpha@9827acc). Click here to learn what that means.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff             @@
##             v2-alpha    #2883   +/-   ##
===========================================
  Coverage            ?   46.35%
  Complexity          ?      304
===========================================
  Files               ?      367
  Lines               ?    14971
  Branches            ?     1191
===========================================
  Hits                ?     6940
  Misses              ?     7614
  Partials            ?      417

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Removed imports that were never used

A few imports were returning warning  'xxx' is defined but never used.

📄 Description
Removed imports that were imported but never used.
🔴 Related Issue

📃 Context
Removes the lint warnings. Not necessarily errors.
🛠 How Has This Been Tested?

Briefly tested on the browser manually.
※ As a site note; we should add some unit tests because non really exist

nice catch.. and thanks for contributing ;D
Well, unit tests can't hurt..
but I see no point in creating them since the website has no actual logic


Could not find test files. Create one and try again

When trying to run tests, I get the following error:
FAIL  src/__type_tests__/pick-deep.test-d.ts
  ● Test suite failed to run

    Could not find test files. Create one and try again

It seems like there isn't anything being found from mlh-tsd in findCustomTestFiles. If, after globby, I output:
    console.log({testFilesPattern, testFiles, cwd})

I get this:
{
  testFilesPattern: [
    'C:\\Users\\crutchcorn\\git\\UnicornUtter\\ts-util-helpers\\src\\__type_tests__\\pick-deep.test-d.ts'
  ],
  testFiles: [],
  cwd: 'C:\\Users\\crutchcorn\\git\\UnicornUtter\\ts-util-helpers'
}

My jest file is this:
module.exports = {
    runner: 'jest-runner-tsd',
    testMatch: ['**/*.test-d.ts'],
};

And my test is indeed in the pick-deep.test-d.ts path
I'm running Windows 11 if that helps

Thanks for report.
Most probably all got fixed in #19. I was wrestling with this error on CI. Here is the problematic line (testFiles must be globs, so slashes had to be normalised): https://github.com/jest-community/jest-runner-tsd/blob/82e6f91f9225a3ec08c3571a144f5d0749e1ee02/src/run.js#L44
Updated code is not yet released.
You can patch the library inside node_modules to see if that helps. Or we could ask @SimenB for a release (;

https://github.com/jest-community/jest-runner-tsd/releases/tag/v1.2.0

That was fast! Thanks y'all!
Don't wanna open a new issue in case this is my fault for not noticing first. Is passing a project to tsd supported in the runner?
Something like this:
https://github.com/SamVerschueren/tsd/issues/32#issuecomment-778650556

Not sure I fully understand your question. Do you try to find out if the workaround will work? It should work. The runner is just a tiny wrapper around tsd.
If something does not work, better open new issue (;

Well, I'm admittedly not sure where to pass the second argument for said workaround 😅 (the project name) in the test runner

Ah.. That’s more simple with the runner. There is no need to create a dummy project. Just point to an empty .d.ts file using a docblock at the top of your test file.
See docs and example (that’s not empty .d.ts, but it should work with empty one too).
Does this help?

Oh! I read that in the docs but didn't make the connection that the two were the same. Apologies!

@crutchcorn Version 2.0.0 is just release. It does not require any work arounds (no more docblock, etc). Just go through the readme carefully. Many details changed.
Give it a try (;


padded dataloader

As discussed in https://github.com/Synthyra/SpeedRunningESM2/pull/3
This removes the issue of a smaller batch size resulting in a larger number of truncations. Should be used for validation / test. Unclear whether it should be used for training - it's more sample efficient, but 4% slower and has fewer tokens per batch.

This code has a few bugs. I think it makes more sense to simply update the data packing script so each .bin file ends with EOS, and begins with CLS.
Does that make sense to you?


Fix imports

The way that manim currently handles imports is far from optimal. This should be one of the first things we change.
In fact, this may be one thing preventing us from having standard manim and community-manim installed side by side on a single machine (see #6).

This causes a problem with people using from manimlib.imports import * at the top of their files.

That is exactly one of the things that I want to fix, and IMHO, using the idiom from ... import * should be discouraged.

Regardless of whether people import * or not, there will be a conflict when people try to import classes from manimlib for their scenes.

What options do we have then? Because importing each file is not feasible given the way most animations make use of mobjects as needed. Having people do something like
from manimlib.mobject.geometry import Square is both time consuming and not really fun to do once you have a lot of mobjects in your scene.

I think we just have to deal with having the conflict.

The alternative is to tidy up the files in such a way that a couple imports will be enough. For most scenes, import manimlib as mn and then doing mn.Square , mn.play, mn.Scene, etc, should work.
For more complicated scenes, it could be necessary to additionally do import manimlib.camera or some such.
@eulertour exactly what kind of conflict are you referring to? I'm not following.

If someone has both versions of manim installed and puts from manimlib.imports import * at the top of their code, then they could end up running one version of manim and importing classes from another, even if our version only uses relative imports internally.

The alternative is to tidy up the files in such a way that a couple imports will be enough. For most scenes, import manimlib as mn and then doing mn.Square , mn.play, mn.Scene, etc, should work.

So like, importing them all into manimlib/__init__.py? Isn't that basically the same thing as importing from manimlib.imports but attaching an alias for manimlib?

Ooooooh gotcha, but I wasn't really thinking about that when I opened that issue. That is being discussed in #6.
For this issue, I think I mean concretely three different things:

It is highly unorthodox to have an imports.py file that just fills the namespace.
All of the internal imports should be local, as at least as many as possible. I just don't think the code base makes good use of python's package architecture at all.
Shift from explicitly encouraging import * to explicitly discouraging it.

I think number 2. here would, maybe a little bit, address the issues of clashing with other manim versions (as discussed in #6), but would certainly not fix it.

1. It is highly unorthodox to have an `imports.py` file that just fills the namespace. Usually, these are in `__init__.py`, as @kilacoda suggests
<snip>
3. Shift from explicitly encouraging `import *` to explicitly discouraging it.

Imo it's an utility that the user can use if they want. It's mostly helpful, even more so with those who are new to manim and are not aware of its structure and just want to get things working. Note that the imports file does import more than just manim classes and whatnot: it also does so for numpy and other math stuff. Of course, the user can just import it by themselves, but again, it's quite helpful for newbies, so it should be an option, just not mandatory, neither discouraged.
To be clear, it is something we'd mention in some sort of tutorial or "Introduction" documentation page.

Fair. I can see the use cases for allowing it / not discouraging it.
I agree this should be very clear in the introduction.
I still think the imports.py file needs to go :)


Fix bug: Property does not work when setting jms pool disabled

Description
Fixes https://github.com/Azure/azure-sdk-for-java/issues/43279
Property does not work when setting spring.jms.servicebus.pool.enabled=false using property source.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

API change check
API changes are not detected in this pull request.

/check-enforcer override

Honestly, this is the most terrible idea of how to fix it. It just complicates everything and looks like you try to fix something broken by something more broken. None of you will understand what you are trying to achieve with this code in a few months. You should be aware that this is a really bad way how to write any autoconfiguration. I am really surprised that it passed through your code review. Isn't it possible to do it simple stupid? I would remove the whole AzureServiceBusJmsPropertiesEnvironmentPostProcessor and use only @ConditionalOnXXX
    @Configuration(proxyBeanMethods = false)
    @ConditionalOnMissingBean(JmsPoolConnectionFactory.class)
    class SimpleConnectionFactoryConfiguration {
      // ...
    }

    @Configuration(proxyBeanMethods = false)
    @ConditionalOnClass({ JmsPoolConnectionFactory.class, PooledObject.class })
    class PooledConnectionFactoryConfiguration {

        @Bean(destroyMethod = "stop")
        @ConditionalOnProperty(prefix = "spring.jms.servicebus.pool", name = "enabled", havingValue = "true", matchIfMissing = true)
        JmsPoolConnectionFactory jmsPoolConnectionFactory(AzureServiceBusJmsProperties properties,
                                                          ObjectProvider<AzureServiceBusJmsConnectionFactoryCustomizer> factoryCustomizers) {
            ServiceBusJmsConnectionFactory factory = createJmsConnectionFactory(properties, factoryCustomizers);
            return new JmsPoolConnectionFactoryFactory(properties.getPool())
                .createPooledConnectionFactory(factory);
        }
    }

Another way is introducing a new configuration property spring.jms.servicebus.connection.factory with three values: POOL, CACHE and SIMPLE.

I think I can explain why we chose to implement this way, but we do want to improve this if there's a better way (sorry for any inconvenience):
Before we used https://github.com/Azure/azure-sdk-for-java/blob/b44403740b282f5c1abbf0c01ccd429aa909a0bb/sdk/spring/spring-cloud-azure-autoconfigure/src/main/java/com/azure/spring/cloud/autoconfigure/implementation/jms/ServiceBusJmsConnectionFactoryConfiguration.java#L35-L81
so the value matrix would be

property/value
spring.jms.servicebus.pool
spring.jms.cache
registered bean type

empty
empty
CachingConnectionFactory

empty
true
CachingConnectionFactory

empty
false
ServiceBusJmsConnectionFactory

true
empty
JmsPoolConnectionFactory

true
true
JmsPoolConnectionFactory

true
false
JmsPoolConnectionFactory

false
empty
CachingConnectionFactory

false
true
CachingConnectionFactory

false
false
ServiceBusJmsConnectionFactory

But we only want to change the default behavior when there's no spring.jms.servicebus.pool or spring.jms.cache provided. If we change the code to
@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(prefix = "spring.jms.servicebus.pool", name = "enabled", havingValue = "false",
    matchIfMissing = false)
static class SimpleConnectionFactoryConfiguration {

    @Bean
    @ConditionalOnProperty(prefix = "spring.jms.cache", name = "enabled", havingValue = "false")
    ServiceBusJmsConnectionFactory jmsConnectionFactory(AzureServiceBusJmsProperties properties,
                                                        ObjectProvider<ServiceBusJmsConnectionFactoryCustomizer> factoryCustomizers) {
        return createJmsConnectionFactory(properties, factoryCustomizers);
    }

    @Configuration(proxyBeanMethods = false)
    @ConditionalOnClass(CachingConnectionFactory.class)
    @ConditionalOnProperty(prefix = "spring.jms.cache", name = "enabled", havingValue = "true",
        matchIfMissing = true)
    static class CachingConnectionFactoryConfiguration {

        @Bean
        CachingConnectionFactory jmsConnectionFactory(JmsProperties jmsProperties,
                                                      AzureServiceBusJmsProperties properties,
                                                      ObjectProvider<ServiceBusJmsConnectionFactoryCustomizer> factoryCustomizers) {
            ServiceBusJmsConnectionFactory factory = createJmsConnectionFactory(properties, factoryCustomizers);
            CachingConnectionFactory connectionFactory = new CachingConnectionFactory(factory);
            JmsProperties.Cache cacheProperties = jmsProperties.getCache();
            connectionFactory.setCacheConsumers(cacheProperties.isConsumers());
            connectionFactory.setCacheProducers(cacheProperties.isProducers());
            connectionFactory.setSessionCacheSize(cacheProperties.getSessionCacheSize());
            return connectionFactory;
        }
    }

}

@Configuration(proxyBeanMethods = false)
@ConditionalOnClass({ JmsPoolConnectionFactory.class, PooledObject.class })
static class PooledConnectionFactoryConfiguration {

    @Bean(destroyMethod = "stop")
    @ConditionalOnProperty(prefix = "spring.jms.servicebus.pool", name = "enabled", havingValue = "true", matchIfMissing = true)
    JmsPoolConnectionFactory jmsPoolConnectionFactory(AzureServiceBusJmsProperties properties,
                                                      ObjectProvider<ServiceBusJmsConnectionFactoryCustomizer> factoryCustomizers) {
        ServiceBusJmsConnectionFactory factory = createJmsConnectionFactory(properties, factoryCustomizers);
        return new JmsPoolConnectionFactoryFactory(properties.getPool())
            .createPooledConnectionFactory(factory);
    }
}

The matrix would be

property/value
spring.jms.servicebus.pool
spring.jms.cache
registered bean type

empty
empty
JmsPoolConnectionFactory

empty
true
JmsPoolConnectionFactory

empty
false
JmsPoolConnectionFactory

true
empty
JmsPoolConnectionFactory

true
true
JmsPoolConnectionFactory

true
false
JmsPoolConnectionFactory

false
empty
CachingConnectionFactory

false
true
CachingConnectionFactory

false
false
ServiceBusJmsConnectionFactory

Yes, we could do that. I admit that the registrar implementation is not as straightforward as auto-configuration, but could you please share what's the current problem function-wise?

It is not a function-wise problem (I have not tested it yet). But if someone checks this code, it is hard to understand and it is not common. It can happen because combinations of JMS factory properties are not clear at first sight. This combination can surprise new users (I understand that you want to be backward compatible):

spring.jms.servicebus.pool
spring.jms.cache
registered bean type

empty
false
ServiceBusJmsConnectionFactory

I just think this code is ugly, hard to maintain and extend. Mainly if there is any better and easier option with a custom @Conditial. But it is just my opinion and it is your code and your decision.


[K32W] Add NXP BLE Adaptation Layer

Add BLE Adaptation Layer for NXP K32W061 platform. Using this PR, K32W can be used for BLE Rendezvous. The Commissioning Parameters are then used for joining a Thread Network.
Note that for K32W we need to manually update OT repo to the latest commit (included documentation in the README). A future PR should update the OT version used by PCHIP - I tried to update the OT stack but then EFR32 stops compiling.
Also, the latest SDK release for K32W is SDK2.6.2. In order to fix the compilation for Project CHIP we are patching this SDK with the files pushed to third_party/k32w_sdk. This is just a temporary step until a newer NXP SDK will be released.

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

Found a workaround using the define "DOXYGEN_SHOULD_SKIP_THIS"

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

Found a workaround using the define "DOXYGEN_SHOULD_SKIP_THIS"

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

Found a workaround using the define "DOXYGEN_SHOULD_SKIP_THIS"

Doxyfile also contains some EXCLUDE_PATTERS and EXCLUDE_SYMBOLS if that helps

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

The Doxygen error seems to be a false positive:
"/home/runner/work/connectedhomeip/connectedhomeip/src/platform/K32W/ble_function_mux.c:54: error: include file gatt_decl_x.h not found, perhaps you forgot to add its directory to INCLUDE_PATH? (warning treated as error, aborting now)"
gatt_decl_x.h is included in the K32W MR2 SDK.

Found a workaround using the define "DOXYGEN_SHOULD_SKIP_THIS"

Doxyfile also contains some EXCLUDE_PATTERS and EXCLUDE_SYMBOLS if that helps

I fixed the review problems. Is there any other change required for having the PR approved?

I fixed the review problems. Is there any other change required for having the PR approved?

Looks like this could be merged, @jelderton @saurabhst can we have final reviews?

Looks like this could be merged, @jelderton @saurabhst can we have final reviews?


Use full url for api.slack.com

Right now if I use your library on a site being served from file:// it tries to hammer file://api.slack.com:80/api/rtm.start

It looks like this is because of https://github.com/substack/https-browserify/pull/2


added mayuri kolhe

Please accept my PR.

Congrats @MayuriKolhe-2003 🎉, your PR got merged. To be eligible for the swags from ClueLess. Follow this steps.

Star this repository. ⭐

SignIn / SignUp in ClueLess Website .

Go to profile section from the top right hand corner.

Comment your profile URL under this comment tagging the person who merged your PR.

That’s it. Follow the leaderboard !
And don’t forget to share your achievement with your friends.

https://www.clueless.tech/profile/admin/100678575974637348384
Merged by @Debajyoti14


fix metricMapsReceived of aggregator

Problem
The aggregator.metricmaps_received values of different workers are always the same due to a typo in the codes.
Fix

Correct the variable typo.
Reset aggregator.metricmaps_received  in the same way with metricsReceived .

This can be 20.1.2, doesn't need a larger bump than that. Can you add a quick update on CHANGELOG.md to note the fix?

Thanks for the quick review. I've updated the metrics doc and change log.


[Feature] Subscribed to link liveness events

Fixes #69
This PR implements the integration with link liveness events as described on this blueprint https://github.com/kytos-ng/kytos/pull/241. The link liveness events from topology's perspective are protocol agnostic, when the time comes to also have liveness via BFD it's expected to be similar as long as we reuse the same event patterns in the future.
Release notes

Topology now reacts to link liveness detection events.
kytos/topology.link_up is only published if link.status is EntityStatus.UP, which takes into account other protocol logical states.

Demo

I'm using a linear topology with 2 switches, once kytos/.*.liveness.up is received liveness goes up, and notice that mef_eline handles it as a link up:

2022-07-08 12:01:44,526 - INFO [kytos.napps.kytos/topology] (thread_pool_app_4) Link liveness up: Link(Interface('s2-eth2', 2, Switch('00:00:00:00:00:00:00:02')), Interface('s1-eth2', 2, Switch('00:00:00:00:00:00:00:01')))
2022-07-08 12:01:44,554 - INFO [kytos.napps.kytos/mef_eline] (thread_pool_app_0) Event handle_link_up kytos/topology.link_up
kytos $>

Also, if you try to find a path via pathfinder it works as you'd expect:
❯ echo '{"source": "00:00:00:00:00:00:00:01:1", "destination": "00:00:00:00:00:00:00:02:1"}' | http http://<IP_ADDRESS>:8181/api/kytos/pathfinder/v2
HTTP/1.0 200 OK
Access-Control-Allow-Origin: *
Content-Length: 196
Content-Type: application/json
Date: Fri, 08 Jul 2022 15:03:21 GMT
Server: Werkzeug/1.0.1 Python/3.9.12

{
    "paths": [
        {
            "cost": 5,
            "hops": [
                "00:00:00:00:00:00:00:01:1",
                "00:00:00:00:00:00:00:01",
                "00:00:00:00:00:00:00:01:2",
                "00:00:00:00:00:00:00:02:2",
                "00:00:00:00:00:00:00:02",
                "00:00:00:00:00:00:00:02:1"
            ]
        }
    ]
}

Now simulating a liveness down event kytos/.*.liveness.down:

kytos $> 2022-07-08 12:03:45,189 - INFO [kytos.napps.kytos/topology] (thread_pool_app_1) Link liveness down: Link(Interface('s2-eth2', 2, Switch('00:00:00:00:00:00:00:02')), Interface('s1-eth2', 2, Switch('00:00:00:00:00:00:00:01')))
2022-07-08 12:03:45,213 - INFO [kytos.napps.kytos/mef_eline] (thread_pool_app_0) Event handle_link_down kytos/topology.link_down
kytos $>

Notice that mef_eline handles it as a link down, and then if try to find a path via pathfinder it wouldn't return any path in this linear topology:
❯ echo '{"source": "00:00:00:00:00:00:00:01:1", "destination": "00:00:00:00:00:00:00:02:1"}' | http http://<IP_ADDRESS>:8181/api/kytos/pathfinder/v2
HTTP/1.0 200 OK
Access-Control-Allow-Origin: *
Content-Length: 13
Content-Type: application/json
Date: Fri, 08 Jul 2022 15:03:52 GMT
Server: Werkzeug/1.0.1 Python/3.9.12

{
    "paths": []
}

@viniarck this PR looks good to me, except for the handling of link up / liveness up strategy. As we discussed in the blueprint PR (kytos-ng/kytos#241 (comment)), it looks like the way it is implemented, the topology will generate a LinkUP event right away after seeing the first liveness_up event. Please check if this is the expected behavior, especially considering the requirements for blueprint EP016 (Link UP Timer).
For the production network, seeing the link as UP may not be sufficient to start using that link immediately: the Link may be in an intermediate status where flaps can still occur, and we would rather wait a bit before actually using that link (some services might be using a backup path at that point).

@italovalcy, thanks for your initial feedback and bringing that up, yes, this will be covered too (I'll map a new issue since it'll need to refactor existing code), I've added more details on this reply https://github.com/kytos-ng/kytos/pull/241#discussion_r922375008, let me know if you have additional questions or any other points that you're seeing.


Submitted "Magical Land of Yeld" Sheet isn't showing up in Roll20's sheet dropdown

I submitted a sheet successfully in early May https://github.com/Roll20/roll20-character-sheets/pull/6340
It seems to have got pulled correctly but the sheet doesn't show up in the dropdown. I have a similar issue with a Mouseguard sheet and just wanted to see if this is a known issue with new sheets.

#6881 has been added as a PR to fix this.


[Project Addition] : Fake and Real Face Detection using DL

Pull Request for DL-Simplified 💡
Issue Title : Fake and Real Face Detection using DL

Info about the related issue (Aim of the project) :  To identify fake and real faces which present a harder challenge to classify them correctly even for the human eye.
Name:  Aaradhya Singh
GitHub ID: https://github.com/kyra-09
Email ID<EMAIL_ADDRESS>Idenitfy yourself: (Mention in which program you are contributing in. Eg. For a JWOC 2022 participant it's, JWOC Participant)  GSSOC-2024 Contributor

Closes: #725
Describe the add-ons or changes you've made 📃

Data Preparation: Collect and preprocess a balanced dataset of real and fake face images, including normalization, resizing, and augmentation.
Base Model Selection: EfficientNetB0,VGG16 , Xception , InceptionV3 like 5 different models excluding its top layers, to leverage its learned features.
Model Construction: Add custom layers on top of the base model for binary classification, compiling with appropriate loss and metrics.
Initial Training: Train the model with the base layers frozen to only update the new layers.
Fine-Tuning: Unfreeze some or all of the base model layers and continue training with a lower learning rate to fine-tune the entire network.
6.) EDA analysis.
7.) Comaprioson using performance matrices such as accuracy scores , confusion matrix etc.

Type of change ☑
What sort of change have you made:

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested? ⚙
Describe how it has been tested
Describe how have you verified the changes made
Checklist: ☑

[x] My code follows the guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly wherever it was hard to understand.
[x] I have made corresponding changes to the documentation.
[ ] My changes generate no new warnings.
[x] I have added things that prove my fix is effective or that my feature works.
[ ] Any dependent changes have been merged and published in downstream modules.

Hi @kyra-09 please update the project folder name as I have updated the issue name. Please do the needful.

okay.. i'll do it right now.

@abhisheks008 Thank you for your time and work . You have been most helpful and consistent with assigning and reviewing our PRs .


Accommodating time series projects in the Lab

I see the biggest design challenge coming up as the "Run" interface on the Lab. I think building tools will actually be pretty straightforward, but populating them with inputs will be really hard. The reason is that Tools want extremely specific inputs (e.g. one band from one scene) while users think more broadly in terms of projects. And there's a more fundamental problem: if one of our primary use cases is change over time (detecting change, documenting change, comparing before/after) then there is additional overhead for the user to create a new project for every point in the time series they are working with.
The intuitive workflow when you're first starting out with the app is to create a single project and add all data in a time series to that single project (e.g. the last four months of Landsat data over an area that had a forest fire to see how recovery is going). But once you start trying to analyze that in the lab, you find that you can only see whichever scene happens to be stacked at the top of the z-order index. If you want to compare slices of time, you have to create separate projects (e.g. Forest Fire - February, Forest Fire - March). This is untenable for long periods of time, you may have to create dozens or hundreds of projects.
One way to skirt this issue is to put more functionality into the Inputs design of a Tool. If you could have the option to select a project, and then optionally select a scene within that project, you could use a single project as multiple time series inputs to different parts of a tool. I could even see this interface looking like the "Browse" view where it's limited only to the scenes already added to the project you've selected: a list on the left ordered by date and a map view on the right where you could select scenes spatially.

You could also apply filters to the scenes within a project. For example, you could put a time filter on the project and then apply the result as an input to a node. Done for each step, this would allow you to accomplish a time-series like result.
This will be a fun problem to solve I think.

@designmatty brought up something I had totally forgotten about--a timeline at the bottom of the map sort of like Planet Labs has on their explorer. So you could scrub forwards/backwards and that would control the Z-order of the scenes stacked on top of each other.

I like the filtering approach; I'm not sure how much of that we'd want to expose to the user because it could get really complicated, but it would allow us to do things like a time scrubber.

It seems to me that we should introduce Alex's scrubber from way back when to handle this issue.

Yeah I agree with that. Is the idea then that when choosing a source there would be some way to access the scrubber to filter the source in some way?
Might be helpful to have context in the scrubber for already selected nodes so you know where to not choose overlap or something.


💂♂ Update (out of date) Sentry Gem

The sentry-raven gem ought to be changed to the sentry-ruby.
The former gem is outdated.
Definition of Done

[ ] Gem updated
[ ] Test functionality still works as expected

Update captured in this PR.
TLDR; this gem was added in 2020 for local testing purposes. It's not used and errors are handled in the DNS and DHCP containers using sentry-ruby.


QCL new cluster terraform config

Add terraform config for new QCL cluster
See #2287

Re kube version I created (and will rerun with fixed node naming having deleted the nb pools)
kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.24.9-gke.3200


Terminal not working in android

Describe the bug
In android tablet the terminal is not opening, so unable to run the nodejs application
Link to the blitz that caused the error
https://stackblitz.com/edit/node-5coxvq
Steps to reproduce
Go to stackblitz and open or create a nodejs project in andriod tablets.  There is no terminal found to run.
Expected behavior
It should allow to run as it is in website
Parity with Local

[ ] I have run the project in my local machine and I could not reproduce the issue.

Screenshots
No response
Platform
Browser name  = Chrome
Full version  = 97.0.4692.98
Major version = 97
navigator.appName = Netscape
navigator.userAgent = Mozilla/5.0 (Linux; Android 11; SM-T733) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.98 Safari/537.36
performance.memory = {
  "totalJSHeapSize": 53500000,
  "usedJSHeapSize": 50400000,
  "jsHeapSizeLimit": 949000000
}
Hash = 5df97b5e61f923919ea927c406a4806b792de8c6

Additional context
No response

Duplicate of https://github.com/stackblitz/webcontainer-core/issues/447


Trino support

Describe the feature
It would be nice if GORM supports Trino
Motivation
Trino is an MPP (Massive Parallel Processing) database that supports connecting to a variety of data sources like Relational Databases, NoSQL, and endpoints like Kafka Topics or Prometheus. This can be a really good addition for Analytical tasks using the Go language.

It's a good idea :stuck_out_tongue:

We are not going implement officially support for Trino, but contributions welcome ;)


React/RCTDefines.h file not found

Summary

Version
0.21.2

Affected OS
iOs

OS Version
10.13.4

Current behavior

When i build in ios, I got the error that 'React/RCTDefines.h' file not found. Actually i don't know if it's a problem with my config or with the extension :(
Expected behavior

It should compile ^^

ok downgrade to v0.20.0 makes it work... Don't know why... Well thanks...


README is too crowded - reorganize into a separate GitHub pages docs site

I suggest keeping the README short and to the point: badges showing status etc. of the package, what is the purpose of the package, how to install, 1-2 basic code examples, and contributing information.
More examples, function API docs, and detailed descriptions can be moved to the docs site.
I also suggest adding a gallery to the docs, similar to Seaborn.

I've split out the advanced configuration details in the README.md into a separate linked-to CONFIGURATION.md file in the repository, keeping the focus on the quickstart as much as possible. I won't do it right now, but yeah, maybe I'll set up a website for this repo eventually. I'll leave this issue open for nowc as a prompt to do so.


How to import L1FilterPruner ?

Environment: VS Code

NNI version: 3.0
Training service (local|remote|pai|aml|etc): remote
Client OS: Windows
Server OS (for remote mode only): Ubuntu
Python version: 3.9
PyTorch/TensorFlow version: PyTorch 1.12
Is conda/virtualenv/venv used?: conda
Is running in Docker?: No

Hi, I am trying to use L1FilterPruner but I can't import it ? Have you removed it ? I have try to import it like other users did in other issues but I am not able to do it.
from nni.compression.pytorch import L1FilterPruner

Hello @gkrisp98 , if you are using nni 3.0rc1, please using https://nni.readthedocs.io/en/v3.0rc1/reference/compression/pruner.html#l1-norm-pruner

Hello @gkrisp98 , if you are using nni 3.0rc1, please using https://nni.readthedocs.io/en/v3.0rc1/reference/compression/pruner.html#l1-norm-pruner

Hi @J-shang , thanks for your response. I want to apply iterative pruning to an object detection model. The code I am using is the following :
def iterative_pruning_finetuning(model, num_iterations=30):
    for i in range(num_iterations):

        print("Pruning and Finetuning {}/{}".format(i + 1, num_iterations))

        print("Pruning...")
#--------------------------------- PRUNING ------------------------------------
        config_list = [{
            'sparsity' : 0.2,
            'op_types' : ['Conv2d'],
        }, {
            'exclude' : True,
            'op_names' : ['loc.0', 'loc.1', 'loc.2', 'loc.3', 'loc.4', 'loc.5',
                        'conf.0', 'conf.1', 'conf.2', 'conf.3', 'conf.4', 'conf.5']
        }]

        pruner = L1NormPruner(model, config_list)
        _, masks = pruner.compress()
        pruner._unwrap_model()  #this is probably not needed - if it is not used, probably the zero's will not get value
        print('Pruning done')

        # Calculate and print the model sparsity
        sparsity = calc_model_sparsity(model)
        print("Model sparsity: {:.2f}%".format(sparsity * 100))

        #torch.save(model.state_dict(), './weights/iter_pruning{}.pth'.format(i))
#--------------------------------- FINE TUNING ---------------------------------
        print("Fine-tuning...")
        train(model, i)
        torch.save(model.state_dict(), './weights/iter_prune{}.pth'.format(i))

        # Use the pruned model as the input for the next iteration
        pruned_model = build_extd('train', cfg.NUM_CLASSES)
        pruned_model.load_state_dict(torch.load('./weights/iter_prune{}.pth'.format(i)))
        model = pruned_model
        #print(model)

        print("New Model sparsity after fine tuning: {:.2f}%".format(sparsity2 * 100))

    return model, masks

Is this algorithm wrong ? At each iteration when I prune and unwrap the model, it's sparsity becomes 16% (I set it to 0.2). After training, some filters are updated and the new sparsity becomes lower (for example now it is 3.17%) but at each iteration the sparsity is getting bigger, which is the wanted outcome although it is a little bit slow. But, when I prune the already pruned model (with some sparsity) the new sparsity becomes again 16%. Is this behaviour normal ? Am I gonna get a good result if I leave it for let's say 200 iterations ?
Thanks agaim for your time, your help and your great work!

hello @gkrisp98, Pruner will not real prune your model, it generates masks and multiply the masks and weights during forward. And after pruner.unwrap_model, the forward will recover to the original forward, that is why zero-weight are updated.
If you want to train your model with simulated pruning, please don't unwrap_model.
If you want to do the real pruning, please using ModelSpeedup https://nni.readthedocs.io/en/stable/tutorials/pruning_speedup.html

hello @gkrisp98, Pruner will not real prune your model, it generates masks and multiply the masks and weights during forward. And after pruner.unwrap_model, the forward will recover to the original forward, that is why zero-weight are updated.
If you want to train your model with simulated pruning, please don't unwrap_model.
If you want to do the real pruning, please using ModelSpeedup https://nni.readthedocs.io/en/stable/tutorials/pruning_speedup.html
The sparsity is 16% maybe because if one more filter is pruned, the sparsity will exceed 20%, nni will not exceed the sparsity set.

Thanks again for your response. Doesn't 'unwrap_model' just multiply the masks with the filters ?
With some iterations already run it seems that the model is learning not to update some zero filters because sparsity at every iteration is increasing. So, doesn't that tell us that after reaching enough iterations the model will have the wanted sparsity?
Also, for the 'ModelSpeedUp' I tried to use it but when I use CPU the kernel crashes, and when I use CPU I get the error CUDA out of memory. Do you have any suggestions how to avoid it ?

Hello @gkrisp98, yes, unwrap_model doesn't multiply the masks with the filters, the sparsity is increasing maybe because the gradient is zero by some dead node chain.
You could show your error message about speedup and the code for using speedup.

hello @gkrisp98 , if you want to always have something like output = xx_op(input, weight * mask), you could forward with the wrapped model, that is, do not unwrap the model.
For the speedup, you could have a try to reduce the dummy input batch size, or just put your model on cpu when speedup.

hello @gkrisp98 , if you want to always have something like output = xx_op(input, weight * mask), you could forward with the wrapped model, that is, do not unwrap the model.
For the speedup, you could have a try to reduce the dummy input batch size, or just put your model on cpu when speedup.

Hi @J-shang, first of all -again- thank you for your time and your useful response. So to get it the right, the following code (used as it is, outside of any for loop) does not zero out parameters in the model ?
from nni.algorithms.compression.v2.pytorch.pruning import L1NormPruner
pruner = L1NormPruner(model, config_list)
_, masks = pruner.compress()
pruner._unwrap_model()

And if it doesn't, why after using the code I provided in my previous answer -for calculating sparsity- it tells me that there are indeed some zero parameters in the model ? Also, the performance of the model drops, so I am guessing that there are zero parameters. Is there any case that these zeros are not actual zeros and they arrise dead node chains as you mentioned earlier ?
Eventually, what I want to do is to let the pruned weights to gain values in the fine tuning proccess, and see if after many iterations the model will learn to not to give values to some weights and finally apply again the pruning criterion to get the final model. I saw in the code that you have a function called mul_mask() . Is this what I am looking for ?
For the speedup, when I try to use dummy_input = torch.rand(1,1,640,640) I get the following error:
RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[1, 1, 640, 640] to have 3 channels, but got 1 channels instead

Hello @gkrisp98 , Sorry for the misunderstanding. Under the code you show, the masked parameters in the model will be set to 0, but if you perform finetuning, these set to 0 values may be updated to non-zero values. But if that's what you want (weight value set to 0 can be updated to a non-zero value again), your code is fine.
For the function mul_mask, it is a util function for the next compression framework, don't mind it.
For speedup, seems that your model need inputs like [n, 3, 640, 640], please make sure n>=2. Note that speedup need an unwrapped model, so make sure you have pruner._unwrap_model before speedup.

Hi @J-shang, I was trying to implement 'soft pruning' as I saw it from this paper:

https://arxiv.org/pdf/1808.06866.pdf

Therefore, I believe I am following the correct procedure.
For the speedup again, after using input with [2,3,640,640] , cuda still runs out of memory. Is there any other way to overcome this ?
I am posting again the error I get :
/usr/local/lib/python3.10/dist-packages/nni/compression/pytorch/speedup/v2/model_speedup.py in speedup_model(self)
    385             self.logger.info('Infer module masks...')
    386             self.initialize_propagate(self.dummy_input)
--> 387             self.propagate_originally()
    388             self.initialize_update_sparsity()
    389             self.update_direct_sparsity()

/usr/local/lib/python3.10/dist-packages/nni/compression/pytorch/speedup/v2/model_speedup.py in propagate_originally(self)
    228                 tree_map_zip(lambda t: t.clone().detach() if isinstance(t, torch.Tensor) else poss_deepcopy(t, self.logger), output)
    229             self.node_infos[node].output_masks = \
--> 230                 tree_map_zip(lambda t: torch.ones_like(t).clone().detach() if isinstance(t, torch.Tensor) else None, output)
    231
    232             if self.garbage_collect_values:

/usr/local/lib/python3.10/dist-packages/nni/compression/pytorch/speedup/v2/utils.py in tree_map_zip(fn, *pytrees)
     72     assert len(pytrees) > 0
     73     if len(pytrees) == 1:
---> 74         return tree_map(fn, pytrees[0])
     75     else:
     76         flat_args_list, spec_list = [], []

/usr/local/lib/python3.10/dist-packages/torch/utils/_pytree.py in tree_map(fn, pytree)
    194 def tree_map(fn: Any, pytree: PyTree) -> PyTree:
    195     flat_args, spec = tree_flatten(pytree)
--> 196     return tree_unflatten([fn(i) for i in flat_args], spec)
    197
    198 Type2 = Tuple[Type[T], Type[S]]

/usr/local/lib/python3.10/dist-packages/torch/utils/_pytree.py in <listcomp>(.0)
    194 def tree_map(fn: Any, pytree: PyTree) -> PyTree:
    195     flat_args, spec = tree_flatten(pytree)
--> 196     return tree_unflatten([fn(i) for i in flat_args], spec)
    197
    198 Type2 = Tuple[Type[T], Type[S]]

/usr/local/lib/python3.10/dist-packages/nni/compression/pytorch/speedup/v2/model_speedup.py in <lambda>(t)
    228                 tree_map_zip(lambda t: t.clone().detach() if isinstance(t, torch.Tensor) else poss_deepcopy(t, self.logger), output)
    229             self.node_infos[node].output_masks = \
--> 230                 tree_map_zip(lambda t: torch.ones_like(t).clone().detach() if isinstance(t, torch.Tensor) else None, output)
    231
    232             if self.garbage_collect_values:

OutOfMemoryError: CUDA out of memory. Tried to allocate 400.00 MiB (GPU 0; 14.75 GiB total capacity; 12.24 GiB already allocated; 288.81 MiB free; 13.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Finally, I would like to thank you one more time for your patience!


Exposing Tethering low level parameters in Popovers

Recently we implemented a popover in Janus that looks like this:

We couldn't make that work with the exposed tethering abstractions from https://github.com/rackerlabs/canon-react/blob/master/src/Popover.jsx#L105.
The positioning on the screenshot is done by tricking the algorithm with a absolutely positioned ghost element.
I believe it would be possible to do that if we could set the tether config to:
     attachment: 'top left',
     targetAttachment: 'bottom left',

That said, I'd expect more scenarios could also benefit from this. I think the abstract tethering named configurations are good, but would be great if we could access/influence the raw settings for Popover positioning.

resolved by #97


Review

Hello Reviewer, 👋
We are a group of two programmers, @Abe1able and @SabaAhmad404, Looking forward to coding an amazing Recipe App.
We have made a Kanban board to divide tasks among ourselves fairly. Below is the link to that project.
https://github.com/users/Abe1able/projects/3

Hi, Team👋,

Highlights💡

A link to the Kanban board is specified.✔
The team member size is stated.✔
Cards contain task descriptions.✔
All project requirements are captured in the Kanban board.✔
All required columns are present.✔
Cards are distributed fairly.✔

STATUS: APPROVED 🕺🥳
Congratulations🍾🥳!! Your project is ready for presentation!  There is nothing else to say other than... it's time to merge it :shipit:

Optional Suggestions:

N/A

Good luck! 🎉

Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me @brytebee in your question so I can receive the notification.
Use @brytebee on slack for faster responses.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Thank you very much @brytebee


immigrate to typescript

@alirezariahi Can you please resolve the merge conflicts?

@alirezariahi Can you please resolve the merge conflicts?

Yes, I work on it

@alirezariahi Can you please resolve the merge conflicts?

please create a branch and get me access to push


Overhaul Basis' documentation

Continuation of https://github.com/godotengine/godot/pull/86664
Related to https://github.com/godotengine/godot/pull/87114, and several other past efforts.
This PR completely overhauls Basis' documentation to the point where it is almost unrecognisable
Why?
The problems with the current documentation are summarised fairly easily:

It is written under the assumption that the reader has a PhD in linear algebra, not for a developer.
It explains the inner workings of Basis too much.
It is just far, far too verbose sometimes, with no actual substance.

"Assuming that the matrix is a proper rotation matrix, slerp performs a spherical-linear interpolation with another rotation matrix."
"This operator multiplies all components of the Basis, which scales it uniformly."
"This can be used to validate if the basis is non-distorted, which is important for physics and other use cases."
"Assuming that the matrix is the combination of a rotation and scaling, return the absolute value of scaling factors along each axis." Guys, it's the length of every axis. That's it!

How does this PR fix it?
On the other hand, this PR does so many changes that, explaining the reasoning behind every single one of them is pretty difficult.
As more notes may be edited latter, let's get these out of the way:

Avoid advanced math terms at all costs:

"Orthonormal", "orthogonal", "transformation". Avoid mentioning them unless it's very appropriate.
Exception to terms like "normalized" can stay. They have a clear and common purpose in Godot.

Say "basis" consistently, avoid "basis matrix" or "matrix" alone.

Most users are reading this class reference for the Basis aspect of the... Basis. Mentioning its matrix a lot can potentially be confusing, unless strictly necessary for a given method.
Advanced users know fully well it is a 3x3 matrix and will do as they please.

Elaborate on oh-so-many methods!

How should I structure these parameters? How is the returned value like? What does the operation I'm doing even mean?
In some cases, this is just not feasible, as it would not just be too complicated, but it would bloat the documentation with information most users wouldn't care about.
This may be frowned upon, but sometimes a link to an external article is a must.

More specific notes:

Give full details on what Euler angles are:

What does each axis represent? What does the default EulerOrder do?
At first I was reluctant to link to Wikipedia article, but I was sold by the simple-to-follow visual examples.

Explain how the UNIFORM basis is structured;
Add a much-needed example for slerp();
Expose the formula for tdotx, tdoty, and tdotz;
Explain the difference between scripting and C++ source code's [] operator;

Should supersede https://github.com/godotengine/godot/pull/84756.

...

Feedback is basically welcome.
This PR may contain multiple versions of the same line I was not sure about, denoted with "??". Reviewing of these lines is heavily appreciated.
Sources:

https://datacadamia.com/geometry/skewing
https://en.wikipedia.org/wiki/Diagonal_matrix
https://en.wikipedia.org/wiki/Euler_angles

Avoid advanced math terms at all costs:
"Orthonormal", "orthogonal", "transformation". Avoid mentioning them unless it's very appropriate.
Exception to terms like "normalized" can stay. They have a clear and common purpose in Godot.

I suggest that these terms be left as needed. This is because explaining things like "whether the angles of all basis vectors are perpendicular" or "whether the scales are all 1's" every time would be redundant.
Instead, I reccomend you write an explanation of the term in Matrices and transforms and link to it.

I suggest that these terms be left as needed. This is because explaining things like "whether the angles of all basis vectors are perpendicular" or "whether the scales are all 1's" every time would be redundant.

I am not sure how to feel about this, because the class reference as is in this PR does read just fine even without them.
Perhaps one solution is to add a note that later explains what those terms mean, in relation to the current method?

I updated the PR with an additional line about the meaning of a few terms, as I wait from more precise feedback from @TokageItLab

I simply point out that there is no definition of what the negative of the basis vector means in visually.
Actually, that definition may not be necessary until rendering, but if you are trying to draw/link a figure, you will need to annotate that the coordinate system of the Godot and the figure are same.
Also, it would be necessary to mention Row-major order and Column-major order. For those moving from other game engines, this can be confusing. See also https://www.mindcontrol.org/~hplus/graphics/matrix-layout.html.

Also, it would be necessary to mention Row-major order and Column-major order. For those moving from other game engines, this can be confusing easily (and I have seen many such people). See also https://www.mindcontrol.org/~hplus/graphics/matrix-layout.html.

I suppose this can be mentioned, yeah... It does sound important.

Actually, that definition may not be necessary until rendering, but if you are trying to draw/link a figure, you will need to annotate that the coordinate system of the Godot and the figure are same.

But I am not sure about. Would it not be for granted that both coordinate systems are the same?

As far as I know, mathematics basically uses the right-hand coordinate system, but in cases such as when a diagram created by someone working in DirectX is referenced, the left-hand coordinate system may be used and the direction of rotation, etc. may be flipped.
In order to counter such a claim from a DirectX user that the rotation is reversed, it is better to specify that Godot is in the right-hand coordinate system.
@aaronfranke previously wrote about the coordinate system for the asset pipeline https://docs.godotengine.org/en/4.1/tutorials/assets_pipeline/importing_scenes.html#d-asset-direction-conventions, but I am not sure if it is appropriate to link to that. Perhaps you can find a more appropriate link.

Linking to more advanced "lessons" does sound good to me. The goal of this PR was to make the Basis class reference more accessible, so talking too much about advanced topics is a bit out of scope. Even my head is spinning about this...

As far as I know, mathematics basically uses the right-hand coordinate system, but in cases such as when a diagram created by someone working in DirectX is referenced, the left-hand coordinate system may be used and the direction of rotation, etc. may be flipped.

This is true... Different programs all use different coordinate systems.
As far as I Godot only uses two: one for 2D and another for 3D. In this PR, you may notice I tried to make the Basis' axes easier to imagine (The IDENTITY constant, x, y, and z descriptions). This looked good enough and in my opinion, this kind of information is MUCH more useful to be included in Transform3D's class reference (coming soon).
But if it indeed is not enough, I could pull my favourite reference sheet as a link for users to see: https://bevy-cheatbook.github.io/fundamentals/coords.html

Thank you @TokageItLab and @aaronfranke very much for your time, here comes a second revision.

I updated the PR to try and address the feedback above. Thank you very much to the both of you for the very detailed explanations and patience
I also slightly changed the first sentence of the description ever so slightly. Thought it was noting.

I updated the PR to try and address the feedback above. Thank you very much to the both of you for the very detailed explanations and patience
I also slightly changed the first sentence of the description ever so slightly. Thought it was noting.

Also would like to note, I have purposely omitted looking_at from this PR because I believe it would be better to address the description of it and all other similar methods in a separate, focused PR.

Thanks!

The late-night merge!!!!! 🌙

Cherry-picked for 4.2.2.
Cherry-picked for 4.1.4.


Fix ascending/descending option for leaderboards

how to fix:

add ascending/descending choice inside leaderboard model
change selector at creation with either a '-ret_field' or 'ret_field' (maybe)
change update.py to order by the selector rather than '-ret_field'

No longer a bug with the leaderboard overhaul.


Update for EnderIO 5.2.60 (Fixes #18)

I went ahead and updated this mod to work with EnderIO version 5.2.60 and above.
I hope a build of this could be released to curseforge.
Thanks!

@Shadows-of-Fire Any updates on this?

Any updates on this?

Why were the mappings moved from stable to a 2017 snapshot...?

@Shadows-of-Fire the stable once are no longer hosted on forge servers.


Incompatible with complex background shorthand

Let's say I have this code:
background: resemble-image(url("foo.png")) no-repeat 75% center / cover;

This would be generated into:
background: url("foo.png"), linear-gradient(90deg, #161e2a 0%, #1a2331 25%, #5c4e52 50%, #584b4f 75%) no-repeat 75% center / cover;

Which would invalidate it.
Now I can think of either making the parser work by detecting commas to know when to put the fallback in (or just whack it at the end of the value with a , prefix, or wrap the whole long shorthand value into the resemble-image function itself. I guess this would be your call on how to proceed 😄

I like the first option (add it to the end of the declaration). Would you like to send a pull request for this? Essentially we need to change it so that this line does not add the gradient to the url function, but instead we should push a new node with the gradient into the AST.
I can help out if you would like to do this!


current limit exceeds maximum limit

When I run diffdock, an error occuring "current limit exceeds maximum limit";
How can I fix it?

I got the same error. Did you manage to fix it?


portaudio.h: No such file or directory

Windows Version
Microsoft Windows [Version 10.0.22621.1413]
WSL Version
WSL version: <IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[X] WSL 2
[ ] WSL 1

Kernel Version
Linux version <IP_ADDRESS>-microsoft-standard-WSL2
Distro Version
Release:        22.04
Other Software
miniconda  conda 23.3.1
pip3           pip 23.0.1
PyAudio
Repro Steps
pip3 install PyAudio

fatal error: portaudio.h: No such file or directory

Expected Behavior
normal PyAudio install as under native Ubuntu 22.04
Actual Behavior
fatal error: portaudio.h: No such file or directory
Diagnostic Logs
No response

The issue is not related to WSL. Please report any on-topic issues.
The portaudio.h file is provided by portaudio19-dev package. The file list of that package can be found here https://packages.ubuntu.com/jammy/amd64/portaudio19-dev/filelist.

It is my understanding that portaudio is provied by standard Ubuntu package and that WSL supplies a non standerd Ubuntu for the WSL support. The files referenced are not present in my Ubuntu 22.04 image under WSL2 with Windows 11.

The portaudio19-dev has to be installed to get that portaudio.h file.

@EdSeymore you need to add the portaudio19-dev package. This package contains the development files and not installed by default. apt install portaudio19-dev should take care of it.

Thanks. Now Successfully installed PyAudio-0.2.13.

Now I am getting a stream of messages from ALSA lib:
ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM sysdefault
ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory
and many more:
This may not be an WSL2 problem; but, I am trying really hard to use WSL2 with Ubuntu in place of a separate Ubuntu system and the complexity of the Operating Systems is making this really difficult.
End users should not have this much trouble setting up a working environment to run python applications. Trying to debug configuration problems with web search has not been productive for me.
I really have no idea how Windows and Ubuntu share devices on a single system and very little desire to know. I just would like for supportted systems to actually work when installed.
WSL2 just happens to be a key component.
Has anyone gotten PyAudio to work on Ubuntu 22.04 with Windows 11 and WSL2 without the ALSA lib messages?
Suggestions would be appreciated.

Ref: https://stackoverflow.com/questions/71945812/audio-recording-script-not-working-on-wsl2-ubuntu

The python3 code to illistrate this conditions is as follows:
#!/usr/bin/env python3
-- coding: utf-8 --
import pyaudio
import wave
import sys
CHUNK = 1024
file_path = "audio.wav"
wf = wave.open(file_path, 'rb') # Modified ets 01/31/2023
t_audio =wf.getnframes() / wf.getframerate()
p = pyaudio.PyAudio()
stream = p.open(format=p.get_format_from_width(wf.getsampwidth()),
channels=wf.getnchannels(),
rate=wf.getframerate(),
output=True)
while len(data := wf.readframes(CHUNK)):
stream.write(data)
stream.close()
p.terminate()
wf.close()

I am having the same issue. Has there been any update regarding this please?

At the time being, has anyone succeeded to have pyaudio working in WSL2?

^^

I am dealing with this same pyaudio situation... I have tried, installing portaudio, wheels, through pip, apt etc none of the workarounds I have found so far have worked for me

I got it to work on wsl-nixos by doing setting hardware.pulseaudio.enable = true
so it is possible but unsure what it would look like for other distros

there's no such cmd on my ubuntu 2204 wsl tho

I got it to work on wsl-nixos by doing setting hardware.pulseaudio.enable = true
so it is possible but unsure what it would look like for other distros

didn't work for me on 2204 tho

For some weird reason I got it working again and I don't know how I did it, perhaps all workarounds added up...

I got it to work on wsl-nixos by doing setting hardware.pulseaudio.enable = true
so it is possible but unsure what it would look like for other distros

didn't work for me on 2204 tho

fixed it again on another wsl of mine, if you having the same pyaudio installation issue, the below instruction should help, especially, FIRSTLY installing portaudio and python3-dev, this additional detail is from the pypi guidelines for pyaudio
Use the package manager to install PyAudio. For example, on Debian-based systems:

sudo apt install python3-pyaudio
Alternatively, if the latest version of PyAudio is not available, install it using pip. Be sure to first install development libraries for portaudio19 and python3.

Now I am getting a stream of messages from ALSA lib:
ALSA lib confmisc.c:855:(parse_card) cannot find card '0' ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM sysdefault ALSA lib confmisc.c:855:(parse_card) cannot find card '0' ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory and many more:
This may not be an WSL2 problem; but, I am trying really hard to use WSL2 with Ubuntu in place of a separate Ubuntu system and the complexity of the Operating Systems is making this really difficult.
End users should not have this much trouble setting up a working environment to run python applications. Trying to debug configuration problems with web search has not been productive for me.
I really have no idea how Windows and Ubuntu share devices on a single system and very little desire to know. I just would like for supportted systems to actually work when installed.
WSL2 just happens to be a key component.
Has anyone gotten PyAudio to work on Ubuntu 22.04 with Windows 11 and WSL2 without the ALSA lib messages?
Suggestions would be appreciated.

Any solutions to this yet?