DB transaction format changed

dude, you're better off submitting requests to my fork https://github.com/jfonte/dbk-tools I fixed this on Oct last year and it still hasn't been accepted... https://github.com/nslater/dbk-tools/pull/4

Got it, thanks, found your fork after submit only :)

hey folks. sorry I missed this. I had notifications for the whole repo turned off for some reason. I have fixed that now
I'm just about to add support for Number26. but no longer use DBK. however, I'm considering keeping the DBK support in
would either of you you be interested in sending a PR to this repo with your changes, and helping me keep that format support up-to-date?

@nslater OK. Submitted new PR and will close old one. Interesting that you switched to number26... are you using it in Germany? If yes, why did you change to it?

in Germany, yeah. switched because I saw my friend using the mobile app and getting real-time transaction notifications. neat experience so far
what do you wanna do with this PR? does #6 supercede this one?

$6 just adds a suggestion for using pyenv to manage environment and has some comments regarding merging beneficiary & details. Other than that, they are equal.

okay cool. I'll do my new work on a branch and I'll wait for @alexole to respond

Hey guys, yes, just use #6.


The dora service is not started. An error is reported execute dora list

Describe the bug
The dora service is not started. An error is reported execute dora list
To Reproduce
Steps to reproduce the behavior:

dora list

Screenshots or Video

Environments (please complete the following information):

System info: Linux jia 5.15.0-69-generic #76~20.04.1-Ubuntu SMP Mon Mar 20 15:54:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Dora version : 0.2.2

Could you clarify what behavior you are expecting? Do you think that the coordinator and daemon should be started automatically on dora list? Or do mean that we should improve the error message to e.g. "no dora coordinator is running" instead of "Connection refused"?

Could you clarify what behavior you are expecting? Do you think that the coordinator and daemon should be started automatically on dora list? Or do mean that we should improve the error message to e.g. "no dora coordinator is running" instead of "Connection refused"?

No, I think that when dora is not started, execute the dora list command, it should not report an error, it should remind the prompt that dora is not started

Thanks for clarifying! I opened https://github.com/dora-rs/dora/pull/254 to fix this.

I may be wrong, but is this issue more geared toward the way the error is reported rather than the actual content of the error @meua ?

I may be wrong, but is this issue more geared toward the way the error is reported rather than the actual content of the error @meua ?

The dora return should clearly explain the cause of the problem, and should not directly throw an exception


Make placeholders configurable

The Code of Conduct will gain more adoption if it's easy for maintainers to adopt it. Some projects may still want to fork it and make more drastic changes, but for a significant portion of the community, the standard template will suffice. This is an attempt to demonstrate what was suggested in #7:

If we want to go this route, there's still some more work to do here:

[x] Clean up the JavaScript
[x] Design it
[ ] ZeroClipboard to make it easier to copy the snippet
[ ] Better way to encode config data

What do you think?
/cc @caniszczyk

I'm a fan of how this was executed. :+1:
I'm not a fan, though, of the user's perspective: how this generates URLs that seem to have robot barf in it. URLs with big fragment values that carry important data generally don't fill me with confidence.

I agree. Here are a few ideas to make this better:

use normal params<EMAIL_ADDRESS>continue using the anchor, but make it cleaner<EMAIL_ADDRESS>

I made a few tweaks to the design. It could use a real designer at some point, but this highlights what I was envisioning.

@mislav I updated this to use this url format: http://opencodeofconduct.com/#ProjectName/email%40example.com. Thoughts?

Looks great (both design and URL path).
Even if query parameters in the URL are human-readable, I think seeing them betrays that this text has been auto-generated and that might make the CoC feel less personalized and less legit. I'm just speaking for myself here.

Here's how to copy contents of a <input> element without Flash in Chrome 43+ and Firefox 41+:
input.select()
document.execCommand('copy')
window.getSelection().removeAllRanges()

Alright, I'm pretty happy with where this is at. I would love feedback or suggestions on what else is needed. There are some changes I need to push to the todogroup.org styles to make this work, but I'll get a PR open for that so we can merge both at the same time.
/cc @caniszczyk @amateurhuman @mislav

If you're going to have magic copy behavior, you have to make the users know. Otherwise they probably won't figure it out by themselves. Maybe it's best to have an explicit “Copy to clipboard” button.

Hell I actually like it! I think it's good enough to start with, we can iterate later and making it a bit fancier. Some people will use this functionality, others will simply copy the index.md

@mislav thanks for all your feedback. It's live: http://todogroup.org/opencodeofconduct/
One thing I think we'll want to tweak is the snippet that gets copy and pasted into the project. Since tons of projects will have this, we'll want to make sure it's well-worded:

This project adheres to the [Open Code of Conduct][code-of-conduct]. By participating, you are expected to uphold this code.
[code-of-conduct]<EMAIL_ADDRESS>


IgnoreCollectionOrder in combination with CustomComparer

Hello Greg,
I have an issue when using IgnoreCollectionOrder in combination with a CustomComparer.
We need a CustomComparer to handle a small difference in a decimal.
I looked into the code and I think the issue occurs in method CompareOutOfOrder.
My objects will never generate the same matchIndex,  because of the possible small difference between decimals.
I thought a possible solution was to use the default comparer while trying to find a matching object.
However this generates a lot of differences.
`                    //if (matchIndex1 == matchIndex2)
                    {
                        CompareParms childParms = new CompareParms
                        {
                            Result = parms.Result,
                            Config = parms.Config,
                            ParentObject1 = parms.Object1,
                            ParentObject2 = parms.Object2,
                            Object1 = enumerator1.Current,
                            Object2 = enumerator2.Current,
                            BreadCrumb = currentBreadCrumb
                        };

                        bool blnIsMatch = _rootComparer.Compare(childParms);
                        if (blnIsMatch)
                        {
                            _alreadyCompared.Add(matchIndex1);
                            found = true;
                            break;
                        }
                    }

`
Hope you can help me with this.
Thanks. Regards, Corné

Corné, sorry, I am just too busy to support this project.  You will have to figure out a fix yourself.

Corné,
I would add an id to your class to match them up or find another identifier that stays consistent.
Thanks,
Greg


Return of the experimental changelog - GitHub API edition (And how to implement it!)

References https://github.com/remyroy/CDDA-Game-Launcher/issues/621

https://github.com/Fris0uman/CDDA-Game-Launcher/pull/6 might be helpful here. Fris just implemented this for their branch.

Fris0uman#6 might be helpful here. Fris just implemented this for their branch.

This is good stuff! Thanks for the heads-up, I will seek to have it merged on this fork too.


initrd: build initial ramdisk instead of using precompiled one

I automated the building of an initial ramdisk as described here. The goal is to be able to deploy further files into it while building HermitCore.
Progress:

[x] Download and build busybox
[x] Create initrd.cpio and make make recognize it as a dependency
[ ] Copy shared libraries
[ ] Collect all the changes and additions to the old initrd and migrate to new build process
[ ] Create a workflow to add further files during build process

I will close this issue since it has been abonded by me for the time being. If we pick this up again, we can reopen this PR. A backup of the latest state can be found here.


[BUG] Liked songs heart isn't centered

Describe the bug
The heart for the liked songs on the left side isn't centered.
Screenshots

Desktop (please complete the following information):

OS: Windows 10 21H1
Version: 2.2
Spicetify Verison: 2.7.0

Logs

Looks like this went by me because I'm using the minimized sidebar, could you try replacing user.css with this, and then running spicetify apply and tell me if it fixes it

I've applied it and it's still apparent.

that's odd. What Spotify version are you using?

that's odd. What Spotify version are you using?

<IP_ADDRESS>2

hm, that's exactly the same as me, are you at all familiar with the chrome devtools and html / css?

hm, that's exactly the same as me, are you at all familiar with the chrome devtools and html / css?

Yes, I am.

Could you maybe try to find out what the problem is then? Since this is not happening with my installation, I can't really do much myself.

Changing the following fixed the position:

The container css class of the container (<div class="GlueDropTarget GlueDropTarget--tracks personal-library" ...):
From GlueDropTarget GlueDropTarget--tracks personal-library to main-navBar-navBarItem
the css class of the  inside the container
From main-collectionLinkButton-collectionLinkButton to link-subtle main-navBar-navBarLink fixes the position.

Btw: Is there a reason why this one is the only div container in the list (other entries all have <li>)?

So from what it looks like in the first screenshot, this might be caused by too much padding on the left. You could try to change the padding-left on the .main-collectionLinkButton-collectionLinkButton. I still don't get why this is only an issue for some people though.
As for the reason some of the sidebar items have different classes / tags, I don't know why that is. But what I do know is that it makes styling it a real hassle.

I tried it with all extensions disabled but dribbblish.js and dribbblish-dynamic.js but the issue is still the same.

I didn't even know you could stick the items to the top. I always hated that the liked songs would scroll. But that explains it. I can push a fix for this now finally. Thank you for your Investigation.

Should be fixed now. It'll be in the next Release. If you want to fix it now just replace your user.css with this


pull request for e2e test at 01/03/2017 21:15:16

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Deploy a Svelte site example required for Durable Objects

Which Cloudflare product(s) does this pertain to?
Durable Objects
Subject Matter
The way a durableobject class is exported in context of a sveltekit project.
Content Location
https://developers.cloudflare.com/pages/framework-guides/deploy-a-svelte-site/#sveltekit-cloudflare-configuration
Additional information
The latest create-cloudflare project generates this code in hooks.ts when you select a sveltekit project.
import { dev } from '$app/environment';

/*
  When developing, this hook will add proxy objects to the `platform` object which
  will emulate any bindings defined in `wrangler.toml`.
*/

let platform: App.Platform;

if (dev) {
	const { getPlatformProxy } = await import('wrangler');
	platform = await getPlatformProxy();
}

export const handle = async ({ event, resolve }) => {
	if (platform) {
		event.platform = {
			...event.platform,
			...platform
		};
	}

	return resolve(event);
};

This is great. So now in npm dev node (using sveltekit dev mode) you can easily use KV and R2 (tested) and probably D1 bindings.
However, there is not proper example of how (or where) to export a Durable Object class.. eg:
Given this class:
export class DOCounter {
    constructor(state: DurableObjectState, env: Env) { }
    async fetch(request: Request) {
        return new Response("Hello World");
    }
}

WHERE do we put this class so that it forms part of the build process.
(It cannot be in a +server.ts or a +page.server.ts file. sveltekit does not like that.
Having this a wrangler.toml config of like this:
[[durable_objects.bindings]]
name = "DO_COUNTER"
class_name = "DOCounter"

migrations
[[migrations]]
tag = "v1"
new_classes = ["DOCounter"]

And a /api/+server.ts file like this:

export const GET: RequestHandler = async ({ request, platform }) => {
    if (!platform) {
        return new Response();
    }
    const env = platform.env;
    if (!env.DO_COUNTER) {
        return new Response('env.DO_COUNTER not found', { status: 404 });
    }
    const id = env.DO_COUNTER.idFromName(new URL(request.url).pathname);
    const stub = env.DO_COUNTER.get(id);
    const response = await stub.fetch(request);
    return response;
};

PLEASE, PLEASE PLEASE create a sveltekit project that works when you do this

npm run build
npx wrangler pages dev .svelte-kit/cloudflare

To prevent this error:
Your worker has access to the following bindings:
- Durable Objects:
  - DO_COUNTER: DOCounter
- KV Namespaces:
  - KVCOUNTER: someKvId
- R2 Buckets:
  - R2_BUCKET: r2-bucket
⎔ Starting local server...
Parsed 2 valid header rules.
✘ [ERROR] service core:user:sveltekit-cloudflare-bindings: Uncaught TypeError: Class extends value undefined is not a constructor or null

    at null.<anonymous> (rrkgcneij6.js:18422:44) in maskDurableObjectDefinition
    at null.<anonymous> (rrkgcneij6.js:18431:18)

✘ [ERROR] MiniflareCoreError [ERR_RUNTIME_FAILURE]: The Workers runtime failed to start. There is likely additional logging output above.

we are soooooo close to getting this in line. But the documentation lacks the full example that provide context of the class.
DurableObjects are easy in a simple cloudflare worker... but NOT in a sveltekit project. Does other frameworks solve this?

@gerhardcit currently creating a DO in a Pages project is technically possible but practically useless since wrangler pages deploy (and the Pages git-integration) doesn't actually deploy your DO.
So basically when using Pages (which is what the C3 SvelteKit template does) you practically cannot define a DO in your application, the only current solution is to define the DO in a worker and then bind it to your Pages application (you can do that both in the dashboard and in the toml file, and getPlatformProxy will pick that up and connect to the local worker's DO)

DurableObjects are easy in a simple cloudflare worker...

Yes... they are currently not for Pages... 😅

Does other frameworks solve this?

AFAIK the do it in the manner I just described, have the DO implemented using a worker and binding (locally and remotely) to your Pages application

Even if Pages wouldn't have this limitation I struggle a bit to imagine how someone could define a DO in a framework and have it exposed in their worker entry point, since it would have to pass through the framework's build process.
I'm not saying that it would be impossible but something that the framework itself would have to have baked-in (unless we introduced some new command/convention/way of uploading DOs...)

Anyways... to summarize... I think that currently the only valid/practical way to use DOs in a SvelteKit (or Pages in general) project is to define the DOs in workers and bind those to the Pages application.
Likely not ideal but I don't imagine there will be any better alternative soon...
Anyways yeah I suspect that we don't really document that properly...

Thanks.. sometimes, when you buy a car or a tool, part of what you want to know is what is CANNOT do.
The Cloudflare docus here:
https://developers.cloudflare.com/pages/framework-guides/deploy-a-svelte-site/#sveltekit-cloudflare-configuration
provides this example:
 env: {
           COUNTER: DurableObjectNamespace;
       };

so naturally the assumption is, ok, that is a thing.
Then it turns on NOT to be thing. It seems someone document it without actually running it in real life?
That is a little disappointing.
At least https://kit.svelte.dev/docs/adapter-cloudflare tells you specifically that it won't work.
Except now it does.. with npm run dev you can test KV, D1, R2 ... all, except... DurableObjects
I'm repeating myself.. but I was hoping documented examples are actually tested. And is some cases provide a repo link where you can run that code and test it in full context.

@gerhardcit sorry for the frustration 😓
yeah I get the argument that
COUNTER: DurableObjectNamespace;

isn't the best example to have there! 😓 I'll bring it to the team and see to remove it.
Alongside making sure that we properly document the extra hurdles that Pages has in regards to DOs.
Anyways, just to clarify, DurableObjects are supported (both in wrangler pages dev, deployed Pages applications and getPlatformProxy), the issue is that you have to basically define them in a separate worker (basically you need to jump through extra hoops to get them working 😓)

I'm repeating myself.. but I was hoping documented examples are actually tested. And is some cases provide a repo link where you can run that code and test it in full context.

100%, these things sometimes slip through the cracks (but we're trying our best to avoid that moving forward) sorry about that 🙇

I would pay good money to see an example of "define them in a separate worker " and how you would structure and run that. (again.. full context)
especially if you want to add websockets into that mix.
the DO webchat samples goes back about 2 - 3 years.. which is really way out of date. If someone can get those samples a rework, or provide more info so we (the devs trying to make it work) do it. It would be great.

Joining the thread. me and my company are working tightly with Pages, D1 and KV and plan to use DO for all of our realtime logic. Our use case is a collaborative 3D platform and DX is a top priority for us to iterate as fast as possible on realtime logic. So far Miniflare worked great with SvelteKit for KV and D1 with persistance and migraitons. I am eager to see working DO .
@gerhardcit  there are some clues already on how to make a worker with DO
https://github.com/JoshAshby/joshashby.github.io/blob/fa69dcdd9e9a81c991c702c5335d5251417af1e1/_drafts/sveltekit-durable-objects-continued.md?plain=1#L4
As well as how to even build the DO source code into a worker from SvelteKit project https://github.com/kalepail/boilerkit/blob/ec77b6cee8b0ed498002b3e4afa9a1fe9c13b8a9/src/helpers/_mf.ts#L33
But I agree , a fully working solution would help a lot. There is not that much material on DO...

I was able to get it working with miniflare.
@gerhardcit remember that poc repo ? https://github.com/jculvey/svelte-cf-bindings-poc
So here is an updated version of it .
in hook.server.ts I added another Miniflare instance specifically for DO's
	const { Miniflare } = await import('miniflare');
	const mf = new Miniflare({
		kvNamespaces: ['KV'],
		kvPersist: '.wrangler/state/v3/kv',
		d1Databases: ['D1'],
		d1Persist: '.wrangler/state/v3/d1',
		modules: true,
		script: ''
	});

	const doMf = new Miniflare({
		modules: true,
		durableObjects: { COUNTER: 'Counter' },
		durableObjectsPersist: '.wrangler/state/v3/do',
		scriptPath: 'durable-objects/counter/index.js'
	});

	env = Object.assign({}, await mf.getBindings(), await doMf.getBindings());

app.d.ts
// See https://kit.svelte.dev/docs/types#app
// for information about these interfaces
// and what to do when importing types
import { DrizzleD1Database } from 'drizzle-orm/d1';

declare global {
	declare namespace App {
		interface Locals {
			userId: string | null;
			userData: UserData | null;
		}
		// interface PageData {}
		// interface Error {}
		interface Platform {
			env: {
				DB: DrizzleD1Database;
				D1: D1Database;
				KV: KVNamespace;
				COUNTER: DurableObjectNamespace;
			};
			context: {
				waitUntil(promise: Promise<any>): void;
			};
			caches: CacheStorage & { default: Cache };
		}
	}
}

index.js looks like that , it has a blank worker . No wrangler pages dev needed .
// Worker

export default {
	async fetch(request, env) {
		return new Response(`}`);
	}
};

// Durable Object

export class Counter {
	constructor(state, env) {
		this.state = state;
	}

	// Handle HTTP requests from clients.
	async fetch(request) {
		// Apply requested action.
		let url = new URL(request.url);

		// Durable Object storage is automatically cached in-memory, so reading the
		// same key every request is fast.
		// You could also store the value in a class member if you prefer.
		let value = (await this.state.storage.get('value')) || 0;

		console.log(url.pathname);

		switch (url.pathname) {
			case '/api/increment':
				++value;
				break;
			case '/api/decrement':
				--value;
				break;
			case '/':
				// Serves the current value.
				break;
			default:
				return new Response('Not found', { status: 404 });
		}

		// You do not have to worry about a concurrent request having modified the value in storage.
		// "input gates" will automatically protect against unwanted concurrency.
		// Read-modify-write is safe.
		await this.state.storage.put('value', value);

		return new Response(value);
	}
}

Now in your SvelteKit projects create api/increment/+server.ts under routes folder.
import { json, type RequestHandler } from '@sveltejs/kit';

export const GET: RequestHandler = async ({ platform, request, cookies }) => {
	let url = new URL(request.url);
	let name = url.searchParams.get('name');
	if (!name) {
		return new Response(
			'Select a Durable Object to contact by using' +
				' the `name` URL query string parameter. e.g. ?name=A'
		);
	}

	// Every unique ID refers to an individual instance of the Counter class that
	// has its own state. `idFromName()` always returns the same ID when given the
	// same string as input (and called on the same class), but never the same
	// ID for two different strings (or for different classes).
	let id = platform?.env.COUNTER.idFromName(name);

	// Construct the stub for the Durable Object using the ID. A stub is a
	// client object used to send messages to the Durable Object.
	let obj = platform?.env.COUNTER.get(id!);

	// Send a request to the Durable Object, then await its response.
	let resp = await obj!.fetch(request.url);
	let count = parseInt(await resp.text());
	let wasOdd = count % 2 === 0 ? 'is odd' : 'is even';

	return json(`Durable Object '${name}' ${count} ${wasOdd}`);
};

run the server and go to http://localhost:5173/api/increment?name=Test observe the counter go up with each page refresh. Restart the server and see the counter persisting.
You can create as many miniflares per DO script as you like or have them split into multiple files (to deploy with wrangler to production) but locally you can just join them under 1 file.

@clibequilibrium , thanks, that looks great.
You said it: "making many people look away..." Which is a shame. Durable Objects by it's design is a super powerful (and I think underestimated) feature.
The key piece of code here is this in hooks.ts
scriptPath: 'durable-objects/counter/index.js'

This is what i've been looking for high and low.
@dario-piotrowicz , would a good clue to add as options to pass to getPlatformProxy?
const { getPlatformProxy } = await import('wrangler');
platform = await getPlatformProxy({
doClassesPath: ".....?"
});

and if you can typescript those, export them somehow so they end up in the vite build ouput, then you have something really powerful.

Also, @dario-piotrowicz , please correct me if I'm wrong. DurableObjects and Pages don't go together correct?
I can run a DO in a worker, but not a raw pages project with a functions directory?
I tried that as a test, but wrangler pages dev with the code from this example
https://developers.cloudflare.com/durable-objects/examples/build-a-counter/ does NOT work.
✨ Compiled Worker successfully
 ⛅ wrangler 3.28.3
-------------------
Your worker has access to the following bindings:
- Durable Objects:
  - COUNTER: Counter
- KV Namespaces:
  - KVCOUNTER: KvCounterID
⎔ Starting local server...
✘ [ERROR] service core:user:do-socket-pages-vite: Uncaught TypeError: Class extends value undefined is not a constructor or null

    at null.<anonymous> (functionsWorker-0.044904672053120764.js:676:44) in
  maskDurableObjectDefinition
    at null.<anonymous> (functionsWorker-0.044904672053120764.js:685:16)

✘ [ERROR] MiniflareCoreError [ERR_RUNTIME_FAILURE]: The Workers runtime failed to start. There is likely additional logging output above.

@dario-piotrowicz it would be very nice if it was possible to deploy Durable Objects with Pages without needing a void Worker.

@clibequilibrium yes indeed it would be! 😅
I do not have any like ETA or idea regarding when/how this could be implemented but I would strongly imagine that with Workers and Pages convergence, defining a DO in a "Pages" app should be possible in the future

@gerhardcit regarding pathScript, yeah I think (unless I'm misremembering) that we did briefly considering having it in getPlatformProxy... but we/I felt that it would be cumbersome and generically not too useful.... (since at the end of the day even if you do have something like that, you do still need to have an actual worker with the DO in production, so it's not really solving much IMO)
I'll open an issue in the workers-sdk repo for this and see what people think 👍
PS: can you clarify to me, do you find this simpler/more convenient than running a worker which exposes the DO and bind that DO with your SvelteKit app? because I personally find the latter much clearer/cleaner... unless I'm missing something
I also still want to provide you with an example of the above (since you said you'd like to see one), I'll get to write one when I have some spare time on my hands

Also, @dario-piotrowicz , please correct me if I'm wrong. DurableObjects and Pages don't go together correct?

DurableObjects and Pages do work together, the issue is that you cannot define a DurableObject inside a Pages application, but you can define the DurableObject inside a worker and use a DurableObject binding to use that durable object inside your Pages application (to clarify, I am not saying to use a service binding to the worker and get the DO, I'm saying that you can bind directly to that DO).

I can run a DO in a worker, but not a raw pages project with a functions directory?

Yes, sort of...
You can declare a DO in a worker but in a Pages project it's different... you can declare the DO in a pages project but it won't get deployed, so basically, you can but there's nearly no point in doing so
Also the Pages' functions/ directory gets bundled so you don't really have that much control over it and I think that it does not allow you to specify a DO there, but you should definitely be able to do that using the advance mode

PS: can you clarify to me, do you find this simpler/more convenient than running a worker which exposes the DO and bind that DO with your SvelteKit app? because I personally find the latter much clearer/cleaner... unless I'm missing something

@dario-piotrowicz , Initially I thought binding in the Sveltekit App is the most obvious way. But understanding DurableObjects better now, with Alarms specifically, it is a true "worker" concept. So I think there must be limits when you mix it in to Pages apps.
The contradiction though, is the way websockets and DurableObjects are mixed.
websockets by design requires a Webclient.. but putting a smart client onto a worker is discouraged here:
https://developers.cloudflare.com/workers/configuration/sites/start-from-scratch/
This feels like someone shot them selves in the foot. You create a true worker component, that fires up and work in the background, but then you mix in with a websocket UI into it.. Somehow that is a weird concept to get my head around.
So for what we want to use DurableObjects for, with Alarms etc.. I'm going to run them completely independent and than hack the CORS thing to get to it from another Pages UI that does ALL the other KV, D1 and R2 stuff.
I thought I solved it with a Vite UI (Pure Svelte) and a functions folder for my api.. but when I throw Durable Objects in there I had the same issue.

@gerhardcit if it helps, here's a quick example of using a DO with Pages in advanced mode (only locally): https://github.com/dario-piotrowicz/cf-pages-advanced-mode-do-example/tree/main

@dario-piotrowicz , thx. that is simple enough..  but I'm not following the practicality of the warning
you can wrangler pages deploy but it won't deploy your DurableObject, for production usage you do need to declare your Counter DurableObject in a worker

When you say production, you mean the pages or the worker?
I'm not sure how a worker code differs in production. How would you reference an DurableObject from pages in production.
Something does not add up somehow. Apologies, but I need to see the link between "here" and "there" which seems to how it will be deployed. It feels like there is a remote URL variable that needs to come in somehow.

it is a true "worker" concept

Exactly! sorry if I skipped through that part, basically DOs are currently only like a "pure worker concept" and Pages can somewhat "borrow" them, but not use them independently

The contradiction though, is the way websockets and DurableObjects are mixed.
websockets by design requires a Webclient.. but putting a smart client onto a worker is discouraged here:
https://developers.cloudflare.com/workers/configuration/sites/start-from-scratch/

Sorry I am not seeing that, could you point me to the specific section that says that? 😅

The wording is: "
Use Cloudflare Pages for hosting full-stack applications instead of Workers Sites. Do not use Workers Sites for new projects."

The wording is: " Use Cloudflare Pages for hosting full-stack applications instead of Workers Sites. Do not use Workers Sites for new projects."

@gerhardcit sorry for being dense.... but I don't see where this relation between that text and "putting a smart client onto a worker is discouraged here" 😕 I guess I'm missing something?
(all I read there is "don't use Workers Sites as it is deprecated, use Pages instead", noting that Pages, especially in advanced mode, can do all that workers sites can)

@dario-piotrowicz , thx. that is simple enough.. but I'm not following the practicality of the warning
you can wrangler pages deploy but it won't deploy your DurableObject, for production usage you do need to declare your Counter DurableObject in a worker

When you say production, you mean the pages or the worker? I'm not sure how a worker code differs in production. How would you reference an DurableObject from pages in production.
Something does not add up somehow. Apologies, but I need to see the link between "here" and "there" which seems to how it will be deployed. It feels like there is a remote URL variable that needs to come in somehow.

You can reference reference the DO in Pages from the dashboard and it will be already available in the bindings. See my example above. After you deploy your DO with a blank worker , you can link the DO to Pages under Functions

What is also amazing is that you get to debug DO and see the metrics

I also disabled the routing for the empty Worker so it is not available. You interact with DO purely from your SvelteKit app

@gerhardcit if you want Typescript with Intellicense and everything I also tried this repo https://github.com/cloudflare/durable-objects-typescript-rollup-esm
Then did wrangler build and instead of scriptPath: 'durable-objects/counter/index.js' I point it to scriptPath: 'durable-objects/counter/dist/index.mjs' since Typescript needs to be compiled first.

Thx @clibequilibrium, I'm digging into it.
I'm not sure I follow the scriptPath part? Where is that applied?

@gerhardcit I've checked with the team and it seems like the general consensus is that something like doClassesPath: ".....?" would be a bit of an awkward property to support and not too worth it to support the Pages use case which is already quite awkward as it is (as we discussed above)
So for now I think we'll just keep the current implementation and things should get much better whenever convergence allows it... I hope that can work for you? 🙏
(PS: I still own you an example of now to set up the binding locally, I haven't forgotten, I'll provide one today or in the next few days 🙇)

Hi !
I made it work without Miniflare entirely via wrangler pages dev and wrangler dev  full hot reload support and websocket support with persistence!!
In your pages wrangler.toml file you need to add
[[durable_objects.bindings]]
name = "DO_COUNTER"
class_name = "DO_COUNTER"
script_name = "worker" // this is a name of a running worker that your DO runs on. navigate to http://localhost:6284/workers to see the name of the worker with wrangler dev running of DO Worker

Then wrangler dev on the DO worker and wrangler pages dev --compatibility-date=2023-11-21 --proxy 5173 -- pnpm run dev and voila full hot reload, persistance and websocket support !

@gerhardcit here's the example I promised 🙂
https://github.com/dario-piotrowicz/sveltekit-durable-object-local-usage-example
Please let me know if this helps

I've created https://github.com/cloudflare/workers-sdk/issues/5164 in the workers-sdk to capture what's been discussed here and maybe get the example moved over there (if we find it useful)
Besides that I don't think there's much we can do in the Cloudflare docs (besides maybe linking to the workers-sdk examples somehow?)
I'm closing this issue as we can continue the discussion in workers-sdk, I hope that works for you @gerhardcit 🙂
(if not please feel free to reopen this)

@dario-piotrowicz Thank you for the helpful repository. This issue would be even more useful if you could provide more details about the comment below. Specifically, we are looking for a clearer understanding of how to make SvelteKit work with Durable Objects in the actual Cloudflare environment. I eventually understood (it's took 4 hours, at least) but it seems undocumented and the future readers must want to know.

The issue being that in the Cloudflare dashboard you will not find your DO anywhere, it simply won't get uploaded/recognized (as that is not supported for Pages).
The only thing you can do to get the binding to work in your deployed Pages application is to define a separate Worker project, define your DO there and deploy the worker. Only then, you will be able to bind that DO to your existing Pages project.
Again, having your Pages project, practically "borrow" the DurableObject declared/owned by your Worker project

@sundaycrafts thanks for your comment 🙂
I'm sorry for the inconvenience the missing documentation caused you 🙇
I'll see what I can do regarding adding the info somewhere in the docs
Also as I mentioned earlier in this thread things should hopefully be much simpler/easier after convergence 🤞 (not a huge relief right now but still 😅)

Thank you, Dario,
I've had some time, so I'd like to share an implementation image of a session store that is probably ideal as a real-world example. First, deploy a worker like the following (please note that this is a code snippet, so for the full implementation, refer to Dario's code or the official documentation). It's important to note that if you want to use it with Pages, the handler of this worker (export default > fetch func) is actually never used, so it should be closed.
// ./sessionstore-worker/src/index.ts
import { Namespace } from './namespace';
import { HTTPResponseCode } from './HTTPResponseCode';
import { getHandler } from './handlers/getHandler';
import { putHandler } from './handlers/putHandler';
import { defaultHandler } from './handlers/defaultHandler';

export interface Env {
	SESSION_KV: KVNamespace;
	SESSION_DO: DurableObjectNamespace;
}

/** This Durable Object and its logic will be shared with your Sveltekit app **/
export class SessionStore {
	private readonly serializedSessions: Record<string, string> = {};

	constructor(
		private readonly state: DurableObjectState,
		private readonly env: Env,
	) {}

	async fetch(request: Request): Promise<Response> {
		const maybeNs = Namespace.tryFromUrl(new URL(request.url));
		if (!maybeNs.success) return new Response(maybeNs.error.message, { status: HTTPResponseCode.BAD_REQUEST });

		console.log(`[${request.method}] ${maybeNs.data.toString()}`);

		switch (request.method.toUpperCase()) {
			case 'GET':
				return getHandler({
					namespace: maybeNs.data,
					serializedSessions: this.serializedSessions,
					kv: this.env.SESSION_KV,
				});
			case 'PUT':
				return putHandler(request, {
					namespace: maybeNs.data,
					state: this.state,
					serializedSessions: this.serializedSessions,
					kv: this.env.SESSION_KV,
				});
			default:
				return defaultHandler();
		}
	}
}

export default {
	async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
                // This stub worker reject all requests
		return new Response('', { status: HTTPResponseCode.FORBIDDEN });
	},
};

// sessionstore-worker/src/handlers/putHandler.ts
import { Namespace } from '../namespace';

export async function putHandler(
	request: Request,
	context: {
		namespace: Namespace;
		state: Pick<DurableObjectState, 'waitUntil'>;
		serializedSessions: Record<string, string>;
		kv: KVNamespace;
	},
): Promise<Response> {
	const newValue = await request.text();
	if (context.serializedSessions[context.namespace.toString()] !== newValue) {
		context.serializedSessions[context.namespace.toString()] = newValue;
		context.state.waitUntil(context.kv.put(context.namespace.toString(), newValue));
	}
	return new Response('ok');
}

After deployment the worker, you will be able to bind the Durable Object defined in this Worker to Pages. Therefore, bind the DO from the Pages settings screen.

After that, you just need to add the DO interface to platform.env as mentioned in Cloudflare's documentation. When a fetch is performed from a DO object bound within Pages, the fetch defined in the worker for that DO (in this example, SessionStore.fetch()) is executed. Honestly, I couldn't have imagined being able to execute functions that appear to be running on another process like an RPC until I actually tried it. I think it's an incredible technology🤯
For local execution, you should refer to the repository shared by Dario. Specifically, the following part is key.
// src/hooks.server.ts
import { dev } from '$app/environment';

/*
  When developing, this hook will add proxy objects to the `platform` object which
  will emulate any bindings defined in `wrangler.toml`.
*/

let platform: App.Platform;

if (dev) {
	const { getPlatformProxy } = await import('wrangler');
	platform = await getPlatformProxy();
}

export const handle = async ({ event, resolve }) => {
	if (platform) {
		event.platform = {
			...event.platform,
			...platform
		};
	}

	return resolve(event);
};

Another point to note is that Dario's code includes a sveltekit-app/wrangler.toml, but this is only respected in a local development environment. The adapter does not include this wrangler.toml as part of the assets during deployment. Only the settings mentioned above in the console are used.
It might seem a bit inelegant to have to define a placeholder for the worker endpoint for deploying the sessionStore, but I think it makes sense for the sessionStore module and the frontend module to be separated, and it's a good level of granularity. I hope this comment will be helpful to future readers.

@gerhardcit here's the (very minimal) example I promised 🙂
https://github.com/dario-piotrowicz/sveltekit-durable-object-local-usage-example
Please let me know if this helps

This seemed to work but I was not able to use WebSockets. The problem seems to be that +server.ts files ignore ws:// requests. So even though I can get hold of the DO in an http:// GET the request created by new WebSocket('ws://') just gets ignored. Any one have a solution for this? I'd be keen to do this in the Svelete app so I can use my apps auth etc.
export const fallback: RequestHandler = async ({
	request,
	params,
	platform
}) => {
	console.log('here');
	const { room } = params;
	const upgradeHeader = request.headers.get('Upgrade');
	if (!upgradeHeader || upgradeHeader !== 'websocket') {
		return new Response('Durable Object expected Upgrade: websocket', {
			status: 426
		});
	}

	let id = platform.env.WEBSOCKET_HIBERNATION_SERVER.idFromName(room);
	let stub = platform.env.WEBSOCKET_HIBERNATION_SERVER.get(id);
	console.log('here2');
	return stub.fetch(request);
};

@gerhardcit here's the (very minimal) example I promised 🙂
https://github.com/dario-piotrowicz/sveltekit-durable-object-local-usage-example
Please let me know if this helps

This seemed to work but I was not able to use WebSockets. The problem seems to be that +server.ts files ignore ws:// requests. So even though I can get hold of the DO in an http:// GET the request created by new WebSocket('ws://') just gets ignored. Any one have a solution for this? I'd be keen to do this in the Svelete app so I can use my apps auth etc.
export const fallback: RequestHandler = async ({
	request,
	params,
	platform
}) => {
	console.log('here');
	const { room } = params;
	const upgradeHeader = request.headers.get('Upgrade');
	if (!upgradeHeader || upgradeHeader !== 'websocket') {
		return new Response('Durable Object expected Upgrade: websocket', {
			status: 426
		});
	}

	let id = platform.env.WEBSOCKET_HIBERNATION_SERVER.idFromName(room);
	let stub = platform.env.WEBSOCKET_HIBERNATION_SERVER.get(id);
	console.log('here2');
	return stub.fetch(request);
};

Hi unfortunately there is only 1 blog post about it but I wasn't able to get it working https://github.com/JoshAshby/joshashby.github.io/issues/22#event-12125121288
What I did instead is deploy a worker to upgrade to websocket connection.


Na values

The following columns are filled with pd.NA in their respective datasets when going through the extract and transform steps of the pipeline.
FERC 1

construction_type

EIA

natural_gas_delivery_contract_type_code
natural_gas_transport_code

The call df.to_sql() in sqlite.py converts the pd.NA values to NoneType since the column dtypes are not preserved. This problem is fixed after the fact by recasting the column to a string type once pudl.sqlite is read into a pandas dataframe. Preserving the column dtype is maybe something that should be addressed at some point.
The following columns are filled with np.nan in the EPA CEMS datasets, not pd.NA. This is because they are categorical type columns and thus do not allow for pd.NA.
EPA CEMS

co2_mass_measurement_code
nox_mass_measurement_code
nox_rate_measurement_code
so2_mass_measurement_code

Note
The column purchase_type is listed in the issue as an EIA column with an empty string in the enum. However this appears to be a FERC 1 column and the enum is based off the dictionary POWER_PURCHASE_TYPES_FERC1 in labels.py. There is no empty string listed in the POWER_PURCHASE_TYPES_FERC1 dictionary, so I didn't do anything about the purchase_type column.

Forgot to address this in above comments - I could also turn "unknown" into NAs for these columns if wanted

The call df.to_sql() in sqlite.py converts the pd.NA values to NoneType since the column dtypes are not preserved. This problem is fixed after the fact by recasting the column to a string type once pudl.sqlite is read into a pandas dataframe. Preserving the column dtype is maybe something that should be addressed at some point.

I think what's happening here is that the null values are getting loaded into SQLite just fine, and stored as appropriate NULL values as far as the database is concerned, but then pandas is doing it's standard datatype conversions when we call pd.read_sql() and it incorrectly converts the SQLite NULL values into None (which is often treated as a "null" object).
With the new metadata system we should be able to give pandas an explicit schema that it can use to type the columns it reads out of the DB appropriately, but that's a later issue.

I made a list of the fields with unknown or undetermined as well as some of the strings that map to these values:
FERC1
construction_type
'', 'automatic operation', 'comb. turb. installn', 'comb. turb. instaln',
'com. turb. installn', 'n/a', 'for detailed info.', 'for detailed info',
plant_kind
'', 'n/a', 'see pgs 402.1-402.3', 'see pgs 403.1-403.9', "respondent's share",
'--', '(see note 7)', 'other', 'not applicable', 'peach bottom', 'none.',
'fuel facilities', '0', 'not in service', 'none', 'common expenses', ..... many more
fuel_type
'', '1265', 'mwh units', 'composite', 'therms', 'n/a', 'mbtu/kg', 'uranium 235',
'oil', 'ccf', '2261', 'uo2', '(7)', 'oil #2', 'oil #6', '\x99å\x83\x90?"', .... many more

construction_type doesn't seem like a primary key and maybe is a good candidate to make NAs?

EPA CEMS
co2_mass_measurement_code
nox_mass_measurement_code
nox_rate_measurement_code
so2_mass_measurement_code
All accept "other", "undetermined", or "unknown code" - these could probably be made into NAs?

FERC 1
The FERC 1 construction_type, plant_kind and fuel_unit "cleaned strings" are things we assigned, so we won't mess up anyone else if we make them NA, but we might mess up our own code. There's an unmapped argument in pudl.helpers.cleanstrings() that we could use to set any unmapped strings to pd.NA but I don't think we want to do that, because we want it to be clear whether we've reviewed and attempted to categorize every one of these nonsense strings, and they add new ones every year. So we may want to go ahead and assign them unknown assert that every value in the column is one of our simplified values (so we catch anything new and unexpected) and then replace unknown with pd.NA.
EIA
We should look at what kinds of utilities are being assigned "unknown" and "other" entity types, and whether they are consistent across years, or if how they're reported has changed. Like did they used to be "unknown" and now they're "other"? Or did they eventually get categorized? What proportion of the overall population has these mystery types? how often does a utility entity type change? Can we fill in the missing values confidently? What does the EIA documentation say about this field (if anything)? Often this is in one of the non-data tabs of the raw input spreadsheets, or in the instructions (PDFs which we've tried to archive in our docs)
EPA CEMS
Similarly we should see what EPA is saying these fields / values mean, and try to understand if "undetermined" and "unknown code" and "other" mean different things, or if they've just changed the coding scheme over the years, and also what proportion of the dataset is affected. There's probably some documentation on web from EPA, but I'm not 100% sure where.  If we can't find it, we've been in contact with some of the EPA people in the past and might be able to ask for help.

Summary of some changes:
FERC 1
I find some strings in construction_type, plant_type, and fuel_unit that hadn't been caught yet by the dictionary or strings and put them in the unknown category of this dictionary. I also added asserts so that we know if there are more uncaught strings. Then I converted all the unknown to pd.NA after this assert is made.
EIA 860
entity_type has unknown as a valid value, however there aren't actually any data points that use this unknown field. I couldn't find much in the documentation about entity_type but other seems to be an EIA label, so I left all the other values in the dataframes. There are some data points (58,000 out of around 109,000 rows for all years) that have np.nan so I converted these to pd.NA
EPA CEMS
I couldn't find much documentation about the nox and co2 measurement codes, but it seems like the Undetermined and Unknown Code are rarely used, so I left them.


[iOS] Update Snapshots with CI is failed.

Overview (Required)

Because main is protected branch, so the CI failed.

Steps To Reproduce

Merge a branch that includes new snapshots.

Expected behavior

After above branch merged, snapshots are updated by CI.

Additional context

Change to process that opens PR is maybe better.
During not fixed this, please ignore snapshot-testing CI failing state.

I'll fix this tomorrow 🙏🏼


Ensure event timer is active

Overview
Ensure that the event timer is active on update_status. Add a log if the timer is not active.
Rationale
We found on one deployment that the event timer was not active / could not be reactivated, resulting in only update status being run and no reconciliation.
Juju Events Changes
The update_status hook now checks for the timer. There is a major refactoring going on in https://github.com/canonical/github-runner-operator/pull/199, the location of the check may be adjusted.
Checklist

[x] The charm style guide was applied
[x] The contributing guide was applied
[x] The changes are compliant with ISD054 - Managing Charm Complexity
[x] The documentation is generated using src-docs
[x] The documentation for charmhub is updated.
[x] The PR is tagged with appropriate label (urgent, trivial, complex)

Moved the check to the update_status hook. Reason: If placed in init, the code will be called before the install hook (first hook is runner_storage_attached), and the code would enable the timer that starts the service directly, and since the charm is not yet prepared (e.g. repo policy is not installed) it will fail and timeout after the reconciliation interval (this will resolve itself but still delay the installation for 10 minutes).


Infracost comment github --tag

Hi,
Context
Got a github repository which manage 2 environment, prod and non-prod. Theses environment are managed by github-action through matrix.
Infracost setup :
      - name: Setup Infracost
        uses: infracost/actions/setup@v1

Got an issue with the following command
infracost comment github --path /tmp/infracost-${{ matrix.environment }}.json \
                                   --repo $GITHUB_REPOSITORY \
                                   --github-token ${{github.token}} \
                                   --pull-request ${{github.event.pull_request.number}} \
                                   --behavior update \
                                   --tag "${{ matrix.environment }}"

Issue
The behaviour of --tag seems wrong.
Because of matrix job is executed twice.

The first one for example non-prod create a new comment which containing the tag "non-prod"
The second one, prod, update the comment created by non-prod and replace the tag

Expected

The first one non-prod create a new comment which contains the tag "non-prod"
The second one, prod, create a new comment  which contains the tag "prod"

@Laucans thanks for creating this issue! Can you please try using the tags "prod" and "dev" to see if that fixes the issue? I suspect the issue here is that "prod" is a substring of "non-prod" so the matching logic is not properly differentiating between them.

@alikhajeh1 thanks for answer.
So I tested with dev and prod and got the same behavior :

I can't show the tag when it has been updated but if I change for behavior new :

Oh wait, the matching logic take the whole message body, dev is in my project name..

Ok I changed dev for development and it's working.
thanks :)
Thank you.

This is fixed in 0.9.23


Feature : Dark Theme

Currently
No dark mode theme in your website which is now widely use by most of the website for better User Integration
Change
I can add dark mode button with sliding feature
I will use google icons for this
Sample Work (its just a sample you can also provide me the colors you want )
https://github.com/SrijanShovit/carbonops-v2/assets/101205226/915c64d7-941c-44b3-9532-13941c433761

Hi @shreya024 Please review it once

@parasss19 Assigned !

I will make it till Saturday as my exams are ongoing.

I want to work on this @shreya024


feat(rollup): private rollup

Context
https://app.asana.com/0/1205770721173533/1207097745265939
https://app.asana.com/0/1205770721173533/1207097745265941

Description
This PR makes it possible to use whitelist for deployment of private rollups. It also changes some log outputs.
Additionally a new runbook was created for deploying a private rollup - https://www.notion.so/trilitech/Runbook-private-rollup-635a2b66ff5048ba9b37967275d1bf53.

Manually testing the PR
Use the runbook to run the rollup.

lgtm -- need to test locally before merging though

ok, feel free to click on the merge button once you have done so.


Split tests into core/tasks

After #378, we should also run the tests in two batches: the ones for the core functions, and the ones for the whole package that also include tasks. This will protect us from issues like "I added a cellpose dependency in the core library". Not the most urgent thing to do, but it's always good to improve the tooling of the repo/package.
Note that pytest does offer some kind of test labels (which we would also use to run the CI locally), but we'll still need to properly set up the GitHub action so that:

It installs the core part
It runs the core tests
It install the tasks extra
It runs the tasks tests
It creates a unique coverage report

Side comment: splitting the tests into core+tasks will also make it clear that some of our core modules require a much more thorough test coverage, e.g. I just realized that lib_masked_loading.py has a very poor one.
For a task, it could be complex to create and maintain test data for all branches, but for the core modules I think we should aim for a very high quality (this is especially true because we plan to have other packages depending on the core part).

Agreed!
I also just realized how complex this can become for the napari workflows task when looking at the different 2D & 3D branches, those are hard to test exhaustively. But we should aim for high test coverage over our library functions.


added web app folder and docs folder for github pages

Hi
I've added the web app folder under eagleeye_web
Also added docs folder which contains the first draft of the web app, which should be added in the repo settings for github pages, so we can see it from the repo link
Let me know if you don't understand part's of the project I've created

Hi,
i have cloned your fork.
Here are a few suggestions:

Can you merge the gitignore from the web folder to the original one in the root directory?
can you rename eagleeye_web to just web?
Please remove docs/dist. What is this for? its just a npm run build right? (cause it looks the same as npm run serve).

Finally: Nice jobs looks good so far.
Regarding the backend: You can already generate a json report using --json  flag which you can then parse and display in vue i guess.
There is a easy way to hide the firefox while running so that shouldnt be the problem but the server still needed xorg since firefox is still not headless (just not shown that way).
I would only merge this pull request after its fully finished since i dont want to merge it right now if its not finished. You can keep me updated if you want. :)

Hi,

Sure
Sure
Docs folder is used by GitHub pages to host the web app. If you don't want the docs folder we can find an alternative for hosting no problems

This pr is only for the front end
I don't have a backend yet,  was thinking to use go or node for that not sure.
Any idea on where we can host backend and frontend?
Do you mean you would only merge with a working backend and frontend?

I dont really want to host that on github docs. I mean if you want to host it on github docs then this should just really be the frontend and you would need a seperate api server somewhere.
Frontend: server or github docs (its free right haha)
Backend: has to be a seperate server (can be selfghosted). Maybe a Python API written in Flask which just runs eagle eye (with the --json option) and parses the file if the job has finished? The flask API would of course also need a queue with uid of the jobs to not get confused.
You can if you want create a simple overview of the plan. For the Backend API i would rather have a seperate repository.
I might also want a seperate repository just for the frontend + backend since it dont want this to get mixed up uin the actual eagle eye program you know?
The web version should be a seperate component and not a glued in piece.

True, i think it should be in a separate repository too.
I'll try to get it working locally, but i'm not sure if I'm going to host it since most of the provider out there charge based on the use of the server.


Separate pipelines for Browser, Node, Expo and RN

Goal
Spilt the JS build into separate pipelines for Browsers, Node, Expo and React Native.
Design
A new pipeline for the React Native CLI will slot in next to these and the main pipeline contains "general" steps such as package/publishing and unit tests.  The package step may eventually disappear if Node/Browsers are switched over to using the NPM published artifacts.
Changeset
As usual with changes like this the diffs look far worse than the real changes.  It was almost copy/paste other than:

Moving the what was RN-specific publish step to the main pipeline.
Adding the build attribute on artifact download steps
Step dependencies reviewed

Testing
Covered by CI passing, combined with a sanity check that no build steps have been lost as part of the refactor.  The browser tests currently fail, as an update to our S3 secrets bucket is needed in order to access the required BrowserStack credentials.

Looks good, only suggestion is that we change the Browser, Node & Expo build steps to use the pre-packaged distro, as that would likely save several minutes from each.

Yes, agreed - I'll raise an internal ticket to address this before too long.


Please upload a wheel package to pypi

Builds using PEP517 and pyproject.toml no longer play nicely with pypi packages that require legacy build tools instead of simply installing a .whl file.
Please upload one.

I'm currently also getting the following warning:

DEPRECATION: docopt is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

@mboersma, @graingert or @keleshev, could one of you look into this?

@EwoutH I don't have access to pypi for the docopt project

You probably want https://github.com/jazzband/docopt-ng

AFACT, none of us have access to the pypi account except docopt's founder @keleshev, and we've failed to contact him for years. So docopt is effectively prevented from publishing a new release.
I very much encourage you to try docopt-ng!

Thanks! I will definitely check out docopt-ng!
FYI, there are options to take over an abandoned PyPI package. This is a bit cumbersome but a lot easier if you’re already maintainer of the project. I 5ink contacting PyPI could go a long way.
See for more details: https://stackoverflow.com/questions/20732179/pypi-can-i-claim-to-be-the-new-maintainer-of-an-unmaintained-package

This would be my last ditch attempt for this: sorry for spamming @keleshev, but can we please have this?
It only requires you to run pip wheel . in the root of the repo and upload the resulting file to pypi.


Switch component attr :rest does not include ~w(checked_value unchecked_value checked hidden_input)

We're seeing the following error when compiling our app with --warnings-as-errors:
Compiling 103 files (.ex)
warning: undefined attribute "checked_value" for component PetalComponents.Form.switch/1
  lib/colife_web/live/profile_live/show.ex:153: (file)

warning: undefined attribute "unchecked_value" for component PetalComponents.Form.switch/1
  lib/colife_web/live/profile_live/show.ex:153: (file)

Compilation failed due to warnings while using the --warnings-as-errors option

I've tracked it down to attr :rest in the switch component https://github.com/petalframework/petal_components/blob/main/lib/petal_components/form.ex#L598 not accepting the same values as the checkbox component https://github.com/petalframework/petal_components/blob/main/lib/petal_components/form.ex#L527-L529, when both call https://hexdocs.pm/phoenix_html/Phoenix.HTML.Form.html#checkbox/3-options.
Cheers.

@periclesd apologies for not getting back to you sooner, this was fixed in 1.4.1.


[decompiler] Incorrectly merged conditionals

(cond
  (a b)
  (c d)
  (else
   <stuff>
    (cond
      (e f)
      (else g)
      )
    )
  )

ends up as
(cond
  (a b)
  (c d)
  ((begin <stuff> e) f)
  (else g)
  )

If <stuff> is nothing (which it sometimes is), then this is pretty harmless.  But if <stuff> happens to contain a variable definition with let, it will make a huge mess.  So far I've only seen one example in (method 20 res-lump).
The plan for fixing this:

confirm that the bad merge happens inside of CfgVtx.cpp
Add a config option to prevent a block from being included as a non-first case within a cond.
Hopefully the CFG pass still completes and everything is good.

I don't see a way to do this automatically. I think both of my examples would end up as the same GOAL code.

fixed in #522


Add mjlshen as a reviewer

Planning to be active in this repo moving forward

Codecov Report

Merging #143 (7e78b44) into main (2919748) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #143   +/-   ##
=======================================
  Coverage   22.08%   22.08%
=======================================
  Files          18       18
  Lines         996      996
=======================================
  Hits          220      220
  Misses        744      744
  Partials       32       32


Unable to use Ruby Docker base image above: 3.2.3

The build fails due to some issue with the version of SQLite.
More details will follow

Have now tested with 3.2.6 and did not observe any issues.
Closing this issue


Add a compatibility note to README

List versions of ElasticSearch supported, related to #18.

fixed in #20 - while i haven't tested on ES 2.3 there appear to be no breaking changes which affect Estella.

Closing via https://github.com/artsy/estella/pull/20.


Issues with Olivetti mode

The modeline does not work correctly whenever Olivetti mode is enabled in a buffer.
Notice the Git status segment dissapearing and the weirdly placed separator.
The issue seems to be when Slim is toggled on.
With slim toggled off, the git segment returns.
Olivetti Disabled:

Olivetti Enabled:

After some more testing, it seems to be linum-mode (or changes to it, like activating git-gutter) that's causing it


Demo site giving the application error does not display anything- Please see the error message below

:( Application Error
If you are the application administrator, you can access the diagnostic resources.

New link has been added


Upgrade to Kryo 5

This includes required changes for use with the recently release kryo-5.0.0-RC1.

we'll update once kryo 5.0.0 has been released


squash plugin: turn from_base on by default

See https://github.com/projectatomic/atomic-reactor/issues/288#issuecomment-136359030

Very nice!


no result when building

Hello,
Thanks for your nice work.
I'm a beginner in sphinx so sorry if I miss obvious points.
I followed your tutorial and was waiting for any error.
Instead, I got a combo no error and and no visible result ... I' not seeing anything new in my html page.
I use Sphinx with separated source and build folder  :  docs/{source,build}
My git repo is a this level (docs)
`
Running Sphinx v3.1.2
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 3 source files that are out of date
updating environment: [new config] 3 added, 0 changed, 0 removed
reading sources... [ 33%] doc_artis/page1
reading sources... [ 66%] doc_artis/page2
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 33%] doc_artis/page1
writing output... [ 66%] doc_artis/page2
writing output... [100%] index
generating indices...  genindexdone
writing additional pages...  searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
The HTML pages are in build/html/3.1.
Running Sphinx v3.1.2
loading pickled environment... failed
failed: source directory has changed
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 3 source files that are out of date
updating environment: [new config] 3 added, 0 changed, 0 removed
reading sources... [ 33%] doc_artis/page1
reading sources... [ 66%] doc_artis/page2
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 33%] doc_artis/page1
writing output... [ 66%] doc_artis/page2
writing output... [100%] index
generating indices...  genindexdone
writing additional pages...  searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
The HTML pages are in build/html/3.2.
Running Sphinx v3.1.2
loading pickled environment... failed
failed: source directory has changed
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 3 source files that are out of date
updating environment: [new config] 3 added, 0 changed, 0 removed
reading sources... [ 33%] doc_artis/page1
reading sources... [ 66%] doc_artis/page2
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 33%] doc_artis/page1
writing output... [ 66%] doc_artis/page2
writing output... [100%] index
generating indices...  genindexdone
writing additional pages...  searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
The HTML pages are in build/html/master.`
Have you met this before ?

Also, I'm using the  RTD theme and using a template with this
`{% extends "page.html" %}
{% block body %}
{% if current_version and latest_version and current_version != latest_version %}

    {% if current_version.is_released %}
    You're reading an old version of this documentation.
    If you want up-to-date information, please have a look at {{latest_version.name}}.
    {% else %}
    You're reading the documentation for a development version.
    For the latest released version, please have a look at {{latest_version.name}}.
    {% endif %}

{% endif %}
{{ super() }}
{% endblock %}%`

I think the html widget is just not displayed despite the fact I followed your recommendations to use the RTD theme.

What HTML widget? If you don't add a custom template, the HTML output will not be modified.
If you're using the RTD Themse, have a look at: https://holzhaus.github.io/sphinx-multiversion/master/templates.html#readthedocs-theme
I'll close this issue for now, as it doesn't look like a bug to me. If the issue persists, please reopen and post some more info, e. g. your config and the custom template.

Yes I just solved it, thanks

What was the solution for this? Followed tutorial and pages do not build with a change in navigation bar for the rtd theme.


AWS STS support for Vault Secrets Backend

Would like to get STS working with the vault secrets backend for aws under spring vault.
Currently, it appears that spring vault only supports IAM User or (/aws/creds/:name) and not other credential types such as STS federation token or Assumed role (/aws/sts/:name). I do not see any configuration options documented for alternate credential types.

Related issues(goes back a few years - 2018):
https://github.com/spring-cloud/spring-cloud-vault/issues/194

Will reopen under the spring cloud vault repo.


Support panic and let assert

Hello!
The upcoming Gleam release as 2 syntax changes. panic (which can go anywhere an expression goes, like todo but without the string argument), and let assert <pattern> = <expression>, which is the new syntax for assert <pattern> = <expression>.
Thanks!

should assert simply be replaced by let assert or should both work?

Both are currently valid Gleam syntax.

https://github.com/gleam-lang/tree-sitter-gleam/pull/46
https://github.com/gleam-lang/tree-sitter-gleam/pull/48

Closed by #46 and #50


Address go type is missing AddressStatus property

Describe the bug
The Address resource has a read only property "status" that isn't represented in the go type.
To Reproduce
Steps to reproduce the behavior:

Be writing a client in golang
Deploy an Address resource
In golang fetch your address
There is no status object on the address type.

Expected behavior
Go types should have the address
Additional context
a example for the go type :
type Address struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   AddressSpec   `json:"spec"`
	Status AddressStatus `json:"status"`
}

type AddressStatus struct {
	IsReady bool `json:"isReady"`
}

I've patched a copy of my types in my vendor directory with this and am able to access the status thusly (apologies for the out of context code)
foundQueue := newQueue(instance, address) //Sets some defaults for my use case
err = r.client.Get(context.TODO(), types.NamespacedName{Name: foundQueue.Name, Namespace: foundQueue.Namespace}, foundQueue) //fetches
/* snip error check*/
if !foundQueue.Status.IsReady {
				reqLogger.Info("Queue Not ready", "Queue.Name", foundQueue.Name)
				requeueCreate = true
			}

@secondsun thanks for the report.  Indeed it is absent.  Please feel free to raise a PR, that would be much appreciated.

https://github.com/EnMasseProject/enmasse/pull/2948


Prefix IOV's chains with iov-*

The BNS is called IOV Name Service now, which is powered by the IOV utility token.
So we should adapt the chain ID spec to prefix IOV's networks with iov-

@webmaster128 Will there be other issues for other BNS renamings? Are these on the MVP track?

@willclarktech Right now I only bother about the things that will end up as in-chain data, which is almost impossible to change later on. Those should be changed before mainnet launch. All the rest can stay as is.


Inconsistent errors on GLTF export from Blender after modifying action

I have a model with 7 animations that exports ok, with or without actions being pushed down into NLA tracks, or saved as data blocks. I introduce a new animation following a workflow identical to the other seven (all rigged and exported in Mixamo, then imported to Blender 2.79b as FBX). The exporter will then typically start throwing errors, even though no settings have changed. A voodoo process ensues, trying things like purging data, removing then relinking actions, appending from a .blend file instead of importing the FBX directly, reinstalling the exporter, clearing keyframes, quitting and restarting, saving as a new file name, etc.
Sometimes one of these things works - or something else happens that makes it appear as though one of these things works. There's no rhyme or reason that I can discern. Establishing a reliable workflow for modifying an animation or introducing a new action has become fraught with fear.
This issue is seemingly related to 39 and 166. So far no workaround I've tried is very predictable.
Occasionally bones import from Mixamo with a -1 orientation on a given quaternion, which I'll correct in Blender. I've also noticed a high degree of weirdness when re-exporting things that play back fine in Blender, then break apart (unexpected rotations) when loaded in a three.js context. Applying all the loc/rot/scale makes no difference. Again, there's not a pattern for why this happens that I can figure out. Unsure if it's somehow related.
The steps that seem to semi-reliably get things exporting properly again include: purging data > clearing all animation data in the scene view, relinking actions by running through each of them in the action editor, saving as a new .blend file, quitting blender, reopening the file, and exporting. When that fails, as a last resort I will clear keyframes (but this often requires a fair amount of cleanup, or redoing an animation from scratch - ugh). Even this last step only works sporadically.
If I had any hair left to pull out I would. This file is an example of one that was broken a bit ago, but started working again after taking the step of clearing keyframes on the root bone of the newly added action.
I'm currently using the most recent build of the gltf 2.0 Import-Export package.
Any feedback/tips much appreciated, and thanks for all the great work.

This is maybe link to issue #461 about NLA that can have some issue in some cases

Hello @aoedipus Can you please test again with a recent build of blender, for exemple with Blender 2.82 beta, that you can download here : https://builder.blender.org/download/


compatible with django 1.8

Coverage decreased (-0.08%) to 54.24% when pulling a5be98d27da68b45a12c13d2c97f0189d9aa26c5 on bashu:patch-3 into 0030c1a3f45f74c8120eb6eb26068a41ef6f6025 on jp74:master.


Provide models with initial data

Sometimes a model needs information at initialization time, that you want to provide in a flexible way, something like:
class MyModel(app.Model):

    def init(self, the_info):
         ...

...

m = MyModel(the_info=...)

Currently, the init() is always argumentless. The required behavior can be achieved by making a property info, the value of which can be used inside the init(). However, a property suggests that the info can be modified/reset after the model is instantiated, which is not always the case.
In other words, there are situations where a Model needs information at initialization time, but for which a property does not make sense conceptually.
Some possible solutions:
A. Any keyword arguments used to instantiate a Model which do not correspond to properties are passed to the init() as kwargs. You'd thus need to write it like def init(self, info=None) to make it a keyword arg. In theory it could also map to positional args by inspecting the signature, but that feels a bit too magical, I think.
I am a bit worried about error reporting if a user makes a type in a property name. We would need to inspect the init() sig to see what kwargs it accepts, so we can be sure that a kwarg is intended for the init(). This should thus be disallowed: def init(self, **kwargs).
B. Provide a new kind of property, similar to the current readonly that must be provided at initialization and is readonly afterwards. This adds less conceptual stuff for the user to learn, as it's just a kind of property.
C. Use positional arguments. The Model.__init__() could catch the positional arguments and pass them to the init(). The only downside is that for Models run as an app via app.launch or app.serve we'd need to provide a way to set these args, but we already do that for the kwargs, so I guess that could work ...
cc @korijn @Maxyme

Added option C (see above): using positional arguments.


Update autorest version to prerelease 3.0.0-alpha.20240801.2

Generated by Autorest Regen Preview build 20240801.2Triggered from pull request: https://github.com/Azure/autorest.csharp/pull/4899

API change check
API changes are not detected in this pull request.


Add deprecation warnings

There is still some work to do:

stop using common/select-action in favor of components/dropdown or icon-dropdown
move application/loading bar to components/layout
change common/progress-bar in loading bar to component/progress-bar

Linked to #1491 #1492


jetbrains EAP products

Project description
The EAP versions of jetbrains products (such as IDEA, Rider, Webstorm, etc). These are experimental/beta versions of various jetbrains products
Metadata

homepage URL: https://www.jetbrains.com/resources/eap/
source URL: Depends on the product
license: unfree
platforms: linux, darwin

@edwtjo @mic92 since you are the maintainers of jetbrains packages

I've started adding these (clion & rider for now) in https://github.com/NixOS/nixpkgs/pull/219902

I just found this issue by accident. Since the EAP updates extremely frequently and is not always available (when a new GA version is just released, there is no EAP version), I think adding them to nixpkgs would bring more problems than solutions.
This can be easily achieved with a simple override. This could be exposed as a function allowing the consumer only to pass version and sha256, but that implies that every version would work. Almost every version works, but sometimes there are changes to the JRE.
I'm doing this on my configurations, if anyone wants to copy that feel free:
https://github.com/Fryuni/config-files/blob/7decca34ea4e90c030708b59406337f4345bac0c/overlay/jetbrains.nix
In that commit pycharm-professional is using the EAP version. The URLs in the comments are where to get the EAP version number, which can be used interchangeably with the release versions.

After some discoussion in the NixOS discord, apparently you can use a netfilter such an EAP IDE can't phone back to jetbrains, skipping the update check.
This is done by @spikespaz in their dotfiles with ja-netfilter:
https://github.com/spikespaz/dotfiles/blob/master/users/jacob/programs/jetbrains.nix
https://github.com/spikespaz/dotfiles/blob/master/packages/ja-netfilter/base.nix
Perhaps it could be convenient to have the netfilter wrappers in nixpkgs?

I don't know if this is against their TOS or not, so just wanted to open up the discussion.

Using this source code requires a subscription service. To me.


[future] django will change the behaviour how to handle/re-render rendered template

this is currently only applies to django master
the frontend menu of django-material is defined in this code. Using django master (1.11-alpha), rendering menu will raise:
  File "/Users/ayik/Repo/Django/django/django/template/context.py", line 285, in make_context
    raise TypeError('context must be a dict rather than %s.' % context.__class__.__name__)
TypeError: context must be a dict rather than RequestContext.
[11/Jan/2017 16:16:29] "GET /employees/departments/ HTTP/1.1" 500 321465

i submitted a bug in django trac, but @timgraham said that it was intended new behaviour according to this discussion

Thanks for the report. There are always a lot of incompatible changes over django releases that affects such low-level dependant libs like django-material. For sure I'll work on this, but only after django 1.11 release.

alpha1 is released

I made a new ticket - https://code.djangoproject.com/ticket/27770#ticket
This is clear new django version mistake, that's usually became fixed till the X.X.1 release.
That's why I usually prefer, not even take a look into even to alpha versions.

From a Django maintainer perspective, I appreciate all the testing we can get with prelease versions so we can fix issues before the final release.

@kmmbvnr drf also added alpha versions testing https://github.com/tomchristie/django-rest-framework/commit/5e9249f42926a238dc03e70f7605a9417de0fc44

Sorry, I'm not in this race.


portability updates and output from speakerDiarization function

Hi,
We looked at your pyAudioAnalysis module for a hackathon last weekend (http://labhack.org). One of the things we found was that the data directory with the classification data is not portable. I updated the audioSegmentation.py file to allow for portability from system to system and calling location.
I also modified the speakerDiarization function to output it's results so that they could be compared with the data we had to see how well your algorithm was working.
AAron

I also noticed this. Solved it by simply setting the path to the location of the PyAudioAnalysis base directory and then tweaking the function a bit:
import numpy
import sklearn.cluster
import time
import scipy
import os
from pyAudioAnalysis import audioFeatureExtraction as aF
from pyAudioAnalysis import audioTrainTest as aT
from pyAudioAnalysis import audioBasicIO
import matplotlib.pyplot as plt
from scipy.spatial import distance
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import sklearn.discriminant_analysis
import csv
import os.path
import sklearn
import sklearn.cluster
import hmmlearn.hmm
import cPickle
import glob
pAA_dir='/Users/PSYC-dap3463/Applications/pyAudioAnalysis'

##SpeakerDiarization function tweak
def speakerDiarization(fileName, numOfSpeakers, mtSize=2.0, mtStep=0.2, stWin=0.05, LDAdim=35, PLOT=False):
    '''
    ARGUMENTS:
        - fileName:        the name of the WAV file to be analyzed
        - numOfSpeakers    the number of speakers (clusters) in the recording (<=0 for unknown)
        - mtSize (opt)     mid-term window size
        - mtStep (opt)     mid-term window step
        - stWin  (opt)     short-term window size
        - LDAdim (opt)     LDA dimension (0 for no LDA)
        - PLOT     (opt)   0 for not plotting the results 1 for plottingy
    '''
    [Fs, x] = audioBasicIO.readAudioFile(fileName)
    x = audioBasicIO.stereo2mono(x)
    Duration = len(x) / Fs

    [Classifier1, MEAN1, STD1, classNames1, mtWin1, mtStep1, stWin1, stStep1, computeBEAT1] = aT.loadKNNModel(os.path.join(pAA_dir,"data","knnSpeakerAll"))
    [Classifier2, MEAN2, STD2, classNames2, mtWin2, mtStep2, stWin2, stStep2, computeBEAT2] = aT.loadKNNModel(os.path.join(pAA_dir,"data","knnSpeakerFemaleMale"))

    [MidTermFeatures, ShortTermFeatures] = aF.mtFeatureExtraction(x, Fs, mtSize * Fs, mtStep * Fs, round(Fs * stWin), round(Fs*stWin * 0.5))

    MidTermFeatures2 = numpy.zeros((MidTermFeatures.shape[0] + len(classNames1) + len(classNames2), MidTermFeatures.shape[1]))

    for i in range(MidTermFeatures.shape[1]):
        curF1 = (MidTermFeatures[:, i] - MEAN1) / STD1
        curF2 = (MidTermFeatures[:, i] - MEAN2) / STD2
        [Result, P1] = aT.classifierWrapper(Classifier1, "knn", curF1)
        [Result, P2] = aT.classifierWrapper(Classifier2, "knn", curF2)
        MidTermFeatures2[0:MidTermFeatures.shape[0], i] = MidTermFeatures[:, i]
        MidTermFeatures2[MidTermFeatures.shape[0]:MidTermFeatures.shape[0]+len(classNames1), i] = P1 + 0.0001
        MidTermFeatures2[MidTermFeatures.shape[0] + len(classNames1)::, i] = P2 + 0.0001

    MidTermFeatures = MidTermFeatures2    # TODO
    # SELECT FEATURES:
    #iFeaturesSelect = [8,9,10,11,12,13,14,15,16,17,18,19,20];                                                                                # SET 0A
    #iFeaturesSelect = [8,9,10,11,12,13,14,15,16,17,18,19,20, 99,100];                                                                                # SET 0B
    #iFeaturesSelect = [8,9,10,11,12,13,14,15,16,17,18,19,20, 68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,
    #   97,98, 99,100];     # SET 0C

    iFeaturesSelect = [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53]                           # SET 1A
    #iFeaturesSelect = [8,9,10,11,12,13,14,15,16,17,18,19,20,41,42,43,44,45,46,47,48,49,50,51,52,53, 99,100];                                          # SET 1B
    #iFeaturesSelect = [8,9,10,11,12,13,14,15,16,17,18,19,20,41,42,43,44,45,46,47,48,49,50,51,52,53, 68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98, 99,100];     # SET 1C

    #iFeaturesSelect = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53];             # SET 2A
    #iFeaturesSelect = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 99,100];     # SET 2B
    #iFeaturesSelect = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98, 99,100];     # SET 2C

    #iFeaturesSelect = range(100);                                                                                # SET 3
    #MidTermFeatures += numpy.random.rand(MidTermFeatures.shape[0], MidTermFeatures.shape[1]) * 0.000000010

    MidTermFeatures = MidTermFeatures[iFeaturesSelect, :]

    (MidTermFeaturesNorm, MEAN, STD) = aT.normalizeFeatures([MidTermFeatures.T])
    MidTermFeaturesNorm = MidTermFeaturesNorm[0].T
    numOfWindows = MidTermFeatures.shape[1]

    # remove outliers:
    DistancesAll = numpy.sum(distance.squareform(distance.pdist(MidTermFeaturesNorm.T)), axis=0)
    MDistancesAll = numpy.mean(DistancesAll)
    iNonOutLiers = numpy.nonzero(DistancesAll < 1.2 * MDistancesAll)[0]

    # TODO: Combine energy threshold for outlier removal:
    #EnergyMin = numpy.min(MidTermFeatures[1,:])
    #EnergyMean = numpy.mean(MidTermFeatures[1,:])
    #Thres = (1.5*EnergyMin + 0.5*EnergyMean) / 2.0
    #iNonOutLiers = numpy.nonzero(MidTermFeatures[1,:] > Thres)[0]
    #print iNonOutLiers

    perOutLier = (100.0 * (numOfWindows - iNonOutLiers.shape[0])) / numOfWindows
    MidTermFeaturesNormOr = MidTermFeaturesNorm
    MidTermFeaturesNorm = MidTermFeaturesNorm[:, iNonOutLiers]

    # LDA dimensionality reduction:
    if LDAdim > 0:
        #[mtFeaturesToReduce, _] = aF.mtFeatureExtraction(x, Fs, mtSize * Fs, stWin * Fs, round(Fs*stWin), round(Fs*stWin));
        # extract mid-term features with minimum step:
        mtWinRatio = int(round(mtSize / stWin))
        mtStepRatio = int(round(stWin / stWin))
        mtFeaturesToReduce = []
        numOfFeatures = len(ShortTermFeatures)
        numOfStatistics = 2
        #for i in range(numOfStatistics * numOfFeatures + 1):
        for i in range(numOfStatistics * numOfFeatures):
            mtFeaturesToReduce.append([])

        for i in range(numOfFeatures):        # for each of the short-term features:
            curPos = 0
            N = len(ShortTermFeatures[i])
            while (curPos < N):
                N1 = curPos
                N2 = curPos + mtWinRatio
                if N2 > N:
                    N2 = N
                curStFeatures = ShortTermFeatures[i][N1:N2]
                mtFeaturesToReduce[i].append(numpy.mean(curStFeatures))
                mtFeaturesToReduce[i+numOfFeatures].append(numpy.std(curStFeatures))
                curPos += mtStepRatio
        mtFeaturesToReduce = numpy.array(mtFeaturesToReduce)
        mtFeaturesToReduce2 = numpy.zeros((mtFeaturesToReduce.shape[0] + len(classNames1) + len(classNames2), mtFeaturesToReduce.shape[1]))
        for i in range(mtFeaturesToReduce.shape[1]):
            curF1 = (mtFeaturesToReduce[:, i] - MEAN1) / STD1
            curF2 = (mtFeaturesToReduce[:, i] - MEAN2) / STD2
            [Result, P1] = aT.classifierWrapper(Classifier1, "knn", curF1)
            [Result, P2] = aT.classifierWrapper(Classifier2, "knn", curF2)
            mtFeaturesToReduce2[0:mtFeaturesToReduce.shape[0], i] = mtFeaturesToReduce[:, i]
            mtFeaturesToReduce2[mtFeaturesToReduce.shape[0]:mtFeaturesToReduce.shape[0] + len(classNames1), i] = P1 + 0.0001
            mtFeaturesToReduce2[mtFeaturesToReduce.shape[0]+len(classNames1)::, i] = P2 + 0.0001
        mtFeaturesToReduce = mtFeaturesToReduce2
        mtFeaturesToReduce = mtFeaturesToReduce[iFeaturesSelect, :]
        #mtFeaturesToReduce += numpy.random.rand(mtFeaturesToReduce.shape[0], mtFeaturesToReduce.shape[1]) * 0.0000010
        (mtFeaturesToReduce, MEAN, STD) = aT.normalizeFeatures([mtFeaturesToReduce.T])
        mtFeaturesToReduce = mtFeaturesToReduce[0].T
        #DistancesAll = numpy.sum(distance.squareform(distance.pdist(mtFeaturesToReduce.T)), axis=0)
        #MDistancesAll = numpy.mean(DistancesAll)
        #iNonOutLiers2 = numpy.nonzero(DistancesAll < 3.0*MDistancesAll)[0]
        #mtFeaturesToReduce = mtFeaturesToReduce[:, iNonOutLiers2]
        Labels = numpy.zeros((mtFeaturesToReduce.shape[1], ));
        LDAstep = 1.0
        LDAstepRatio = LDAstep / stWin
        #print LDAstep, LDAstepRatio
        for i in range(Labels.shape[0]):
            Labels[i] = int(i*stWin/LDAstepRatio);
        clf = sklearn.discriminant_analysis.LinearDiscriminantAnalysis(n_components=LDAdim)
        clf.fit(mtFeaturesToReduce.T, Labels)
        MidTermFeaturesNorm = (clf.transform(MidTermFeaturesNorm.T)).T

    if numOfSpeakers <= 0:
        sRange = range(2, 10)
    else:
        sRange = [numOfSpeakers]
    clsAll = []
    silAll = []
    centersAll = []

    for iSpeakers in sRange:
        k_means = sklearn.cluster.KMeans(n_clusters = iSpeakers)
        k_means.fit(MidTermFeaturesNorm.T)
        cls = k_means.labels_
        means = k_means.cluster_centers_

        # Y = distance.squareform(distance.pdist(MidTermFeaturesNorm.T))
        clsAll.append(cls)
        centersAll.append(means)
        silA = []; silB = []
        for c in range(iSpeakers):                                # for each speaker (i.e. for each extracted cluster)
            clusterPerCent = numpy.nonzero(cls==c)[0].shape[0] / float(len(cls))
            if clusterPerCent < 0.020:
                silA.append(0.0)
                silB.append(0.0)
            else:
                MidTermFeaturesNormTemp = MidTermFeaturesNorm[:,cls==c]            # get subset of feature vectors
                Yt = distance.pdist(MidTermFeaturesNormTemp.T)                # compute average distance between samples that belong to the cluster (a values)
                silA.append(numpy.mean(Yt)*clusterPerCent)
                silBs = []
                for c2 in range(iSpeakers):                        # compute distances from samples of other clusters
                    if c2!=c:
                        clusterPerCent2 = numpy.nonzero(cls==c2)[0].shape[0] / float(len(cls))
                        MidTermFeaturesNormTemp2 = MidTermFeaturesNorm[:,cls==c2]
                        Yt = distance.cdist(MidTermFeaturesNormTemp.T, MidTermFeaturesNormTemp2.T)
                        silBs.append(numpy.mean(Yt)*(clusterPerCent+clusterPerCent2)/2.0)
                silBs = numpy.array(silBs)
                silB.append(min(silBs))                            # ... and keep the minimum value (i.e. the distance from the "nearest" cluster)
        silA = numpy.array(silA);
        silB = numpy.array(silB);
        sil = []
        for c in range(iSpeakers):                                # for each cluster (speaker)
            sil.append( ( silB[c] - silA[c]) / (max(silB[c],  silA[c])+0.00001)  )        # compute silhouette

        silAll.append(numpy.mean(sil))                                # keep the AVERAGE SILLOUETTE

    #silAll = silAll * (1.0/(numpy.power(numpy.array(sRange),0.5)))
    imax = numpy.argmax(silAll)                                    # position of the maximum sillouette value
    nSpeakersFinal = sRange[imax]                                    # optimal number of clusters

    # generate the final set of cluster labels
    # (important: need to retrieve the outlier windows: this is achieved by giving them the value of their nearest non-outlier window)
    cls = numpy.zeros((numOfWindows,))
    for i in range(numOfWindows):
        j = numpy.argmin(numpy.abs(i-iNonOutLiers))
        cls[i] = clsAll[imax][j]

    # Post-process method 1: hmm smoothing
    for i in range(1):
        startprob, transmat, means, cov = trainHMM_computeStatistics(MidTermFeaturesNormOr, cls)
        hmm = hmmlearn.hmm.GaussianHMM(startprob.shape[0], "diag")            # hmm training
        hmm.startprob_ = startprob
        hmm.transmat_ = transmat
        hmm.means_ = means; hmm.covars_ = cov
        cls = hmm.predict(MidTermFeaturesNormOr.T)

    # Post-process method 2: median filtering:
    cls = scipy.signal.medfilt(cls, 13)
    cls = scipy.signal.medfilt(cls, 11)

    sil = silAll[imax]                                        # final sillouette
    classNames = ["speaker{0:d}".format(c) for c in range(nSpeakersFinal)];

    # load ground-truth if available
    gtFile = fileName.replace('.wav', '.segments');                            # open for annotated file
    if os.path.isfile(gtFile):                                    # if groundturh exists
        [segStart, segEnd, segLabels] = readSegmentGT(gtFile)                    # read GT data
        flagsGT, classNamesGT = segs2flags(segStart, segEnd, segLabels, mtStep)            # convert to flags

    if PLOT:
        fig = plt.figure()
        if numOfSpeakers>0:
            ax1 = fig.add_subplot(111)
        else:
            ax1 = fig.add_subplot(211)
        ax1.set_yticks(numpy.array(range(len(classNames))))
        ax1.axis((0, Duration, -1, len(classNames)))
        ax1.set_yticklabels(classNames)
        ax1.plot(numpy.array(range(len(cls)))*mtStep+mtStep/2.0, cls)

    if os.path.isfile(gtFile):
        if PLOT:
            ax1.plot(numpy.array(range(len(flagsGT)))*mtStep+mtStep/2.0, flagsGT, 'r')
        purityClusterMean, puritySpeakerMean = evaluateSpeakerDiarization(cls, flagsGT)
        print "{0:.1f}\t{1:.1f}".format(100*purityClusterMean, 100*puritySpeakerMean)
        if PLOT:
            plt.title("Cluster purity: {0:.1f}% - Speaker purity: {1:.1f}%".format(100*purityClusterMean, 100*puritySpeakerMean) )
    if PLOT:
        plt.xlabel("time (seconds)")
        #print sRange, silAll
        if numOfSpeakers<=0:
            plt.subplot(212)
            plt.plot(sRange, silAll)
            plt.xlabel("number of clusters");
            plt.ylabel("average clustering's sillouette");
        plt.show()
    return cls

flags = speakerDiarization(fileName, numOfSpeakers, mtSize=2.0, mtStep=0.2, stWin=0.05, LDAdim=35, PLOT=False)


semantics of inverse_throughput_cycles  and prefix_inverse_throughput_cycles

Hello, is it accurate to say that inverse_throughput_cycles refers to CPI (cycles per instruction)?
I do not understand the description of prefix_inverse_throughput_cycles in throughput.py: "The number of cycles of inverse throughput of the prefixes of the basic block.". There can be prefixes like REP for instructions, but what are the prefixes of a basic block?
Thanks!
Hongbo

It's not correct to say that inverse_throughput_cycles refers to CPI. It refers to the number of cycles needed per block execution (reciprocal of throughput), where the block is measured in steady state. I think the BHive/UICA papers have the exact formula (essentially doing two measurements, one twice as long as the other, and then finding the difference to remove startup effects).
The definition in the code could probably use a bit of improving though.
Not sure what prefix_inverse_throughput_cycles is used for. I believe it is referring to a previous block. @ondrasej will have more context on that. To me it kind of seems like support for an experiment at some point that didn't end up panning out (it doesn't show up in any of the importers), but I could be wrong there.

I see. Thanks, Aiden!

I'm sorry for the late response.

I do not understand the description of prefix_inverse_throughput_cycles in throughput.py: "The number of cycles of inverse throughput of the prefixes of the basic block.". There can be prefixes like REP for instructions, but what are the prefixes of a basic block?

The basic block prefixes have nothing to do with instruction prefixes (REP, LOCK, ...). At one point, we tested the idea that the models would be easier to train if we had per-instruction contributions for each basic block, and we could use them all at the same time during the training process. So, instead of measuring only
        lea     eax, [rdi-1]
        imul    eax, edi
        shr     eax

and training on the block as a whole, we'd measure also
        lea     eax, [rdi-1]
        imul    eax, edi

and
        lea     eax, [rdi-1]

and take the increment from N instructions to N+1 as the inverse throughput contributed by the one additional instruction. We'd then train the model to predict the inverse throughput as a sum of these increments.
This was more interesting for the LSTM-based (Ithemal-like) models, because in theory it could allow them to generalize to much longer sequences of instructions than can handle now. But in the end, it didn't bring the expected improvements, and in practice, models trained only with the sum performed just as well, but they were faster to train. But we didn't remove the code, and it's still there for people to experiment with.
For the graph models (GRANITE), this approach (compute per-instruction increments, and train them only with the sum) proved to work the best as well.
Btw. I've opened the discussion section here in the repository. Feel free to use it for any other questions.

Yes, this makes a lot of sense. Computing the delta contribution of each instruction should be able to enable longer sequences and more precise prediction. I guess this might be applied to identify performance bottlenecks, e.g. build a critical path.


fix: fix dependency mismatch

Summary
There was a dependency mismatch between @typescript-eslint packages. This resolves the issue
Testing

Make sure github actions pass

LGTM


Example calculation does not make sense in the general notes section

On page: https://library.threefold.me/info/threefold#/manual_tfgrid3/threefold__grid3_components
The general notes section does not make sense:
Each price defines a price for a single UNIT/SECOND. So for example SU is the price of a single Storage Unit per second where a Storage Unit can be 1 Gigabyte
The price is defined as mil of the currency. 1 UNIT = 10,000,000 mil
Example:

Currency: TFT SU: 1000 (mil TFT)

(Price for 1 Gigabyte of ssd storage costs 1000 / 10 000 000 TFT per second)

So let’s assume capacity is created at Time = T

Node send first report at T+s with SU consumption C = (10 gigabytes)
price = C/(102410241024) * s * SU
price = 10 * s * SU
Assume s is 5 min (300 seconds)
Then price = 10 * 300 * 1000 = 3000000 mil = 3 TFT
Same for each other unit EXCEPT the NU. The NU unit is incremental. It means the node will keep reporting the total amount of bytes consumed since the machine starts. So to correctly calculate the consumption over period S it has to be now.NU - last.NU

https://library.threefold.me/info/manual/#/grid3_components
still does not make sense

@rkhamis @scottyeager could we please take a look and perhaps reem, point me to someone who could fix this

https://github.com/threefoldfoundation/info_manual3/pull/193