Does Prisma offer Transactions across multiple files?

Discussed in https://github.com/prisma/prisma/discussions/12974

Originally posted by jaequery April 23, 2022
Coming from TypeORM, I am surprised to see that Prisma does not have any way to perform transactions in a similar way TypeORM does.
With TypeORM and the transactional-cls-hooked decorator, we can do such as:
@Transactional()
async updateManyThings() {
   await this.productService.updateStuff();
   await this.categoryService.updateAnotherStuff();
   await this.customerService.changeOtherStuff();
}

This allows so that if there is an error on any of the service calls, it'd safely rollback.
Does Prisma offer anything close to that? Only thing that I've come across is to perform the operation all inside a single method, which is not all that useful for us, as we rely heavily on using service files to perform mutable operations respectively.

https://www.prisma.io/docs/concepts/components/prisma-client/transactions
I think this is what you need.

https://www.prisma.io/docs/concepts/components/prisma-client/transactions
I think this is what you need.

I am not sure if any of them works across different files but do correct me if I'm wrong.

Any thought on doing this @jaequery, @salvajavier: Returning $transaction to the user and pass it to the API interface as a argument. Something like this:
const transactionObj = prisma.createTransaction();
const user = await prisma.user.create({
  data: {
    email<EMAIL_ADDRESS>    name: 'Elsa Prisma',
  },
  { transactions: transactionObj }
})
const temp = prisma.temp.update({ where: { id: 1}, data: {f1: 1},  { transactions: transactionObj }})

@jaequery, @kasir-barati I think the point is that you cannot return a transaction built in another function because if you do it just run. So as the prisma.transaction$ functions receives an array as argument I do this:
async function mainFunction(){

    //This function can be anywhere an returns an array of transactions.
    let prismaExternalOperations = await getPrismaOperation()

    let prismaLocalOperation= prisma.product.update()

    // Use the spread operator to pass to prisma.transactions$ function all the arguments
    // returned from getPrismaOperation
    prisma.transactions$([prismaLocalOperation, ...prismaExternalOperations])
}

function prismaOperation (data) {
    let transactions$ = []

    let firstTransaction = prisma.customer.create({data})
    let secondTransaction = prisma.local.create({data})

    transactions$.push(firstTransaction, secondTransaction)

    // Instead of returning the prisma operation you wrap them into an array so they can be returned without executing.
    return transactions$
}

I'm facing this same problem. I created repositories for each of my database entities, to be able to isolate the prism of my use cases. But in services where I need to interact with different repositories, I get stuck in not being able to control the transaction of the repositories' methods, in a centralized way.
Is there any existing way to resolve this?

Hi, I'm from the Issues https://github.com/prisma/prisma/issues/12458 @millsp said I should look there for other issues.
I am working in a NestJs project with the CleanArchitecture. That means, we have our business logic in a domain layer and call single accessors from there, which then call prisma functions around a model (e.g. user).
Since we have business logic that needs to access multiple accessors, we can't use the callback transaction at all.
Here is a super simplified example:
@Injectable()
export class UserAccessor {
  constructor(private prismaClient: PrismaClient) {
  }

  async create(...): Promise<User> {
    return this.prismaClient.user.create(...)
  }
}

@Injectable()
export class SettingsAccessor {
  constructor(private prismaClient: PrismaClient) {
  }

  async create(user: User): Promise<Settings> {
    return this.prismaClient.userSettings.create(...)
  }
}

@Injectable()
export class UserDomain {
  constructor(private readonly userAccessor: UserAccessor,
              private readonly settingsAccessor: SettingsAccessor,
  ) {
  }

  async createUser(...) {
    // start Transaction
    const user = await this.userAccessor.create(...)
    // maybe more stuff done between this accessor calls
    const settings = await this.settingsAccessor.create(user)
    // end Transaction
  }
}

As @sijakubo pointed this out in the other Issue, Java/Spring/Hibernate uses this too in form of the @Transactional() Annotation around a service or function and also in tests. Spring Transaction Management: @Transactional In-Depth
This Topic is also Discussed in https://github.com/prisma/prisma/issues/13004

I have the same problem. I use CleanArchitecture and I can't work with transactions.

I don't have the time to understand the internals of Prisma but I believe such feature can work by utilizing a series of hooks and event emitters. Has anyone from Prisma attempted to replicate the @Transactional() decorator? I'm curious if there is some kind of a technological blocker that makes it impossible atm.

I'll add onto this. Utilizing NestJS and cannot utilize transactions across multiple services.

Also looking for a solution to this. No way, it seems, to create a transaction which contains Prisma client invocations inside other classes. Also using NestJS

What i ended up doing was for each function passing in an optional PrismaClient parameter. If the parameter exists, it uses that context instead of the one native to the controller/service.
It's tedious for sure, had a couple hundred -_-.

I am using NestJS and have considered replacing it with TransactionClient for the inability to share transactions between services. Doing this would eliminate the need to pass PrismaClient as a method argument. (This is useful for processing outside of the application, e.g., batch processing.)
An example is below.
import { Test } from '@nestjs/testing';
import { Prisma } from '@prisma/client';
import { AppModule } from 'src/app.module';
import { PrismaService } from 'src/prisma/prisma.service';

export const getModuleWithTransactionClient = async (
  prisma: Prisma.TransactionClient,
) => {
  return await Test.createTestingModule({
    imports: [AppModule],
  })
    .overrideProvider(PrismaService)
    .useValue(prisma)
    .compile();
};

use this
    await prisma.$transaction(async (prisma) => {
      const module = await getModuleWithTransactionClient(prisma);

      const service = module.get<StudentAdminService>(StudentAdminService);
      const service2 = module.get<ClassMemberAdminService>(
        ClassMemberAdminService,
      );

However, I don't think this is the right approach and hope a better way is implemented.

I wanted to use Prisma, migrating from TypeORM, until I researched this, and it's basically a deal breaker. We require a way to keep all our resource DB logic in their respective services while being able to get executed in single transactions.

Yes, dealbreaker here as well.

Related: https://github.com/prisma/prisma/issues/12458

@janpio this is important feature for us and would be great if this is prioritized.

Here is a workaround with prisma.$extends
Creating request context
import { RequestContext } from 'nestjs-request-context';

export type PrismaTransactionClient = Omit<PrismaClient, ITXClientDenyList>;

export class AppRequestContext extends RequestContext {
  public prismaTransactionClient: PrismaTransactionClient;
}

export class RequestContextService {
  public static getContext(): AppRequestContext {
    const ctx: AppRequestContext = RequestContext.currentContext.req;
    return ctx;
  }

  public static setPrismaTransaction(
    prismaTransactionClient: PrismaTransactionClient,
  ): void {
    const ctx = this.getContext();
    ctx.prismaTransactionClient = prismaTransactionClient;
  }

  public static getPrismaTransaction(): PrismaTransactionClient {
    const ctx = this.getContext();
    return ctx.prismaTransactionClient;
  }

  public static cleanPrismaTransaction(): void {
    const ctx = this.getContext();
    ctx.prismaTransactionClient = undefined;
  }
}

Creating extended prima client
const extendedPrismaClient = (client: PrismaClient) =>
  client.$extends({
    client: {
      // async onModuleInit() {
      // Uncomment this to establish a connection on startup, this is generally not necessary
      // https://www.prisma.io/docs/concepts/components/prisma-client/working-with-prismaclient/connection-management#connect
      // await Prisma.getExtensionContext(this).$connect();
      // },
      enableShutdownHooks(app: INestApplication) {
        Prisma.getExtensionContext(client).$on('beforeExit', async () => {
          await app.close();
        });
      },
      async $transaction<T>(handler: () => Promise<T>) {
        return Prisma.getExtensionContext(client).$transaction(async (tx) => {
          if (!RequestContextService.getPrismaTransaction()) {
            RequestContextService.setPrismaTransaction(tx);
          }

          try {
            const result = await handler();
            console.debug(`[${new Date()}] transaction committed`);
            return result;
          } catch (e) {
            console.debug(`[${new Date()}] transaction aborted`);
            throw e;
          } finally {
            RequestContextService.cleanPrismaTransaction();
          }
        });
      },
    },
    query: {
      $allModels: {
        $allOperations({ model, operation, args, query }) {
          const tx = RequestContextService.getPrismaTransaction();

          if (tx) {
            // eslint-disable-next-line @typescript-eslint/no-unsafe-call
            return tx[model][operation](args);
          }

          return query(args);
        },
      },
    },
  });

export type IExtendedPrismaClient = new (
  options?: Prisma.PrismaClientOptions,
) => ReturnType<typeof extendedPrismaClient>;

export const ExtendedPrismaClient = <IExtendedPrismaClient> class {
  constructor(options?: Prisma.PrismaClientOptions) {
    const client = new PrismaClient(options);

    // eslint-disable-next-line no-constructor-return
    return extendedPrismaClient(client);
  }
};

Extending in prisma service
@Injectable()
export class PrismaService extends ExtendedPrismaClient {
  constructor() {
    super({
      datasources: {
        db: { url: DatabaseConfig.databaseUrl },
      },
    });
  }
}

Using transaction
await this.prisma
      .$transaction(async () => {
        const user = await this.prisma.user.create({
          data: {
            firstName: 'fsdf',
            email: 'huil',
          },
        });
      })

+1
I'm passing the prisma instance as the function parameter, my code soon looks ugly for sure

I'm developing a clean architecture application in Nest.js and I am tired of using Prisma's TransactionClient between modules and I came across this problem looking for a good good solution.
I implemented the PrismaClientManager by AsyncLocalStorage with nestjs-cls that makes it easier to use transactions, although it is implemented using Nest.js, so please take a look if you have the same problem.
https://github.com/uki1014/nestjs-prisma-client-manager

@uki1014 i have implemented similar solution. And works as expected until now. Thanks for sharing. I believe this solution can be used whoever is seeking to solve this problem.

@yigitahmetsahin is it publicly available?
@uki1014 Just a quick feedback after just a glance (😅): It would be much much better if you could implement it using decorators. Not sure how though. But in Typeorm you only need to pass the entity's ref to the transaction. And as such you can just create a module and pass them to that module in order to know which module you should have in order to add that decorator.

@kasir-barati no i don't have it available publicly, but my solution is very similar to @uki1014. I believe much better to bound in functions so that you will have more control over where transaction start and end. Decorator wraps whole method which might not be ideal for all cases.

@kasir-barati @yigitahmetsahin
yes, I do not think decorators are a good solution to this problem.
For example, if you annotate @Transaction for a service method, it would be easy to exceed the timeout limit of the transaction if the in-memory logic or read SQL is taking a long time.
It would encourage stretching the timeout in that case, but I don't think that is a good idea because it would stretch the database locks.
Better would be to be able to initiate a TRANSACTION when you finally write with the read data and parameters.
keeping the scope of the transaction to a minimum is an important option.
@Transaction
class SomeService {
  async create() {
    // slow query
    const someData = await prisma.some.findMany(...)

    // Slow queries increase the probability of timeout.
    await prisma.other.create(...);
  }
}

class SomeService {
  async create() {
    // slow query
    const someData = await prisma.some.findMany(...)

    await this.prismaClientManager.transaction(async () => {
      await prisma.other.create(...);
    })
  }
}

@uki1014 Acknowledged, But still I believe that you need to decorate your method with @transaction and not your class. Besides if you only wanna only wrap some part of your login in a transaction you can alway opt to have a new method called someSelfDescriptiveName and decorate it with @transaction. This way you are illuminating cognitive complexity.
Also I guess we could have both 🤔, right?
TBF IDK much about how easy each one of these two would be in unit test too. So frankly speaking I guess I am just too much influenced by decorators 😂

@uki1014 hey, I have implemented similar above and we both have issue with background jobs. This solution only works in context of a request. I have implemented it to work anywhere, might share it later if interested

@Kainar01 would be nice if you can also share your solution so that we can get benefit of the alternative approach to this problem rather than limiting it to request context.

@Kainar01 any update from your solution which can work without context of a request?

Happy new year all!
@yigitahmetsahin sorry for taking long. Here is the snippet I use.
I created context manager with async_hooks
import { AsyncLocalStorage } from 'async_hooks';
import { PrismaClient } from '@prisma/client';
import { ITXClientDenyList } from '@prisma/client/runtime/library';

export type PrismaTransactionClient = Omit<PrismaClient, ITXClientDenyList>;

type Context = {
  transaction: PrismaTransactionClient;
};

export class TransactionContextService {
  private static readonly cls = new AsyncLocalStorage<Context>();

  static getTransaction() {
    const context = this.getContext();

    return context?.transaction;
  }

  private static getContext(): Context | undefined {
    return this.cls.getStore();
  }

  static async runInContext<T>(tx: PrismaTransactionClient, handler: () => Promise<T>): Promise<T> {
    return this.cls.run({ transaction: tx }, handler);
  }
}

And created extended prisma client
import { Prisma, PrismaClient } from '@prisma/client';

import { TransactionContextService } from './transaction-context.service';

/**
 * This function creates a new transaction if there is none in the current context.
 * It will commit the transaction if it completes successfully, or abort if an error is encountered.
 * @param {PrismaClient} client - The PrismaClient instance.
 * @param {Function} handler - The function that will be performed within the transaction.
 * @return {Promise<T>} - The result of the transaction.
 */
async function performTransaction<T>(client: PrismaClient, handler: () => Promise<T>) {
  if (TransactionContextService.getTransaction()) {
    return handler();
  }

  return client.$transaction(async (tx) => TransactionContextService.runInContext(tx, handler));
}

function createExtendedPrismaClient(options?: Prisma.PrismaClientOptions) {
  const client = new PrismaClient(options);

  return client.$extends({
    client: {
      async $transaction<T>(handler: () => Promise<T>) {
        return performTransaction(client, handler);
      },
    },
    query: {
      $allModels: {
        $allOperations({ model, operation, args, query }) {
          const tx = TransactionContextService.getTransaction();

          if (tx) {
            // eslint-disable-next-line @typescript-eslint/no-unsafe-call
            return tx[model][operation](args);
          }

          return query(args);
        },
      },
    },
  });
}

export type IExtendedPrismaClient = new (options?: Prisma.PrismaClientOptions) => PrismaClient & {
  $transaction: <T>(handler: () => Promise<T>) => Promise<T>;
};

export const ExtendedPrismaClient = class {
  constructor(options?: Prisma.PrismaClientOptions) {
    // eslint-disable-next-line no-constructor-return
    return createExtendedPrismaClient(options);
  }
} as IExtendedPrismaClient;

Initiate as regular prisma client and use it in code as this:
 this.prisma.$transaction(async () => {
  await this.prisma.account.create(); // transaction taken from context
})

@Kainar01 thanks for sharing this detailed answer. I have one question, you are using AsyncLocalStorage for storing the transaction context. How this can work without request? Because as far as i know async local storage is only present if there is a request.

@yigitahmetsahin from the nestjs docs

It is similar to a thread-local storage in other languages.

It is not necessarily bound to request

I just wanted to give me 2 cents on this topic because the absence of @Transactional decorator is also being very damaging to my clean architecture application here.
The reality is that every single solution presented here "solves" one or another use case, the ideal world is where we could control the transactions on method level by passing properties like Spring Framework does, it's the gold standard.
TypeORM folks have developed a literal copy of this functionality here:
https://www.npmjs.com/package/typeorm-transactional
The idea is to simply copy their idea and do this via AsyncLocalStorage or nest-cls package, considering all the options.
Unfortunately I'm running against time else I'd spend some days trying to implement this myself.

@renatoaraujoc can you share some link to the way that "Spring Framework" handles transaction in the way you have explained it. Btw I don't remember that typeorm-transactional decorator gives you the ability to pass properties to it.
I also didn't get what you meant by properties, do you mean like passing entities as prop to indicate which tables should be included in the transaction. 🤔

by looking at the source code of the TypeORM transactional repo, it looks like they are using EventEmitters to signal start and end of transaction.
I just used ChatGPT to help write something up here but i think something like this can work:
import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

// Transactional decorator
function Transactional() {
  return function (target: any, propertyName: string, descriptor: PropertyDescriptor) {
    const originalMethod = descriptor.value;

    descriptor.value = async function (...args: any[]) {
      return await prisma.$transaction(async (transaction) => {
        // Add transaction to the Prisma context for this execution
        const contextWithTransaction = { ...this, prisma: transaction };

        // Call the original method with the transaction context
        return await originalMethod.apply(contextWithTransaction, args);
      });
    };

    return descriptor;
  };
}

class MyService {
  // The service methods don't need to be aware of transactions
  @Transactional()
  async myMethod() {
    // Perform database operations using 'this.prisma'
    const data1 = await this.otherService.method();
    const data2 = await this.otherService2.method();
    const data3 = await this.otherService3.method();

    return this.findById(x);
  }
}

@renatoaraujoc can you share some link to the way that "Spring Framework" handles transaction in the way you have explained it. Btw I don't remember that typeorm-transactional decorator gives you the ability to pass properties to it.
I also didn't get what you meant by properties, do you mean like passing entities as prop to indicate which tables should be included in the transaction. 🤔

Their @Transactional decorator is very much like Springs, they accept the same isolation methods, etc, see:
https://github.com/Aliheym/typeorm-transactional/blob/master/src/decorators/transactional.ts

by looking at the source code of the TypeORM transactional repo, it looks like they are using EventEmitters to signal start and end of transaction.
I just used ChatGPT to help write something up here but i think something like this can work:
import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

// Transactional decorator
function Transactional() {
  return function (target: any, propertyName: string, descriptor: PropertyDescriptor) {
    const originalMethod = descriptor.value;

    descriptor.value = async function (...args: any[]) {
      return await prisma.$transaction(async (transaction) => {
        // Add transaction to the Prisma context for this execution
        const contextWithTransaction = { ...this, prisma: transaction };

        // Call the original method with the transaction context
        return await originalMethod.apply(contextWithTransaction, args);
      });
    };

    return descriptor;
  };
}

class MyService {
  // The service methods don't need to be aware of transactions
  @Transactional()
  async myMethod() {
    const data1 = await this.otherService.method();
    const data2 = await this.otherService2.method();
    const data3 = await this.otherService3.method();

    return this.findById(x);
  }
}

@jaequery it's a good start but it's not as simple as that since we can have @Transactional methods that call other @Transactional annotated methods, that's where the propagation kicks in.
In reality, you have to work with a proxied prisma object that will return the proper prisma object or the transaction object given if we're in a transaction and it's propagation.

any update on this issue guys?? i am facing the same problem with clean architecture.


Enable SQL Query Editor to pass data to SandDance extension

This is the pull request by Garage interns working on the data visualization experience in Azure Data Studio using SandDance.
Added a 'visualizer' query event and enabled the visualizer icon to pass data to the SandDance extension.
Once SandDance is installed, when a user clicks on the visualizer icon, the SandDance extension will take over to load the query data in SandDance.

The SandDance extension will add a new top-level tab and renders the visualization.

@joslevin

You're still going to be working out of the garagedev2 branch? How big are your changes in that branch vs master? If you can you should consider trying to merge changes into master whenever you can (i.e. ones that don't affect other stuff and are deemed stable enough). Otherwise you end up with a massive PR at the end into master which makes it really difficult to review.

Charles - we need to sort out the entry point and onboarding story to Karl's satisfaction before merging to master. We should chat with him on this so we can think about the best way to get agreement quickly & drive this to master for sure.

@Charles-Gagnon there are a couple issues that need to be worked out before merging this feature to master.  Specific concerns are:
(1) why do we need a visualizer and a charts button, can't we just use one button that is configurable to do both, these seem like basically the same use-case as far as I can tell.
(2) we need to determine what is the long-term plan to get this feature to GA.  Specifically is the new UI accessible?  If there are no plans to GA and fix accessibility bugs then we need to carefully consider how to integrate the extension into the product  Basically we can't have any fixed UI in the default product (i.e. a Visualize button) that is primarily bound to a feature that doesn't have a GA road-map.  Experimental"always-preview" extensions need to contribute all their UI elements via extension install.

Yeah that's fine - just suggesting that if there are any parts that can be merged in earlier that it might be useful to do that soon rather than try to fit everything into one huge PR.
If that's not really possible then continue on as you were =)

@Charles-Gagnon yep, fully agree.  In this case there are some fundamental design\direction issues that are still unresolved.


Adds test with HTTP connect followed by parser failure

Fixed by https://github.com/OISF/suricata/pull/5449

Merged in https://github.com/OISF/suricata-verify/commit/7045dfaaefbd85edb4a2566293cd61c80160c384


Fix the issue where an error message is printed twice.

Problem
 The error message is printed twice.

Issue number, if available:
Changes

This line of code in the ampx.ts file lacks a try catch and then does not catch the error, so the error is caught by attachUnhandledExceptionListeners. that's why the error is printed twice.so just add try catch and then can solve it.
Corresponding docs PR, if applicable:
Validation

Checklist

[ ] If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
[ ] If this PR requires a change to the Project Architecture README, I have included that update in this PR.
[ ] If this PR requires a docs update, I have linked to that docs PR above.
[ ] If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This is still kinda papering over the root cause, which seems to be that yargs invokes the command error handler and also throws the error if the validation / middleware / handler fails. I played around a bit locally and it doesn't seem like there's a convenient way to modify this behavior so I think what you've got here is okay. Just left a couple small comments

This makes me wonder if original solution wasn't better. I.e. memorize already handled exceptions in the error handler.

This is still kinda papering over the root cause, which seems to be that yargs invokes the command error handler and also throws the error if the validation / middleware / handler fails. I played around a bit locally and it doesn't seem like there's a convenient way to modify this behavior so I think what you've got here is okay. Just left a couple small comments

This makes me wonder if original solution wasn't better. I.e. memorize already handled exceptions in the error handler.

The root cause (i.e. expectation that yargs.parseAsync won't throw if error handler is attached) is the yargs behavior which unfortunately we don't have any control over. The best way to fix is to ensure we don't propagate errors multiple times for a single occurrence.
We should also not leave any unhandled exceptions lying around intentionally. The original solution to memorize errors was hiding the root cause, even hiding other bugs in the future.


fix(update_epsagon): update epsagon version

:tada: This PR is included in version 1.27.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


can't change azure cloud in Azure Extention

I am using the Azure extension in Visual Studio Code I have the extension configured to an Azure Cloud Subscription but I want to reconfigure the extension to Azure US Government. there is no where in the extension to do that. I used VSC command pallet to log off of azure cloud and into azure us government and I am still seeing azure cloud resources in the extension

You can change the cloud environment using the microsoft-sovereign-cloud.environment setting.


Don't clone in mutable set

I think this was an oversight - the whole point of UnsafeMutableSet is to overwrite the value supplied, so cloning and returning a different one is wrong, and causes extra memory allocations.

We should consider proposing this upstream and get rid of this repository (I think they are aligned now)

I have proposed https://github.com/mndrix/ps/pull/6
For some reason GitHub does not recognize that the two repos are in sync - https://github.com/mndrix/ps/compare/master...weaveworks:master shows three commits different.  Maybe this repo should do a merge commit?

This breaks scope report rendering. Seems to go into an infinite loop for me running
GOMAXPROCS=1 ./prog/scope --mode=app --weave=false --app.collector=file:///home/matthias/tmp/reports

where /tmp/reports contains Adam's prod reports, with .msgpack.gz extensions
and then running
curl -o /dev/null -s http://localhost:4040/api/topology

which never completes, and even ctrl-c'ing that scope consumes a full core.

Sorry, I had not understood the way nilMap is constructed

And I had clearly forgotten
On Mar 13, 2017 18:36, "Bryan Boreham"<EMAIL_ADDRESS>wrote:

Sorry, I had not understood the way nilMap is constructed
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/weaveworks/ps/pull/2#issuecomment-286183895, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACQOJPoJ5IIcAQn6P_DJRpj9YNy4-XUYks5rlX6dgaJpZM4MaqPM
.


adding semversion runtime version

Adding a semver runtime version.

actually I reordered the fields so it fits 32 bit binary format


netrw: Opening a URI-encoded URL by gx will open corrupted URL

Steps to reproduce

Write some URL-encoded URL such as https://www.example.com/#:~:text=%E3%81%82 in some buffer.
Put cursor on the URL, and run gx shortcut (= NetrwBrowseX) in normal mode.
It tries to open the target URL in the web browser, but the opening page URL is corrupted
like https://www.example.com/#:~:text=/%E3/%81/%82.

Here it is calling extra escape() that causes this.
Due to this, it would fails with paths including # or %, the characters escaped here.
The comment for it says they've added the escape() since shellescape(..., 1) in netrw#Open() is insufficient.
Is the escape() really required now?
Expected behaviour
It opens the correct URL.
Version of Vim
9.1.866
Environment

Operating system: Windows 11 Pro 64bit 24H2 (Version 10.0.26100.2605)

MSYS2 (msys2-runtime 3.5.4-8)

Terminal: mintty '3.7.6' 2024-09-24_05:10 (x86_64-pc-msys)

Value of $TERM: xterm-256color
Shell: GNU bash, version 5.2.37(1)-release (x86_64-pc-msys)

Vim g:netrw_browsex_viewer value: start

Browser: Google Chrome v131.0.6778.140

Logs and stack traces
No response

I remember when that comment was made, I wondered myself about it whether this is true. At least I cannot reproduce it:
:exe '!echo ' .. shellescape('one#two%three four',1)

seems to work just fine. So perhaps we can just get rid of the double escape here? @Konfekt

I wondered myself about it whether this is true

Everybody is wondering about what's happening.
Testing was not exhaustive, but it made common URLs with hashes such ashttps://github.com/vim/vim/issues/16252/#issue-2749168683 work, so I assumed % (current buffer) needed similar treatment to # (alternate buffer).
On Ubuntu 22.04 with Vim 9.1.919, removing that escaping of # breaks it.
It may be that only # is causing trouble, as with
" runtime/autoload/netrw.vim (line 5202)
call netrw#Open(escape(fname, '#'))

gx opens the URL https://www.example.com/#:~:text=あ fine.
What's curious is that @KSR-Yasuda 's URL is escaped with forward instead of backward slashes, though: https://www.example.com/#:~:text=/%E3/%81/%82.
On Ubuntu 22.04 with Vim 9.1.919 backward slashes are added at % but a forward slash at #: https://www.example.com//#:~:text=\%E3\%81\%82.

Since @KSR-Yasuda uses MSYS on Microsoft Windows, it also looks like [shellescape(..., 1)](https://github.com/vim/vim/blob/21c37d7f695077efe6df57806ff35da79adce1d5/runtime/autoload/netrw.vim#L5114) does not work as intended here? Maybe there are two issues at play.

What's curious is that @KSR-Yasuda 's URL is escaped with forward instead of backward slashes, though: https://www.example.com/#:~:text=/%E3/%81/%82. On Ubuntu 22.04 with Vim 9.1.919 backward slashes are added at % but a forward slash at #: https://www.example.com//#:~:text=\%E3\%81\%82.

Probably it is because of path conversion of MSYS2 env.
Manually executing start 'https://www.example.com/\#:~:text=\%E3\%81\%82' opens
https://www.example.com//#:~:text=/%E3/%81/%82.
Every \ chars are replaced with /.
Even if you drop %-escape, #-escape may cause the problem in MSYS2.
For https://www.example.com/#:~:text=%E3%81%82#a,
it opens the URL https://www.example.com/#:~:text=%E3%81%82/#a (extra / for the latter #).

Even if you drop %-escape, #-escape may cause the problem in MSYS2.

Maybe  "" //b are not required here if you use the Git Bash version of MSYS2
" autoload/netrw.vim (line 5032)
exe 'silent !start "" //b' a:args s:redir() | redraw!

or cygpath is installed in MSYS2

Probably it is because of path conversion of MSYS2 env.

One could prepend MSYS_NO_PATHCONV=1on Git Bash according to https://github.com/git-for-windows/build-extra/blob/main/ReleaseNotes.md#known-issues or MSYS2_ARG_CONV_EXCL='*' on MSYS2 according to https://www.msys2.org/docs/filesystem-paths/
I wonder if the authors @vim-jp of https://github.com/vim-jp/vital.vim maybe already worked out all the edge cases long ago and could give definite replies.

@KSR-Yasuda If you could find a combination of above knobs that suits you?

shellescape(..., 1)

Since @KSR-Yasuda uses MSYS on Microsoft Windows, it also looks like shellescape(..., 1) does not work as intended here?

shellescape(..., 1) itself looks fine in MSYS2 env.
shellescape('https://www.example.com#:~:text=%E3%81%82#a', 1)

" => 'https://www.example.com\#:~:text=\%E3\%81\%82\#a'

Probably it runs
the vim command like :exe 'silent !start ' . 'https://www.example.com\#:~:text=\%E3\%81\%82\#a',
that implies the bash command start https://www.example.com#:~:text=%E3%81%82#a.
Rather Ubuntu env. is exceptional to need extra escape(), isn't it?
There defines netrw#Launch() for each env.
If still extra escape() is needed, shouldn't you call it there?
(The one for Ubuntu would be this?)
Otherwise, instead of escape(), you might had better quote the path.

Even if you drop %-escape, #-escape may cause the problem in MSYS2.

Maybe  "" //b are not required here if you use the Git Bash version of MSYS2
" autoload/netrw.vim (line 5032)
exe 'silent !start "" //b' a:args s:redir() | redraw!

No, without   "" //b, it opened command prompt... (start command would have went wrong.)
My env is pure MSYS2, if you mean Git for Windows.

Rather Ubuntu env. is exceptional to need extra escape(), isn't it?
...  If still extra escape() is needed, shouldn't you call it there?

Not so sure how special an issue of Ubuntu that is. Same phenomenon on Debian 12 and thus likely its forks such as Linux Mint and others.

No, without  "" //b, it opened command prompt...

Thank you, that's good to know.

My env is pure MSYS2

Okay, does it come with cygpath? The above link  https://www.msys2.org/docs/filesystem-paths/ claimed so, but I think mine didn't. So you are using the start script?

so I assume we can get rid of the double escape() ?

Strange, but we can try and time will tell

The path converstion thingy is tricky however. Not sure we should try to open that can of worms.

One could prepend MSYS_NO_PATHCONV=1on Git Bash according to<EMAIL_ADDRESS>or MSYS2_ARG_CONV_EXCL='*' on MSYS2 according to msys2.org/docs/filesystem-paths
If @KSR-Yasuda has success with either of them, why not? Git Bash and MSYS2 can be distinguished by environment variables

One could prepend MSYS_NO_PATHCONV=1on Git Bash according to<EMAIL_ADDRESS>or MSYS2_ARG_CONV_EXCL='*' on MSYS2 according to msys2.org/docs/filesystem-paths
If @KSR-Yasuda has success with either of them, why not? Git Bash and MSYS2 can be distinguished by environment variables

Anyway, I think it is not the point.
Appearance of extra / or \ itself would be wrong already.
MSYS_NO_PATHCONV is just an option to stop path separator conversion, isn't it?

Even if you drop %-escape, #-escape may cause the problem in MSYS2. For https://www.example.com/#:~:text=%E3%81%82#a, it opens the URL https://www.example.com/#:~:text=%E3%81%82/#a (extra / for the latter #).

As I said, even escape(fname, '#') introduces extra / before #.
For the first #, it does not make a problem.
But the latter #, the anchor part #ANCHOR#a and #ANCHOR/#a are completely different things.

Anyway, I think it is not the point.
Appearance of extra / or \ itself would be wrong already.
MSYS_NO_PATHCONV is just an option to stop path separator conversion, isn't it?

Yes, that's the second issue, that eventually should also be fixed if additional escaping has to be done.
I think what caused confusion is that despite
" autoload/netrw.vim (lines 5112-5114)
fun! netrw#Open(file) abort
  call netrw#Launch(s:viewer() .. ' ' .. shellescape(a:file, 1))
endf

:Open https://github.com/vim/vim/issues/16252#issuecomment-2556009926 still does not work, but requires escaping #. However, Itt's better to force the user to escape it if this causes issues elsewhere.

:Open https://github.com/vim/vim/issues/16252#issuecomment-2556009926 still does not work, but requires escaping #. However, Itt's better to force the user to escape it if this causes issues elsewhere.

So, you've checked via :Open command?
Isn't it an issue of vim command execution, that # is replaced with alternate file name,
as :e https://github.com/vim/vim/issues/16252#issuecomment-2556009926 behaves the same.
Then, if you need to fix the :Open case, shouldn't you change its command definition?
But I'm not sure if they might want to call :Open for the alternate file by #.

Isn't it an issue of vim command execution, that # is replaced with alternate file name

Yes, that's what supposedly 1 is for in above

shellescape(a:file, 1))

:Open https://github.com/vim/vim/issues/16252#issuecomment-2556009926 still does not work, but requires escaping #. However, Itt's better to force the user to escape it if this causes issues elsewhere.

OK.  It is reproduced.  # seems considered alternate file.

Hm...  gx with https://github.com/vim/vim/issues/16252#issuecomment-2556009926 works in current code.

I have removed https://github.com/vim/vim/blob/21c37d7f695077efe6df57806ff35da79adce1d5/runtime/autoload/netrw.vim#L5202 's escape() and it work for me.
Please provide the failed example?

As above, I vaguely remember it was somewhat inscrutably the only place where :Open would accept # as well while keeping gx working. Alas, this is not the case using MSYS2 on Windows, so I agree that breaking :Open is preferable to breaking gx, as the user can fix the latter and is less likely to use it (hitting gx on a URL is more common than pasting/writing a URL in the command-line)

OK.  So the escape should be removed.
:Open may be broken in some cases, but users can escape it manually.

Can you create the PR?  If not, I can create it instead.


Implement Cache-Control header for umami.js

Set cache for 30 days, for flexibility in case of changes.

Please ignore the last commit, I only hope for the Cache-Control header implementation.


22758: update hanna twig templates to use new storybook path

This PR add changes in some twig templates used in hanna repository. The changes are needed due to storybook Drupal path has changed to match the new hanna-twig folder used in  hanna repository. components and twig templates are not anymore grouped by name

@fran
Hi and thanks for the PR.
I've got one question: Can we drop the @storybook part from the {% include ... %} paths, and make them into file-sysem relative paths?
i.e. turn {% include<EMAIL_ADDRESS>… into something like {% include "./RelatedLinks.twig" …
??

@fran Hi and thanks for the PR. I've got one question: Is it possible for us/me to drop the @storybook part from the {% include ... %} paths, and make them into file-system relative paths?
i.e. turn {% include<EMAIL_ADDRESS>… into something like {% include "./RelatedLinks.twig" …
??

Hi @maranomynet , Yes I think we can do it and taking into account the template is out of Drupal system, it makes more sense and will not add regressions in other system using it.
I will do the change.

@fran check out https://github.com/rvk-utd/hanna/commit/0456f0aa and let me know if something looks wrong.

@fran I'd also love to get rid of the hard-coded content in .Layout__nav and .Layout__footer

			{{ page.breadcrumb }}
			{% include<EMAIL_ADDRESS>%}
			{{ page.secondary_menu }}

			{% include<EMAIL_ADDRESS>%}
			{% include "./FooterBadges.twig" %}

IMO, this should be injected at the point where you use Layout.twig.
Additionally the contents of FooterBadges.twig should be injected via props/parameters as well.
You guys should be able to update that content without making a PR here.
…ugh, same for FooterInfo.twig. 😬


Abstract class not bring generated in OAS.yaml

I am using SwaggerAPI to generate my OAS.yaml.
OpenAPI generator version 6 is used to generate pojos from generated OAS.yaml
The abstract classes in my module are not being generated as abstract.
example :
public abstract class example
is being generate as
public class example.
Is there any annotation / specific extension / property that can be used to mention the class as abstract in Open API Specification ?
I have tried
x-abstract : true
also
abstract : true
none worked.
Any help is appreciated.
Thanks in advance.

Please raise tooling-related issues on the relevant repository. This repository is for the OpenAPI specification itself. You may also receive more answers on a general support forum such as StackOverflow. Thanks.


Add Support for BeautyQuests

Hi I wanted to ask if you could add support for BeautyQuests.
I have already asked BeautyQuests and was instructed to ask you if you could set this up on your end.
Here is the link to the original issue of BeautyQuests.
https://github.com/SkytAsul/BeautyQuests/issues/303#issuecomment-1872153972
It would be great if you could set this up.

You misunderstood. They did not ask for me to set it up on my end, but to implement the support for them into their plugin. It's not high on my priority list so there's a chance I won't get around to it, but i'll take a look eventually.
Closing this issue since it's not technically specific to pl3xmap's codebase.


Additional error logs during platform initialization.

Additional error logs during platform initialization.
Example output from a failure.
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_core::worker","Fields":"{\"message\":\"failed to start VM\",\"error\":[\"failed to initialize platform security\",\"failed to request vmgs encryption keys\",\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"13.348678604s\",\"level\":\"ERROR\",\"target\":\"underhill_core::worker\",\"related_activity_id\":\"0829af09-0000-003e-0100-000040000000\",\"fields\":{\"message\":\"failed to start VM\",\"error\":[\"failed to initialize platform security\",\"failed to request vmgs encryption keys\",\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation","Fields":"{\"message\":\"failed to retrieve key-encryption key\",\"attestation_type\":\"Tdx\"}","Message":"{\"timestamp\":\"13.348415765s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"failed to retrieve key-encryption key\",\"attestation_type\":\"Tdx\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"13.348302718s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":8,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"12.161429040s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":8,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"12.161313919s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":7,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"10.983605364s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":7,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"10.983426485s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":6,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"9.476082382s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":6,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"9.476005517s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":5,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"8.324955956s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":5,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"8.324882401s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":4,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"7.145144612s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":4,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"7.145070978s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":3,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"5.937921373s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":3,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"5.937846387s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":2,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"4.423430944s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":2,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"4.423358113s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":1,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"3.264854191s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":1,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"3.264712644s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"underhill_attestation::secure_key_release","Fields":"{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":0,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}","Message":"{\"timestamp\":\"2.056434015s\",\"level\":\"ERROR\",\"target\":\"underhill_attestation::secure_key_release\",\"related_activity_id\":\"6cd39487-0000-003e-0400-000040002000\",\"fields\":{\"message\":\"VMGS key-encryption key request failed due to error\",\"retry\":0,\"error\":[\"failed to make an IgvmAttest KEY_RELEASE GET request\",\"IGVM agent returned an error\"]}}"}
{"VmId":"70970671-31e9-4b7a-b0c8-4de489492d8b","VmName":"fc5daf76-04f4-4fc3-b377-9dcd78b3f4ba","CorrelationId":"00000000-0000-0000-0000-000000000000","Name":"","Target":"guest_emulation_transport::process_loop","Fields":"{\"message\":\"Agent returned an error\"}","Message":"{\"timestamp\":\"2.056257894s\",\"level\":\"ERROR\",\"target\":\"guest_emulation_transport::process_loop\",\"fields\":{\"message\":\"Agent returned an error\"}}"}

I talked with Sergio offline, and worked with him to understand more of the rationale behind John's suggestion on the previous PR (https://github.com/microsoft/openvmm/pull/180#issuecomment-2433328109), and why having a fully enumerated match statement is preferable to an Any based approach, as well as the pitfalls of having the error "code" identifier be tied to the particular backing Rust type.
He expressed broader concerns about how our current structured logging from OpenHCL is being ingested upstream in Kusto, and while I redirected him to some folks who might know more about that infrastructure, I pointed out that those concerns are somewhat orthogonal to the particular code structure we are discussing in this PR.

It seems this PR, much like the previous PR, has morphed into a set of logging related changed (as opposed to a larger overhaul of error handling infrastructure)?
That's fine, I suppose? Looking through these changes, I don't see anything seriously concerning (outside of a few minor comments).
As long as you're confident that you've both

correctly annotated the confidentiality level of these various new tracing statements
correctly specified the log level of these various tracing statements (i.e: I'm seeing lots of info-level logs - that might get "loud" in kusto?)

...these changes seems reasonable to me.

Thanks.  This change is critical to help debugging CVM issues.  For monitoring, I think using the "target" field would be enough I think.


Mac App

Hello WebRTC devs,
thank you for all the effort you put into the card.
It works in most cases, but not in the Mac App, while this is similar (visually) to the iOS one.

Is there any plan to support the MacApp?
Is there an option to detect the problem and maybe fall back to the still image of the video?

PS: This integration is relatively popular in due to the fuatugan/eufy_security plugin, which integrates eufy cameras and encourages the installation of this card to render the stream.
Thank you.

Support only depends on the guys from Cupertino #76 #248

Thank you for pointing that out, will try to check if there is a workaround. It works perfectly on the iOS App.


Bug: ReferenceError: intToBuffer is not defined

Describe the bug
ReferenceError: intToBuffer is not defined
Steps

Go to
https://github.com/coinbase/coinbase-wallet-sdk/blob/84ddf4a910d820319a255a75f34d11fdfddffe3c/packages/wallet-sdk/src/vendor-js/eth-eip712-util/util.js#L70

Expected behavior
intToBuffer is defined
Version
No response
Additional info
No response
Desktop
No response
Smartphone
No response

hi. sorry for the belated response. we are marking stale issues closed for now. please try our latest version 3.6.0. feel free to open a new one if needed. thanks.


Adjacent pixel drill results have different time series dates

We recently ran the command line pixel drill on 5 sites of 9 pixels each (5 3x3 spots) to return NBAR and FC reading across all available bands between 1987 and 2015. We specified that the drill use PQA filtering. The command used was:
while read lat lon; do echo "Retrieving time series for $lat / $lon" ; retrieve_pixel_time_series.py --lon $lon --lat $lat --acq-min 1987-06 --acq-max 2015-10 --satellite LS5 LS7 LS8 --mask-pqa-apply --dataset-type ARG25 --hide-no-data --output-directory $PWD --overwrite ; done <pixels.txt
We have found that the pixel drill results for adjacent pixels have quite different dates in their time series. We expect some, but are finding that there is much more than we expected. Can we check that the drill is retrieving all results in the time series repeatably?
Norman

pixels.txt

I've had a quick look at the file list returns for the locations:
(145.38525, -16.85219)
(145.38525, -16.85469)
And the lists contain the same records.
My guess is it has to do with what constitutes a "has_data".
https://github.com/GeoscienceAustralia/agdc/blob/master/api/source/main/python/datacube/api/tool/retrieve_pixel_time_series.py#L228
This line will exclude pixels not satisfying the "has_data" condition.


ci: delete workspace before checkout

Delete the files in the workspace before make the checkout.

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #1681 opened

Start Time: 2021-03-02T10:51:41.712+0000

Duration: 57 min 42 sec

Commit: dd04ef6778f2292a0b80944c696d33c0b4982486

Test stats :test_tube:

Test
Results

Failed
0

Passed
1848

Skipped
14

Total
1862

Trends :test_tube:

:green_heart: Flaky test report
Tests succeeded.

Expand to view the summary

Test stats :test_tube:

Test
Results

Failed
0

Passed
1848

Skipped
14

Total
1862


specify namespace for test cases

https://github.com/istio/istio/issues/16639
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

@irisdingbj: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

istio-racetest-master
d611f6df89899548b70b0b828f51932cd728db98
link
/test istio-racetest-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test istio-racetest-master

@irisdingbj: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

istio-racetest-master
d611f6df89899548b70b0b828f51932cd728db98
link
/test istio-racetest-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@irisdingbj I don't want this to merge until I understand the problem better.  Working on it.

I think we should close this in favor of my PR #16752 . Putting the namespace into the test was a good idea and I used that in my PR. This PR did not have a test case for creating a ServiceEntry. The problem only manifested when a ServiceEntry was created by the mock client. This PR blocked that creation but did not fix the underlying problem between add-to-mesh and the Kubernetes fake client-go.

Thanks @esnible Let me close this PR now.


Confusing arg name

Hi, Zhang. Nice work!
When inserting your net to my model, some details confuse me. In SepFlow's forward API, the two images to be aligned are 'image1' and 'image2'. I wonder which one is the reference frame, and which one is the supporting frame? (reference one is the destination frame, and supporting frame is the frame that will be aligned to the reference one with the computed flow).

嗨，Zhang. 读完您的工作，对我产生了一些启发，我想基于您的工作展开一些拓展。在部署代码的时候，我发现很多forward接口中，使用image1和image2来代表两帧。这让我有点疑惑，我不知道接口是计算image1对齐到image2的光流，还是计算image2到image1的光流。请问您能解释一下吗


Semantic Highlighting: fontStyle doesn't seem to function correctly

I can't find a way to replace a fontStyle that is underline in the selected theme with one that is only italic (no underline).  It appears a user-specified fontStyle merges with the default fontStyle to create a style that is the sum of all the parts, with no way to selectively remove part of the style.  If the user definition replaced the default

VSCode 1.44.2 ff915844119ce9485abfe8aa9076ec76b5300ddd and 1.45.0-insider ec0d19789f89d4c25895f4a0cf364276cfb0c6ed
OS Version: macOS 10.15.4 (19E287)

Steps to Reproduce:

Select the monokai theme
Install Rust on your system, and the rust-analyzer plugin (yes when it prompts you to download the thing)
Create a new rust project (cargo new scratch) and open it in VSCode
Add the following code to the top of src/main.rs:

trait MyTrait {}
struct MyStruct;
impl MyTrait for MyStruct {}

Notice that MyTrait is green and underlined (once Semantic Highlighting has kicked in).

Attempt to replace the underline with italics in settings.json.

    "editor.tokenColorCustomizationsExperimental": {
        "interface": {
            "fontStyle": "italic"
        }
    },

It adds italics but leaves the underline, when it should have replaced the underline.

Leaving fontStyle blank is supposed to clear the style entirely, but by itself without an accompanying foreground it does absolutely nothing (style remains underlined).
    "editor.tokenColorCustomizationsExperimental": {
        "interface": {
            "fontStyle": ""
        }
    },

However, if you add a foreground color then a blank fontStyle suddenly starts clearing the style like it is supposed to:
    "editor.tokenColorCustomizationsExperimental": {
        "interface": {
            "foreground": "#ff8800",
            "fontStyle": ""
        }
    },

But even with a new foreground, an italic fontStyle does not replace the underline:
    "editor.tokenColorCustomizationsExperimental": {
        "interface": {
            "foreground": "#ff8800",
            "fontStyle": "italic"
        }
    },

Does this issue occur when all extensions are disabled?: N/A - if I disable extensions, then Semantic Highlighting won't work for Rust.

I fixed that "fontStyle": "" doesn't not clear all styles.
Also, now fontStyle sets/clears all styles, like it does in in TextMate theme.
To set or unset individual styles, I added properties bold, italic, and underline in addition to fontStyle. Note: fontStyle wins against bold, italic, and underline

@aeschli Thanks!


Predict improvement

the main idea of this PR is keeping cache locality for predicted rows.
Results:

Data set
predict time (master), s
predict time (this PR), s
Speedup

higgs1m
4.12
1.32
3.12

msrank
13.9
6.07
2.29

without Dmatrix create:

Data set
predict time (master), s
predict time (this PR), s
Speedup

higgs1m
4.12 - 0.25
1.32 - 0.25
3.62

msrank
13.9 - 2.6
6.07 - 2.6
3.25

Data set
logloss (master)
logloss (this PR)

higgs1m
0.4036675
0.4036675

msrank
0.8550638
0.8550638

Codecov Report

Merging #6127 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6127   +/-   ##
=======================================
  Coverage   78.52%   78.52%
=======================================
  Files          12       12
  Lines        3069     3069
=======================================
  Hits         2410     2410
  Misses        659      659

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6bc9b9d...cd1707d. Read the comment docs.

@hcho3  not sure that failed steps are related to current changes
https://travis-ci.org/github/dmlc/xgboost/jobs/731946554#L8110

I'll review today. Thanks for removing the unrolling variable kUnroll.

I'll review today. Thanks for removing the unrolling variable kUnroll.
@hcho3  Thanks  for review in advance :)
With new approach there is no significant performance gain from unrolling.

@hcho3 Thanks a lot for your review.
this construction
 (*this)[pid].LeftChild() + !(fvalue < (*this)[pid].SplitCond());
brings up to 35% of speedup, it was reflected in comment: https://github.com/dmlc/xgboost/pull/6127/files#diff-d7a927cebbc6e99b6432fdba2b54231cR48
it helps to reduce miss branch prediction.

Then please add a comment directly on top of the statement (*this)[pid].LeftChild() + !(fvalue < (*this)[pid].SplitCond()); to explain what it does.

Then please add a comment directly on top of the statement (*this)[pid].LeftChild() + !(fvalue < (*this)[pid].SplitCond()); to explain what it does.

Done.
Thanks

@hcho3 seems some checks were not started, at least half of it

@ShvetsKS See #6214

The test farm is now back to online.

@hcho3 , @SmirnovEgorRu  Thanks a lot for review!

@hcho3 could take a look at failed tests, please
I think similar failed tests were discovered before the current PR.
Here [Doc] Add info on GPU compiler (#6204): https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/master/525/pipeline fail is
similar to https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-6127/16/pipeline
should we restart it?


question or feature reuqest -- grouping resutls for multi - line entries

[x] I have read through the manual page (man fzf)
[x] I have the latest version of fzf
[x] I have searched through the existing issues

Info

OS

[x] Linux
[ ] Mac OS X
[ ] Windows
[ ] Etc.

Shell

[x] bash
[ ] zsh
[ ] fish

Problem / Steps to reproduce
PROBLEM

Search query produces multi line parings
Searching for keywords within the paring
Primary result inst in that line but the line before it but line is cleared during the filtering for the keyword

EXAMPLE
for commands in "$(atp search "$1" | fzf --preview="apt show {} -m -e")";
do
sudo apt install "$commands" || exit 1
done

Results Produced

zfs-initramfs/focal-updates 0.8.3-1ubuntu12.4 amd64
  OpenZFS root filesystem capabilities for Linux - initramfs

zfs-test/focal-updates 0.8.3-1ubuntu12.4 amd64
  OpenZFS test infrastructure and support scripts

zfs-zed/focal-updates 0.8.3-1ubuntu12.4 amd64
  OpenZFS Event Daemon

zfsutils-linux/focal-updates 0.8.3-1ubuntu12.4 amd64
  command-line tools to manage OpenZFS filesystems

During the fzf search often what im searching for is located in the description but the results i want to act on are in the first line. when fzf searching i get my result and i can mark it, however, i then have to clear query and search for the mark.
So A. is there a way to group up multi lines as one "line" of (i don't really understand the --nth and --with-nth so this might be the solution but i suspect it wont produce a working group) . I'm hoping for a solution beyond this example as i don't use apt often but this is just the example that came to mind.

I decided not to implement multi-line entry support due to technical difficulties. See #2174.
In this case, I'm not sure fzf is the best tool for your requirement, but if I had to use fzf which is strictly a line-based solution, I would consider squashing the lines in each entry so any text in the entry is searchable, while showing the original multi-line form in the preview window.
cat > /tmp/source << EOF
zfs-initramfs/focal-updates 0.8.3-1ubuntu12.4 amd64
  OpenZFS root filesystem capabilities for Linux - initramfs

zfs-test/focal-updates 0.8.3-1ubuntu12.4 amd64
  OpenZFS test infrastructure and support scripts

zfs-zed/focal-updates 0.8.3-1ubuntu12.4 amd64
  OpenZFS Event Daemon

zfsutils-linux/focal-updates 0.8.3-1ubuntu12.4 amd64
  command-line tools to manage OpenZFS filesystems
EOF

perl -pe 'BEGIN{undef $/;} s/\n\n/\x0/g; s/\n$//g' /tmp/source |
  fzf --height 50% --reverse --read0 --no-hscroll --preview 'echo {}' --preview-window up:2 |
  awk 'NR == 1 {print $1}'

Hey thanks ill check that out I think upon consideration I might also be able to save each set into an array and call them ""line"" by line ill have to check it out and post back this weekend but for now thanks.

I think actually i might have solved this as ive written a script that more or less reverses the functionality of the preview window main search i just need to figure out a way to extend it to this case use, currently you fuzzy query the content of files and it shows files that match the fzf query in the preview window and i think i could do it with process substitution
fvar="$1"
echo "$fvar"
if test -n "$1"; then
  fvar="$1"
else fvar=''
fi
echo "$fvar" | \
  fd -uu -t f \
  -x rg . {} --heading --no-filename | \
  fzf --print-query \
  --preview="rg {+} --files-with-matches -H" \
  --bind="change:reload(export fvar={q} && echo \"$fvar\" | fd -uu -t f \
  -x rg . --no-filename --heading)"

See https://junegunn.github.io/fzf/tips/processing-multi-line-items/


D1.4 - Stacks

Tower of Hanoi example should be built upon - given the tower of Hanoi, how would the solution be calculated automatically? There should be code for that
example of one interview question (medium difficulty) pertaining to stacks + solutions

idea: https://leetcode.com/problems/implement-stack-using-queues/

checkpoints
concepts

Week of 3/2 Goal:

Build upon the tower of hanoi example with how it could be automated
move concepts into folder
Issues: #786, #783

Week of 3/9 Goal:

Create interview Question
Begin making checkpoints
issues: #785, #751

Week of 3/16 Goals:

Over all goal is to have the activity completely done - that way we can focus on finishing slides for break and preparing for the workshop

That means:

Finish all check points! - Make sure to look at the readme's for the last one as it will need a test case
Interview questions need to be done
Check grammar, sentence structure, etc and any other last minute changes

Issues: Any issues from the previous 2 weeks which are open, as well as #782
Hope your finals went well! :)

Week of 3/22 Goals:

You should be working on making the slides for your assigned activity


Plan the December Meetup

[ ] Location - RIAT
[ ] Sponsor -
[ ] Talks -
[ ] Ideas -

RIAT can host us (added link and location above).

@robeerob  for the date: 8th or 15th of Dec? I would vote for 8th. I think it's a holiday :)

@DDCreationStudios the 8th does not work for me, unfortunately :( Shall we go for the 15th? Will check with RIAT.

@DDCreationStudios Looks like 15th will work for RIAT. That date is ok for you as well?
They have some cryptocurrency events/series of events that week with international visitors, so we talked about either one of the guests or Matthias having a beginner friendly talk about blockchain, or for instance Monero (a cryptocurrency). What do you think?

@robeerob Thanks for managing! Yes works good 👍 I think this time we should again aim for a classical meetup (rather than another coding workshop). So yes, great idea! 👏  I would prefer a talk about blockchain, because I am personally interested in it :P. But otherwise feel free to decide, you know the community best :)
Maybe we should also add a Christmas special program point? Or end the program with a christmas special?  I have many exams in December so I can't schedule a "christmas & drinks" like we had last year. :(

@DDCreationStudios Good point about talk focused meetup, and we can definitely think of something xmas-ey to end with ;) :tada:
Just checked with Svetlozar, and he is not in Vienna on the 15th. Do you have someone in mind that we can ask to have a talk, should one of us have a talk, or maybe we can announce a "call for talk proposals" in "Developers Vienna" FB group and such?

@DDCreationStudios 15th does not seem to work for RIAT after all. They suggested 14th or 16th. 14th is not good for me, but I think 16th could work. How about you?
Or should we try another space for this month?

@robeerob hmmm a pity :/. The 14th and 16th are both not really good for me.  Shall we ask Zi8 or Stockwerk? Or I think you also had another place in mind and have been in contact with them. Whats the status of that?

@DDCreationStudios yeah, indeed :( Maybe we can take the opportunity to try out Zi8? If they can't host us, I am also fine with Stockwerk. My other place idea never replied, so ruling them out for now.
Do you have contact details for Zi8?

RIAT confirmed, one talk proposal so far, date for 16.12 confirmed


bin size error

Environment info
Operating System:WINDOWS 10
CPU:Intel i7
C++/Python/R version: python 2.7 and 3.5
Error Message:
D:\coding\Python\Anaconda\envs\tf_with_lgbm\lib\site-packages\lightgbm\engine.py:99: UserWarning: Found num_rounds in params. Will use it instead of argument
warnings.warn("Found {} in params. Will use it instead of argument".format(alias))
D:\coding\Python\Anaconda\envs\tf_with_lgbm\lib\site-packages\lightgbm\basic.py:642: UserWarning: max_bin keyword has been found in params and will be ignored. Please use max_bin argument of the Dataset constructor to pass this parameter.
'Please use {0} argument of the Dataset constructor to pass this parameter.'.format(key))
[LightGBM] [Warning] Met negative value in categorical features, will convert it to NaN
[LightGBM] [Warning] Met negative value in categorical features, will convert it to NaN
[LightGBM] [Warning] Met negative value in categorical features, will convert it to NaN
[LightGBM] [Warning] Met negative value in categorical features, will convert it to NaN
[LightGBM] [Warning] Met negative value in categorical features, will convert it to NaN
[LightGBM] [Warning] Met negative value in categorical features, will convert it to NaN
[LightGBM] [Fatal] bin size 16948 cannot run on GPU
Traceback (most recent call last):
File "D:/coding/Python/Projects/Recommendation/lightgbm_test.py", line 264, in
lgbm_model = lgb.train(params, train_set = lgb_train, valid_sets = lgb_val, verbose_eval=5)
File "D:\coding\Python\Anaconda\envs\tf_with_lgbm\lib\site-packages\lightgbm\engine.py", line 178, in train
booster = Booster(params=params, train_set=train_set)
File "D:\coding\Python\Anaconda\envs\tf_with_lgbm\lib\site-packages\lightgbm\basic.py", line 1296, in init
ctypes.byref(self.handle)))
File "D:\coding\Python\Anaconda\envs\tf_with_lgbm\lib\site-packages\lightgbm\basic.py", line 48, in _safe_call
raise LightGBMError(_LIB.LGBM_GetLastError())
lightgbm.basic.LightGBMError: b'bin size 16948 cannot run on GPU'
Reproducible examples
params = {
'objective': 'binary',
'boosting': 'gbdt',
'learning_rate': 0.1 ,
'verbose': 0,
'num_leaves': 120,
'bagging_fraction': 0.95,
'bagging_freq': 1,
'bagging_seed': 1,
'feature_fraction': 0.9,
'feature_fraction_seed': 233,
'max_bin': 256,
'num_rounds': 800,
'lambda_l1': 0.3,
'lambda_l2': 0.3,
'metric' : 'auc',
'is_unbalance' : 'true',
'device' : 'gpu',
'gpu_platform_id' : 0,
'gpu_device_id' : 0,
'gpu_use_dp' : 'false'
}
Other information
visual studio 2017
lightgbm Version: 2.0.11
CUDA 8.0
I installed LightGBM GPU Version and my GPU is GTX 1070 8GB, I thought it should be able to run my program but failed, there are some pages I found that seems have the same problem like me:
https://www.kaggle.com/c/instacart-market-basket-analysis/discussion/37836
https://github.com/Microsoft/LightGBM/issues/1003  (the last commented)

GPU version cannot support categorical features with high cardinality.
You can fix it by split one categorical feature into multi categorical features.


Parallel replay train

So, I am really trying my best to speed this up, as our current projection for 10,000 training iterations is about 500 days. Which is insane.
I don't expect this to be an insane improvement, but I am hoping it helps a decent amount.
I will be running tests to figure out the sweet spot for the number of workers to use for this, since this is different than normal, where throwing more workers increases speed. There is a number of workers that will maximize the value of this, since there are locks to prevent

I am getting mighty annoyed. Windows is perfectly respecting the boundaries of the keyboard interrupt, but WSL is completely ignoring any attempts that I make to try to get it to recognize that I am telling it to stop.
I recognize that it works a bit differently because the threads are working and need to be told to stop, but its like the main program isn't even recognizing the valid keyboard interrupt.

Alright, I did manage to get the interrupt working for both platforms. It kinda feels like WSL just decided it felt like working suddenly.
Now I am going to test out some values for different worker numbers for the training. My guess is that all that this is really gonna benefit from is 2 workers, but I want to make sure we are making the most of this extra training potential.

Starting new run for parallel linux training with 2 workers now

It is saying that the run took 20 minutes to train... That is an insanely high speed increase, and it leads me to believe that there were some shortcuts made to the training.
Looks like the majority of scores were negative, and the fall off from initial testing was much higher compared to normal training iterations.

Starting regular training now to get a better idea of the contrast between the two.

I can think of zero reason that this speed up would improve the training time by like, 14 times, other than it is not doing what it should be. And the training graph seems to prove it, I am going to run some more tests and see if any of them look similar to what we expect out of a normal training graph.

Alright, really obvious problem time. The reason its so much faster is because its not doing anything. This method that I used while NEAT training worked because each child process was independent and didnt need to communicate with its siblings. It also worked because the result each process found was returned to its parent process.
So I need to find some way of allowing each child (and the parent) processes to share that model with each other. This may end up just being more overhead than its worth, but if its possible to get a performance boost from this small improvement, I will be super happy.

Starting another run now using thread based parallelism. This means it should share the same memory space, without needing a bunch of overhead that might result in no improvement at all.

Alright, this run took 16:15 hours.

The only thing I find curious is that there were 0 timeouts in this testing... its really hard to understand why there would be none at all, but I am gonna run another with 2 threads and if that doesnt show any then something is strange with how I am identifying them.

Now that I think about it, we just recently made it so we were using our GPU rather than CPU to process these model updates. If that is the case, then wouldn't it not matter at all if we had multiple threads, because they are going to be limited on the GPU requests that they can make?
This was already a cheesy strategy to try and milk a little bit more performance out of the training before I had to just bite the bullet and do a major improvement to implement a large scale solution to improve the training method.

Alright it seized up.

It got good results, but I cant think of why it wouldve stopped. Might be something to do with my interpreter or something like that.
From the timeline of modified files, it looks like the run took about 15 hours. This is slower than the normal runs take. I am going to try on windows, where GPU usage is not supported, and see if the thread based processing speeds it up at all.

Here is the standard time delta: 14:44:14.146766
Here is the processed time: 19:52:24.828125

Alright, this is significantly disappointing, so I will not be moving forward with these changes.


🐞 [Bug]: Dialog style is not consistent in dashboard

Is there an existing issue for this?

[X] I have searched the existing issues

which package/s did you face the problem with?
Dashboard
What happened?
We need to unify the styling of dialogs across dashboard, something like headers and font size should be unified
Steps To Reproduce
No response
which network/s did you face the problem on?
Dev
version
2.4.0
Twin ID/s
No response
Node ID/s
No response
Farm ID/s
No response
Contract ID/s
No response
Relevant screenshots/screen records

Relevant log output
.

all delete dialogs should also have 'delete' button not 'confirm'

style fixed with https://github.com/threefoldtech/tfgrid-sdk-ts/pull/2796

Work Completed:
Update the Delete button Action from 'Confirm' to 'Delete'

verified on devnet
version : db2daf4
dialog style is now unified


[BUG] SeLinux on EC2 RHEL 8,4 block pod to access serviceaccount mount

What happened:
From a workload pod, it has the following logs showing the pod doesn't have permission to access the mounted serviceaccont.
"unable to find leader election namespace: error checking namespace file: stat /var/run/secrets/kubernetes.io/serviceaccount/namespace: permission denied"

What you expected to happen:
Workload pod can access the mounted serviceaccont.
How to reproduce it (as minimally and precisely as possible):

run microshift on ec2 RHEL
deploy some pods which have a serviceaccount mounted, then exec into the pod
then run stat /var/run/secrets/kubernetes.io/serviceaccount/namespace (assuming the service account has a namespace field)

Anything else we need to know?:
After disable selinux(setenforce 0), the pod is able to access the mounted service account.
Slack conversation is at: https://microshift.slack.com/archives/C025AQ0QD8B/p1632421234103900
Environment:

Microshift version (use microshift version): Microshift Version: 4.7.0-0.microshift-2021-08-31-224727
Hardware configuration: t2.xlarge
OS (e.g: cat /etc/os-release): Red Hat Enterprise Linux 8.4 (Ootpa)
Kernel (e.g. uname -a): Linux ip-172-31-32-38.ec2.internal 4.18.0-305.el8.x86_64 #1 SMP Thu Apr 29 08:54:30 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
Others:

Relevant Logs

@cooktheryan can you please help on this one?

@ianzhang366 can you send me some YAML to recreate this?

yup

/assign cooktheryan

It's an issue that happens within the directories of emptydir as well. Im hoping this https://danwalsh.livejournal.com/81269.html leads me down the right path. There may be deeper concern if the controllers need to automatically append selinux values at launch time of instances but I haven't had the opportunity to deep dive yet.
@rootfs going to leave this up to @fzdarsky and you to decide if we should roll back selinux while I patch this or keep in place while awaiting PR.
It appears to be a small but regular use case of items that have ephemeral dirs


Nextcord iinstalled but not importing

I've installed nextcord on pip, but when I try to import it using: from nextcord.ext import ipc import nextcord it says missing installation. I have the folders in my site-packages folder. I don't know what I am doing wrong

Have you installed nextcord-ext-ipc also?

It successfully installed it but it is still saying missing imports

Have you installed nextcord-ext-ipc also?

It successfully installed it but it is still saying missing imports

There are many potential problems:

You're IDE's language server is out of date. Restarting the IDE should fix that issue.

You have multiple Python installations and the installation you installed nextcord-ext-ipc on is different from the one you're using for running your code

You have a virtual enviroment enabled, and nextcord-ext-ipc is installed outside of the virtual enviroment.

You installed nextcord-ext-ipc in a virtual environment, but your virtual environment is not activated.

There are many potential problems:

You're IDE's language server is out of date. Restarting the IDE should fix that issue.
You have multiple Python installations and the installation you installed nextcord-ext-ipc on is different from the one you're using for running your code
You have a virtual enviroment enabled, and nextcord-ext-ipc is installed outside of the virtual enviroment.
You installed nextcord-ext-ipc in a virtual environment, but your virtual environment is not activated.

I've restarted my IDE multiple times, and I'm not sure how to fix any of the other issues

Alright, what is saying the error, your IDE or when running the file?
What IDE/editor do you use?
How do you run your bot?

i'm having the same issue.
i've uninstalled all other versions of python
i've restarted my ide multiple times
i've uninstalled and reinstalled nextcord multiple times
i've installed nextcord-ext-ipc
i've uninstalled and reinstalled nextcord-ext-ipc
i don't have a virtual environment
i'm running my bot on idle shell and i use idle 3.9 if that is helpful
what is going on!?


lib for nutrition pull

https://github.com/Nelson-Gon/pyfdc

https://fdc.nal.usda.gov/fdc-app.html#/food-details/746766/nutrients


Refactor request handling

Dear maintainers,
Added/Changed the followin

type hints where possible
base class is now abstract
added getPrefix method
removed the need for duplicate constructor calls
added .editorconfig and changed the format to PSR-2 (https://www.php-fig.org/psr/psr-2/)
added dotenv as required dev package
added .env.example
added some things to git ignore

Greetings


Support multiple array index access methods

Use case:
this is how @proddata wanted (https://github.com/crate/crate/issues/10110) to access an array:
CREATE TABLE test (a OBJECT);
insert into a values({ id = 1, name = 'foo', tags = ['apple', 'banana', 'pear'], size = 3.1415, valid = true });

SELECT a['tags'][1] FROM test LIMIT 100;
-- SQLActionException[UnsupportedFeatureException: Nested array access is not supported]

however, this works:
SELECT a[1]['tags'] FROM test limit 100;
-- SELECT a[1]['tags'] FROM test limit 100; --> apple

with an explicit cast, the original method would work:
cr> select a['tags']::text[][1] from test;
+--------------+
| a[1]['tags'] |
+--------------+
| apple        |
+--------------+

Feature description:
we should support both methods of accessing an array
cc @mfussenegger

@mechanomi thx :)


Delete interests module

I don't think it's used anymore.
https://github.com/squeaknode/squeaknode/blob/ee9d68d3e2b60a140b50fba637d6a7f36f67aa34/squeaknode/core/interests.py#L22-L75

https://github.com/squeaknode/squeaknode/pull/2073


Regarding Run time / tello-gesture-control/running/ main.py

Reading configuration
usage: ipykernel_launcher.py [-h] [-c MY_CONFIG] [--device DEVICE]
[--width WIDTH] [--height HEIGHT]
[--is_keyboard IS_KEYBOARD]
[--use_static_image_mode]
[--min_detection_confidence MIN_DETECTION_CONFIDENCE]
[--min_tracking_confidence MIN_TRACKING_CONFIDENCE]
[--buffer_len BUFFER_LEN]
ipykernel_launcher.py: error: unrecognized arguments: -f C:\Users\mayank\AppData\Roaming\jupyter\runtime\kernel-97c73e9e-2e91-4deb-9b71-3f6e8e7a2fca.json
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2

Please, provide more details. What are you trying to run?


Update pulse sink polybar is reading from

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
With two pulseaudio sinks/outputs polybar can only display one at a time. Currently mine is set to display the default sink but that changes based on if I'm using the speakers-hdmi sink or rear audio jack-sink.
Why does polybar need this feature?
Describe why this feature would be useful to a large percentage of users (You need to convince us).
Guess this would only apply to users with multiple speakers/sinks there's some who may only have speakers or one audio sink they use so they just read levels from the one sink.
Describe the solution you'd like
A clear and concise description of how your solution would work. This includes possible config options that should be added.
Perhaps on mouse click the module could switch the index its reading from? But then the user would only be able to switch sinks or mute on click not both.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered, if any.
Just let polybar read from one sink as it currently needs to be stopped and restarted if the default sink is changed to update right?

Did you set the sink in your config? If you leave it unset, it should always follow the default sink.

Closing due to inactivity

I currently use polybar-pulseaudio-control for this. It allows me to switch between different audio sinks (headphones, speakers) as well as increment/decrement/mute the current sink. Is this functionality something that could be included in this repository?

Is this functionality something that could be included in this repository?

This is certainly something we would accept contributions for. Changing the volume is already supported, the only thing that would need to be done is allowing users to change the sink.

@patrick96 Great! I'll try and create a pull request when I have the time :)

Thanks :smiley:
I assigned this issue to you.


 Try quoting the trailing text.
  kprobe:$1asdf { 1 }
  ~~~~~~~~~~~~~

  # bpftrace -e 'kprobe:$1"asdf" { 1 }' -- f
  Attaching 1 probe...
  cannot attach kprobe, probe entry may not exist
  Error attaching probe: 'kprobe:fasdf'

This closes #606 .
Checklist

[x] Language changes are updated in docs/reference_guide.md
[x] User-visible and non-trivial changes updated in CHANGELOG.md

This is definitely useful; I found that the followings do not work.

Use as int

% ./src/bpftrace -e 'i:ms:$1 { printf("hi\n"); }' 100
stdin:1:1-8: ERROR: Failed to parse '': stoi
i:ss:$1 { printf("hi\n"); }
~~~~~~~

Use as offset

sudo ./src/bpftrace -e 'k:vfs_open+$1 { printf("hi\n"); }' 5
stdin:1:1-14: ERROR: Invalid offset
k:vfs_open+$1 { printf("hi\n"); }
~~~~~~~~~~~~~

% sudo ./src/bpftrace -e 'BEG$1 { printf("hi\n"); }' IN
stdin:1:1-6: ERROR: Unrecognized probe type: BEG
BEG$1 { printf("hi\n"); }
~~~~~

% sudo ./src/bpftrace -e 'BEGIN$1 { printf("hi\n"); }' IN
stdin:1:1-8: ERROR: Unrecognized probe type: BEGININ
BEGIN$1 { printf("hi\n"); }
~~~~~~~

@mmisono I fixed 1-3 with:
commit d925f154416d28d5a5bf8acb9a9748702db1c796 (HEAD -> params_in_aps)
Author: Daniel Xu<EMAIL_ADDRESS>Date:   Mon May 18 14:27:39 2020 -0700

    Load positional parameters before driver runs

    Before, we were loading positional parameters into bpftrace after the
    driver ran and before the AST was processed. This was fine for
    positional parameters used inside the probe body but not ok for
    positional parameters parsed in the driver (ie attach point defns).

diff --git a/src/main.cpp b/src/main.cpp
index cafe0b4..d497585 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -346,6 +346,13 @@ int main(int argc, char *argv[])
   BPFtrace bpftrace(std::move(output));
   Driver driver(bpftrace);

+  // positional parameters
+  while (optind < argc)
+  {
+    bpftrace.add_param(argv[optind]);
+    optind++;
+  }
+
   bpftrace.usdt_file_activation_ = usdt_file_activation;
   bpftrace.safe_mode_ = safe_mode;
   bpftrace.force_btf_ = force_btf;
@@ -453,12 +460,6 @@ int main(int argc, char *argv[])

   cap_memory_limits();

-  // positional parameters
-  while (optind < argc) {
-    bpftrace.add_param(argv[optind]);
-    optind++;
-  }
-
   // defaults
   bpftrace.join_argnum_ = 16;
   bpftrace.join_argsize_ = 1024;

What is the issue with 4? That seems like reasonable behavior to me.

@danobi
oops, there is no issue with 4.

Ah I figured out why my tests passed initially. driver.parse() runs twice. My tests only had string arguments and the AttachPointParser accepts an empty string as an argument, but cannot accept an empty string as an numeric parameter (stoi fails). But before second driver.parse runs the positional params are added.
I think the fix I put above is correct (modulo a small change).


Absent Page

Create Absent Page For later deverlop. Page is for student in class to check if who absent or not. And nav section in sidebar component to navigate to absent page.

Can you update to lastet commit before merge request

Change the title of the absent page

Updated The title of Absent Page


feat(groq-codegen): misc updates

This PR:

adds new test fixtures to allow us to write the expected type and have the typescript compiler throw if those types don't match
fixes a few errors regarding undefines and nulls with attribute access
support for object expressions outside of projections
supports dereferencing ->
removes nested array wraps and re-wraps (i.e. removes normalizeArrayTransform) in favor of conditional types

Superseded by #149


Feature Request / Question - Record Screenshots at Runtime

My team is currently leveraging Shot to perform screenshot tests in Firebase Test Lab. So far it works alright, but we are running into issues where the recorded screenshot fails when compared to the golden screenshot even when the screenshots appear to be identical. I've heard that this issue can arise when the environment used to record the golden screenshots is different than the environment that is used to verify screenshots for tests. We are ensuring that the emulator configuration used is the same, but the only difference is the host machine that runs the tests. i.e. We record golden screenshots on our developers local MacBook Pros and verify the screenshots in Firebase Test Lab which I'm assuming is running a docker instance running Linux.
So, my ask here is if there is a way currently in the library to control whether to record or verify screenshots at runtime without using the gradle commands? We'll need to be able to record screenshots at runtime so that we can ensure both the recording and verifying of screenshots happens both in Firebase Test Lab. The reason we need this functionality is because it is not possible (from my understanding) to pass gradle commands to be executed in Firebase Test Lab. Let me know if more details are needed here. Thank you!

@Alton09 For this you really just want to make sure you're always recording screenshots in the same environment that you verify it (i.e. don't record on developer laptops)


Add gitter badge to README.md

Looks like a spurious failure. I have no idea how an update such as this would cause build failure :)