Restore Debian-specific support for /etc/redis/redis-{server,sentinel}.{pre,post}-{up,down}.d

This reverts commit 6a9e4d0142b45195a0d55945bbc558df4c48707b.
Devuan downstream needs this run-parts functionality for service state change hooks.
Please consider allowing it to remain? :bowing_man:

@blinkdog Please can you elaborate more on your use-case? :) Thanks!

@lamby Sure, no problem. My use-case is systems that have migrated downstream, from Debian to Devuan.
Some of these could have scripts that hook state changes in the redis service. An example might be (re)populating the cache from a local or remote source on service start. Another example might be triggering a backup of the persistent store on service stop if the current backup is too old.
In Debian (>= Jessie), one can use the hooks provided by systemd (ExecStartPre, ExecStartPost, ExecStopPre, ExecStopPost) and point them at the script(s). In Devuan, for better or for worse, there is no systemd. Devuan systems rely on the init script to call run-parts to kick off the script(s).
Thank you for considering my pull request. :smile:

Hi @blinkdog, thanks for your message.
However do you have any concrete examples of users that are are using this? I have never heard of anyone actually using it, with any initsystem.
If a user does 1000% require this feature, then they can simply modify the /etc/init.d/redis-server initsecript:

They marked as conffiles for this reason
This is what users would have to do with any other daemon in Debian

ps. It would be nice if the thread on DNG would calm down and stop emitting such FUD.

However do you have any concrete examples of users that are are using this? I have never heard of anyone actually using it, with any initsystem.

No, unfortunately I don't have any information about redis usage beyond the Debian popularity contest:
#<name> is the package name;
#<inst> is the number of people who installed this package;
#<vote> is the number of people who use this package regularly;
#<old> is the number of people who installed, but don't use this package regularly
#<recent> is the number of people who upgraded this package recently;
#<no-files> is the number of people whose entry didn't contain enough information (atime and ctime were 0).
#rank name                            inst  vote   old recent no-files (maintainer)
2157  redis-server                    4791  3527   952    310        2 (Chris Lamb)

If a user does 1000% require this feature, then they can simply modify the /etc/init.d/redis-server init script:

They marked as conffiles for this reason
This is what users would have to do with any other daemon in Debian

Indeed they could, and honestly I'm not so worried about users coming at it de novo. They are likely to know the change they want to make and will figure it out how to make it happen. I'm more worried about the users with existing scripts who upgrade from Devuan Beowulf to Devuan $(Beowulf+1) who might get caught out without an easy migration path.
I'm just asking since the functionality is already there, and works well in both systemd and sysvinit systems, will you leave it in place to allow these users a smooth upgrade?

ps. It would be nice if the thread on DNG would calm down and stop emitting such FUD.

Yeah, the thread on DNG runs a bit hot. Unfortunately, 6a9e4d0142b45195a0d55945bbc558df4c48707b touches on many people's darkest fears about systemd in Debian. They just knew it was going to kick their puppy sooner or later, and was that a dog yelping just now?
The truth is closer to puppy's leash was moved to a different hook. My request is aimed toward the low energy solution; can we move the leash back, or is that wall part of the remodeling plans?
If you are committed to keeping 6a9e4d0142b45195a0d55945bbc558df4c48707b, then Devuan will need to supplement or fork the packaging for redis, which something we'd rather not do if we can help it. So, I thought it would be best to ask first.
And, no hard feelings either way. When I go back to the DNG list, I will tell them that you are reasonable and not a puppy kicker, even if we have to install a new leash hook.

I'm just asking since the functionality is already there, and works well in both systemd and sysvinit systems, will you leave it in place to allow these users a smooth upgrade?

The smoothest thing to do at this point is remove it early on in the buster release cycle. It doesn't work well, is Debian-specific, is misleading, has no documentation. I also have had zero reports of anybody using it whatsover, in Debian or any downstream distribution.

If you are committed to keeping 6a9e4d0, then Devuan will need to supplement or fork the packaging for redis, which something we'd rather not do if we can help i

This is not a Devuan or init system specific issue thing at all, so I have no idea why 6a9e4d0 would touch on anyones fears any more than any other package in Debian changing behaviour. Forking the packaging would just be creating unnecessary work for literally zero benefit.
I'm afraid I don't follow (or enjoy) your dog kicking, dog leash or remodelling metaphors.

I'm just asking since the functionality is already there, and works well in both systemd and sysvinit systems, will you leave it in place to allow these users a smooth upgrade?

The smoothest thing to do at this point is remove it early on in the buster release cycle. It doesn't work well, is Debian-specific, is misleading, has no documentation. I also have had zero reports of anybody using it whatsover, in Debian or any downstream distribution.

If you are committed to keeping 6a9e4d0, then Devuan will need to supplement or fork the packaging for redis, which something we'd rather not do if we can help it.

This is not a Devuan or init system specific issue thing at all, so I have no idea why 6a9e4d0 would touch on anyones fears any more than any other package in Debian changing behaviour. Forking the packaging would just be creating unnecessary work for literally zero benefit.

Okay. Sounds to me like you're committed to 6a9e4d0.
I thought I'd ask, just in case. Again, no hard feelings.

I'm afraid I don't follow (or enjoy) your dog kicking/leash metaphors.

I apologize. I thought the metaphors would be enjoyable and build rapport. Please accept my direct translations below, and my assurances that I won't stray into metaphor again.
"They just knew it was going to kick their puppy sooner or later, and was that a dog yelping just now?"
--> People get worked up easily about things they are afraid of. The DNG folks are afraid of things like 6a9e4d0.
"The truth is closer to puppy's leash was moved to a different hook."
--> 6a9e4d0 is not as bad as the people on the DNG list thought.
"can we move the leash back, or is that wall part of the remodeling plans?"

"I will tell them that you are reasonable and not a puppy kicker, even if we have to install a new leash hook."
--> Even if you are committed to 6a9e4d0, I don't think it makes you a bad person or maintainer, it just means we have different goals. I will tell the people on DNG that you are a good person, even if we have to make changes downstream.

@blinkdog: this is why it's best to wait until an actual user reports this.  Cutting down a feature no one uses means less maintenance work, both for the Debian maintainer and for any downstreams.
As for the DNG thread, beside (too many) opinion pieces, it did include also some real info.  It surprises me you did file this report without reading the thread in its entirety.
(I haven't ever used redis thus I have no monkey in this fight, but it'd be good to not cause noise without doing the research.)

@blinkdog: this is why it's best to wait until an actual user reports this. Cutting down a feature no one uses means less maintenance work, both for the Debian maintainer and for any downstreams.

As for the DNG thread, beside (too many) opinion pieces, it did include also some real info. It surprises me you did file this report without reading the thread in its entirety.

(I haven't ever used redis thus I have no monkey in this fight, but it'd be good to not cause noise without doing the research.)

Thank you for your kind advice, it is appreciated.

@blinkdog I think I just don't understand what goal you are aiming for. AIUI Devuan is mostly about a non-systemd version of Debian but — as I've mentioned countless times — this change is completely initsystem agnostic - I actually believe in that.
Anyway, can only see myself rolling this back at this point if I got a large number concrete reports from users saying they require this feature and adjusting the sysvinit script (or systemd unit) was literally impossible for some reason or reasons I have overlooked.
I note that my response to the DNG thread has been moderated away.

@blinkdog I think I just don't understand what goal you are aiming for. AIUI Devuan is mostly about a non-systemd version of Debian but — as I've mentioned countless times — this change is completely initsystem agnostic - I actually believe in that.

Anyway, can only see myself rolling this back at this point if I got a large number concrete reports from users saying they require this feature and adjusting the sysvinit script (or systemd unit) was literally impossible for some reason or reasons I have overlooked.

The change itself is init system agnostic; the service state change hooks are removed across the board (sysvinit, systemd, etc.).
However, the impact on different init systems is also different. For example, let's take user Alice who has a script phosphoromancy.sh which loads photonics data into the redis cache when the service starts.
When Alice upgrades from Debian Stretch to Debian Buster, her script will break. She will turn to Debian forums and ask: "Hey, how do I make my script work again?" The answer will be "Go to the systemd unit for redis, and add this line."
ExecStartPost=/path/to/phosphoromancy.sh

When Alice upgrades from Devuan Beowulf to Devuan $(Beowulf+1), her script will break. She will turn to Devuan forums and ask: "Hey, how do I make my script work again?" The answer will be, "First, find /etc/init.d/redis-server, and add a run-parts hook". In the sysvinit case, Alice is in for a little more work.
And, believe me, I understand this is our downstream problem, not your problem. I was only asking in case you're comfortable leaving the functionality in place. I have zero concrete reports of this functionality being used, so you're probably not interested in keeping it for the sake of a phantom downstream user. No worries, and no hard feelings.

I note that my response to the DNG thread has been moderated away.

I was unaware that you had responded. I asked a moderator to please release your response to DNG.

When Alice upgrades from Debian Stretch to Debian Buster, her script will break

Well, this kinda assumes Alice actually exists, which I actually highly doubt. ;)

"First, find /etc/init.d/redis-server, and add a run-parts hook". In the sysvinit case, Alice is in for a little more work.

Not at all. She does not need to add a run-parts hook, she can simply call the script directly or inline it in /etc/init.d/redis-server like she would have to do for literally any other Debian package :)  Special-casing Redis with these hooks is thus highly misleading and not very useful for users. :)

When Alice upgrades from Debian Stretch to Debian Buster, her script will break
Well, this kinda assumes Alice actually exists, which I actually highly doubt. ;)

This point is well made. I'm going to close the pull request. If we get any users downstream who need this functionality, I'll come back and re-open it, and we can discuss on concrete cases rather than theoretical users.


help

please help

also this

From your error message, you have to run xcode-select --install.

help

Try running sudo xcode-select --reset and then xcode-select --install.

Still

But installation completed now with no error

Oh, good. Problem solved?

Nope

What's the problem?

Read the README, then run python3 -m hackq_trivia.bearer_finder to obtain your bearer token.

Ok trying

Enter your phone number, including + and country code (e.g. +14155552671)
Example:<PHONE_NUMBER> (U.S. number) -> +14155550171
Alternatively, enter a previous verification ID:
? +14142869827
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/devanshugarg/HackQ-Trivia/hackq_trivia/bearer_finder.py", line 65, in
main()
File "/Users/devanshugarg/HackQ-Trivia/hackq_trivia/bearer_finder.py", line 38, in main
verify_resp = hq_post('verifications', {'phone': phone, 'method': 'sms'})
File "/Users/devanshugarg/HackQ-Trivia/hackq_trivia/bearer_finder.py", line 26, in hq_post
raise HQResponseError(f'Error code {resp["errorCode"]}: {resp["error"]}')
main.HQResponseError: Error code 102: not authorized

Try using a VPN.

saying this

ok

wait a minute installing vpn

got my bearer token

pasted it in config file

what about google api and others

Follow the instructions in the README to obtain either a Bing/Google API key. If I remember correctly, you'll need a valid credit card but you won't be charged.

you mean google console??

Name your custom search engine and type in any valid URL in Sites to search
what is this

Name it, and add anything (e.g. https://google.com/) in sites to search. The site you add will be deleted later as part of the process.

ok

if i dont setup bing then??

As long as you have one of Google or Bing set up, it is OK.

ok

any test py??

Feel free to run any of the tests in the test folder, such as python3 -m hackq_trivia.tests.test_question_handler_answers.

As it says, your API key is invalid.

i got from official website

Did you paste it in hq_config.conf?

yup

if i rum main py

i resterted the terminal and guess what it worked.    thanks a lot

No problem!

Best not share your API key online :-)
If you like the project, a star would be much appreciated!

I started it because I loved this project

This project should be appreciated


AllowPickingOriginalPhoto option

Thank you first.
It is different from TZImagePicker that there is no allowPickingOriginalPhoto option.
will you open imagePicker option ? such as allowPickingOriginalPhoto , allowPickingVideo

Hi, I will do it this weekend if possible. Thanks!


Change scaling mode to FixedHorizontal

Objective

Fixes the issue with orthographic camera imported from glTF not displaying anything (mentioned in #4005).

Solution

This was due to wrong scaling mode being used. This PR simply changes WindowSize scaling mode to FixedHorizontal.

Important Note
Currently, othographic scale in Blender, three.js, and possibly other software does not translate to Bevy (via glTF) because their developers have misinterpreted the spec. The camera parameters have been clarified in glTF 2.0, which was released on October of 2021. In Blender 3.0.1 this issue has not been fixed yet. If you are importing orthographic cameras from Blender, you have to divide the scale by 2.

I don't think it really makes sense to use gltf for 2D...

bors r+


Internal error occurred during message handling. Please check your implementation. Error: The receiver for "xxx/xxx/xxx" is already receiving messages.

Dear @abdomohamed first of all thank you for open sourcing this.
Unfortunately I get an error when having more than one subscription.
My implementation is
import {
  ServiceBusPubSub,
  IServiceBusOptions,
} from '@talema/graphql-azure-servicebus-subscriptions';

const options: IServiceBusOptions = {
  connectionString: process.env.SERVICEBUS_CONNECTION_STRING,
  topicName: process.env.SERVICEBUS_TOPIC,
  subscriptionName: process.env.SERVICEBUS_SUBSCRIPTION_NAME,
  filterEnabled: false,
  messageLabelKeyName: null
};
export const pubsub = new ServiceBusPubSub(options);

Do you have any idea what's going wrong there? Or did you even came across this error before?
The full error log is this:
Internal error occurred during message handling. Please check your implementation. Error: The receiver for "my-topic/Subscriptions/my-subscription" is already receiving messages.
    at ServiceBusReceiverImpl._throwIfAlreadyReceiving (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/@azure/service-bus/src/receivers/receiver.ts:326:21)
    at ServiceBusReceiverImpl.subscribe (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/@azure/service-bus/src/receivers/receiver.ts:504:10)
    at ServiceBusPubSub.subscribe (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/@talema/graphql-azure-servicebus-subscriptions/src/ServiceBusPubSub.ts:169:35)
    at /Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/src/pubsub-async-iterator.ts:113:32
    at Array.map (<anonymous>)
    at PubSubAsyncIterator.subscribeAll (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/src/pubsub-async-iterator.ts:112:41)
    at PubSubAsyncIterator.<anonymous> (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/src/pubsub-async-iterator.ts:55:65)
    at step (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/dist/pubsub-async-iterator.js:33:23)
    at Object.next (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/dist/pubsub-async-iterator.js:14:53)
    at /Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/dist/pubsub-async-iterator.js:8:71
Internal error occurred during message handling. Please check your implementation. Error: The receiver for "my-topic/Subscriptions/microservice-gateway-ws" is already receiving messages.
    at ServiceBusReceiverImpl._throwIfAlreadyReceiving (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/@azure/service-bus/src/receivers/receiver.ts:326:21)
    at ServiceBusReceiverImpl.subscribe (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/@azure/service-bus/src/receivers/receiver.ts:504:10)
    at ServiceBusPubSub.subscribe (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/@talema/graphql-azure-servicebus-subscriptions/src/ServiceBusPubSub.ts:169:35)
    at /Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/src/pubsub-async-iterator.ts:113:32
    at Array.map (<anonymous>)
    at PubSubAsyncIterator.subscribeAll (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/src/pubsub-async-iterator.ts:112:41)
    at PubSubAsyncIterator.<anonymous> (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/src/pubsub-async-iterator.ts:55:65)
    at step (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/dist/pubsub-async-iterator.js:33:23)
    at Object.next (/Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/dist/pubsub-async-iterator.js:14:53)
    at /Users/reventwork/Code/microservice-playground/microservice-gateway-ws/node_modules/graphql-subscriptions/dist/pubsub-async-iterator.js:8:71

I'd be happy to extend the project with you.
Best regards

If it might help. I changed the code to:
import {
  ProcessErrorArgs,
  ServiceBusClient,
  ServiceBusMessage,
  ServiceBusReceivedMessage,
  ServiceBusReceiver,
  ServiceBusSender,
  delay,
  isServiceBusError,
} from '@azure/service-bus';
import { Subject, Subscription, filter, tap, map } from 'rxjs';
import { PubSubEngine } from 'graphql-subscriptions';
import { IEventBody, IEvent, IEventResult } from<EMAIL_ADDRESS>
/**
 * Represents configuration needed to wire up PubSub engine with the ServiceBus topic
 * @property {string} connectionString - The ServiceBus connection string. This would be the Shared Access Policy connection string.
 * @property {string} topicName - This would be the topic where all the events will be published.
 * @property {string} subscriptionName - This would be the ServiceBus topic subscription name.
 * @property {boolean} filterEnabled - This would be used to enable a channeling concept for the connected clients, so they would receive only the events they've asked to subscribe to.
 *                                     All the published events will be pushed to the same configured topic, This means all clients would be receive all the events unless this field set as false.
 *                                     To enable message channeling when the message get published, the ServiceBusPubSub would applicationAttribute called eventName, this would be the eventName from the publish method.
 */
export interface IServiceBusOptions {
  connectionString: string;
  topicName: string;
  subscriptionName: string;
}

/**
 * An override for the in-memory PubSubEngine which connects to the Azure ServiceBus.
 */

export class ServiceBusPubSub extends PubSubEngine {
  publish(eventName: string, payload: any): Promise<void> {
    throw new Error('Method not implemented.');
  }

  private client: ServiceBusClient;
  private handlerMap = new Map<number, Subscription>();
  private options: IServiceBusOptions;
  private eventNameKey: string = 'sub.eventName';
  private subject: Subject<IEventResult>;

  constructor(options: IServiceBusOptions, client?: ServiceBusClient) {
    super();
    this.options = options;
    this.client = client || new ServiceBusClient(this.options.connectionString);

    this.subject = new Subject<IEventResult>();

    const subscription = this.client
      .createReceiver(this.options.topicName, this.options.subscriptionName)
      .subscribe({
        processMessage: async (message: ServiceBusReceivedMessage) => {
          this.subject.next({
            ...message,
          });
        },
        processError: async (args) => {
          console.log(
            `Error from source ${args.errorSource} occurred: `,
            args.error
          );

          if (isServiceBusError(args.error)) {
            switch (args.error.code) {
              case 'MessagingEntityDisabled':
              case 'MessagingEntityNotFound':
              case 'UnauthorizedAccess':
                console.log(
                  `An unrecoverable error occurred. Stopping processing. ${args.error.code}`,
                  args.error
                );
                await subscription.close();
                break;
              case 'MessageLockLost':
                console.log(`Message lock lost for message`, args.error);
                break;
              case 'ServiceBusy':
                await delay(1000);
                break;
            }
          }
        },
      });
  }

  /**
   * Subscribe to a specific event updates. The subscribe method would create a ServiceBusReceiver to listen to all the published events.
   * The method internally would filter out all the received events that are not meant for this subscriber.
   * @property {eventName | string} - published event name
   * @property {onMessage | Function} - client handler for processing received events.
   * @returns {Promise<number>} - returns the created identifier for the created subscription. It would be used to dispose/close any resources while unsubscribing.
   */
  async subscribe(eventName: string, onMessage: Function): Promise<number> {
    const id = Date.now() * Math.random();
    this.handlerMap.set(
      id,
      this.subject
        .pipe(
          filter(
            (e) =>
              (eventName && e.body.name === eventName) ||
              !eventName ||
              eventName === '*'
          ),
          map((e) => e.body),
          tap((e) => e)
        )
        .subscribe((event) => onMessage(event))
    );
    return id;
  }

  /**
   * Unsubscribe method would close open connection with the ServiceBus for a specific event handler.
   * @property {subId} - It's a unique identifier for each subscribed client.
   */
  async unsubscribe(subId: number) {
    const info = this.handlerMap.get(subId) || undefined;
    if (info === undefined) return;
    info.unsubscribe();
    this.handlerMap.delete(subId);
  }
}

but beware: publish is not filled out.

Hey @paulbijancoch, for some reason, I just saw this message. Sure, I'd appreciate any help :) can you please send a PR with the change you're proposing, so it's easier for me to follow your proposal?

@abdomohamed take a look at https://github.com/abdomohamed/graphql-azure-servicebus-subscriptions/pull/3


Custom Networking does not reflect changes

Setting up custom Networking does not reflect changes for generated CLI commands.

@sbugalski thank you very much for logging the issue, this was an oversight, PR submitted

Fixed


Add the ability to transform an existing non-partitioned to a partitioned topic without deleting it.

Context:
At the moment we cannot change the topic type from non-partitioned to partitioned for topics that already exist and contain data. Such topics can now only be deleted and created as partitioned.
In this case, we lose data.
In order to keep the existing data in topics, it is highly desirable to have a way to transform the topic type from non-partitioned to partitioned and vice versa.
Desirable solution:
For example, for the command - pulsar-admin topics, create a key - transform-type, with flags:
--to-non-partitioned
--to-partitioned

which will allow us to transform the topic type.

The issue had no activity for 30 days, mark with Stale label.


Support Sequel ORM

hello. Thanks for your PR. Would you mind adding a test that demonstrates the usage?
Also an entry to README would be nice, that explains usage.

Would you mind adding a test that demonstrates the usage?

I wouldn't mind, but I think you should accept #64 first, in order to avoid conflicts here.

Just in case I'll say that this is ready for review


RDC-robust credit initialization

It would be useful to define a credit exchange protocol for credit/valid style channels that cross a reset domain boundary.
The baseline credit_stall based protocol is a good start, but really only handles initial reset release:

sender drives a "stall" signal to the receiver when it is in reset
if receiver comes out of reset first, it won't release credits to the sender until the sender deasserts its stall signal

What is needed is to handle arbitrary re-entry into reset of one or both sides of the channel. The desired behavior is:

data can get lost (unavoidable)
channel always reaches a consistent state where both sides have a correct view of credits (and the correct amount of credit) after either (or both) sides undergo any arbitrary reset
either side get a positive notification when the other side goes through a reset, so that it can take any cleanup action (if necessary/desired)
data in flight at time of reset assertion will never be delivered at the destination after the re-initialization completes (i.e. any data after the notification can be assumed valid from the current reset epoch)

I haven't mapped out in detail, but think this should need like a 3-4 state FSM on each side, the state of which is exported to the other link partner.

I would not want this common component to be too opinionated about a larger reset architecture. Surely it could provide useful capabilities to that end, but I don't think that is necessarily the point.
I think the desired behavior here is good from a resilience POV generally. If two independent components are connected by a credited channel, and one of them encountered an error/problem that can only be recovered from via a reset, it is generally possible to reset just that component (perhaps in combination w/ some additional action to quiesce traffic at a higher level) and recover.
That is a good property that generally opens up a lot of options from a survivability POV.

Summarizing some offline discussion. Not trying to design a general reset architecture here. The goal is to support credit/valid flow control where there can be a reset domain crossing or a reset skew boundary in between sender and receiver.
Currently the credit modules support only a reset skew boundary, i.e., both sender and receiver must enter and exit reset (with some skew in entry and exit times). We want to generalize it so that sender and receiver can independently enter and exit reset without breaking the flow control correctness, i.e., no credits should be permanently lost.
We think this can be done by changing credit_stall to mean sender_in_reset (wire going from sender to receiver) and then adding a second signal called receiver_in_reset (wire going from receiver to sender). These wires will control the reinitialization of credit counters on either end and also "transmission gate" the datapath as needed.
This scheme won't actually quiesce traffic or do transaction-level reset protocol. But we feel having a reset-robust flow control mechanism is a necessary (but insufficient) condition for general reset architecture.

I would suggest we keep the name credit_stall and name the new signal credit_reinit. This will make the name match its function (reinitializing credit count at the sender) rather than its origin.

credit_dump?

The name on the regular counter is reinit, so I think credit_reinit would be most consistent.

Took a stab at it in #244 . The signals are named credit_reset_active_fwd (sender to receiver) and credit_reset_active_rev (receiver to sender). When the sender gets credit_reset_active_rev, it reinitializes its credit counter and blocks sending valid. When the receiver gets credit_reset_active_fwd, it reinitializes its credit counter and stalls sending any credit. That should be sufficient to allow either side to be reset without the other.


Rename blob and blobref type variables to b

Follow-up to #469. It's a rather pervasive change, so I'm keeping this PR in draft mode until more important PRs have landed on main

Should be ready to review now!


Create an instructional beginner map with audio playbacks

From @IAmNotHanni in #75:

Maybe we could script a beginner map with audio playbacks. I am inspired by Portal in this aspect:
https://youtu.be/eFe3eKaKeF0?t=59s
This would be so awesome!

In my opinion this issue is in the wrong repository. Technically it's already possible, so somebody needs just to build such a map and that belongs into inexor-game/data.


Bad line snapping (python)

Take the following code snippet;

If I now press "Option + Left" to jump back to the previous "word", it jumps to the wrong position in the line, as you can see below.

It jumped onto the end of the email_address string, rather than the closing bracket. My guess is that it not detecting the code context and subsequently doesn't know to jump to the end of the bracket rather than the end of the word.
Instead, it should have jumped here;

Here's another one.

When pressing "Option + Left" on the start of the first word of the line, it should jump to the start of the line before the whitespace, but instead it jumps up to the next line beyond the colon.

I can see how these behaviors could be surprising, but both of them are actually intentional. For the editor:move-to-beginning-of-word and editor:move-to-end-of-word commands, a "word" is defined as either a string of consecutive non-whitespace characters that are either all "word characters", or all "non-word characters". This is similar to what Vim and several other text-editors do.
Also, the definition of "non-word characters" is configurable through the editor.nonWordCharacters configuration setting, which can be set differently for different languages.
Also, if you have a different behavior in mind for alt-left and alt-right, you can map those keys to commands that you define.
@foxx I'm going to close this out, since I don't think this is a bug in Atom. Let me know if you do still think it's a bug; I can reopen it.

You can also remap to use the editor:move-to-*-word-boundary commands which might be more what you're looking for:
'.platform-darwin atom-workspace atom-text-editor':
  'alt-left': 'editor:move-to-previous-word-boundary'
  'alt-right': 'editor:move-to-next-word-boundary'
  'alt-shift-left': 'editor:select-to-previous-word-boundary'
  'alt-shift-right': 'editor:select-to-next-word-boundary'

Thank you for the quick and detailed reply, appreciated

@maxbrunsfeld, while this might not be a bug. It could be a glitch in UX, especially for the users migrating from other modern editors.

+1. Please reopen

So how do i configure atom to jump in a way @foxx described in the beginning of the discussion?


Escalating areas with large number of linting issues

A common issue raised with microtasking is that one does not see the larger picture of what is being fixed. This is a problem when there is bad changeset with multiple errors that is better to be reverted than fixed fixed by micro tasking.

An example is when this accidental changeset introduced a large number of overlapping connectivity errors throughout Cambridge. When the original mapper and DWG initiated a cleanup by reverting the changesets it ran into conflicts because of fixes that happened to the objects via to-fix.
@Rub21 @geohacker Fixing these issues via to-fix made the data harder to cleanup, how can we better handle issues like this to avoid such a scenario? Could we flag this separately somehow?
cc @manoharuss

@planemad

This is a problem when there is bad changeset with multiple errors that is better to be reverted than fixed fixed by micro tasking.

I completely agree with that, We are working a lot with to-fix fixing many issues per day. from now on we are going report or revert this changest in osmcha.
right now something automatic which we can do in to-fix  is not possible, i think the only way so far is to check the changeset and set as harmful or not or just an unintentional edition
cc. @calfarome


cols prop having no effect on layout

I am trying to set up a grid with only 7 columns, and one row: They are generated dynamically like so:
import styled from 'styled-components';
import { useTableContext } from '../reducers-contexts/table_context';
import { DayColumn } from './DayColumn';
import moment from 'moment';
import GridLayout from 'react-grid-layout';

export const TasksContainer = () => {
  const { dayColumns, startTime } = useTableContext();

  const layout = [];

  return (
    <Wrapper>
      {dayColumns.map((day, index) => {
        layout.push({
          i: index.toString(),
          x: index,
          y: 0,
          w: 1,
          h: 1,
        });
        return (
          <GridLayout
            className='layout'
            layout={layout}
            cols={7}
          >
            <DayColumn
              key={index.toString()}
              date={moment(startTime + ((index + 1) * 86400000 - 1)).format(
                'l'
              )}
              day={day.id}
            />
          </GridLayout>
        );
      })}
    </Wrapper>
  );
};

const Wrapper = styled.section`
  width: 100%;
  height: 100%;
`;

The cols={7} attribute is having no effect and is only deisplaying one column like so:

What I tried:

Creating the layout with the data-grid prop on the DayColumn
Changing the width of the day column to less than the width of the container, hoping they were just wrapping
Changing the width of the Wrapper at the bottom to less than 100%, hoping they were just wrapping

You're generating 7 grid layouts.
On Tue, May 4, 2021 at 9:44 AM Conner Linzy @.***>
wrote:

I am trying to set up a grid with only 7 column, and no rows: They are
generated dynamically like so:
import styled from 'styled-components';
import { useTableContext } from '../reducers-contexts/table_context';
import { DayColumn } from './DayColumn';
import moment from 'moment';
import GridLayout from 'react-grid-layout';
export const TasksContainer = () => {
const { dayColumns, startTime } = useTableContext();
const layout = [];
return (

{dayColumns.map((day, index) => {
layout.push({
i: index.toString(),
x: index,
y: 0,
w: 1,
h: 1,
});
return (

<DayColumn
key={index.toString()}
date={moment(startTime + ((index + 1) * 86400000 - 1)).format(
'l'
)}
day={day.id}
/>

);
})}

);
};
const Wrapper = styled.section  width: 100%;   height: 100%;;
The cols={7} attribute is having no effect and is only deisplaying one
column like so:
[image: Screenshot 2021-05-04 094057]
https://user-images.githubusercontent.com/74991878/117012377-e2924d00-acbc-11eb-9d58-318ab0dbc1fe.png
What I tried:

Creating the layout with the data-grid prop on the DayColumn
Changing the width of the day column to less than the width of the
container, hoping they were just wrapping

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/react-grid-layout/react-grid-layout/issues/1464, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJEKPZ32GTUPO5CDNWWNA3TL72ZXANCNFSM44CXVQKQ
.

@STRML  How so? the prop is cols, doesn't that imply your're controlling columns?

You've rendered 7 grids, not 1 grid with 7 items.


Error after deploying

Forge CLI Version: 0.1.5

PHP Version: 8.0.2
Operating System: WINNT
Error Message: Unable to connect to remote server. Maybe run [ssh:configure] to configure an SSH Key?

Description:
When I deploy to the site, the deployment actually works and the site is deployed. Just a weird thing after that. I get an error, Unable to connect to the remote server. I have the connection working hence I can deploy.
Steps To Reproduce:
forge deploy site

Before performing any tasks using the Forge CLI, you should ensure that you have added an SSH key for the forge user to your servers so that you can securely connect to them: https://forge.laravel.com/docs/1.0/accounts/cli.html#ssh-key-authentication.

The issue is... I have already done that. I have just confirmed again the ssh key is there. I have do it a fresh again now, am still getting the same error now.

Can you try forge ssh:test?

On my system, forge ssh:test gets the same results:
Unable to connect to remote server. Maybe run [ssh:configure] to configure
an SSH Key?
I noticed when I generate SSH keys via forge-cli, their format is different than those I've generated manually.
Public keys I've successfully used before, generated w/o Forge CLI:
ssh-rsa xxxxx== user@machinename
Public keys Forge CLI generates:
-----BEGIN PUBLIC KEY-----
xxxxx
-----END PUBLIC KEY-----

Private keys successfully used before, generated w/o Forge CLI:
-----BEGIN OPENSSH PRIVATE KEY-----
xxxxxxxxxx
-----END OPENSSH PRIVATE KEY-----

Private keys generated by Forge CLI:
-----BEGIN PRIVATE KEY-----
xxxxx
-----END PRIVATE KEY-----

With the old one having quite a few more lines than the new one. I don't know enough about SSH protocol and how these things work to say what the tangible difference is between these formats.

Dumping around in RemoteRepository.php:ensureSshIsConfigured while running ssh:test, the SSH command that results is:
ssh -o ConnectTimeout=5 -o ControlMaster=auto -o ControlPersist=100 -o ControlPath=C:\Users\ethan/.laravel-forge/sockets/%h-%p-%r -o LogLevel=QUIET -t<EMAIL_ADDRESS>-t exit 0

It's result is an empty string, the output array is empty, and the exit code is 255.

Can you try forge ssh:test?

Yah, when I run ssh:test, it passes. I get "SSH Key Based Secure Authentication Is Configured". But after deploying, I still get the same error

Hm, that suggests the issues are unrelated I suppose.
Forgoing forge-cli altogether to connect, but still using the SSH keys it generated, I get permission denied anyway:
ssh -i C:\Users\ethan\.ssh\ethan_rsa.pub<EMAIL_ADDRESS>Load key "C:\\Users\\ethan\\.ssh\\ethan_rsa.pub": invalid format
<EMAIL_ADDRESS>Permission denied (publickey).


Data at Rest Encryption

Data at Rest Encryption
Issue
Riak as a whole does not encrypt data at rest.
https://www.tiot.jp/riak-docs/riak/cs/2.1.1/cookbooks/faqs/riak-cs/#does-riak-cs-encrypt-data-at-rest
https://quabase.sei.cmu.edu/mediawiki/index.php/Riak_Security_Features
Problem
The problem has three fronts:

Many competing systems are offering encryption of data at rest i.e. marketing.
Industries are starting to make encryption of data at rest a requirement i.e. usage viability.
Hackers are becoming better at accessing file systems where non-encrypted data is at risk i.e. security.

Potential Pitfalls

Partitions are moved between servers so an encryption key would have to be standardised between all nodes in the cluster to allow encryption and decryption i.e. if a partition is encrypted by one node with one key and then transferred to a node with a different key, it is now useless.
As Riak runs inside an ErlangVM, the encryption key would most probably need to be stored in plain text format.
Being open source, it should not be long before a potential attacker learns the location of the encryption key, copies it and is able to use it to decrypt data again.
Keeping the encryption key on one server and sharing with the cluster at run time works until the server that stores the key goes offline.

Security Concepts
Security works in three ways:

Who you are
What you have
What you know

"Who you are" is based on the user you are accessing the server as. If the attacker is root or riak then they can just ask Riak for the full database in human readable form, so let's ignore this one.
"What you have" would apply to the encryption key. Anybody with the encryption key and a bit of technical know how should be able to retrieve human readable data from the file system (data directory) of Riak even if it is only partial data.
"What you know" would be where real users can be differentiated from attackers. Real users will know things such as bucket names and key names. Attackers are thieves of opportunity and will be unlikely to know this information especially if all they have to work with is the file system of a single node.
Proposed Solution
Combining the above Security Concepts, we can see that the only difference between an attacker and a genuine user is the knowledge of the bucket and key names. I would like to propose the following solution to encrypting data at rest within Riak KV:

Share an encryption key between all nodes in the cluster (probably user specified text file saved per node).
Have an option in riak.conf to allow encryption to be used (some people might not want it yet).
Have a start up check and possibly a polled interval that makes sure all nodes in the cluster are using the same key and have encryption turned on.
When writing data, use the encryption key as discussed but salt the key with the bucket name and the key name to encrypt the value stored.
When reading data, decrypt with the same salted encryption key which should work as the user will be providing the bucket name and key name.
Buckets are tagged as encrypted so that we can check what percentage of the data is encrypted

By salting the key with the bucket and key names, it would render the plain encryption key nearly worthless to a potential attacker as they would have to brute force all potential bucket/key name combinations to access the data.
Legacy and Migration
Assuming that the above functionality can be implemented, users may wish to encrypt an already existing Riak install or may decide they no longer wish to keep their data encrypted and wish to decrypt everything. I propose that a default read/re-write mode be created. This mode would check whether data is meant to be encrypted or not and every time Riak reads data that is not of the desired encryption type, it re-writes the exact same data but in the desired encryption type.
Through this method, users with high load clusters could wait for their data to naturally become encrypted/decrypted or users in a hurry could run a script to read every single bucket to force the encryption. Just in case clusters are subject to performance spikes, potentially add a user definable config flag e.g. 70% where read/re-write is temporarily disabled while it waits for the cluster to be under less than 70% load before resuming.
We might want to add an output to a command such as riak-admin cluster status that would include the "Percentage encrypted" information for the cluster.
Additional Security
As security is all about layers, it might be worth looking at storing the key with extremely limited permissions i.e. to access the encryption key you would have to be the riak user or preferably just root but this would depend on how easy it would be to get the ErlangVM to access said encryption key.

To be blunt this seems like a lot of work for no result. If the encryption key lives on a text file on the host, copying the filesystem means copying the key and all needed information as:

The key is on the same node - an attacker has it
The bucket names need to be stored unencrypted for bucket listing/bucket metadata (or encrypted with the key only but as we have the key it's the same thing) - an attacker has them
The key names have to be stored unencrypted (or encrypted with key + bucket name but since we got those on the last two steps it's the same thing)

The only possible solution for this I see is having a user provide the encryption key on request, it still lives in memory for a while so a crafty attacker would be able to obtain it but at least it'll be a bit safer. This could be somewhat mitigated by using asymetric encyption, i.e. have a public key assigned to a bucket (that lives on the nodes w/o issue) that would mean writes do not need an additional key, and either return encrypted data (which would mean reworking how repair & CRDTs work) or providing the secret key on reads (which introduces another security risk).

When it comes to just encryption at rest, filesystem-level encryption will probably do a better job then what we can come up with, those systems are often build by experts and audited for security.

I think the point has been missed here. The idea of salting the encryption key with the bucket name and key name is to render the encryption key alone useless.
From Riak's point of view, it doesn't care whether your bucket is called "myBucket" or "as789AW!233fh84325qhfdsuy9876", so encrypting the bucket name with the encryption key salted with the bucket name is a perfectly valid solution. This way, the attacker cannot retrieve the bucket name from the file system without significant brute force work or some other non-Riak related method to get the list of bucket and key names a different way.
This method does have the caveat that you cannot ask Riak for a list of buckets and keys but even Basho said that this should not be done in a production environment:
https://www.tiot.jp/riak-docs/riak/kv/2.2.3/developing/api/http/list-buckets/
https://www.tiot.jp/riak-docs/riak/kv/2.2.3/developing/api/http/list-keys/
I do not argue that having filesystem-level encryption would be beneficial. However, security is all about layers and if the attack is against a live server where the attacker successfully gains riak or root then filesystem-level encryption will be worthless. Thinking about it, with the bucket and key names encrypted, the attacker would not be able to run a bucket list or key list to retrieve the buckets or keys as the returned lists would be the encrypted names. Of course, the attacker could brute force them but still, this would further slow them down and that is the entire idea of layered security. Nothing is totally secure but every step to hinder the attacker is worthwhile.
At the end of the day, we need to look at this with a regard to the future suitability of Riak for corporate customers which means we need to address the problem listed at the start of the post:

Many competing systems are offering encryption of data at rest i.e. marketing.
Industries are starting to make encryption of data at rest a requirement i.e. usage viability.
Hackers are becoming better at accessing file systems where non-encrypted data is at risk i.e. security.

Riak already has a surprisingly wide array of security options https://www.tiot.jp/riak-docs/riak/kv/2.2.3/using/security/basics/ which have been nicely summarised in this blog post https://gist.github.com/lucperkins/8920660 and, in the corporate world, where ticking boxes is often as far as senior management looks towards IT security, having this feature available would fill that requirement nicely.
It may be a lot of work that many people do not see as a priority right now but as regulations start rolling out, I am not sure whether Riak will be able to survive outside of obscurity for long if its only remaining users are hobbyists and one or two die hard corporate fans.

My 2¢ - better to encrypt the data at the middleware layer - if it were confidential I'd never even send it to the storage system in the first place.


Navigine Licence Agreement

Hi Alex,
I am just wondering what software licence agreement you are distributing the Navigine under, I cannot seem to find it anywhere (the docs site just has a licence for the documentation itself).
I would like to incorporate this into an open source project (with full recognition back to this project), but just want to make sure the licence is compatible before proceeding.
Regards,
Blair

Re: [Indoor-navigation-algorithms] Navigine Licence Agreement (#4)
Hi Blair,
Thnx for the question,
we are happy to help you with your project.
Let's shortly discuss it via Skype.
Best regards,
Alexey Panyov
PhD, CEO of Navigine,
www.navigine.com
skype: alexey.panev
phone: +79266599850
Hi Alex,
I am just wondering what software licence agreement you are distributing the Navigine under, I cannot seem to find it anywhere (the docs site just has a licence for the documentation itself).
I would like to incorporate this into an open source project (with full recognition back to this project), but just want to make sure the licence is compatible before proceeding.
Regards,
Blair
—
Reply to this email directly or view it on GitHub.


Ignored folders

There should be a way to any user add more ignored folders

I don't remember for sure but I think it respects .gitignore.
On Tue, Feb 9, 2016 at 2:35 PM, Rafael Mello Campanari <
<EMAIL_ADDRESS>wrote:

There should be a way to any user add more ignored folders, not only
node_modules
—
Reply to this email directly or view it on GitHub
https://github.com/mark-hahn/line-count/issues/14.

but some files should not be counted but should in git，such as hacked lib..

@mark-hahn unfortunately it doesnt ignore the .gitignore paths either.

Adding 'lib' and 'env' would be a big help alongside node_modules

Yea, a little box in the UI to add a list of ignored folders could be a solution. I like this plugin but my project was counted for over 669,000 lines of code, which is totally wrong since it included my virtualenv Python libraries and MeteorJS build folder... Does it really use .gitignore? I have these in my .gitignore but it still counted them.


import react

https://github.com/kata-ai/aksara-ui/blob/a1244c2c84859fb701537dea151ac3b4ccd062f0/packages/aksara-icons/src/chart/IconChartGraph.tsx#L1
Wouldn't it be better to call react this way?
Because in a post I read that the correct way to call react is this.
import * as React from 'react'

@wdavidcalsin Yeah, I used an automatic svg -> react converter to generate the icon files. Will have to look through them again and redo the exports.


Add some graphic elements to README.md

the readme is just a giant text wall right now and needs a bit of tweeking to be more appealing to user's eyes.
These include a banner on top and some emojis in the text.

Please use this issue thread to suggest/discuss any ideas  regarding the possible graphical elements we can add to the document to make it more appealing to read.


Module Metadata not displaying EXPORTS correctly

I think your Module decorator is not creating the EXPORT metadata correctly. I believe it should be setting the attribute with self.exports instead of imports.
class Module:
    def __init__(
        self,
        imports=None,
        controllers=None,
        providers=None,
        exports=None,
        is_global: bool = False,
    ):
        self.controllers = controllers or []
        self.providers = providers or []
        self.imports = imports or []
        self.exports = exports
        self.is_global = is_global

    def __call__(self, cls):
        setattr(cls, ModuleMetadata.CONTROLLERS, self.controllers)
        setattr(cls, ModuleMetadata.PROVIDER, self.providers)
        setattr(cls, ModuleMetadata.IMPORT, self.imports)
        setattr(cls, ModuleMetadata.EXPORTS, self.imports)  # <- This seems like it should be self.exports
        setattr(cls, "__is_module__", True)
        setattr(cls, "__is_global__", self.is_global)

        return cls

@JEub Hi! Sorry for the delay, not sure how I missed so an important issue. I can see that there is a mix in the keywords. the problem is that no functionality uses these exports, so there was no way to know that this is broken. Do you want to open a patch PR for that? this could be a real quick win

@JEub issue fixed in this commit - https://github.com/PythonNest/PyNest/pull/60/commits/69771ac01d26ff29c8da088ed3a2e9da36b85db1
Thanks again for raising this issue!


Test data from demo sessions

is it our Roadmapper Trello board or some demo board?

Doesn't seem to be our trello, probably some demo board then.
I could make all the task "internal" (not imported)


Speckle don’t send all data point cloud from lidar mobile

Prerequisites
Need insert file point cloud to revit model

[x] I read the contribution guidelines
[x] I checked the documentation and found no answer.
[x] I checked existing issues and found no similar issue.
[x] I checked the community forum for related discussions and found no answer.
[x] I'm reporting the issue to the correct repository (see also speckle-server, speckle-sharp, specklepy, speckle-docs, and others)

What package are you referring to?

Describe the bug
Bug when push point cloud from Revit Model, something missing point
https://speckle.community/t/speckle-dont-send-all-dada-point-cloud-from-lidar-mobile/4034/6
We are missing it here:
https://github.com/specklesystems/speckle-sharp/blob/b56aa04e789652ecb7d4ba944ca7ad613e898254/Objects/Converters/ConverterRevit/ConverterRevitShared/Partial Classes/ConvertGeometry.cs#L105-L112
To Reproduce
Steps to reproduce the behavior:

Go to Revit
Insert file point cloud
push to speckle
See error

Expected behavior
All point cloud will be show in speckle
Screenshots

Related to this thread reported by @chuongmep
Description
User links a point cloud file to Revit. He then selects this file and sends it to a stream. However, points below a certain Z value do not appear in the 3D viewer. You can confirm this by checking the stream below.
This can be reproduced. I changed the location of the linked point cloud file and rotated it 90 and 180 degrees. Nothing changed. It only sends the upper half of all the points (needs confirmation).
Also, this is not a viewer issue. When the same commit is received in Rhino, only points in the viewer show up.
Expected Behaviour
Send all points to Speckleverse.
Steps to reproduce

Open a new blank project in Revit
Link attached point cloud file
Enable Speckle connector and check Send Linked Models option under Advanced Settings
Select the linked point cloud file and Send to Speckleverse.
Open newly made commit and examine it.

Streams/Commits Used
https://latest.speckle.dev/streams/32953abf86
Files Used
Point Cloud File
https://drive.google.com/drive/folders/14r8YRzcOSq3FtICNu77kuDIUMSc--n8a?usp=share_link
Screenshots
Point cloud file viewed in Revit

Send to Speckle

Same commit received in Rhino👇

Other relevant conversation here:

(private PM): https://speckle.community/t/youre-a-new-user-of-the-year/3967/15


Fix Clang ASAN issue by disabling float-cast-overflow check in sanitizer

Explanation can be found in the link: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
-fsanitize=float-cast-overflow: Conversion to, from, or between floating-point types which would overflow the destination. Because the range of representable values for all floating-point types supported by Clang is [-inf, +inf], the only cases detected are conversions from floating point to integer types.


Where is this contract? ethPriceSource of DAI vaults

Would like to know where to find the source, or verified contract of ethPriceSource for:

polygon Stake DAO USD Strategy (0x57cbf36788113237d64e46f25a88855c3dff1691)
camDAI Vault (0xD2FE44055b5C874feE029119f70336447c8e8827)

These are the contracts I am looking to know the source:

https://polygonscan.com/address/0x1505ec6dd93d4a829415c543225b03ea4b64a280 (Stake DAO ethPriceSource)
https://polygonscan.com/address/0xcc7db9865db54caadc5ac88952c75734b6060439 (camDAI ethPriceSource)

I looked here on github but cant find it, and would be important to have all the maths to avoid being liquidated.

could derive camDAI from the logic on other cam Vaults.
But would appreciate if someone can share the maths hidden on Stake DAO USD Strategy's oracle

bump


[bug]: ModuleNotFoundError: No module named 'controlnet_aux'

Is there an existing issue for this problem?

[X] I have searched the existing issues

Operating system
Windows
GPU vendor
Nvidia (CUDA)
GPU model
RTX 3070
GPU VRAM
8GB
Version number
4.2.0
Browser
Google Chrome
Python dependencies
No response
What happened
I have been trying to run invoke ai for weeks now i keep getting this error: "ModuleNotFoundError: No module named 'controlnet_aux' ". I tried the dev option and used the command "pip install controlnet-aux," and after that, I even activated the virtual environment and downloaded the library. After installation, it says the requirement is satisfied. But still, whenever ever i try to run it again, it doesnt work and gives me the same error. I tried many things and saw a post saying to reinstall Python with Cuda because sometimes it helps. I did that as well, but nothing worked. I even deleted and installed the whole invoke ai multiple times both by using installer and git but i am still getting this error
What you expected to happen
How can i fix this error and run the program?
How to reproduce the problem
No response
Additional context
No response
Discord username
No response

Hi, did you fix this?
If not, I think I've figured out the a workaround.
The controlnet_aux package has had some issues with how it is distributed. Sometimes a broken version of it is installed. We don't have control over this, it's the python packaging system (pip) that is causing the issue.
When this happens, the broken version of controlnet_aux is cached by pip and subsequent reinstalls will re-use the pip cache. We can clear the pip cache for controlnet_aux and/or all packages:

Run the Invoke launcher.
Select the option for developer console.
Run pip cache remove controlnet_aux to remove it from the cache.
Close the launcher window.
Run the installer. A fresh controlnet_aux package will be downloaded and used.

If this doesn't fix the issue, do the same thing but at step 3, run this command to empty the whole pip cache:
pip cache purge


Shim ES6 methods for IE11 close #27

To solve problems in IE 11 and make this lib easy to use we need to shim ES6 methods:

[x] Object.assign - polyfilled by next.js
[x] Array.find - ponyfilled by this PR
[x] Array.includes - ponyfilled by this PR

I am not a big fan of this because I believe it is the job of preset-env to add such polyfills.
Nevertheless right now it is pretty complex to configure it via next.config.js
What do you think about this PR @fridays?
Status: ⚠  Not tested

Thanks @HaNdTriX for taking a stab at this.  I was contemplating doing the same today.

Many thanks @HaNdTriX ! It's probably the best way to solve this for now. I'll publish in a moment.

It's now there in 1.0.25

I'm testing and seeming to have issues with Object.assign. Do you have any suggestions on how to definitively test?


support ol.source.ImageStatic

ol3-cesium  does not ol.source.ImageStatic

the Cesium.SingleTileImageryProvider  can solve this problem.


Nuxt generate adding all styles inline

Everything working fine except stylesheets.
Nuxt adding all styles inline inside

Why do you need them separate?

@roslaneshellanoo have you read this - https://github.com/nuxt/nuxt.js/issues/12

#475 #471


Set-DbaLogin - the -MustChange param requires check_policy and check_expiration to be enabled

Report
Set-DbaLogin - the -MustChange param requires check_policy and check_expiration to be enabled
While working on #7004 it was noticed that the -MustChange param requires the policy related params to be set to true. The MS documentation has a mention of this:  https://docs.microsoft.com/en-us/sql/t-sql/statements/alter-login-transact-sql?view=sql-server-ver15#remarks
Suggested code changes for this issue are here:
Validation
https://github.com/sqlcollaborative/dbatools/pull/7087/files/7127b1c70d9ad43de2d5fa412ab4e9ea485f57f5#diff-419523c18ac6bcaa3c3134c4b1996958ae756dbd7a59abec0193bfb8dc804381R221
https://github.com/sqlcollaborative/dbatools/pull/7087/files/7127b1c70d9ad43de2d5fa412ab4e9ea485f57f5#diff-419523c18ac6bcaa3c3134c4b1996958ae756dbd7a59abec0193bfb8dc804381R226
Ensuring the policy params are set and the login object is refreshed
https://github.com/sqlcollaborative/dbatools/pull/7087/files/7127b1c70d9ad43de2d5fa412ab4e9ea485f57f5#diff-419523c18ac6bcaa3c3134c4b1996958ae756dbd7a59abec0193bfb8dc804381R273
https://github.com/sqlcollaborative/dbatools/pull/7087/files/7127b1c70d9ad43de2d5fa412ab4e9ea485f57f5#diff-419523c18ac6bcaa3c3134c4b1996958ae756dbd7a59abec0193bfb8dc804381R284
Host used

[x] powershell.exe
[ ] ISE
[ ] VS Code
[ ] Other (please specify)

Errors Received
WARNING: [15:31:44][Set-DbaLogin] Something went wrong changing the password for [SQLUser1] | The MUST_CHANGE option cannot be used when CHECK_EXPIRATION is OFF.

Steps to Reproduce
$securePassword = ConvertTo-SecureString -String 'strong password' -AsPlainText -Force
Set-DbaLogin -SqlInstance $server -Login SQLUser1 -MustChange -Password $securePassword
Expected Behavior
Set-DbaLogin needs to ensure the must_change is handled correctly.
Option 1: The command can validate the policy params and ensure check_policy and check_expiration are set to 'on'. A warning message can be returned to the caller describing that it was required to change the policy options.
Option 2: The command can enforce a two-step process. The first step the caller has to enable check_policy and check_expiration explicitly. Then, in the second step they can do the must_change.
Actual Behavior
An error is returned as noted above.
Environmental information
PowerShell Version : 5.1.18362.1171
dbatools latest installed : latest from the development branch
Culture of OS : en-US
SQL Server:
Microsoft SQL Server 2019 (RTM-CU8-GDR) (KB4583459) - 15.0.4083.2 (X64) Nov 2 2020 18:35:09 Copyright (C) 2019 Microsoft Corporation Developer Edition (64-bit) on Windows 10 Pro 10.0 (Build 18363: ) (Hypervisor)
us_english

if ((Test-Bound MustChange) -and ((Test-Bound PasswordPolicyEnforced -Not) -or (Test-Bound PasswordExpirationEnabled -Not))) {
The fact that the settings are required to be enabled on the login is fine, but we can't enforce this by requiring them to provide the additional parameters in the same call. They may not know it's required and the password they are setting could not work if the policy is enforced.
                            Write-Message -Message "Setting check_policy and check_expiration to true for $l because the must_change option requires it." -Level Warning
                            $l.PasswordPolicyEnforced = $PasswordPolicyEnforced
                            $l.PasswordExpirationEnabled = $PasswordExpirationEnabled
                            $l.Alter()

In this area we cannot forcefully enable those properties on a login without the user explicitly requesting it. We can provide a more precise message because if both are enabled the only error SQL Server spits out is that one of the two is not enabled:

We can help in that regard if SMO requires them both to be enabled that we check for them to be, and if we find them not then spit out the warning as such and not attempt the change password call.
Something like:

user submits MustChange
Validate checks for policy and expiration are enabled on the login
One or both are found not enabled: spit warning stating which ones are not enabled (Stop-Function with continue)
continue processing on the next login object

if ((Test-Bound MustChange) -and ((Test-Bound PasswordPolicyEnforced -Not) -or (Test-Bound PasswordExpirationEnabled -Not))) {
The fact that the settings are required to be enabled on the login is fine, but we can't enforce this by requiring them to provide the additional parameters in the same call. They may not know it's required and the password they are setting could not work if the policy is enforced.
                            Write-Message -Message "Setting check_policy and check_expiration to true for $l because the must_change option requires it." -Level Warning
                            $l.PasswordPolicyEnforced = $PasswordPolicyEnforced
                            $l.PasswordExpirationEnabled = $PasswordExpirationEnabled
                            $l.Alter()

In this area we cannot forcefully enable those properties on a login without the user explicitly requesting it. We can provide a more precise message because if both are enabled the only error SQL Server spits out is that one of the two is not enabled:

We can help in that regard if SMO requires them both to be enabled that we check for them to be, and if we find them not then spit out the warning as such and not attempt the change password call.
Something like:

user submits MustChange
Validate checks for policy and expiration are enabled on the login
One or both are found not enabled: spit warning stating which ones are not enabled (Stop-Function with continue)
continue processing on the next login object

ok, thanks, I agree that sounds much better.

ok, thanks, I agree that sounds much better.


Debug viewlet stays in "welcome" mode even after launch.json creation

Version: 1.50.0-insider
Commit: cca20eba8974b1d6b6a9b71bf4e60a29c8f26648
OS: Darwin x64 19.6.0

create new folder aynwhere
open folder in VS Code
goto debug viewlet
observe: debug viewlet is in "welcome" mode
click link "To customize Run and Debug create a launch.json file" to create launch.json
select a debug environment, create a launch config, and save editor

Observe:
debug viewlet is stuck in "welcome" mode

I can not reproduce this. What are you doing differently than me?

VS Code insiders, open new empty folder
Open debug viewlet, click link "To customize Run and Debug create a launch.json file" to create launch.json
Debug viewlet is no longer in Welcome mode

I also tried out of source and vscode stable and I can not reproduce.
If you still see this, do you get file events in that VS Code instance? For example make a change outside of VS Code to some file and see if VS Code is picking up the change.

I was seeing this:

A launch.json exists but the debug viewlet shows the welcome view (even after a restart).
Disabling all extensions made the problem disappear.
After re-enabling all extensions the problem didn't come back...
Strange...


[All mobile Devices] Page is not responsive

Page is not responsive it moves horizontally
URL : https://thenewboston.com/guide/root-account-file
Screenshots :
http://recordit.co/d9MskJ2t2S

Its due to this table. Make this table responsive by giving the table itself a horizontal scroll

This page also https://thenewboston.com/guide/resync-triggers


how to get more mana

i am checking the page and looking thrue things and i cant find how to get more mana help please

figurd it out


Fix issues with GitLab CI

Description
It's time to test this project against real world CI/CD Pipelines!
Since I already have a ready-to-use GitLab instance, I will try with this one. All required fixes will be documented here.

Things that have been fixed:

Strongly type Run's property projectGraph
Strongly type Run's property vcsContext
Add missing Run's property sha
Update HashDetails with some nullable properties


lua help

Describe your problem here.
i'm not sure if i'm doing this right but i'm trying to change a variable at a certain step in the song (i'm still fairly new to lua)

Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
No response

If the variable already exists then don't use local in that line


Link to previous version

The link Introduction to Collections in Configuration Manager (link is http://go.microsoft.com/fwlink/p/?LinkID=259433) point to a previous version documentation. Please update the link.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d8a45367-1771-6f86-1cfa-8fbd62837ae4
Version Independent ID: 0a3ba2c4-2bd5-9659-7243-39d206b02871
Content: Get-CMDeviceCollectionQueryMembershipRule - Configuration Manager
Content Source: sccm-ps/ConfigurationManager/Get-CMDeviceCollectionQueryMembershipRule.md
Product: configuration-manager
Technology: configmgr-other
GitHub Login: @aczechowski
Microsoft Alias: aaroncz

Thanks @gbeaudoin78 for the feedback. We had a lot of old and crusty links in this doc set. I just replaced all old fwlinks and previous-version references, including this one. The changes will publish live soon with the 2006 content release.


Add docker-compose to run as standalone container

Fixes #31

Added docker-compose to run both sunsynk-multi as well as mbusd
Cleaned up sunsynk-multi Dockerfile, as list of Python dependencies should be in a requirements.txt file

Thanks! What do you use this for if I may ask, do you deploy it locally without HASS OS?

Indeed. I find it really annoying how HASS OS / HA Supervisor hijacks your entire machine. I've been through many different setups where the host machine is shared between Home Assistant and other services, and each time either HA is unhappy, or it kills my other containers. Yesterday after a reboot HA didn't want to start up, so I gave up and decided to go clean HA Core in a container, and run all the add-ons manually in containers next to it.
I prefer running everything in docker containers on my server, allowing me to update the host without affecting any of the running services. So very similar reason as pinning library versions. Having to run a VM for HA is a lot of unnecessary overhead.
Regarding this PR: I'll test the yaml config file approach, apply your requested changes and push a new commit.

Thank you @jpmeijers 🚀

Can I ask a question here?

Can I ask a question here?

You can, but always better to ask in a new/active discussion, or new/open issue (like #183)


please add global hotkey support

dear developers of the fantastic ebiten,
thank you very much for your work and the easy to use GUI creation package.
could you please add global hotkey support?
i would like to react to hotkeys even when the ebiten window does not have focus.

Hi,
I'm not sure what is 'hotkey'.

i would like to react to hotkeys even when the ebiten window does not have focus.

Hmm, I think this is not Ebiten's responsibility but OS's responsibility.

please see https://github.com/glfw/glfw/issues/1204, it is explained better there

OK so let's revisit after GLFW implements this since Ebiten depends on GLFW.
Thank you for using Ebiten!

Ebiten is a great way to have a gui and everything else. All the thanks to you. I just consume. :-)
Here I found the way to register global hotkeys for windows: https://play.golang.org/p/osbinz7DmN
I will use this way for now since I only target windows atm.

@hajimehoshi please take a look at this: https://github.com/robotn/gohook
this exactly does what i need. a bit over-powered (a complete eventlogger for inputs) but already for three platforms.
i understood, you said you will take a look at it when GLFW implements something...
maybe someone else is hyped by ebiten and needs global inputs before that point of time. ;-)

So you can use this without waiting for Ebiten change?

Yes, did some testing yesterday on Win10 - no issues. tested today on Win7 in a very strict network and was very suprised that none of the security systems prevented this global hook... Not even local admin rights were needed.
Virustotal and Hybrid-Analysis also do not rate this way of realizing "global hotkeys" as suspicious.
Long answer shot: Yes.


Functionality to Solve Quadratic Equation

Issue #112
Implemented the logic to solve quadratic equation when a POST request is made to the /math/quadraticequation endpoint. Additionally, write appropriate test cases within the test/test_app.py file to validate this functionality.

@AimenYaseen ,Thank you for your valuable contribution!!, Do check out other repositories from IEEE-VIT.


feat: openapi version 3 support non-auth changes

Issue #, if available:
#191 #627
Description of changes:
Adding OpenApi 3.0 support for non-auth use cases. This adds a global/api flag called OpenApiVersion, which when specified -

with value 2.0 or 3.0, fixes the issue #191
With value 3.0, changes the swagger to openapi 3.0 format.

Description of how you validated changes:
Added tests, and end to end tests.
Checklist:

[x] Write/update tests
[x] make pr passes
[x] Update documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

#191

#627

Codecov Report

Merging #932 into develop will increase coverage by 0.14%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop     #932      +/-   ##
===========================================
+ Coverage    94.73%   94.87%   +0.14%
===========================================
  Files           69       69
  Lines         3018     3044      +26
  Branches       562      572      +10
===========================================
+ Hits          2859     2888      +29
  Misses          84       84
+ Partials        75       72       -3

Impacted Files
Coverage Δ

samtranslator/model/apigateway.py
97.72% <ø> (ø)
:arrow_up:

samtranslator/model/sam_resources.py
95.5% <ø> (ø)
:arrow_up:

samtranslator/model/api/api_generator.py
95.81% <100%> (+1.52%)
:arrow_up:

samtranslator/plugins/globals/globals.py
100% <100%> (ø)
:arrow_up:

samtranslator/plugins/globals/globals_plugin.py
100% <100%> (ø)
:arrow_up:

samtranslator/swagger/swagger.py
97.66% <100%> (+0.06%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ccf4fc...b89713e. Read the comment docs.

Looks good. Have you deployed any of these OpenApi 3.0 templates to verify that they work in CFN?
@keetonian
I could deploy this successfully

Globals:
  Api:
    OpenApiVersion: '3.0.1'

Resources:
  ImplicitApiFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: s3://praneep-test-2/thumbnails.zip
      Handler: index.gethtml
      Runtime: nodejs8.10
      Events:
        GetHtml:
          Type: Api
          Properties:
            Path: /
            Method: get


Add an option to exclude insufficient measurements

Brief description
An enhancement to import is required to exclude any measurements which, alone, do not sufficiently permit the estimation of station coordinates in two dimensions.
Background [CHANGE OR DELETE IF NOT APPLICABLE]
A network may contain stations to which only a single measurement (e.g. a single distance or angle) is available.  This may be the result of an unintentional error in database extraction, or automatically ignoring a measurement without checking the consequential impact of leaving an insufficient number of measurements behind.
For network adjustments that require the estimation of coordinates in two dimensions, it will not be possible to estimate the parameters.  If the adjustment operator is unaware of such cases, the adjustment will fail to converge and it will not be clear why.
Basic requirements
To help diagnose and remedy such cases, an option is required to identify and remove any single measurements to a station.
Priority
priority 1 (high)

Addressed by #219


Support Abbreviations

http://fletcher.github.io/MultiMarkdown-4/abbreviations
Abbreviations should be working, but they are not:
*[HTML]: HyperText Markup Language
*[W3C]:  World Wide Web Consortium

Then, wherever you use the words HTML or W3C in your document, the abbr markup will be added:
The HTML specification
is maintained by the W3C.

becomes:
The HTML specification
is maintained by the W3C.

Closing this. Please open a new ticket if the problem still exists on v5 (https://stackedit.io/app)


[Transform] Operator Lattice for Mapping

 Issue by maheeka
Monday Oct 30, 2017 at 10:08 GMT
Originally opened as https://github.com/ballerina-lang/composer/issues/4297

[x] Removing operator mappings
[ ] Create mapping
[ ] Casting mappings

Transformer syntax is remove, hence closing this issue


Stop supporting 32-bit platforms

We are considering dropping support for 32-bit platforms and only supporting 64-bit ones. We are a small group of volunteers and supporting 32-bit platforms is a bit difficult.

We can only do CI via QEMU which is far from awful but is also not so easy.
Getting development hardware for doing 32-bit support is non-trivial in some cases.
We experience what seem like bugs in dependencies far more on 32-bit than 64-bit platforms
We have a limited amount of time and devoting it to 32-bit platforms that we suspect aren't being used, seems wasteful when we could devote that time to other tasks

We'd like to hear if anyone is actively using Pony for serious work on any 32-bit platforms and if yes, what you are doing with it. To that end, we have opened this issue. If you can't talk about what you are doing publicly, please reach out to me privately via email.

One counter-example we may want to support in the future (though we don't today) is WebAssembly, where 32-bit is still relevant.


Display proper names in error message

Description
Display the proper account and other names rather than the
uuid representation of the object

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[X] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[X] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[X] Trivial

Screenshots (if appropriate):
Error: (HTTP 531, error code 4365) Account testuser does not have permission to launch instances from template testubuntu

How Has This Been Tested?

Merging based on travis pass.


chore: changed max-width

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 82.128%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
8260

Relevant Lines:
9212

💛  - Coveralls


cmake(clear warning):toolchain file do not need set parent scope

Summary
The warning printout of the CI log will slow down our ability to locate the real problem
toolchain file variable is global scope
dont need set parent scope
clear warning:
CMake Warning (dev) at /github/workspace/sources/nuttx/arch/arm/src/cmake/gcc.cmake:69 (set):
  Cannot set "GCCVER": current scope has no parent.
Call Stack (most recent call first):
  /github/workspace/sources/nuttx/arch/arm/src/cmake/Toolchain.cmake:56 (include)
  /github/workspace/sources/nuttx/build/CMakeFiles/3.26.0/CMakeSystem.cmake:6 (include)
  /github/workspace/sources/nuttx/build/CMakeFiles/CMakeScratch/TryCompile-ZJVOZO/CMakeLists.txt:5 (project)
This warning is for project developers.  Use -Wno-dev to suppress it.

Impact
clear generator warning
Testing
cmake -B build -DBOARD_CONFIG=qemu-armv7a:nsh

@xuxin930 rebase to the last master to fix the ci error.


⚠ Bofrost.at has degraded performance

In f45d07a, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1795 ms

Resolved: Bofrost.at performance has improved in e8a887b after 8 minutes.


Added support for emailing attachments

Bot example:
const binary = await createPdf(repo, "report.pdf", {
  content: [
    "Hello world",
  ],
});

await sendEmail({
    to<EMAIL_ADDRESS>    subject: 'Email from medplum code',
    text: 'This email is from a bot',
    attachments: [{
        filename: 'report.pdf',
        path: "Binary/" + binary.id
    }]
});

Coverage decreased (-0.02%) to 95.222% when pulling d06b3b453d410ce5d0732b5bfd0878888dfb7952 on cody-email-attachments into 66fdf9e1192b7ef0f51ff8517f791391c30656e5 on main.


Additional DLL information

(Imported from private todo list)
Add "Explorer file property" DLL information to DLL pages, e.g., file version.

Details (imported from Trello):
- DLL metadata
  - Version (that shows in Explorer)
  - Size
  - Date modified
  - x86/64/AnyCpu
  ? .NET-based Version
  ? AssemblyCompany
  ? AssemblyDescription
  ? TargetFramework


TransformerEngine FP8 is slower & more memory intensive than FlashAttention FP16?

I'm running some benchmarks on TransformerEngine MHA FP8 versus FlashAttention MHA FP16. However, I'm consistently getting that TE FP8 is not only slower by 50-60% than FlashAttention; it also uses much more memory (11GB vs 27GB).
I'm scratching my head because FP8 should use less memory for the same sequence length + MHA parameters. I'm using the latest TE build from source on 1xH100 and cudnn installed.  Here's the benchmarking code:
TransformerEngine:
b = 32
n = 1024 #w * h
nhead = 32
nhead_k = 8
head_dim = 128

hidden_dim = 4096
n_iters = 1000

print("BASELINE TEST, %d TOKENS" % n)

#seq = torch.randn((b, n, hidden_dim)).cuda()
seq = torch.randn((n, b, hidden_dim)).cuda()
mha = te.TransformerLayer(hidden_dim, ffn_hidden_size = hidden_dim*4, num_attention_heads = nhead, num_gqa_groups = nhead_k, hidden_dropout=0, attention_dropout=0, kv_channels=head_dim, self_attn_mask_type='no_mask', bias=True).cuda()
params = list(mha.parameters())
opt = torch.optim.AdamW(params, lr=1e-4)

fp8_recipe = recipe.DelayedScaling(margin=0, fp8_format=recipe.Format.HYBRID)#, amax_history_len=32, amax_compute_algo="max")

num_params = sum(p.numel() for p in mha.parameters())
print(f"Number of parameters: %f M" % (num_params/1e6))

start = perf_counter()
for _ in tqdm(range(n_iters)):
    opt.zero_grad()

    with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
        out = mha(seq)
        loss = out.sum()

    loss.backward()
    opt.step()

torch.cuda.synchronize()
print('perf: %f' % (1000*(perf_counter() - start)/n_iters))

FlashAttention Benchmark:
from flash_attn import flash_attn_qkvpacked_func, flash_attn_func
from flash_attn.ops.fused_dense import FusedDense
from flash_attn.modules.mha import MHA

# Baseline flash_attn test

b = 32
n = 1024
nhead = 32
nhead_k = 8
head_dim = 64

hidden_dim = 4096
n_iters = 1000

print("BASELINE TEST, %d TOKENS" % n)

seq = torch.randn((b, n, hidden_dim)).cuda().to(torch.bfloat16)
class TransformerLayer(nn.Module):
    def __init__(self, hidden_dim, nhead, nhead_k, ffn_hidden_size):
        super().__init__()
        self.mha = MHA(hidden_dim, nhead, nhead_k, use_flash_attn=True, fused_bias_fc=True, causal=False)

        self.ffn = nn.Sequential(
            FusedDense(hidden_dim, ffn_hidden_size, bias=True),
            nn.GELU(),
            FusedDense(ffn_hidden_size, hidden_dim, bias=True),
        )

    def forward(self, x):
        x = x + self.mha(x)
        x = x + self.ffn(x)
        return x

#mha = MHA(hidden_dim, nhead, nhead_k, use_flash_attn=True, fused_bias_fc=True, causal=False).cuda()
mha = TransformerLayer(hidden_dim, nhead, nhead_k, ffn_hidden_size = hidden_dim*4).cuda()

params = list(mha.parameters())
opt = torch.optim.AdamW(params, lr=1e-4)
num_params = sum(p.numel() for p in mha.parameters())
print(f"Number of parameters: %f M" % (num_params/1e6))

a = perf_counter()
for _ in tqdm(range(n_iters)):
    with torch.cuda.amp.autocast(dtype=torch.bfloat16):
        opt.zero_grad()
        out = mha(seq)
        loss = out.sum()

        loss.backward()
        opt.step()

print('flash_attn_func: %f' % (1000*(perf_counter() - a)/n_iters))

Any ideas?

I tested the scrips you posted. The reason why TE is slower in your example is due to the fact that fp8_autocast is only affecting the FP8 execution, but we do preserve the precision of the other parts of the model. In this case it is FP32, since you did not use AMP or cast the model to FP16/BF16 there. In the default recipe in TE attention is not being computed in FP8, and so without casting the model it uses the original precision (FP32). Neither flash attention nor cuDNN attention backends in TE support FP32 execution, and so what you got was the slowest, unfused attention (which additionally uses the most memory), which resulted in the poor performance you observed.
Changing your TE script to this:
import torch
import transformer_engine.pytorch as te
from transformer_engine.common import recipe
from time import perf_counter
from tqdm import tqdm

b = 32
n = 1024 #w * h
nhead = 32
nhead_k = 8
head_dim = 128

hidden_dim = 4096
n_iters = 1000

print("BASELINE TEST, %d TOKENS" % n)

#seq = torch.randn((b, n, hidden_dim)).cuda()
seq = torch.randn((n, b, hidden_dim)).cuda().bfloat16()
mha = te.TransformerLayer(hidden_dim, ffn_hidden_size = hidden_dim*4, num_attention_heads = nhead, num_gqa_groups = nhead_k, hidden_dropout=0, attention_dropout=0, kv_channels=head_dim, self_attn_mask_type='no_mask', bias=True).cuda()
mha = mha.bfloat16()
params = list(mha.parameters())
opt = torch.optim.AdamW(params, lr=1e-4)

fp8_recipe = recipe.DelayedScaling(margin=0, fp8_format=recipe.Format.HYBRID)#, amax_history_len=32, amax_compute_algo="max")

num_params = sum(p.numel() for p in mha.parameters())
print(f"Number of parameters: %f M" % (num_params/1e6))

start = perf_counter()
for _ in tqdm(range(n_iters)):
    opt.zero_grad()

    with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
        out = mha(seq)
        loss = out.sum()

    loss.backward()
    opt.step()

torch.cuda.synchronize()
print('perf: %f' % (1000*(perf_counter() - start)/n_iters))

(I added the missing imports and cast the model and input to bf16) I get following performance results on H100 PCIe (just fwd/bwd - omitting the optimizer, which gives the constant overhead for all cases):

TE (original script): 104 ms
Flash: 95 ms
TE (new script): 44.6 ms
TE (new script, fp8 disabled, so pure BF16 execution for comparison): 65.6 ms

One additional note is that in your FA script you did not include LayerNorm, but that is a small difference in this case.

Very helpful, thank you

hi @ptrendx I have a follow-up question: what is actually cast to fp8 with TE using the code above? First, we cast model to bfloat16, so model weights are 16-bit. Then, we use the fp8_autocast to convert the TE Linear layer activations to FP8. Is that correct? So the activations & gradients for the TE linear layers are in FP8, but the weights, the optimizer states are all bfloat 16?
On the other hand, if we don't directly cast to bfloat16, the model weights are in fp32 and stay that way throughout training? What happens if we use torch.cuda.amp.autocast(Bfloat16)?
I am trying to wrap my head around how the FP8 is implemented with TE

Functionality-wise, you can think of fp8_autocast as changing just the internal execution of the operators, so a functionally equivalent execution of the forward pass would be:
x = x.to(fp8).to(fp32)
weight = weight.to(fp8).to(fp32)
y = linear(x, weight).to(original input type)
y = y + bias

What is different in actual execution is that the linear layer actually takes the x and weight in fp8 and only the internal accumulator is in FP32, but the result is the same.
So in your table:

with AMP:

weights and weight gradients are FP32;
activations and data gradients are BF16;
optimizer state is FP32;
GEMM casts things to FP8 if in fp8_autocast just in time but outputs BF16/FP32

with bf16 cast:

weights and weight gradients are BF16;
activations and data gradients are BF16;
optimizer state depends on your optimizer implementation - regular optimizers would make it BF16 and you would need master weights support to keep the state in FP32;
GEMM casts things to FP8 if in fp8_autocast just in time but outputs BF16

We also have an additional context manager fp8_model_init which makes the layers actually hold FP8 parameters rather than higher precision ones. It is not the default behavior though since the user needs to make sure that they do have a high precision copy of the parameters somewhere else (or that those parameters are not actually trainable, like in inference or Lora), otherwise the convergence would suffer.


Update tf.nn.rnn.md

It is hard to read some parts of markdown with gfm newline grammar.
I think adding code block is more good for reading.

Can one of the admins verify this patch?

I signed it!

This is a generated file, please change the code comments instead

Thanks. I changed it by your comment

You have to edit https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/rnn.py#L75 -- all of the .md files are generated from the source code documentation, so this won't change anything.

Oh, I see. I misunderstood comment.
I fixed rnn.py code documentation for code block which shown as one line

The changes you made don't look like what you wanted. Can you re-check?

I comment what's my purpose in file changed tab.

Thanks your comment. Could you recheck it?

Great! Thanks!

Jenkins, test this please.


Failed to connect google spanner database

Please help, I got error below when try to connect and read from spanner database.
It is clearly stated in the error response that access token scope is insufficient, but I've already use service account JSON file that has role "google spanner admin", so it must be enough right?
I've also set environment variable to my json file with "export" command on the console server (is this right?). I suspect that this "json" is not readable (?). Please advise, thanks.
{
    "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
    "domain": "googleapis.com",
    "errorInfoMetadata": {
        "method": "google.spanner.v1.Spanner.BatchCreateSessions",
        "service": "spanner.googleapis.com"
    },
    "message": "Request had insufficient authentication scopes.",
    "code": 7,
    "status": "PERMISSION_DENIED",
    "details": [
        {
            "@type": "google.rpc.errorinfo-bin",
            "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
            "domain": "googleapis.com",
            "metadata": {
                "method": "google.spanner.v1.Spanner.BatchCreateSessions",
                "service": "spanner.googleapis.com"
            }
        },
        {
            "@type": "grpc-status-details-bin",
            "data": "<Unknown Binary Data>"
        }
    ]
}

sorry that was my bad. Got it right by setting scope in the instance level.


As a customer, I can view technical documentation on the filtering and sorting

Updated gitbook technical documentation on filtering and sorting
Must also update for searching

Story points estimation: 1

Need other's input on how we implemented filtering and sorting!

I've updated the filtering and searching documentation to reflect the backend implementation and API endpoints. Sorting is handled by the frontend and @clairebingham  and @elmdecoste are handling updating that documentation. This took me about .5 hours.