Should Maybe codec return Nothing for nulls?

If you're writing a codec for a value that cannot have a DecodeJson instance declared for it, it seems sensible to write something like this:
traverse decodeCustom =<< obj .? "prop"

To get a Maybe Custom success result. The idea being obj .? "prop" result is a Maybe Json and then decodeCustom does the Json -> Custom step. However, the decodeJsonJson instance always succeeds, as null is a decodeable value, so you end up with Just null as the response, and then decodeCustom fails as it's probably expecting something else.
If you're using instances everywhere this works fine, as the custom decoder would run "inside" the Maybe decoder, and failing there gives you a Nothing result.
Maybe it was a faulty assumption on my part that null values would be considered Nothing? It took me a really long time to track this down though. :sob:

Oh, just to point out, this was found while trying to test a encode/decode roundtrip - Nothing does get encoded as null, so that didn't help my assumption that it would be decoded as such also.

Yeah, I think null -> Nothing, Nothing -> null makes sense.


Added Feature: status

This PR caters to adding a new feature called status.
This command tells you the active processes running in the background. Also adds information related to what issues does each repository cater to.

[X] Built successfully
[X] Ran gofmt successfully.

Sample output:

@yashvardhan-kukreja is there anything additional to be done?


Linktree clone

Description 📜
Project title
Linktree clone
Project Description
The idea behind this project
We all have created lInktree as per collecting all social links at the same place so this is just a copy of this project.
How does it work
I have tried to copy every part of Linktree of the website with the help of HTML and CSS
Fixes #1538

Type of change 📝
Please delete options that are not relevant. And in order to tick the check box just put x inside them for example [x] like this

[x] Add a new project

Domain of Contribution 📊
Please delete options that are not relevant. And in order to tick the check box just put x inside them for example [x] like this

[x] Project using HTML, CSS & JS

Checklist ✅
Please delete options that are not relevant. And in order to tick the check box just put x inside them for example [x] like this

[x] I have performed a self-review of my own code or work.
[x] I have commented on my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GSSoC'22 contributor

## Screenshots / Gif (Recommended) 📸
https://user-images.githubusercontent.com/73521123/167454608-b431bd49-da12-4b25-bbc4-51c7b36b7e92.mp4
https://user-images.githubusercontent.com/73521123/167454060-7dc478e9-5d6a-4e37-b853-94b07d1b94bf.mp4

No JS is Used.


No module named 'transformers' on initial Arm MacOS setup

Describe your environment
MacOS MBP 2021
ZSH
Anaconda
Python (latest version)
Just ran through the Mac Arm installation instructions and cannot get past this error.
(ldm) ➜  stable-diffusion git:(main) python scripts/preload_models.py
Traceback (most recent call last):
  File "/Users/comp/dev/stable-diffusion/scripts/preload_models.py", line 6, in <module>
    from transformers import CLIPTokenizer, CLIPTextModel
ModuleNotFoundError: No module named 'transformers'

Has anyone experienced this or know what went wrong?

As I remember, I just install them from conda.

@netsvetaev  thanks that helped. I've had the same issue:
RuntimeError: Failed to import transformers.models.clip.modeling_clip because of the following error (look up to see its traceback):
dlopen(/Users/$USER/miniconda3/envs/ldm/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_)

solved it by installing transformers manually:
conda install transformers

However, I switched to pytorch nightly before, don't know if that makes a difference:
conda install pytorch torchvision torchaudio -c pytorch-nightly

Hm, I checked out the develop branch recreated the conda env, but I still get the No module named 'transformers' error.

That's strange, because about 3-4 hours ago I completley removed my conda env and the anaconda virtual environment from pyenv and reinstalled it from the ground, had no problems 😮
Did you uninstall protobuf via brew uninstall protobuf?

I just retried it and made sure the brew protobuf is also uninstalled. Still get the same error.  Sorry to not be of more help!

Have you tried to clean the Repository
git clean -f
conda clean \
  --yes \
  --all

and to reset the conda base env
conda update \
  --force-reinstall \
  -y \
  -n base \
  -c defaults conda

I ran those commands, but they didn't help.

is rust installed via brew?
can you do a pip list | grep "protobuf" and a conda list -n ldm | grep "transformers"
my results:
pip list | grep "protobuf"
protobuf                                  3.20.1

conda list -n ldm | grep "transformers"
taming-transformers       0.0.1                     dev_0    <develop>
transformers              4.21.2                   pypi_0    pypi

Yes, I have Rust via brew. pip list | grep "protobuf" seems to be much higher than your version, not sure if that is a problem:
protobuf                      4.21.6

conda list -n ldm | grep "transformers" gets me this:
CondaError: Expected exactly one `egg-info` directory in '/Users/wukerplank/Desktop/stable-diffusion', via egg-link 'lib/python3.10/site-packages/stable-diffusion.egg-link'. Instead found: ('stable_diffusion.egg-info', 'latent_diffusion.egg-info').  These are often left over from legacy operations that did not clean up correctly.  Please remove all but one of these.

I guess at this point I've unsuccessfully followed so many different guides that my environment is totally messed up 😭

remove latent_diffusion.egg-info (I also only have stable_diffusion.egg-info)

Now, with conda list -n ldm | grep "transformers" I get:
taming-transformers       0.0.1                     dev_0    <develop>
transformers              4.21.2                   pypi_0    pypi

Alright, thank you! I'll take another stab at following the guide. 🤞

I also installed stable diffusion yesterday from scratch following the guide. After installing you will have to close and reopen the shell, otherwise conda will fail. The conda installation mentions that, but the guide not ... and first time I overlooked the message. I also switched to pytorch nightly, before figuring out that transformers was missing. Afterwards I got a protobuf error. And overcame it by conda update protobuf. I don't know if that causes any side effects, but I could successfully use SD

can you do a pip list | grep "protobuf" and a conda list -n ldm | grep "transformers"
my results:
pip list | grep "protobuf"
protobuf                                  3.20.1

as you can see protobuf now installs 3.20+ version and there is related github issue on protobuf github regarding that version: https://github.com/protocolbuffers/protobuf/issues/10571
solution seems as simple as downgrading to some other version, and for my case I just downgraded protobuf to 3.19.4 version by running this command:
pip install protobuf==3.19.4
and after that completes, I continued by running this command again:
python scripts/preload_models.py
and it completed successfully and the issue seems solved for me.

is protobuf installed via brew as well? @burakkurkcu
conda list protobuf -n ldm
# packages in environment at /Users/mauwii/.pyenv/versions/anaconda3-2022.05/envs/ldm:
#
# Name                    Version                   Build  Channel
libprotobuf               3.20.1               hb5ab8b9_4    conda-forge
protobuf                  3.20.1          py310hb07a4bc_0    conda-forge

pip list | grep "protobuf"
protobuf                                  3.20.1

I don't have protobuf installed via brew (brew uninstall protobuf) and it is working pretty fast with version 3.20.1 👨🏻💻

I ran into this same issue; conda install transformers didn't solve the issue.  I did not try uninstalling the brew version (since the install instructions for macOS install protobuf via brew), but downgrading protobuf as suggested by burakkurkcu worked.

I just ran into this but realized I had rebooted and had not activated my conda environment with conda activate ldm

Not sure if this helps anyone, but in the instructions it says run python ./scripts/preload_models.py which didn't work for me, but running ./scripts/preload_models.py worked fine

Having this issue on Ubuntu 22.04 as well.

for me what fixed the issue was pip install protobuf==3.19.4 to downgrade

This is a bit jumbled for an issue and it looks like most issues were resolved on macOS. Closing; if you still have a problem like this, please create a new issue for focused troubleshooting.


add appveyor setting and cmake files

Fix #16

成果物

https://ci.appveyor.com/project/y-yu/article-template/build/1.0.24/artifacts


Update version to 1.1.13-4 on staging

Update version to 1.1.13-4

🚀 Deployed to staging by @OSBotify in version: 1.1.13-4 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by @Jag96 in version: 1.1.14-4 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Create a method to delete or invalidate the token

To delete or invalidate toke like
The method name is invalid or deleted.
const data = { test: 1 };
const token = jwt.sign(data, 'shhhh', { expiresIn: 100 });
const invalidate = jwt.invalidate(token, privateKey);
const old_token = jwt.verify(token, 'shhhh');

// { test: 1 }
console.log(data);
// { error: "Invalid/deleted token, create a new" }
console.log(invalidate);

or something like this.

Most of developers doesn't understand that there is no way to invalidate the token in a stateless system. To do that you must introduce a... state. For example you can implement white- or blacklisting of tokens and check every request against this list. Or introduce a session. But well, this is never a part of an jwt library like this one. Look for example at Single Sign Out.

https://github.com/auth0/node-jsonwebtoken/issues/626#issuecomment-519813568

Tks for replying to me about this issue.
I didn't know jwt and didn't know its features completely.


screenshots

Hello! Can u tell me what's plugins u used in the last screen? Left seems like default netrw, but right side?

@EternalWinter Sure!
Right side: Tagbar.
Left side is actually NerdTree with some custom parameters ;)
let NERDTreeRespectWildIgnore=1
let NERDTreeAutoDeleteBuffer=1
let NERDTreeMinimalUI=1
let NERDTreeShowHidden=1
let g:NERDTreeHighlightFolders=0
let g:NERDTreeHighlightFoldersFullName=0
let g:NERDTreeExactMatchHighlightFullName=0
let g:NERDTreeFileExtensionHighlightFullName=0
let g:NERDTreePatternMatchHighlightFullName=0

@EternalWinter Sure!
Right side: Tagbar.
Left side is actually NerdTree with some custom parameters ;)
let NERDTreeRespectWildIgnore=1
let NERDTreeAutoDeleteBuffer=1
let NERDTreeMinimalUI=1
let NERDTreeShowHidden=1
let g:NERDTreeHighlightFolders=0
let g:NERDTreeHighlightFoldersFullName=0
let g:NERDTreeExactMatchHighlightFullName=0
let g:NERDTreeFileExtensionHighlightFullName=0
let g:NERDTreePatternMatchHighlightFullName=0

ty so much!


Support for other services

kube-rclone has only been tested with Plex for playing media so far. As the project develops, it would be nice to support other services like Sonarr and Radarr for example.

Is there any reason why this wouldn't work with services like Sonarr/Radarr?

Is there any reason why this wouldn't work with services like Sonarr/Radarr?

I think I would have some concerns around the stability for mounting volumes for sonarr and radarr if rclone failed. Saying that, I’ve not officially tested it yet but my assumption would it will work but be aware of any stability issues


Clarification in Tutorial: Arguments and Accounts

had a bit of trouble with the arguments and accounts example javascript code:
https://project-serum.github.io/anchor/tutorials/tutorial-1.html#update-instruction.
Specifically the provider and SystemProgram variables kinda come out of no where. Here is the source I ended up with for this sample.
// test.js
const anchor = require('@project-serum/anchor');

describe('mnemonic_sol', () => {

  // Configure the client to use the local cluster.
  anchor.setProvider(anchor.Provider.env());

  it('initializes with an account', async () => {
    const program = anchor.workspace. Basic1;
    const myAccount = anchor.web3.Keypair.generate();
    await program.rpc.initialize(new anchor.BN(1234), {
      accounts: {
        myAccount: myAccount.publicKey,
        user: anchor.getProvider().wallet.publicKey,
        systemProgram: anchor.web3.SystemProgram.programId
      }
    })
  })
});

Apologies for not opening a PR. I am short on time, but I wanted to post my code suggestion.

Nevermind, found the source.
I'd recommend linking to the source in the docs.


Block subsidy

Part of https://github.com/ZcashFoundation/zebra/issues/338

Thanks @yaahc , that was great, specially the combinator suggestions that i have big troubles to implement on my own :) I will take care of all the changes.

Hi @oxarbitrage, just wondering if you'd like a detailed review here?
It seems like the Canopy Funding Streams implementation isn't finished yet - do you want to focus on the Founders Reward in this PR, and do Funding Streams in a separate PR?
I also noticed a few things that could improve the code:

replace activation height constants with the NetworkUpgrade::activation_height function, so the code works on mainnet and testnet
use Amount<NonNegative>, rather than u32 or f32, so that all the calculations are accurate (and don't overflow)

let me know if you need help converting from f32 to Amount

write a short doc comment for each function and constant, explaining what it does, and how it is used

for most functions, you can just write one sentence that explains what it does with each argument

Thanks @teor2345 . Yea, we can do funding streams in a separated PR so we can merge faster. Feel free to make a detailed review, i will try to make the changes tomorrow. In addition to what you mentioned this PR needs a rebase now that https://github.com/ZcashFoundation/zebra/pull/1057 is merged. I removed the draft state.

@teor2345 i was thinking on adding the formulas in latex from https://zips.z.cash/protocol/protocol.pdf#subsidies into the comments of the subsidy functions. let me know what do you think.
In order to do this i was expecting https://github.com/ZcashFoundation/zebra/pull/832 to work but in some small test i made it seems the output documents are not rendering the latex content.

In regards to the use of Amount<NonNegative> in constants and function return values i see some problems that i will like to be clarified before implementing. In addition to Add and Sub there are several functions or part of them that will need Div and Mul. To avoid conversions back and forth one option can be to add this operators to the Amount type. For the functions that return Amount<NonNegative>(and maybe all functions) we can wrap in Result<Amount<NonNegative>, Err> so we can have ? at hand.
Let me know what do you think about that @teor2345 as i maybe have something missing. Thanks.

In regards to the use of Amount<NonNegative> in constants and function return values i see some problems that i will like to be clarified before implementing. In addition to Add and Sub there are several functions or part of them that will need Div and Mul. To avoid conversions back and forth one option can be to add this operators to the Amount type. For the functions that return Amount<NonNegative>(and maybe all functions) we can wrap in Result<Amount<NonNegative>, Err> so we can have ? at hand.
Let me know what do you think about that @teor2345 as i maybe have something missing. Thanks.

Adding Div/Mul and returning Result is the right way to go imo.

In regards to the use of Amount<NonNegative> in constants and function return values i see some problems that i will like to be clarified before implementing. In addition to Add and Sub there are several functions or part of them that will need Div and Mul. To avoid conversions back and forth one option can be to add this operators to the Amount type. For the functions that return Amount<NonNegative>(and maybe all functions) we can wrap in Result<Amount<NonNegative>, Err> so we can have ? at hand.

Adding Div/Mul and returning Result is the right way to go imo.

How should we handle inexact divisions?
(That is, a division that has a remainder.)
I'd like to make them errors, so that we explicitly handle remainders in every calculation.
We can use expect for calculations that will never have a remainder, and match otherwise.
Here's the list of operators I think we'll want for amount:

Add<Amount, Rhs=Amount, Output=Result<Amount>>
Sub<Amount, Rhs=Amount, Output=Result<Amount>>
Mul<Amount, Rhs=u64, Output=Result<Amount>>
Mul<u64, Rhs=Amount, Output=Result<Amount>>

Implement using amount * self

Div<Amount, Rhs=u64, Output=Result<Amount, DivisionError>>

with enum DivisionError { DivideByZeroError, InexactDivisionError(remainder: Amount) }
overflow and underflow can't happen when dividing by a positive integer

We should also implement the corresponding ...Assign traits, using the non-...Assign trait.
Add, Sub and Mul can have these errors:

overflow: result greater than MAX_MONEY
underflow: result less than -MAX_MONEY or 0, depending on the Amount type

Things we might want to implement - but let's wait and see if we need them:

Rem<Amount, Rhs=u64, Output=Result<Amount>>

Implement using self - ((self / rhs) * rhs)

Neg<Amount, Output=Result<Amount>>

Implement using Amount(0) - self

Things we don't want to implement, because they are probably coding errors:

multiplying two amounts
multiplying an amount by a negative integer
dividing two amounts (to get an integer)
dividing an amount by a negative integer

The rules table I originally posted was incomplete and incorrect. Here's my second attempt:
Here is my understanding of the block subsidy rules from the spec - rules that we don't need to handle are in ~strikethrough~:

Height
Slow Start Multiplier
Target Spacing
Distribution Rules

~Genesis~
~(no coinbase)~
~(none)~
~(none)~

~1..Slow Start Shift~
~height~
~150 seconds~
~Founders Reward~

~Slow Start Shift..Slow Start Interval~
~height + 1~
~150 seconds~
~Founders Reward~

~Slow Start Interval~ Sapling..Blossom

150 seconds
Founders Reward

Blossom..Canopy

75 seconds
Founders Reward

Canopy..Second Halving

75 seconds
Funding Streams

Second Halving..

75 seconds
Only Miners

This rules summary table can go in the module doc comment of the block subsidy module.
In this PR, we'll implement the rules from:

Sapling
Blossom
Heartwood

Here's how we'll handle the rules from the other network upgrades:

the Genesis, BeforeOverwinter, and Overwinter blocks are verified by the CheckpointVerifier, so we can:

[ ] delete any code for those network upgrades
[ ] panic! if we are asked to verify a block from those network upgrades

[ ] delete the post-Canopy code from this PR, and move it to another PR

I think it would be helpful to have functions for each different set of rules, and group those functions into modules. Then the top-level subsidy module can select the appropriate rules using a match statement.
For example, we could have modules for:

[ ] general subsidy: maximum subsidy, halvings
[ ] ~slow start subsidy modifiers (before and after shift)~ - not required, delete this code
[ ] target spacing subsidy modifiers
[ ] founders reward distribution
[ ] funding streams distribution - delete code from this PR, and move to next PR

Hey @oxarbitrage  let me know when you're ready for another review.
Can I do the review before you rebase on the latest main branch?
It will be easier to see the new changes.

@oxarbitrage now that we have a stable design for Mul and Div, do you also want to implement:

MulAssign<u64> for Amount<NonNegative>
DivAssign<u64> for Amount<NonNegative>

I think we are going to be comparing amounts more sooner than later so added the operators to do it at https://github.com/ZcashFoundation/zebra/pull/1051/commits/7b0f1664b5aac0aa018f2627e3999a0f005b19f5

@oxarbitrage now that we have a stable design for Mul and Div, do you also want to implement:
MulAssign for Amount
DivAssign for Amount

I implemented in https://github.com/ZcashFoundation/zebra/pull/1051/commits/453ad1f31400cf6425273b2c848cdc6e657fb14b however having some issues to make them safe. I cant return a Result in those operators so i am silently returning the Amount unmodified if overflow or division by zero. Probably some workaround is needed but i was not able to find any that convince me yet.

@oxarbitrage now that we have a stable design for Mul and Div, do you also want to implement:
MulAssign for Amount
DivAssign for Amount

I implemented in 453ad1f however having some issues to make them safe. I cant return a Result in those operators so i am silently returning the Amount unmodified if overflow or division by zero. Probably some workaround is needed but i was not able to find any that convince me yet.

Try copying the implementations from:
https://github.com/ZcashFoundation/zebra/blob/main/zebra-chain/src/amount.rs#L76
And changing the types.

Try copying the implementations from:
https://github.com/ZcashFoundation/zebra/blob/main/zebra-chain/src/amount.rs#L76

And changing the types.

I tried this already however abandoned because of error:
216 | impl std::ops::MulAssign<u64> for Result<Amount<NonNegative>> {
    | ^^^^^------------------------^^^^^---------------------------
    | |    |                            |
    | |    |                            `std::result::Result` is not defined in the current crate
    | |    `u64` is not defined in the current crate
    | impl doesn't use only types from inside the current crate
    |

So it seems we cant do it with u64 there. We can change the operator signature.

Try copying the implementations from:
https://github.com/ZcashFoundation/zebra/blob/main/zebra-chain/src/amount.rs#L76

And changing the types.

I tried this already however abandoned because of error:
216 | impl std::ops::MulAssign<u64> for Result<Amount<NonNegative>> {
    | ^^^^^------------------------^^^^^---------------------------
    | |    |                            |
    | |    |                            `std::result::Result` is not defined in the current crate
    | |    `u64` is not defined in the current crate
    | impl doesn't use only types from inside the current crate
    |

So it seems we cant do it with u64 there. We can change the operator signature.

I think we should just delete these implementation.
They aren't that important. We can always use let a = (a * b)?; as a workaround.
Let's leave a comment explaining that we can't implement them, due to trait coherence issues. (The warning you are seeing is due to a trait coherence rule. You can't implement a foreign trait on a foreign type, because traits are global - so they might clash.)

I think we're almost done on this PR.
What do you think we need to do before we merge, @oxarbitrage ?
I'd like to make sure we've tested this code on all the block test vectors. Are there any other tests you think we should do?
Here's my draft TODO list:

[ ] Split zebra_test::vectors::TEST_BLOCKS into two BTreeMap<u32, Vec<u8>>s, one  for Mainnet and one for Testnet
[ ] Add a test for Sapling to Heartwood Mainnet blocks
[ ] Add a test for Sapling to Heartwood Testnet blocks

Here are some open questions:

[ ] Do we want to make Ord for Amount generic over NonNegative and NegativeAllowed?

I implemented in 453ad1f however having some issues to make them safe. I cant return a Result in those operators so i am silently returning the Amount unmodified if overflow or division by zero. Probably some workaround is needed but i was not able to find any that convince me yet.

I would suggest just panicking in those cases.

Here what i think we can do:

For the tests we need to rebase and pass all the new vectors from https://github.com/ZcashFoundation/zebra/pull/1096 through https://github.com/oxarbitrage/zebra/blob/block_subsidy/zebra-consensus/src/block/subsidy.rs#L213 We can do this in this PR or in a new one but the rebase is needed to fix conflicts with current main.
The Ord for Amount, etc for negatives was added at https://github.com/ZcashFoundation/zebra/pull/1051/commits/be5645b8981433c7b35486d79c8dc6c96a8bf633
In regards to founders reward addresses lets do it in a new PR as discussed.
In regards to structure i think i don't follow the idea suggested. I think we can discuss it more by chat or you can send some code to my branch(https://github.com/oxarbitrage/zebra/tree/block_subsidy) as a PR.

In other issues, we have a dead code warning currently by not using miner_subsidy() anywhere. I had plans to add a validation check for it with what we have now, something as: sum_of_outputs_in_coinbase >= miner_subsidy + funders_reward however the protocol dont specify this so unsure if it is always true.

Here what i think we can do:
* For the tests we need to rebase and pass all the new vectors from #1096 through https://github.com/oxarbitrage/zebra/blob/block_subsidy/zebra-consensus/src/block/subsidy.rs#L213 We can do this in this PR or in a new one but the rebase is needed to fix conflicts with current main.

I'd like to do tests in this PR - I can do the zebra_test refactor today.
(We try to have good tests with every PR, so we don't merge bugs.)
Feel free to rebase whenever you need to, I think we've covered all of the outstanding comments in the list here:
https://github.com/ZcashFoundation/zebra/pull/1051#issuecomment-699941294

* The `Ord for Amount`, etc for negatives was added at [be5645b](https://github.com/ZcashFoundation/zebra/commit/be5645b8981433c7b35486d79c8dc6c96a8bf633)

👍

* In regards to founders reward addresses lets do it in a new PR as discussed.

👍

* In regards to structure i think i don't follow the idea suggested. I think we can discuss it more by chat or you can send some code to my branch(https://github.com/oxarbitrage/zebra/tree/block_subsidy) as a PR.

Yeah I'll also ask some of the other developers for advice.
It's reasonably easy to split code into modules later, so we might not need to do it now.

In other issues, we have a dead code warning currently by not using miner_subsidy() anywhere. I had plans to add a validation check for it with what we have now, something as: sum_of_outputs_in_coinbase >= miner_subsidy + funders_reward however the protocol dont specify this so unsure if it is always true.

The protocol specifies:

Like Bitcoin, Zcash creates currency when blocks are mined. The value created on mining a block is called the block subsidy . It is composed of a miner subsidy and a Founders’ Reward . As in Bitcoin, the miner of a block also receives transaction fees.
...
The first (and only the first) transaction in a block is a coinbase transaction, which collects and spends any miner subsidy and transaction fees paid by transactions included in this block.
-- 3.9 Block Subsidy and Founders’ Reward, 3.10 Coinbase Transactions

So the consensus rules from Bitcoin are:

the transaction fees are the sum of each transactions inputs, minus its outputs, for all non-coinbase transactions in a block
the sum of the coinbase transaction outputs must be less than or equal to the block subsidy plus transaction fees (because miners can destroy coins)

So the rule is actually:
sum_of_outputs_in_coinbase <= miner_subsidy + founders_reward + transaction fees
Since we aren't calculating the transaction fees yet, we can't do any useful tests on miner_subsidy yet.
You can mark it as #[allow(dead_code] - we can remove that in the next PR.

I think we just have these two things left:

add tests that use MAINNET_BLOCKS and TESTNET_BLOCKS
format errors as "error kind: conditions that caused the error"

I'll go through all the error messages in the PR today, and make suggestions.

@teor2345 your changes are all great so i brought them here. I think is ready for someone else to take a look

Ok, I'll ping the team for a review, and also add shielded coinbase test vectors to the tests.
I think the code might fail on shielded coinbase - if it does, we will need to panic on heartwood blocks. (Or shielded coinbase in heartwood blocks.) And we'll need to make shielded coinbase our next priority.

Hey @oxarbitrage, I just realised that we're missing failure tests. Let's write a test for each error:

[ ] NoCoinbase
[ ] FoundersRewardNotFound

It should be easy to make these tests using generate, or by deleting transactions/outputs from valid blocks. Can you do that today?
It would also be good to make sure that we allow duplicate outputs, as long as at least one output passes - I'll put that on the tracking issue.
I also:

did some fixes for @yaahc's review in https://github.com/oxarbitrage/zebra/pull/30/
added shielded coinbase blocks in https://github.com/ZcashFoundation/zebra/pull/1116
tested that this PR works with the shielded coinbase blocks

Oops, I think we missed this PR - we still need a review from someone who isn't me or @oxarbitrage.

The last fix looks good, i think i already checked the rest. We should merge this one and IMHO, continue with the block subsidy design(https://github.com/ZcashFoundation/zebra/pull/1129) and implement the rest of it in separated PRs.


test discord hooks

Is your feature request related to a problem? Please describe.
nope
Describe the solution you'd like
discord hooks to work
Describe alternatives you've considered
IRC lol
Additional context
testing testing

quiet you


Top nav not saving changes

Category

[x] Question
[x] Bug
[ ] Enhancement

Expected or Desired Behavior
The nav menu at the top of the page should save changes
Observed Behavior
When editing the nav menu it does not save any changes
Steps to Reproduce
Click edit, make a change, save, refresh the page

Is there any specific documentation for this nav menu?

@iainl the top navigation you are referring to is standard OOTB hub navigation. The edit feature is a part of SharePoint Hub sites and not specific to the starter kit. I have found the edit feature for hub navigation can take a few minutes to stick sometimes, but that is unusual.
https://support.office.com/en-us/article/customize-the-navigation-on-your-sharepoint-site-3cd61ae7-a9ed-4e1e-bf6d-4655f0bf25ca#hubnav

Ahh ok, Thanks. Do you happen to know how to enable this on other modern pages?

That is a hub site navigation. If you make some site collection as a hub site, it will have this hub navigation for you to edit and modify.

@iainl the hub navigation (top nav) you are referring to should appear on all content pages within a site that is joined to a hub. You can manually join other sites to a hub and those new sites should inherit the hub navigation.


Fix ROI Table choice being reset on every run

Currently, clicking run resets the ROI table choice. I added a workaround in the run function to set it back to the value before, but I should figure out what triggers this reset. The _get_roi_table_choices doesn't get called, but still it somehow resets the choices to the default [""].

This also has the annoying side-effect that all other choices are reset on run. Currently also added a workaround to set them back to their selected value, but should get rid of this after bug fix here

Tracked this down a bit further: It's the adding of channels that triggers this weird behavior. Whenever I add or remove a channel, the choices of ROI Table are reset to [""]. I don't understand what triggers this.

It seems to come down to this issue here: https://github.com/pyapp-kit/magicgui/issues/306

Closed by using _default_choices and not initializing the choices when creating the dropdowns anymore. Not optimal, but seems to work reliably now


What license is the code provided under?

Hi Amod,
Great work with this tool! It looks really useful - thank you for sharing it.
Are you able to provide a license for the code?
Thanks

Hi Ben,
Thank you! Thank you for pointing out regarding the license. Just added the MIT license.


Possible transient IT failures with 2.0.0-alpha-2-rc1

Results summary.
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ScanIdIT.testScanId:184 Expected at least 8 scanIds, but saw 0
[ERROR]   SessionBlockVerifyIT.run:169 Must have ten sessions. Failure indicates a synchronization block within the sweep mechanism expected:<10> but was:<1>
[ERROR] Errors:
[ERROR]   ScanFlushWithTimeIT.test:80 » IllegalArgument Scanner is closed
[ERROR]   VerifySerialRecoveryIT.testSerializedRecovery:101 » FileNotFound /home/etcolem...
[ERROR]   ConcurrencyIT.run:102->runTest:147  Thread 1 did not see 50, saw 0
[ERROR]   DeleteTableDuringSplitIT.test:104 » Execution java.lang.RuntimeException: Dele...
[ERROR]   GarbageCollectorIT.gcLotsOfCandidatesIT:159 » StringIndexOutOfBounds String in...
[ERROR] org.apache.accumulo.test.functional.GarbageCollectorIT.testInvalidDelete(org.apache.accumulo.test.functional.GarbageCollectorIT)
[ERROR]   Run 1: GarbageCollectorIT.testInvalidDelete:229 » TestTimedOut test timed out after 3...
[ERROR]   Run 2: GarbageCollectorIT.testInvalidDelete »  Appears to be stuck in thread Time-lim...
[INFO]
[ERROR]   SslIT.concurrency:57 »  Thread 1 did not see 50, saw 0
[ERROR]   SslWithClientAuthIT.concurrency:56->SslIT.concurrency:57 »  Thread 1 did not s...
[ERROR]   CyclicReplicationIT.dataIsNotOverReplicated:330 » TestTimedOut test timed out ...
run1.txt

I was able to reproduce ScanIdIT, and am issuing a fix now. The rest are either already fixed or perhaps failed due to host resource constraints, but I wasn't able to reproduce.


Improve docs for AMP

Description
Improve docs for AMP. Add docs on how to override whitelist .
How to cast params to FP16.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@mxnet-label-bot add [doc, pr-awaiting-review]

Number of commits seems wrong - you should probably rebase this PR before merging. Otherwise, LGTM.


Age and sex missing data

It seems that about more than 80% of age and sex data is missing in the raw data. Is there any possibility to get it in your database. Also, when the raw data with all demographic information post 26 th April 2020 will be available in your database.

The data is released by state govts. So there's nothing much to do on that front. The demographics work is going on as it needs to adapt to new data format post 26th April.


Interservice calls do not correctly set request headers

When making interservice calls no headers are set - i.e content-type as well.
Because of which when request reaches a gofr server without content-type body is not binded.

@aryan-mehrotra-zs Can you please assign this issue to me

@ssshekhu53 assigned the issue to you.


[Hub Generated] Review request for Microsoft.Authorization to add version preview/2021-04-01-preview

This is a PR generated at OpenAPI Hub. You can view your work branch via this link.
Changelog
Please ensure to add changelog with this PR by answering the following questions.

What's the purpose of the update?

[ ] new service onboarding
[x] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When you are targeting to deploy new service/feature to public regions? Please provide date, or month to public if date is not available yet.
When you expect to publish swagger? Please provide date, or month to public if date is not available yet.
If it's an update to existing version,  please select SDKs of specific language and CLIs that require refresh after swagger is published.

[ ] SDK of .NET (need service team to ensure code readiness)
[ ] SDK of Python
[ ] SDK of Java
[ ] SDK of Js
[ ] SDK of Go
[ ] PowerShell
[ ] CLI
[ ] Terraform
[x] No, no need to refresh for updates in this PR

Contribution checklist:

[x] I commit to follow the Breaking Change Policy of "no breaking changes"
[x] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

[x] Ensure to check this box if one of the following scenarios meet updates in the PR, so that label “WaitForARMFeedback” will be added automatically to involve ARM API Review. Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs, all “removals” and “adding a new property” no more require ARM API review.

Adding new API(s)
Adding a new API version
[ ] Ensure to copy the existing version into new directory structure for first commit (including refactoring) and then push new changes including version updates in separate commits. This is required to review the changes efficiently.
Adding a new service

[ ] Please ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If there are following updates in the PR, ensure to request an approval from Breaking Change Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in stable version
[ ] Removing properties in stable version
[ ] Removing API version(s) in stable version
[ ] Updating API in stable or public preview version with Breaking Change Validation errors
[ ] Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
Please follow the link to find more details on PR review process.

@rounakbhansali
Calls with pagination does not look fundamentally different from regular role assignment calls. Please integrate your change to the existing roleAssignmentCalls swagger or provide a reason as to why it should be a new file.
You can copy-paste 2020-10-01-preview/roleAssignmentCalls to use as a base.

Please go through the contribution checklist added as part of the review and mark them completed. It will help you fix most of the issues and improve the quality of swagger and help close the PR faster. This is a requirement before review can be continued.


Getting ETIMEDOUT errors, does anyone experience these errors?

Hello,
First of all I want to congratulate you @chrisleekr for your work. I really like your bot and started using it for 2 weeks now.
I am using 5 pairs with the cron set up every second (default). So each second it makes 3 api calls to binance, get account info, get open orders and get candles. I am experiencing ETIMEDOUT errors each 2-3 minutes after a while. Sometimes I get the errors pretty fast, after a few minutes, sometimes after 30 minutes. Once the errors appear they kind of repeat every 2-3 minutes.
All api's give the ETIMEDOUT so it is not related to one api.
I know that binance could maybe experience issues from time to time but last night I left the bot running and got the errors every 2-3 minutes for 8 hours straight.
I have very good internet connection, fiber optics 500Mbps.
If I add more pairs it seems the errors start to appear faster to the point where the interface does not update for whole minutes.
I also increased the cron time to */5 (each 5 secs) and it was better but I still get the errors after a longer time.
I ran the bot on windows and also on mac and it is the same.
My thoughts are that somehow binance blocks the bot?
My question is if anyone experienced these errors or is it related to my setup?
I am pasting one example below.
request to https://api.binance.com/api/v3/klines?interval=30m&symbol=ADAUSDT&limit=30 failed,
reason: connect ETIMEDOUT <IP_ADDRESS>:443```Stack:```FetchError: request to https://api.binance.com/api/v3/klines?interval=30m&symbol=ADAUSDT&limit=30 failed,
reason: connect ETIMEDOUT <IP_ADDRESS>:443
    at ClientRequest.<anonymous> (/srv/node_modules/node-fetch/index.js:133:11)
    at ClientRequest.emit (events.js:315:20)
    at TLSSocket.socketErrorListener (_http_client.js:469:9)
    at TLSSocket.emit (events.js:315:20)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)

Sorry for jumping in here, but I have some questions and comments from my usage experience.

Where do you get crontab? I have a docker deploy on Linux and never saw there's a configurable crontab.

How do you manage to see those errors? There are logs? How do you access them?

I am no expert but Binance have a limit of 1200 API requests per minute, you can see a mention of it in #52 - so even doing 3 request per second you should be good to go.
Maybe it's a issue with the tunnel, have you trying disabling it? Please see #32

Hei @chopeta, yes, saw #32 and #52

There is no crontab, there is a nodejs module that simulates a cron. You need to change this:
https://github.com/chrisleekr/binance-trading-bot/blob/4de7391940064f9c987c076dc9181483ac3cdef7/config/default.json#L41
and also change in Mongo db -> collection trailing-trade-common -> field cronTime
and restart

The errors are also in logs and on slack.
 docker logs binance-bot | grep ETIMEDOUT

I don't think it is related to the localtunnel, the errors come from calls to binance api. The localtunnel works ok.
Also, I don't think the bot reaches the limit because it just gives a timeout after 2-3 minutes of working ok. If it would reach the limit it would stop working. I think it may be flagged by the ML algorithm but I can't be sure, it may be from my setup. This is why I am asking if someone else encountered this.
Info on Binance limits:
https://www.binance.com/en-AU/support/faq/360004492232

@traian13
Hm, doesn't sound like the bot issue. It must be some connection issue between you and Binance API.
Where are you located?
Are you able to run in a cloud server such as DigitalOcean?

OK, still a lot to learn here... I didn't have Slack notifs enabled... now I see ETIMEDOUT errors as well inside my Slack channel. Only 3 errors so far, so in theory nothing to be worried about. These are the errors:

request to https://api.binance.com/api/v3/account?
request to https://api.binance.com/api/v3/openOrders?
request to https://api.binance.com/api/v3/account?

From the logs I can't see anything (too many info in there), I guess I have to use bunyan for that.
I also had issues when accessing the bot remotely (over a VPN) using a internet connection with a very high latency... other than that it's working good.
@chrisleekr I am curious about the Slack integration, is there a list of msg types that are defined to be sent to Slack? What else can I expect to see inside Slack?

@chrisleekr Thanks for the reply.
I am located in Buchares/Romania.
I didn't run the bot in cloud yet because I am still testing it. But I have to try in another environment.

@chopeta
If you enable Slack, you will get notifications such as buy/sell order like below:

@traian13 Yeah, please try with another environment (or location).
Timeout is happening when you didn't get the response from API within a specific time.

@traian13 if you have no more issue, please close the issue.

Sorry, I didn't have time to test in another environment.
I'll close the issue and let you know if I have updates.

Sorry, I didn't have time to test in another environment.
I'll close the issue and let you know if I have updates.

Did you find a solution?
I have the same issue...


python3Packages.pyls-black: 0.4.6 -> 0.4.7

Remove patch included in upstream release.

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 125759 at efda027a run on x86_64-linux 1

2 packages built successfully:

python38Packages.pyls-black
spyder (python38Packages.spyder)

1 suggestion:

warning: python-explicit-check-phase
Consider using pytestCheckHook in checkInputs rather than checkPhase = "pytest";.
Near pkgs/development/python-modules/pyls-black/default.nix:18:3:
   |
18 |   checkPhase = ''
   |   ^


Add hsaputra for Henry as entry in the master pom, as well as adding for Albert and Poorna.

Create PR to test my karma for Apache Twill repo.

LGTM

+1

Thanks guys, will merge this ...


fix: rename dnsExternalAlias

:tada: This PR is included in version 1.0.24 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Latency for pruned transducer stateless5 streaming models

We are trying out the pruned transducer stateless5 streaming models and we're experiencing unstable/increasing latency in streaming when sending 0.1s chunks of audio.
What is the expected latency for these types of models?

Could you please be a bit more specific about what the issue is.  E.g. how many clients, how does the latency increase?

1 client sending 0.1s chunks of 48k 16-bit int audio. Server is run on a single Tesla K80. This is the model used: https://huggingface.co/luomingshuang/icefall_asr_wenetspeech_pruned_transducer_stateless5_streaming
Parameters used:
  --max-batch-size 50
  --max-wait-ms 5
  --nn-pool-size 4

Case 1: Latency increases towards the end of the audio. In the beginning the latency is about 3s, and as we keep streaming audio to the server (~10s long speech), the latency between the recognised words and our speech perceivably increases.

Case 2: Sometimes the latency is ~3s before seeing the first recognized word, and sometimes the latency is ~10s

Only modification to the server code is changing this line
https://github.com/k2-fsa/sherpa/blob/eef89876fa1aa5820f2513ef292a2dcde1284252/sherpa/bin/streaming_pruned_transducer_statelessX/streaming_server.py#L509
into this:
in_array = np.frombuffer(message, dtype=np.int16)
converted_array = self.tfm.build_array(input_array=in_array, sample_rate_in=48000).astype(np.float32) / 32768.0

does this happen even with only 1 client?

Perhaps you could use py-spy to get some stack traces from the server at the time when it is experiencing latency?  that would help Fangjun debug.

Latency increases towards the end of the audio. In the beginning the latency is about 3s, and as we keep streaming audio to the server (~10s long speech)

This does not sound right to me.
Could you use py-spy  to get an image like the following one from
https://github.com/benfred/py-spy/blob/master/images/console_viewer.gif

Here's a screen recording. I spoke to the microphone for about 10s and client received the first recognised word 10s after I started speaking.
https://user-images.githubusercontent.com/31003116/183807172-c038088d-7583-47aa-9cc7-d848e6b9bd9f.mov

Some additional information about Case 2 latency after more experiments:
Every time we restart the server, the first client call to the server will result in a 10s latency, and all subsequent calls with have 1 to 2s latency.
Is there some sort of module loading going on only after the first call?

Did you change the code?

Some additional information about Case 2 latency after more experiments:
Every time we restart the server, the first client call to the server will result in a 10s latency, and all subsequent calls with have 1 to 2s latency.
Is there some sort of module loading going on only after the first call?

If you use the original code of sherpa, you should not have such an issue. If you change the code, please read the code comments carefully in sherpa carefully.

Only made minimal changes, but I'll have a closer look.
What is the expected latency for the sherpa streaming server? Is 1 to 2 seconds expected?

Did you miss the following lines?
https://github.com/k2-fsa/sherpa/blob/eef89876fa1aa5820f2513ef292a2dcde1284252/sherpa/bin/streaming_pruned_transducer_statelessX/streaming_server.py#L554-L563

No, I can confirm they are present in our script.

Could you show your code?

No, I can confirm they are present in our script.

Are they executed before all other functions?

Is 1 to 2 seconds expected?

No, that is unexpected.

Could you use the master code for testing? If you experience the same latency with the master code,
we can have a closer look at it.

Every time we restart the server, the first client call to the server will result in a 10s latency, and all subsequent calls with have 1 to 2s latency.

That is why we add
https://github.com/k2-fsa/sherpa/blob/eef89876fa1aa5820f2513ef292a2dcde1284252/sherpa/bin/streaming_pruned_transducer_statelessX/streaming_server.py#L561-L563
If you vary the batch size in the subsequent calls, you will suffer additional delays. The reason is that
torchscript will compile its graph dynamically by default to optimize the computation, which takes time.
That is why we disable the optimization.

Here's a screen recording. I spoke to the microphone for about 10s and client received the first recognised word 10s after I started speaking.

Thanks! Could you also use
watch -n 0.5 py-spy dump --pid 12345  --native

?

#!/usr/bin/env python3
# Copyright      2022  Xiaomi Corp.        (authors: Fangjun Kuang,
#                                                    Wei Kang)
#
# See LICENSE for clarification regarding multiple authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
A server for streaming ASR recognition. By streaming it means the audio samples
are coming in real-time. You don't need to wait until all audio samples are
captured before sending them for recognition.

It supports multiple clients sending at the same time.

Usage:
    ./streaming_server.py --help

    ./streaming_server.py
"""

from datetime import datetime
import time
import sox
import math
import ssl
import sys
import requests
import json
import argparse
import asyncio
import http
import logging
import math
import warnings
from concurrent.futures import ThreadPoolExecutor
from typing import Optional, Tuple

import k2
import numpy as np
import sentencepiece as spm
import torch
import websockets
from scipy.io.wavfile import write
import torchaudio

from beam_search import FastBeamSearch, GreedySearch
from stream import Stream
from sherpa import RnntConformerModel, add_beam_search_arguments

# TODO: hardcoded crt and pem file path
PATH_TO_CRT = ""
PATH_TO_PEM = ""

def get_args():
    beam_search_parser = add_beam_search_arguments()
    parser = argparse.ArgumentParser(
        parents=[beam_search_parser],
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )

    parser.add_argument(
        "--port",
        type=int,
        default=6006,
        help="The server will listen on this port",
    )

    parser.add_argument(
        "--lang",
        type=str,
        required=True,
        help="Server language. Supports 'zh_tw' and 'en_us'",
    )

    parser.add_argument(
        "--nn-model-filename",
        type=str,
        required=True,
        help="""The torchscript model. You can use
          icefall/egs/librispeech/ASR/pruned_transducer_statelessX/export.py \
                  --jit=1
        to generate this model.
        """,
    )

    parser.add_argument(
        "--bpe-model-filename",
        type=str,
        help="""The BPE model
        You can find it in the directory egs/librispeech/ASR/data/lang_bpe_xxx
        where xxx is the number of BPE tokens you used to train the model.
        Note: You don't need to provide it if you provide `--token-filename`.
        """,
    )

    parser.add_argument(
        "--token-filename",
        type=str,
        help="""Filename for tokens.txt
        For instance, you can find it in the directory
        egs/aishell/ASR/data/lang_char/tokens.txt
        or
        egs/wenetspeech/ASR/data/lang_char/tokens.txt
        from icefall
        Note: You don't need to provide it if you provide `--bpe-model`
        """,
    )

    parser.add_argument(
        "--decode-chunk-size",
        type=int,
        default=8,
        help="The chunk size for decoding (in frames after subsampling)",
    )

    parser.add_argument(
        "--decode-left-context",
        type=int,
        default=32,
        help="""left context can be seen during decoding
        (in frames after subsampling)""",
    )

    parser.add_argument(
        "--decode-right-context",
        type=int,
        default=2,
        help="""right context can be seen during decoding
        (in frames after subsampling)""",
    )

    parser.add_argument(
        "--nn-pool-size",
        type=int,
        default=1,
        help="Number of threads for NN computation and decoding.",
    )

    parser.add_argument(
        "--max-batch-size",
        type=int,
        default=50,
        help="""Max batch size for computation. Note if there are not enough
        requests in the queue, it will wait for max_wait_ms time. After that,
        even if there are not enough requests, it still sends the
        available requests in the queue for computation.
        """,
    )

    parser.add_argument(
        "--max-wait-ms",
        type=float,
        default=10,
        help="""Max time in millisecond to wait to build batches for inference.
        If there are not enough requests in the stream queue to build a batch
        of max_batch_size, it waits up to this time before fetching available
        requests for computation.
        """,
    )

    parser.add_argument(
        "--max-message-size",
        type=int,
        default=(1 << 20),
        help="""Max message size in bytes.
        The max size per message cannot exceed this limit.
        """,
    )

    parser.add_argument(
        "--max-queue-size",
        type=int,
        default=32,
        help="Max number of messages in the queue for each connection.",
    )

    parser.add_argument(
        "--max-active-connections",
        type=int,
        default=500,
        help="""Maximum number of active connections. The server will refuse
        to accept new connections once the current number of active connections
        equals to this limit.
        """,
    )

    return (
        parser.parse_args(),
        beam_search_parser.parse_known_args()[0],
    )

class StreamingServer(object):
    def __init__(
        self,
        nn_model_filename: str,
        bpe_model_filename: str,
        token_filename: str,
        decode_chunk_size: int,
        decode_left_context: int,
        decode_right_context: int,
        nn_pool_size: int,
        max_wait_ms: float,
        max_batch_size: int,
        max_message_size: int,
        max_queue_size: int,
        max_active_connections: int,
        beam_search_params: dict,
        lang: str
    ):
        """
        Args:
          nn_model_filename:
            Path to the torchscript model
          bpe_model_filename:
            Path to the BPE model. If it is None, you have to provide
            `token_filename`.
          token_filename:
            Path to tokens.txt. If it is None, you have to provide
            `bpe_model_filename`.
          decode_chunk_size:
            The chunk size for decoding (in frames after subsampling)
          decode_left_context:
            The left context for decoding (in frames after subsampling)
          decode_right_context:
            The right context for decoding (in frames after subsampling)
          beam:
            The beam for fast_beam_search decoding.
          max_states:
            The max_states for fast_beam_search decoding.
          max_contexts:
            The max_contexts for fast_beam_search decoding.
          decoding_method:
            The decoding method to use, can be either greedy_search
            or fast_beam_search.
          nn_pool_size:
            Number of threads for the thread pool that is responsible for
            neural network computation and decoding.
          max_wait_ms:
            Max wait time in milliseconds in order to build a batch of
            `batch_size`.
          max_batch_size:
            Max batch size for inference.
          max_message_size:
            Max size in bytes per message.
          max_queue_size:
            Max number of messages in the queue for each connection.
          max_active_connections:
            Max number of active connections. Once number of active client
            equals to this limit, the server refuses to accept new connections.
          beam_search_params:
            Dictionary containing all the parameters for beam search.
        """
        if torch.cuda.is_available():
            device = torch.device("cuda", 0)
        else:
            device = torch.device("cpu")

        self.model = RnntConformerModel(nn_model_filename, device=device)

        self.subsampling_factor = self.model.subsampling_factor

        # Note: The following 3 attributes are in frames after subsampling.
        self.decode_chunk_size = decode_chunk_size
        self.decode_left_context = decode_left_context
        self.decode_right_context = decode_right_context

        # We add 3 here since the subsampling method is using
        # ((len - 1) // 2 - 1) // 2)
        # We plus 2 here because we will cut off one frame on each side
        # of encoder_embed output (in conformer.py) to avoid a training
        # and decoding mismatch by seeing padding values.
        # Note: chunk_length is in frames before subsampling.
        self.chunk_length = (
            self.decode_chunk_size + 2 + self.decode_right_context
        ) * self.subsampling_factor + 3

        if bpe_model_filename:
            self.sp = spm.SentencePieceProcessor()
            self.sp.load(bpe_model_filename)
        else:
            self.token_table = k2.SymbolTable.from_file(token_filename)

        self.context_size = self.model.context_size
        self.blank_id = self.model.blank_id
        self.vocab_size = self.model.vocab_size
        self.log_eps = math.log(1e-10)

        self.initial_states = self.model.get_encoder_init_states(
            self.decode_left_context
        )

        # Add these params after loading the RNN-T model
        beam_search_params["vocab_size"] = self.vocab_size
        beam_search_params["context_size"] = self.context_size
        beam_search_params["blank_id"] = self.blank_id

        decoding_method = beam_search_params["decoding_method"]
        if decoding_method.startswith("fast_beam_search"):
            self.beam_search = FastBeamSearch(
                beam_search_params=beam_search_params,
                device=device,
            )
        elif decoding_method == "greedy_search":
            self.beam_search = GreedySearch(
                self.model,
                beam_search_params,
                device,
            )
        else:
            raise ValueError(
                f"Decoding method {decoding_method} is not supported."
            )

        if bpe_model_filename:
            self.beam_search.sp = spm.SentencePieceProcessor()
            self.beam_search.sp.load(bpe_model_filename)
        else:
            self.beam_search.token_table = k2.SymbolTable.from_file(
                token_filename
            )

        self.nn_pool = ThreadPoolExecutor(
            max_workers=nn_pool_size,
            thread_name_prefix="nn",
        )

        self.stream_queue = asyncio.Queue()
        self.max_wait_ms = max_wait_ms
        self.max_batch_size = max_batch_size
        self.max_message_size = max_message_size
        self.max_queue_size = max_queue_size
        self.max_active_connections = max_active_connections

        self.current_active_connections = 0

        # sox converter to handle 48k input
        self.tfm = sox.Transformer()
        self.tfm.set_output_format(rate=16000)

        # ITN server setting
        self.lang = lang
        self.punctuation = PUNCTUATION
        self.itn_server_url = ""

    async def stream_consumer_task(self):
        """This function extracts streams from the queue, batches them up, sends
        them to the RNN-T model for computation and decoding.
        """
        while True:
            if self.stream_queue.empty():
                await asyncio.sleep(self.max_wait_ms / 1000)
                continue

            batch = []
            try:
                while len(batch) < self.max_batch_size:
                    item = self.stream_queue.get_nowait()

                    assert len(item[0].features) >= self.chunk_length, len(
                        item[0].features
                    )

                    batch.append(item)
            except asyncio.QueueEmpty:
                pass
            stream_list = [b[0] for b in batch]
            future_list = [b[1] for b in batch]

            loop = asyncio.get_running_loop()
            await loop.run_in_executor(
                self.nn_pool,
                self.beam_search.process,
                self,
                stream_list,
            )

            for f in future_list:
                self.stream_queue.task_done()
                f.set_result(None)

    async def compute_and_decode(
        self,
        stream: Stream,
    ) -> None:
        """Put the stream into the queue and wait it to be processed by the
        consumer task.

        Args:
          stream:
            The stream to be processed. Note: It is changed in-place.
        """
        loop = asyncio.get_running_loop()
        future = loop.create_future()
        await self.stream_queue.put((stream, future))
        await future

    async def process_request(
        self,
        unused_path: str,
        unused_request_headers: websockets.Headers,
    ) -> Optional[Tuple[http.HTTPStatus, websockets.Headers, bytes]]:
        if self.current_active_connections < self.max_active_connections:
            self.current_active_connections += 1
            return None

        # Refuse new connections
        status = http.HTTPStatus.SERVICE_UNAVAILABLE  # 503
        header = {"Hint": "The server is overloaded. Please retry later."}
        response = b"The server is busy. Please retry later."

        return status, header, response

    async def run(self, port: int):
        task = asyncio.create_task(self.stream_consumer_task())

        ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
        path_cert = PATH_TO_CRT
        path_key = PATH_TO_PEM
        ssl_context.load_cert_chain(path_cert, keyfile=path_key)

        async with websockets.serve(
            self.handle_connection,
            host="",
            port=port,
            max_size=self.max_message_size,
            max_queue=self.max_queue_size,
            process_request=self.process_request,
            ssl=ssl_context
        ):
            await asyncio.Future()  # run forever

        await task  # not reachable

    async def handle_connection(
        self,
        socket: websockets.WebSocketServerProtocol,
        path = "/client/ws/speech"
    ):
        """Receive audio samples from the client, process it, and send
        decoding result back to the client.

        Args:
          socket:
            The socket for communicating with the client.
        """
        try:
            await self.handle_connection_impl(socket)
        finally:
            # Decrement so that it can accept new connections
            self.current_active_connections -= 1

            logging.info(
                f"Disconnected: {socket.remote_address}. "
                f"Number of connections: {self.current_active_connections}/{self.max_active_connections}"  # noqa
            )

    async def handle_connection_impl(
        self,
        socket: websockets.WebSocketServerProtocol,
    ):
        """Receive audio samples from the client, process it, and send
        deocoding result back to the client.

        Args:
          socket:
            The socket for communicating with the client.
        """
        logging.info(
            f"Connected: {socket.remote_address}. "
            f"Number of connections: {self.current_active_connections}/{self.max_active_connections}"  # noqa
        )
        stream = Stream(
            context_size=self.context_size,
            initial_states=self.initial_states,
        )

        self.beam_search.init_stream(stream)

        while True:
            samples = await self.recv_audio_samples(socket)
            if samples is None:
                break
            # TODO(fangjun): At present, we assume the sampling rate
            # of the received audio samples is always 16000.
            stream.accept_waveform(sampling_rate=16000, waveform=samples)

            while len(stream.features) > self.chunk_length:
                await self.compute_and_decode(stream)
                result = self.beam_search.get_texts(stream)
                package = {
                    "status": 0,
                    "segment": 0,
                    "result": {
                        "hypotheses": [
                            {"transcript": result}
                        ],
                        "final": False,
                        "id": "tmp"
                    }
                }
                await socket.send(json.dumps(package))

        stream.input_finished()
        while len(stream.features) > self.chunk_length:
            await self.compute_and_decode(stream)

        if len(stream.features) > 0:
            n = self.chunk_length - len(stream.features)
            stream.add_tail_paddings(n)
            await self.compute_and_decode(stream)
            stream.features = []

        result = self.beam_search.get_texts(stream)

        package = {
            "status": 0,
            "segment": 0,
            "result": {
                "hypotheses": [
                    {"transcript": result}
                ],
                "final": True,
                "id": "tmp"
            }
        }
        await socket.send(json.dumps(package))
        await socket.send("EOS")

    async def recv_audio_samples(
        self,
        socket: websockets.WebSocketServerProtocol,
    ) -> Optional[torch.Tensor]:
        """Receives a tensor from the client.

        Each message contains either a bytes buffer containing audio samples
        in 16 kHz or contains b"Done" meaning the end of utterance.

        Args:
          socket:
            The socket for communicating with the client.
        Returns:
          Return a 1-D torch.float32 tensor containing the audio samples or
          return None.
        """
        message = await socket.recv()
        if isinstance(message, str):
            if message == "EOS":
                return None
            else:
                logging.warn(f"UNKNOWN MESSAGE: {message}")

        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            # PyTorch warns that the underlying buffer is not writable.
            # We ignore it here as we are not going to write it anyway.
            if hasattr(torch, "frombuffer"):
                # Note: torch.frombuffer is available only in torch>= 1.10
                return torch.frombuffer(message, dtype=torch.float32)
            else:
                in_array = np.frombuffer(message, dtype=np.int16)
                converted_array = self.tfm.build_array(input_array=in_array, sample_rate_in=48000).astype(np.float32) / 32768.0
                return torch.from_numpy(converted_array)

@torch.no_grad()
def main():
    args, beam_search_parser = get_args()
    beam_search_params = vars(beam_search_parser)
    logging.info(vars(args))

    port = args.port
    nn_model_filename = args.nn_model_filename
    bpe_model_filename = args.bpe_model_filename
    token_filename = args.token_filename
    decode_chunk_size = args.decode_chunk_size
    decode_left_context = args.decode_left_context
    decode_right_context = args.decode_right_context
    nn_pool_size = args.nn_pool_size
    max_batch_size = args.max_batch_size
    max_wait_ms = args.max_wait_ms
    max_message_size = args.max_message_size
    max_queue_size = args.max_queue_size
    max_active_connections = args.max_active_connections
    lang = args.lang

    server = StreamingServer(
        nn_model_filename=nn_model_filename,
        bpe_model_filename=bpe_model_filename,
        token_filename=token_filename,
        decode_chunk_size=decode_chunk_size,
        decode_left_context=decode_left_context,
        decode_right_context=decode_right_context,
        nn_pool_size=nn_pool_size,
        max_batch_size=max_batch_size,
        max_wait_ms=max_wait_ms,
        max_message_size=max_message_size,
        max_queue_size=max_queue_size,
        max_active_connections=max_active_connections,
        beam_search_params=beam_search_params,
        lang=lang
    )
    asyncio.run(server.run(port))

torch.set_num_threads(1)
torch.set_num_interop_threads(1)

# See https://github.com/pytorch/pytorch/issues/38342
# and https://github.com/pytorch/pytorch/issues/33354
#
# If we don't do this, the delay increases whenever there is
# a new request that changes the actual batch size.
# If you use `py-spy dump --pid <server-pid> --native`, you will
# see a lot of time is spent in re-compiling the torch script model.
torch._C._jit_set_profiling_executor(False)
torch._C._jit_set_profiling_mode(False)
torch._C._set_graph_executor_optimize(False)
"""
// Use the following in C++
torch::jit::getExecutorMode() = false;
torch::jit::getProfilingMode() = false;
torch::jit::setGraphExecutorOptimize(false);
"""

if __name__ == "__main__":
    formatter = "%(asctime)s %(levelname)s [%(filename)s:%(lineno)d] %(message)s"  # noqa
    logging.basicConfig(format=formatter, level=logging.INFO)
    main()

I'm not sure if this is the result you are looking for with that command:
https://user-images.githubusercontent.com/31003116/184046419-16d9abf0-058b-4003-907c-12903a01ea41.mov

Is 1 to 2 seconds expected?

No, that is unexpected.
Could you use the master code for testing? If you experience the same latency with the master code, we can have a closer look at it.

I tested with the master code (server: ./sherpa/sherpa/bin/streaming_pruned_transducer_statelessX/streaming_server.py and client ./sherpa/sherpa/bin/streaming_pruned_transducer_statelessX/streaming_client.py) and I'm seeing the same latency, see video:
https://user-images.githubusercontent.com/31003116/184049939-d741daa3-9872-49c9-a2c5-af94013f9902.mp4

Thanks. Will look into your code.
Could you try the master code and see whether it can reproduce your problem?

Here's the exact command I used to run the server:
./sherpa/sherpa/bin/streaming_pruned_transducer_statelessX/streaming_server.py \
  --port 6006 \
  --max-batch-size 50 \
  --max-wait-ms 5 \
  --nn-pool-size 4 \
  --nn-model-filename "${en_nn_model}" \
  --bpe-model-filename "${en_bpe_model}"

And the command for the client:
./sherpa/sherpa/bin/streaming_pruned_transducer_statelessX/streaming_client.py \
  --server-addr localhost \
  --server-port 6006 \
  "${wav_file}"

I ran another test with this model: https://huggingface.co/Zengwei/icefall-asr-librispeech-conv-emformer-transducer-stateless2-2022-07-05
and this time the initial latency seems to be shorter, but it is still significantly longer than all the subsequent client calls, see video:
https://user-images.githubusercontent.com/31003116/184050701-8cca1658-3127-426a-9e84-e6f1eb76975f.mp4
Command for the server:
./sherpa/sherpa/bin/conv_emformer_transducer_stateless2/streaming_server.py \
  --port 6006 \
  --max-batch-size 50 \
  --max-wait-ms 5 \
  --nn-pool-size 4 \
  --nn-model-filename "${en_nn_model}" \
  --bpe-model-filename "${en_bpe_model}"

(command for the client is the same as above)

@shaynemei  Try smaller decode-chunk-size? like:
--decode-chunk-size=8 \
--left-context=32 \
--right-context=0

After some testing, I think the cause could be related to torchscript building the graph after the first request is sent in and saving it to memory.

Upon loading the model, GPU usage is ~1000MB (9732MB -> 10767MB)
Sending the first request, GPU usage increases by ~100MB (10767MB -> 10825MB)
After the first request, GPU usage stays at 10825MB
Sending another request, GPU usage doesn't change.
Sending yet another request, GPU usage doesn't change.

See video:
https://user-images.githubusercontent.com/31003116/184257157-80ab0ccb-3eea-4523-b1b9-303a89f25ffe.mp4
Maybe there is a way to build the graph in advance if we can pre-determine the batch size?

Thanks!

Maybe there is a way to build the graph in advance if we can pre-determine the batch size?

Does the latency depend on the batch size? If not, we can send a faked batch to the model while it is loaded.

Initial latency does seem to be smaller when sending in a very large batch. (speaking into microphone vs. sending in full file at once.)

The initial latency should be a quick fix, but this 1-2s latency is more worrisome. Can you look into this?

Made another discovery, it looks like GPU cards have quite an influence on the initial latency.
K80: initial call latency ~10s, all subsequent calls ~0.6s
T4: initial call latency ~4s, all subsequent calls ~0.5s

The initial latency should be a quick fix, but this 1-2s latency is more worrisome.

Could you try the latest master?
#100 should fix the issue of the initial latency.

but this 1-2s latency is more worrisome.

Sorry, I cannot reproduce this issue. In my test, the initial latency for the first request is nearly the same as
all other requests with the master code after using #100


Input data format

Thanks very much for your work Ardavan!
I have noticed sHDP requires a picked data file and a pickled word embedding in dict format. I have a bunch of tweets that I want to test on your model (one tweet per line). May I ask how do I come about transforming it to the required format please?
In texts, e.g. does ('time', 2) mean the token 'time' occurred twice in the whole corpus? and each list of tuples is a tokenised document? Un-ordered?
Is vectors_dict merely subsampled pre-trained word2vec embeddings as it only has 4768 items?
Thanks again!

Thank you for your interest :)
As you said there are two main pickle files in the code.
wordvec.pk contains the subset of the words from glove.word2vec that are
available in your corpus (i.e. tweets). You may want to do some standard
preprocessing to make sure you have a good quality corpus and then only
make the wordvec.pk for the words in the clean dataset.
texts.pk converts each document to a "bag of words" list. That is, for each
document you have a list containing pairs of words and their frequency in
that document. You can convert a document to this format using the NLTK
package in python.
Let me know if this doesn't answer your question or you have any other
questions.
Best,
Ardavan
On Thu, Apr 20, 2017 at 6:17 PM, bluemonk482<EMAIL_ADDRESS>wrote:

Thanks very much for your work Ardavan!
I have noticed sHDP requires a picked data file and a pickled word
embedding in dict format. I have a bunch of tweets that I want to test on
your model (one tweet per line). May I ask how do I come about transforming
it to the required format please? Is vectors_dict merely subsampled
glove.twitter.word2vec.27B.50d ?
Thanks again!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Ardavans/sHDP/issues/2, or mute the thread
https://github.com/notifications/unsubscribe-auth/ACkD5b9t6wwI9I9O_iMPLTc5WP8oDrwjks5rx9lpgaJpZM4NDpL6
.

Thanks for you reply Ardavans! Jus want to re-confirm on the two points.
Hmm I originally thought each document of texts.pk contains pairs of words and their frequency in the whole corpus as it is usually the case right? I have a collection of tweets and each tweet is considered as a document. But I think in my case each word should be paired with its frequency in the whole collection do you agree?
For wordvec.pk, did you mean if I want to train my own word2vec model I should preprocess my training corpus (tweets in this case) beforehand right?
Best,
Bo

No problem!
Hmm I originally thought each document of texts.pk
http://texts.pk/ contains pairs of words and their frequency in the whole
corpus as it is usually the case right? I have a collection of tweets and
each tweet is considered as a document. But I think in my case each word
should be paired with its frequency in the whole collection do you agree?
I think it should be the frequency of the words in each document. See for
instance the nips dataset available in the repo. There are 1566 documents
and for instance in the first document word 'weights' occurs 7 times.
For wordvec.pk http://wordvec.pk/, did you mean if I want to train my
own word2vec model I should preprocess my training corpus (tweets in this
case) beforehand right?
I meant to say even after training there might be some words that are less
important for you; you can remove those with some preprocessing. However,
if you've already taken care of those words in training then you won't need
to that again.
Let me know if you have any questions.
Best,
Ardavan
On Sat, Apr 22, 2017 at 1:13 PM, bluemonk482<EMAIL_ADDRESS>wrote:

Thanks for you reply Ardavans! Jus want to re-confirm on the two points.
Hmm I originally thought each document of texts.pk contains pairs of
words and their frequency in the whole corpus as it is usually the case
right? I have a collection of tweets and each tweet is considered as a
document. But I think in my case each word should be paired with its
frequency in the whole collection do you agree?
For wordvec.pk, did you mean if I want to train my own word2vec model I
should preprocess my training corpus (tweets in this case) beforehand right?
Best,
Bo
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Ardavans/sHDP/issues/2#issuecomment-296387822, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACkD5WxEU3QPuJktXWZfcY2FBMeYFXEJks5ryjU_gaJpZM4NDpL6
.

Thanks very much Ardavans!
Indeed, you're right. But is the frequency of words used for normalising its word vectors here or is it being used for something else? I have quickly checked your code and I can't see where such frequency is used in any way?
Also, do I need to normalise my glove vectors to unit L2 norm beforehand or this normalisation is already integrated in your code? Thanks :)

As far as I remember, the frequency is used for the variational inference
updates and not for normalizing the word vectors.
Also I think we did normalize the glove vectors in the code but there is no
harm in doing it again if you are not doing it already :)
On Sun, Apr 23, 2017 at 10:17 AM, bluemonk482<EMAIL_ADDRESS>wrote:

Thanks very much Ardavans!
Indeed, you're right. But is the frequency of words used for normalising
its word vectors here or is it being used for something else? I have
quickly checked your code and I can't see where such frequency is used in
any way?
Also, do I need to normalise my glove vectors to unit L2 norm beforehand
or this normalisation is already integrated in your code? Thanks :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Ardavans/sHDP/issues/2#issuecomment-296446347, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACkD5Y1x6x8OKHp3ovbZ4Z56vFJDsNq8ks5ry11sgaJpZM4NDpL6
.

Thanks !! :+1:
Final question :P , is there any necessity or caveat for parameter tuning for tweets? apart from Nmax and
mbsize which I need to adjust for my use.. I would prefer to use default settings obvs if these are empirically found to be the most effective and no significant change will be seen on a different data
./runner.py -is 1 -alpha 1 -gamma 2 -Nmax 40 -kappa_sgd 0.6 -tau 0.8 -mbsize 10 -dataset twitter

No problem :)
Unfortunately, that's a really hard question to answer as I haven't tried
the code on very short documents like tweets. I guess alpha and gamma
parameters may also need to be changed since you don't expect to see so
many topics in a single tweet. However, I might be wrong and it may also
work with the default setting.
On Sun, Apr 23, 2017 at 11:52 AM, bluemonk482<EMAIL_ADDRESS>wrote:

Thanks !! 👍
Final question :P , is there any necessity or caveat for parameter tuning
for tweets? apart from Nmax and
mbsize which I need to adjust for my use.. I would prefer to use default
settings obvs if these are empirically found to be the most effective and
no significant change will be seen on a different data
./runner.py -is 1 -alpha 1 -gamma 2 -Nmax 40 -kappa_sgd 0.6 -tau 0.8
-mbsize 10 -dataset twitter
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Ardavans/sHDP/issues/2#issuecomment-296452701, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACkD5RQK5S28DGFFRNyAqPIyYkCc_55Zks5ry3O_gaJpZM4NDpL6
.

Thanks for all the help Ardavans!! :)
I shall close this issue now.

No problem at all! Good luck with your project!
On Sun, Apr 23, 2017 at 12:06 PM, bluemonk482<EMAIL_ADDRESS>wrote:

Thanks for all the help Ardavans!! :)
I shall close this issue now.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Ardavans/sHDP/issues/2#issuecomment-296453534, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACkD5W3qnK6dsTOvyHLMjMeMHcDPLqoeks5ry3b1gaJpZM4NDpL6
.


New code doesn't migrate old db:

$ go run . env list --db-path ~/.config/envelope.db
could not create env service: could not init db: migrate: migration error: name="migrations/001_create.sql" err=migrate file err: txFunc err: SQL logic error: table env already exists (1)
exit status 1

Why is it trying to run this migration? Maybe need to check what's in the table now.

Ok the issue is that I changed the name of the migrations folder:
┌──────────────────────────────────────┐
│                 name                 │
├──────────────────────────────────────┤
│ embedded_migrations/001_create.sql   │
│ embedded_migrations/002_env_ref.sql  │
│ embedded_migrations/003_refactor.sql │
└──────────────────────────────────────┘

And now it's migrations/ instead of embedded_migrations


RE2 enemy skins issues

Version
BioRand v3.1.1
Seed
all seeds
Scenario
RE 1 (Chris)
Player Character
all characters
What happened?
A lot of enemy skins are not randomizing no matter the seed i use.
jill(RE3),Kendo, Kendo(RE2R) Kevin, Krauser, Leon(RE2R) Leon(RE4) Lisa Trevor(RE1R) Marvin, Marvin(RE2R) Mikhail, MrX, Murphy, Nemesis, Nemesis(2) Nemesis(RE3) Piers, Rebecca, Regina, Richard, sherry(RE2R) Sheva, Steve, Tyrant(CV) Tyrant(RE1) Tyrell, Wesker, Wesker(CV) Wesker(RE1R)    Wesker(RE5) William, Worm(RE3) Yawn(RE1) Zombies(RE3)

This is unfortunately not enough information for me to investigate. Have you tried a seed with just one enemy skin selected such as Nemesis (RE 3)?

I tried that. all the skins i named did not changed any letters or numbers in any seed I tried.  this for an example. R720-WW5F-WXPE806YWU431BNVH3021600C000000000H0Y00000. I see when we are making a seed, whatever we choose to randomize makes changes to those letters or numbers. the skins i listed are the only ones not working for me. I've checked and unchecked every thing in the rando and I see the change in the seed.  But the skins I listed  jill(RE3) on down did not change anything. Alexia is the only enemy skin that has changed tyrant 1 (MRX)

Can you show me a screenshot or video of what you are seeing?
Did you install biorand to a clean folder? Overwriting an old version may have left some unwanted old files in the data folder.

I just made a new folder and everything is good. I forgot about making a new folder. Thank you so much for the help and the best rando ever! now, time to try it out.


FastNoise2 is taking more time to generate a 2D grid than FastNoiseLite

Hello! First of all, thanks for your awesome libraries.
Testing noise generation with the same parameters using FastNoise2 and FastNoiseLite gives me much better performance with the latter. I wonder if I'm doing something wrong. Here's the code:
FastNoiseLite:
  const float frequency = 0.005f;
  const int octaves = 4;
  const float lacunarity = 3.2f;
  const float gain = 0.32f;
  const float weighted_strength = 0.25f;

  auto noise = FastNoiseLite{seed};
  noise.SetSeed(seed);
  noise.SetNoiseType(FastNoiseLite::NoiseType::NoiseType_OpenSimplex2S);
  noise.SetFrequency(frequency);
  noise.SetFractalType(FastNoiseLite::FractalType::FractalType_FBm);
  noise.SetFractalOctaves(octaves);
  noise.SetFractalLacunarity(lacunarity);
  noise.SetFractalGain(gain);
  noise.SetFractalWeightedStrength(weighted_strength);

  for (int j = 0; j < 1024; ++j)
  {
    for (int i = 0; i < 1024; ++i)
    {
      const float noise_value = noise.GetNoise(static_cast<float>(i), static_cast<float>(j));
    }
  }

FastNoise2:
  const float frequency = 0.005f;
  const int octaves = 4;
  const float lacunarity = 3.2f;
  const float gain = 0.32f;
  const float weighted_strength = 0.25f;

  std::vector<float> height_map(1024 * 1024);

  const auto& simplex = FastNoise::New<FastNoise::OpenSimplex2S>();
  const auto& fractal = FastNoise::New<FastNoise::FractalFBm>();
  fractal->SetSource(simplex);
  fractal->SetOctaveCount(octaves);
  fractal->SetLacunarity(lacunarity);
  fractal->SetGain(gain);
  fractal->SetWeightedStrength(weighted_strength);
  fractal->GenUniformGrid2D(height_map.data(), 0, 0, 1024, 1024, frequency, seed);

Running NoiseTool gives me the following information:
Renderer: Intel HD Graphics 4000 OpenGL Engine by Intel Inc.
OpenGL version: 4.1 INTEL-14.7.23
Using optional features:
    GL_ARB_vertex_array_object
    GL_ARB_ES2_compatibility
    GL_ARB_separate_shader_objects
    GL_ARB_texture_storage
    GL_EXT_texture_filter_anisotropic
    GL_EXT_debug_label
    GL_EXT_debug_marker
Using driver workarounds:
    no-layout-qualifiers-on-old-glsl
    apple-buffer-texture-unbind-on-buffer-modify
FastSIMD detected max CPU SIMD Level: AVX

Are you testing using a debug or release build, unless you test using a release build with optimisations enabled you won't get true performance numbers

That was the issue, I was using a debug build. Thanks for your help!


add implode template function

fixes #21

I was about to close the laptop for the night when I saw this PR come in. I'll give it a closer read tomorrow, but here's what I was thinking. I started this section in the template.rs file where we could define some built-in template functions: https://github.com/fermyon/bartholomew/blob/main/src/template.rs#L163-L220
I was thinking it would be nice to add some functions that would be generally useful to template developers, and since the Rust code is compiled, it's a bit faster than the Rhai code.
I hadn't considered just doing all the template functions in Rhai code. That definitely has some pros and cons. For example, while it might be slower to execute, it means that the Bartholomew user could edit functions without having to recompile.
Maybe the right thing division point would be that "utiltiies" (string manipulation, math, formatting, etc) would go in the Rust code while things that dealt with objects like site or page or env would be rhai scripts. What are your thoughts?
I'll look in on this issue tomorrow. Thanks again! This is fabulous!

it totally makes sense. (may be we could even implement something like you had with sprig).  I've updated the PR with implode implemented in template.rs
kindly let me know if this looks ok.


Reduce PDF size by merging layer images

Closes https://github.com/mapfish/mapfish-print/issues/391

@tsauerwein does it looks good to you?

Your solution to only merge layers with the same size looks reasonable to me. Thanks for taking care of this!

Thanks :-)


Multiple Connector Contradiction

https://learn.microsoft.com/en-us/azure/active-directory/hybrid/connect/how-to-connect-password-hash-synchronization#how-password-hash-synchronization-works
In this section you state "However, if there are multiple connectors, it is possible to disable password hash sync for some connectors but not others using the Set-ADSyncAADPasswordSyncConfiguration cmdlet."
But your own documentation on how to configure the Azure AD Connect client says that you can't use multiple connectors for one tenant and even mentions this scenario: "Having multiple Azure AD Connect sync servers connected to the same Azure AD tenant is not supported, except for a staging server. It's unsupported even if these servers are configured to synchronize with a mutually exclusive set of objects."
https://learn.microsoft.com/en-us/azure/active-directory/hybrid/connect/plan-connect-topologies#single-forest-multiple-sync-servers-to-one-azure-ad-tenant
So is it possible to have multiple connectors with different sync settings or not?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 2b5d859b-57c8-a09b-d21b-2d533542fc65
Version Independent ID: 09946fbd-cfcd-c30a-1f0d-6e3423813022
Content: Azure AD Connect: User sign-in - Microsoft Entra
Content Source: articles/active-directory/hybrid/connect/plan-connect-user-signin.md
Service: active-directory
Sub-service: hybrid
GitHub Login: @billmath
Microsoft Alias: billmath

@r0bulus
Thanks for your feedback! We will investigate and update as appropriate.

@r0bulus
I'm going to assign this to the document author so they can take a look at it accordingly
@billmath
Could you Please review this and update as appropriate.

Verifying with engineering.

I reviewed this with engineering and here is the response:
This cmdlet takes the source AD Connector as a parameter (“SourceConnector”) and disables PHS for the source connector. So the git issue in question is a non-issue – we only sync to one AAD tenant and we can turn off PHS on any given AD Connector.
Thanks!
#please-close

When we state "However, if there are multiple connectors, it is possible to disable password hash sync for some connectors but not others using the Set-ADSyncAADPasswordSyncConfiguration cmdlet.", we are referring to a single AADConnect server running multiple AD Connectors running on the same ADSync service instance, i.e. when you have multiple AD Forests synchronizing to Azure AD.
So, you can have multiple Connectors (i.e. multi-forest topology) but you can't have multiple Servers, except for a staging servers.
Please see: https://learn.microsoft.com/en-us/azure/active-directory/hybrid/connect/plan-connect-topologies
Hope this helps.


Row Level Security vs Role Level Security

Both this page and the Row-level security page¹ reference "RLS". In this whitepaper, it denotes Role level security, while the other page denotes Row level security (Row vs Role, but use the same acronym).
Are these the same things?

https://docs.microsoft.com/en-us/power-bi/service-admin-rls

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 13b73941-7a77-b7e6-01a5-6d088f7c50ec
Version Independent ID: ca5a44b2-88c7-e03d-4e94-a0a727160fff
Content: Power BI security whitepaper - Power BI
Content Source: powerbi-docs/whitepaper-powerbi-security.md
Service: powerbi
Sub-service: powerbi-service
GitHub Login: @davidiseminger
Microsoft Alias: davidi

I opened one issue for each page. I hope that was correct.

Hi, @bfrancom -- thanks for your question(s)! I'll assign both to the authors.


Delete unnecessary spaces

When copying from the web page, there is an unnecessary space after the code.

@changeworld : Thanks for your contribution! The author, @ericrad, has been notified to review your proposed change.

@changeworld Thank you for this submission. We’ve consulted with the repo admin and verified that removing double spaces at the end of lines changes the rendering, so we’re closing this PR. If there are non-whitespace changes in this PR, you can submit them in another PR. Thanks!
#please-close


Appstudio update test-component-pac-mvnn

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-mvnn-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
4 minutes
clone-repository

✅ Succeeded
30 seconds
build-container

✅ Succeeded
9 seconds
deprecated-base-image-check

✅ Succeeded
10 seconds
inspect-image

✅ Succeeded
8 seconds
clair-scan

✅ Succeeded
33 seconds
clamav-scan

✅ Succeeded
8 seconds
sbom-json-check

✅ Succeeded
14 seconds
label-check

✅ Succeeded
13 seconds
optional-label-check

✅ Succeeded
6 seconds
show-sbom

✅ Succeeded
6 seconds
show-summary


[BUG]: u.getCurrentUser is not a function

Describe the bug | BUG描述
TypeError: u.getCurrentUser is not a function
at discordExperiments.start (DeveloperExperiments.plugin.js:23:9)
at A.startPlugin (renderer.js:5:34290)
at A.startAddon (renderer.js:5:34083)
at A.loadAddon (renderer.js:5:6780)
at A.loadAddon (renderer.js:5:31500)
at A.reloadAddon (renderer.js:5:7276)
at AsyncFunction. (renderer.js:5:4400)
Version | 版本
V1.5.0
Code of Conduct | 行為守則

[X] I have confirmed that what I have BUG Report has not been reported. | 我已確認我所回報的BUG並沒有被回報過

It working fine for me.
Note: This plugins just support discord stable version


Make Svelte's Switch component in sync with React version

Linked Issue
Closes #2882
Description
The Svelte version of the Switch component seems to not do anything previously. I basically copied the React version of the implementation here.
Changsets
Instructions: Changesets automate our changelog. If you modify files in /packages, run pnpm changeset in the root of the monorepo, follow the prompts, then commit the markdown file. Changes that add features should be minor while chores and bugfixes should be patch. Please prefix the changeset message with feat:, bugfix: or chore:.
Checklist
Please read and apply all contribution requirements.

[x] This PR targets the dev branch (NEVER master)
[x] Documentation reflects all relevant changes
[x] Branch is prefixed with: docs/, feat/, chore/, bugfix/
[x] Ensure Svelte and Typescript linting is current - run pnpm ci:check
[x] Ensure Prettier linting is current - run pnpm format
[x] All test cases are passing - run pnpm test
[x] Includes a changeset (if relevant; see above)

Thanks @phamduylong, but can you elaborate on this point:

The Svelte version of the Switch component seems to not do anything previously

The component seems functional on the documentation, so I'm not sure I follow:
https://next.skeleton.dev/docs/components/switch/svelte
Plus the issue you've linked to was referring to console log warning. Can you confirm the goal of this PR and that this does resolve those errors please.
Additionally, since this issue touches a project in /packages, you are required to submit a Changeset per the instructions provided in the original message above. Make sure to include that please.

@endigo9740 Yeah I completely forgot that. The console warnings I was able to do nothing about those, the attempt was just made to have onCheckedChange to be available as props. Sorry for the confusion.

For some reason I don't encounter the console warnings anymore after testing this again with the repo provided in the linked issue. And a changeset is now provided.