Allow for embedding terraform in the marketplace

Because the deployment of a unity application is specific to unity and MCP, it doesn't make sense to embed that terraform deployment with the application itself.
This request is to add the ability to embed the terraform scripts directly into the application marketplace. The management console would then be updated to be able to run terraform from the local marketplace directory, and not from a remote repository.

@maseca Can we use your MMGIS case as a guinea pig for this?  In other words, can you move the directory you have here:
https://github.com/unity-sds/unity-cs-infra/tree/main/MMGIS-IAC/terraform
to a new sub-directory under here:
https://github.com/unity-sds/unity-marketplace/tree/main/applications/mmgis
?
then of course we would have to modify the metadata.json file to point to the new location.


Added simple macro for Que

Here is a macro that can be used for the Que job queue that we created out of necessity and would like to share it with the community if it will be useful.

Thanks @mossgross-cf!
Is it for this lib? https://github.com/chanks/que

@prognostikos I wouldn't mind merging this back into master and including it in the official gem if it's still applicable. Could you check if @mossgross-cf's branch works with your app?
gem "hirefire-resource",
  github: "CrowdFlower/hirefire-resource",
  branch: "task/add-que-macro"

@mrrooijen I've tested this in production for our app and it does work

@prognostikos thanks! Just released v0.4.0 which includes Que.


Tagless Final Example

Question
Is it technically possible to use tagless-final with fp-ts? I fried my brain all day to make it work. If it is, can you showcase an example?
PS: Thank you for getting functional bits to typescript.

Please
https://gist.github.com/gcanti/85bb38c5cf2cd6fb5a1decf7c04624c2

Check this out https://github.com/gcanti/fp-ts/blob/master/examples/mtl.ts
Also https://gist.github.com/gcanti/453e5419fbcabe078d933ab21f0df8bf

I added this question and the linked documents to the FAQ section of the documentation, closing this now.


$parent.$parent not defined

See https://github.com/markfirmware/audit/blob/efb45657ce91a7368b5a8a6cf7364bff60e9e815/docs/index.html#L13-L16
These lines detect the completion of four digits and then attempt to change the state of the fsm.
Before moving on to a better design, is chaining back more than one frame supported by $parent?

Hi,
If you're using the $parent helper, you can't use $parent.$parent at all, but instead can use $component(id).
However, I think the devtools doesn't pick up changes made via $component/$parent yet but I'll look into that this weekend (just noticed this yesterday actually)

Unrelated, but I wrote a pin type component awhile back you might find useful.
https://codepen.io/KevinBatdorf/pen/Vweprqg

Thanks. I will close this and study your pin component. I see it uses tick which is on my list.


CI Build Fix

Updating older NuGet references.
May take a few iterations, I just wanted to try this on the CI machine.

@dotnet-bot test this please

I can semi-successfully build at this commit, so I'm going to push to see if I can persuade the build+package machine to do so in the interests of getting an updated package out. We still need to fix the CI builds of course.


[Feature Request] Compatibility with Tomb Many Graves 2 via API

Hello!
I have recently made an update to Tomb Many Graves 2 which allows mod authors other than myself to add compatibility between their mod and mine. This update uses a new API I created; I have also (temporarily) created a plug-in to allow your mod to work with TMG using this API. I was hoping/wondering if you would consider using the API in order to take compatibility handling out of my hands in case future updates change the way things work for your mod? You can use the plug-in I created as an example, and I can answer any questions you may have.
Link to API project: https://minecraft.curseforge.com/projects/tombmanygraves-2-api
(A link to the specific plugin's project can be found on that page as well)
Thanks!
-M4thG33k

The GNU GPL may require other mods to be GPL to link to its API; we're not. See https://github.com/TheIllusiveC4/ConstructsArmory/issues/61


Intellisense is Broken for Packagees

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.88.0
OS Version: Windows_NT x64 10.0.22631

Steps to Reproduce:

I have NPM Intellisense installed in my VS Code.
I am developing a ReactJS project.
I have installed Material UI packages through NPM
Use the components available in the Material UI
When I typed a part of the name of a component, the intellisense gave me the suggestions from different packages
I then updated the VS Code to the latest release (v1.88)
And after that when I tried to use the components, the intellisense does not gives me the correct suggestions.

Probably https://github.com/microsoft/TypeScript/issues/58113#issuecomment-2043273180

I have no intellisense for anything at all. Not tsconfig, not package.json, not even for the console object in a js file. The same project intellisense works fine in visual studio and webstorm.

Does this reproduce in the latest VS Code insiders build with all extensions disabled?

@mjbvz & @gjsjohnmurray
We started an issue in MUI about this, because it seems to only affect certain packages.
Not sure if there is already an existing issue for this or if this is where we should tag to but for now the easiest workaround is to downgrade to 1.87.2 we found
I can confirm that the issue still is persistent and not fix currently in the 1.89.0 (Insiders) the MUI imports still have the same issues.
Imports work in ts/tsx but do not resolve properly in js/jsx for MUI Specifically at least
.ts / .tsx works correctly as expected:

.js / .jsx does not work correctly:

Would love to find a main issue to tag for this to get fixed quickly! Thank you! 😄
mui/material-ui#41802

@lyricalcasanova https://github.com/microsoft/TypeScript/issues/58113 but this user isn't seeing any intellisense so possible not related

Curious @mjbvz what would be the easiest way for me to test the solution proposed in that issue "You should be able to use the nightly TS extension to work around this for now." if you can offer a few more details on this I could see if it works unless that's just switching from .js to .ts instead as I show above?

You can either install https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-typescript-next or install typescript@next in your workspace and run Select typescript version to use it

You can either install https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-typescript-next or install typescript@next in your workspace and run Select typescript version to use it

This is exactly what is needed to be updated can confirm works as intended after installed
I would put this at the top of the list on priority as there will probably be A LOT more people soon if their VS Code auto updates like mine did and they don't have the extension installed and are facing the same issue
It is VERY hard to search for this issue as I didnt know if this was VS or MUI originally and most people won't

That's why were releasing a recovery release for it

Where can the progress for that be viewed? I assume there's a PR but thank you once again !

https://github.com/microsoft/vscode/issues/209915

Thank you for all the help!
I assume this can be closed as duplicate because I know the OP has the exact same issue as me he just typed the full component name and then waited for suggestions as shown:

PLEASE add those same details you gave me above to the original issue as I actually saw that but it was too vague and didn't seem related since I was using .JS / .JSX

Hi @mjbvz & @lyricalcasanova
It seems it is an issue with the latest release of VS Code v1.88.0. I have gone through all the comments @lyricalcasanova  provided, and thanks so much for giving many details of the issue.
I have installed the JavaScript and TypeScript Nightly extension into my VSCode and selected the VSCode version. But I am still having the same issue, it does not show me the correct suggestions.  Have I done it incorrectly? Or do we have workaround for this issue? Or do we have to wait until the next release?

If all else is failing you can also attempt to revert back to 1.87.2 of VS Code which is confirmed to be working without the extension (Don't forget to disable automatic updates)
If the issue still persists, most likely you need to install the packages again in the root of your project where your package.json is located execute the command below and it will install all the packages required for mui if not already install and give an error if there are any:
npm install @mui/material @emotion/react @emotion/styled
(I accidentally did this a few times while pulling my original code and forgetting to reinstall the deps so some of my tests I realized were failing no matter what because the packages didn't exist in my project)

@chamalperera Does any IntelliSense work? Try opening an empty javascript or typescript file and triggering suggestions
Also did you test in the latest insiders build?

Hi @mjbvz & @lyricalcasanova,
I have updated the VS Code to the lastest release v1.88.1 which resolves the issue. I can confirm the auto imports are now working. I have tested it aftre uninstalling the TypeScript & JavaScript Nightly extension, which @lyricalcasanova suggested earlier as a workaround.
If VS Code is updated to v1.88.1, issue should be gone


add app, app bar and back button

fix #122
I also removed UsedTheme to use the existing ThemeMode

I have added a defaultStyle on the theme, the idea is to have a global one that you can merge/override later by providing a specific one. But it will need changes on all components that are currently using NeumorphicStyle, so I'll update those components on another PR to avoir a huge one. That will allow user to have less style to apply across their app.
I also need to add the boxShape on the app bar fields for the button to use that one automatically if specify like we talk earlier, but same I'll do it in another PR.
This one is good to me, let me know what you think

I have removed defaultStyle because I finally understand that all field was already on the theme ^^ but not sure those theme fields are used on all components by default, or I didn't understand how it works that's possible

I see you're doing a lot of changed in this PR, I'll wait until you tell me to review before opening it :)

Yes sorry for that :) I'll not touch it more except to fix your comments. So you can review it.
For now buttons under appBar need manual tweak to look good (basically need to remove button's padding), I plan on fixing that on another PR later by providing an AppBarTheme or something to override the default style when inside an app bar.

Ok :) I start review it so ! do you have created a sample ?

yes : main_home.dart

I think the sample should stay in light mode, my computer is in dark mode & it's ugly here :)
we'll add themeMode: ThemeMode.light, in the root app
then add
 Scaffold(
        backgroundColor: NeumorphicTheme.baseColor(context),

into playground, because the scaffold background color is wierd

if the appBar uses automaticallyImplyLeading we should add a way to provide an argument of type NeumorphicBoxShape to have for example a circle back button, without defining entierely the leading widget
appBar: NeumorphicAppBar(
     title: "my app",
     leadingBoxShape: NeumorphicBoxShape.circle(),
     ...

for example (i'm not sure of the arg name)

we'll add themeMode: ThemeMode.light, in the root app

Right I didn't test in dark mode ^^

Scaffold(
backgroundColor: NeumorphicTheme.baseColor(context),

There is no need as it's what is done under NeumorphicApp already :)

if the appBar uses automaticallyImplyLeading we should add a way to provide an argument of type NeumorphicBoxShape to have for example a circle back button, without defining entierely the leading widget

Yeah that's the non finished part I was talking about, the idea here is to have a NeumorphicAppBarTheme where you can configure a NeumorphicStyle and a shape and the buttons will retrieve those automatically. But I'll do that in another PR if you don't mind because there is a lot going on here I'm getting a bit lost ^^

Oh i forgot, thanks for this MR

No problem it was cool to do :)

I have pushed themeMode: ThemeMode.light, in the root app.
Do you need me to do anymore changes on this ?

Nope I try again & will merge this :)

I merge it :) If we have futher work we will do this on master or others PR


Better address field

closes https://github.com/dobtco/screendoor-v2/issues/1400

Nice.
ZIP Codes are U.S. only... pretty sure we can just say "Postal Code" for all other countries.


Update Used on: for app.kubernetes.io labels on 'Well-Known Labels, Annotations, and Taints' page

Page: https://kubernetes.io/docs/reference/labels-annotations-taints/
The app.kubernetes.io labels' Used on: fields added by https://github.com/kubernetes/website/pull/32923 could be clearer - they typically apply to Workload Resources as pointed out by @sftim in https://github.com/kubernetes/website/pull/32923#discussion_r850513602_ - I'll address this in a PR shortly, opening this issue to link.

How about adding a text note that this is usually used as a label on a workload object, or on something that an workload object directly depends on.
Originally posted by @sftim in https://github.com/kubernetes/website/pull/32923#discussion_r850513602

/assign

/reopen


fix: Prevent scrap from revealing the last-played card unless the last move was a 3

Now that the lastEvent object has been flattened into its individual properties, it's important that various consumers of lastEvent data be able to tell which type of move (change) the lastEvent was immediately rather than waiting for the change-specific logic to apply.
Additionally, consumers of lastEventCardChosen etc should specifically check if the lastEvent is a particular change i.e. was a three played vs a four in order to only display their content for the appropriate moves.

Issue number
Relevant issue number

Resolves #828

Please check the following

[ ] Do the tests still pass? (see Run the Tests)
[ ] Is the code formatted properly? (see Linting (Formatting))
For New Features:

[ ] Have tests been added to cover any new features or fixes?
[ ] Has the documentation been updated accordingly?

Please describe additional details for testing this change

Closing as dupe of #832


for rebase

Codecov Report

Merging #26 (4cce779) into rebase (ff29b02) will decrease coverage by 2.53%.
The diff coverage is 26.39%.

@@            Coverage Diff             @@
##           rebase      #26      +/-   ##
==========================================
- Coverage   80.96%   78.42%   -2.54%
==========================================
  Files         318      328      +10
  Lines       28072    29217    +1145
==========================================
+ Hits        22728    22913     +185
- Misses       5344     6304     +960

Impacted Files
Coverage Δ

espnet2/asr/encoder/rnn_encoder_mix.py
0.00% <0.00%> (ø)

espnet2/asr/encoder/transformer_encoder_mix.py
0.00% <0.00%> (ø)

espnet2/asr/encoder/vgg_rnn_encoder_mix.py
0.00% <0.00%> (ø)

espnet2/asr/espnet_enh_asr_model.py
0.00% <0.00%> (ø)

espnet2/asr/espnet_model_mix.py
0.00% <0.00%> (ø)

espnet2/bin/asr_mix_inference.py
0.00% <0.00%> (ø)

espnet2/bin/asr_mix_train.py
0.00% <0.00%> (ø)

espnet2/bin/enh_asr_inference.py
0.00% <0.00%> (ø)

espnet2/bin/enh_asr_train.py
0.00% <0.00%> (ø)

espnet2/bin/enh_inference.py
96.38% <ø> (ø)

... and 95 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff29b02...4cce779. Read the comment docs.


The organisation is an integer

While rebasing https://github.com/fishtown-analytics/dbt-spark/pull/85 against the latest master, I've encountered an issue with the connection:
root@f425d97e576a:/dbtlake# dbt docs generate
Running with dbt=0.17.0-rc2
Encountered an error while reading profiles:
  ERROR Runtime Error
  Credentials in profile "default", target "dev" invalid:<PHONE_NUMBER>234567 is not of type 'string'
Defined profiles:
 - default

With the following config (slightly modified of course):
 default:
   target: dev
   outputs:
     dev:
       method: http
       type: spark
       schema: fokko
       organization: "1285381041234567"
       host: "westeurope.azuredatabricks.net"
       port: 443
       token: "..."
       cluster: "..."
       connect_retries: 20
       connect_timeout: 60
       threads: 8

 config:
   send_anonymous_usage_stats: False

The big question here is, can an organization start with a zero 🤔

Hey @Fokko, check out this similar issue: https://github.com/fishtown-analytics/dbt/issues/2384
The resolution there was to introduce a special Jinja filter as_text (https://github.com/fishtown-analytics/dbt/pull/2395). I'm hopeful that would work for you here, and enable you to include a leading 0 as well.

I just checked, and this works at the latest master. However, I think it would be better to change it to an integer. the as_text solution feels a bit awkward.

I'm all for switching this to be an integer if we feel good about that going forward. I'm cognizant of the README note added by this older commit: https://github.com/fishtown-analytics/dbt-spark/commit/9ab1da326c48d9d674d31f8c0623bc6ec4bb0fdb. (I've since removed the full comment only to summarize it a bit more succinctly.) Have you ever seen a non-numeric organization ID, or one that requires a leading 0?
If you feel good about this change, could you update the "Configuring your profile" section of the README accordingly?

For a current project, we have many (20+) workspaces, and I haven't seen any with a leading zero, so I think we're safe on this one. The Python ints are of unlimited size, so I think we're safe there as well. @poidra02 any thoughts on this one?

@jtcohen6 Great work on the 0.17 release! Would it be possible to include this in 0.17 dbt-spark? :)

hey @Fokko - sorry for the delay in getting back to you.  I could get on board, but there are a couple things that I suggest we consider:

this org id is a field entirely owned by Databricks, and I haven't seen any guidance around how it's generated.  whether it will stay the same now or into the future is an open question.  the original choice to keep it a string was to avoid chasing any changes given we were reverse engineering it.
it looks like Azure Databricks will keep the workspace IDs in the URL, but they're changing the URL to be region independent (which is a good thing): [https://docs.microsoft.com/en-us/azure/databricks/workspace/migrate-workspace-urls].  while there's no timing the old URL will go away, if we do make changes, we should consider moving to the new URL format.

if we feel that the risk of chasing changing workspace ids/orgs is small (or the level of effort to change and test that is small) then I could get behind the change.  though the choice to make it a strings was deliberate to avoid that obviously, and I'm not sure I'm understanding just yet what challenge a string is introducing here to weigh the trade off myself.
but happy to talk about it further...

this org id is a field entirely owned by Databricks, and I haven't seen any guidance around how it's generated.

I agree with this hesitation. I asked some folks from Databricks with whom I've been in contact whether an organization ID can ever contain non-numeric characters or a leading 0. I'm waiting on a response from them.
Given that this is functional (albeit clunky) with organization ID as a string, and that switching it to be an integer has potential risk—likelihood unknown—of being a breaking change for some future accounts, I'm inclined to keep it as a string for now until we receive more explicit direction from Databricks.

This will be resolved by the change we made to native env rendering in v0.17.1 (discourse)


Proper slugify

More languages, anyone? ...anyone?

Hi, thanks for reaching, but this slugify is just plain slugify not for advance usage, but if you can write something better I am looking forward to integrating it. I just do not want to introduce 3rd party libraries.

just, please, add more rules

:+1: ok, I will add them later tomorrow maybe and will be released with 1.1.0 on Friday. But those rules will get not to slugiffier but to the latinizer :)


Players Don't Die in the void

When a player jumps into the void, they don't die.
Plugin List:
(10)
Essentials, FAWE, LuckPerms, MissileWarsX, PlaceholderAPI, ProtocolLib, ServerSelectorX-Connector, ViaBackwards, ViaRewind, ViaVersion

Fixed in 260f947!
Thanks for the report :)


Async reader may return wrong results on MacOS, Linux, WSL, Docker

Describe the bug
Sometimes the query may return wrong results under high load, on Non-Windows clients.
To reproduce

.NET Core
MSSQL instance
Client is not running on Windows (reproduced on MacOS or Linux)
Query uses async reader methods
High load that saturates the connection pool and causes timeouts.

Bug is seen more often when MultipleActiveResultSets=True (MARS) is in the connection string. It might be prevented by disabling MARS (unclear at this point).
The issue is hard to reproduce and rare, therefore this project was created with the code that attempts to simulate the situation.
This program starts 2000 concurrent connections that runs select @Id as Id
statement. Each Id is different, and the query result should always return
the id that was queried, yet under certain circumstances that's not the case. It looks like
sometimes the reader returns the stale data from random previous connection that experienced the timeout.
There are more details in the linked project.
Network latency can be more easily simulated using this method here: https://github.com/trainline/SqlClient659
Expected behavior
Under no circumstances a select statement should return a different result.
Further technical details
Microsoft.Data.SqlClient version: 4.8.1 or earlier
.NET target: netcoreapp2.0 - netcoreapp3.1
SQL Server version: SQL Server Web Edition 13.00.5426.0.v1, SQL Server Express Edition 14.00.3281.6.v1
Operating system: macOS 10.15.5, Docker container
Workaround
Do not use this SQL client on linux/docker.
Always check the id of the returned result and retry or crash if it is wrong.
To decrease the likelihood of this or maybe prevent it (unclear at this point), do not use MARS feature, if you are using .NET Core with MSSQL database, ensure that the connection string does not contain MultipleActiveResultSets=True.

@Nercury thanks for reaching out. Issue #422 is similar to the problem you have and is under investigation.

Hi @Nercury,
Thank you for producing the sample app and describing the issue well.
I ran your sample app multiple times at a virtual machine running Ubuntu 18.0, .Net core 3.1, and SQL Server 19 with MARS enabled and connection pool equals 100. I didn't get any Wrong results as you expected.
Do you have any comments?

I didn't get any Wrong results as you expected.
Do you have any comments?

Yes, this matches my experience. As I mentioned, the issue seems to be related to timeouts, and they can not happen if the DB server and the code is on the same machine, and the query is doing basically nothing.
Try to introduce some latency between the db server and the application.
I reproduced it using a bit crude approach, by running DB server in cloud and the application on my laptop with a router in another room (so there is some considerable latency between the two). Maybe there are tools to do it a bit more cleanly.
In production environment the AWS starts throttling read operations when they go above allowed IOPS limit, that's how it originally happened when running on AWS.

Thanks for checking it out. I am not sure I completely understand what you mean by "added a local index inside the QueryAndVerify method to keep the index parameter value and used this local variable instead of that parameter inside the function body", because the int index is already local to the QueryAndVerify function. Can you provide a changeset (in the form of PR) so that I am sure I am doing the same thing?

While I was preparing the requested changes I found out it is happening again. Maybe the network conditions simultaneously had affected my test by the last change.
The issue intermittently happens either with System.Data.Sqlclient & Microsoft.Data.SqlClient when ExecuteReaderAsync blends with exceptions.

In between investigation, I found these mitigations besides a workaround that has mentioned as disabling the MARS:

Set Pooling = false
Removing a connection that encountered an exception from the connection pool: SqlConnection.ClearPool(cn)
Attempting to execute a SQL command without attaching a transaction to the command (remove the BeginTransaction & CommitAsync) :

var command = new SqlCommand(sql, cn);
command.Transaction = null;

Using ExecuteScalarAsync instead of ExecuteReaderAsync

@Nercury Can you give them a try one by one?

Set Pooling = false
MultipleActiveResultSets=True;Pooling=false; painfully slow and many timeouts, but no wrong results, which is good
Removing the connection that encountered an exception
            SqlConnection cnp = null;
            try
            {
                using (var cn = new SqlConnection(connectionString))
                {
                    cnp = cn;

                    // ...
                }
            }
            catch (Exception e)
            {
                if (cnp != null)
                {
                    SqlConnection.ClearPool(cnp);
                }

                // ...
            }

Got wrong results on a third attempt.
Attempting to execute a SQL command without attaching a transaction to the command
This was attempted in production when it was still unclear if this is a MARS issue. It makes wrong results more rare, but they still happen. I was not able to reproduce it in this test project with a simplistic select statement.
Using ExecuteScalarAsync instead of ExecuteReaderAsync
Tried it 5 times and wrong results did not happen.
@DavoudEshtehari I hope this helps.

Same bug experienced when MARS is disabled:
We are experiencing a similar problem in our code base, although we have MARS disabled explicitly in the connection string and still see the bug.
We run this method across multiple incoming requests from an API Controller:
public async Task<T> GetAsync(string id)
        {
            var sql = $"SELECT Data FROM { _configuration.TableName } {(_configuration.InMemory ? "WITH (SNAPSHOT)" : "")} WHERE Id=@id";

            using (var connection = _connectionFactory.Create())
            {
                var record = await connection.QueryAsync<string>(sql, new { id }).ConfigureAwait(false);

                var data = record.FirstOrDefault();

                return data == null ? default(T) : _serializer.Deserialize<T>(data);
            }
        }

The connectionFactory simply returns a new SqlConnection:
return new SqlConnection(_connectionString);
id is a GUID
data will be an encrypted blob of data that decrypts to a JSON payload.
_serializer handles the decryption (if needed) then serialisation.
Things will run normally, then a single instance will experience a SQL Exception - previously there was an unexpected (and denied) request to perform a SHOWPLAN, sometimes it's a SQL Timeout exception.  Following that exception, serialisation bugs will continue to appear on that instance until it is taken out of service.
The source of the serialisation failure is that the value stored in data is an id from an earlier request, rather than the expected value stored on the SQL DB.  We've had this happen both on the in-memory DB and without it.
Once the instance has been out of service for a while, subsequent attempts to access the DB via this method complete successfully, so it seems to have the ability to resolve itself.  Even when the instance is misbehaving, it appears that not all requests get jumbled up, as some manage to succeed.
Previously this service was hosted on Windows without any issues.  Since the port to Linux this problem crops up every 7-30 days or so.  We've also seen it on more than one service (the above code is in a shared library).  This bug seems to be the closest thing to what we are experiencing, except that we have explicitly disabled MARS in the connection string and are still seeing it from time to time.
Libraries/Infrastructure involved:
NetCore 3.1
Dapper 2.0.35
Microsoft.Data.SqlClient 2.0.1
This is being run on an AWS T3 instance hosting Ubuntu 18.04
We are going to attempt changing the call from QueryAsync to ExecuteScalarAsync and are attempting to capture some more information to help us debug the issue.  If the change to ExecuteScalarAsync resolves the problem for at least a month or so then I'll add an extra comment here indicating that it's a possible fix.

Hi @Nercury @gyphysics
Could you please try your use-cases with PR #796 as I tried to run repro provided and am able to see issue fixed.
Kindly confirm.

This issue remained unfixed for now, due to colliding changes in different async paths.

@cheenamalhotra All right, it makes sense to wait until all fixes are in.

Following on from @gyphysics (some company, different team):
Upgraded to 2.1.0 and still seeing this issue in multiple services, all on Linux.
Definitely no MARS in connection string (though we haven't actively disabled it).
We are using Dapper and will try to remove it from one project to prove it's not related.
In the meantime we're using guard clauses to prevent wrong data being used
if (!thingId.Equals(thing.Id))
{
    throw new ContextConflictException($"Asked for thing [{thingId}], got thing [{thing.Id}]");
}

We have also seen some problems on INSERT, where no exception was raised, but checking later the data was not written to the table. There were gaps in the IDENTITY column, suggesting a possible silent transaction rollback (exception didn't come back to originating async context?). Possibly unrelated, but seems close enough to link not here.

I have just set up a test case using @Nercury 's test program, deliberately creating network latency towards the sql server using iptables
sudo iptables -A INPUT -p tcp --sport 1433 -m statistic --mode random --probability 0.1 -j DROP

This is with a connection string with no MARS, and I was able to get a wrong result.
Connectionstring:
Connection string Server=SERVERADDRESS;Database=DATABASE;Integrated Security=False;User Id=USERNAME;Password=PASSWORD!;MultiSubnetFailover=true;Max Pool Size=100

Hello!
I've created a new repro putting together items of this discussion using docker-compose so it can be easily execute locally also simulating network failures.
I can confirm the problem can be reproduced even without MARS, although it is less frequent.

.NET version
SQL Server Version
MARS
Failures rate

.NET Core 3.1
2017 Enterprise
Yes
High

.NET Core 3.1
2017 Enterprise
No
Low

.NET 5.0
2017 Enterprise
Yes
High

.NET 5.0
2017 Enterprise
No
Low

.NET Core 3.1
2019 Enterprise
Yes
High

.NET Core 3.1
2019 Enterprise
No
Low

.NET 5.0
2019 Enterprise
Yes
High

.NET 5.0
2019 Enterprise
No
Low

@cheenamalhotra Any suggestion on how to try to pinpoint the origin of the problem? Is there a diagnostic source available in the SqlClient?

Hi; I see a mention of Dapper here. It is entirely possible that this is a Dapper issue, using an unintended and unsupported feature that only kinda works by accident (and only when MARS is enabled). Ultimately, "MARS enabled" is not the same as "performing evilness via overlapped unawaited operations", and I will be the first to say: any time you don't immediately await something, you're effectively doing the same thing as multi-threading, and race conditions become obvious.
So; if there's a problem, and this API doesn't work (or even claim to work) the way in which Dapper has used (abused) it: maybe we should just deprecate that feature from Dapper - or at least see if we can re-work it to be less evil.
(I'm one of the Dapper maintainers)

The system where the bug was discovered was using Dapper too. But the bug remained after the code was rewritten without Dapper. We will know for sure once the underlying issue is fixed.

@Nercury is it worth updating the issue title as it is not only happening with MARS enabled?

Both repros were tracking together queries returning invalid result with queries returning no result (but no exception).
I've now updated https://github.com/trainline/SqlClient659 to track those separately.
Without MARS I am not able to get invalid results with this repro so I will try to create a new one.
However not returning records where it should and not throwing an exception is a bug as well.

@alefranz let me know how it goes, because I have removed the MARS as a possible workaround now based on your previous report. I should add it back if not using MARS does not return wrong result. Returning no results is not that bad (but not good either).

Posting latest update for investigations so far:
For scenario:

This is what is happening:

We are receiving Data on pooled connection AFTER connection is sent back to pool, and that data gets added to Read Packet Queue in SNI Mars Handle, from where it's picked up in next read cycle for next query, leading to "wrong data".
Lack of effective tracing is making the investigations slow, but we're working on improving and adding more meaningful traces to entire driver that can help track end-to-end connection activity in complex applications.
Thanks for your patience!

Posting latest update for investigations so far:
For scenario:

This is what is happening:

We are receiving Data on pooled connection AFTER connection is sent back to pool, and that data gets added to Read Packet Queue in SNI Mars Handle, from where it's picked up in next read cycle for next query, leading to "wrong data".
Lack of effective tracing is making the investigations slow, but we're working on improving and adding more meaningful traces to entire driver that can help track end-to-end connection activity in complex applications.
Thanks for your patience!

@cheenamalhotra continuing here from out discussion of this on an unrelated PR

We should be cleaning up MarsHandle or packets which are still connected to StateObj on the connection in pooling.
I also always hit the Debug assert in SNIPacket's destructor, and we have a lot of unreleased packets, maybe it's related?

Hitting the finalizer on SNIPackets regularly is worrying. At the top of the file there's this comment:
// #define TRACE_HISTORY // this is used for advanced debugging when you need to trace the entire lifetime of a single packet, be very careful with it

try enabling that and see if you can replicate the problem, if you can then you'll be able to look at the history of the packet to identify what allocated it and then with some work you can trace back to the thing that should have, but didn't, release it.
Every managed packet should be allocated and released through the pool and the pool forces the data to be cleared by calling Release() on it. If the packet is dropped to the GC it can't be recycled and the array rented from the pool would be lost as well so it shouldn't be possible for a dropped packet to be re-used.
The packet could be partially reconstructed while the disconnect happens on another thread. The packet smux header contains the connection id used to find the connection to dispatch it to. There's a dictionary of SNIMarsHandle in SNIMarsConnection (iirc) and that's where the dispatch happens so at that point the connection must have the correct handler in the dictionary to find it and send the packet to it. If there's a queue on the mars handle then that should be cleared, though really as I understand the TDS protocol the server won't send any packets after an end message so it'd be weird for there to be any pending.

@cheenamalhotra continuing here from out discussion of this on an unrelated PR

We should be cleaning up MarsHandle or packets which are still connected to StateObj on the connection in pooling.
I also always hit the Debug assert in SNIPacket's destructor, and we have a lot of unreleased packets, maybe it's related?

Hitting the finalizer on SNIPackets regularly is worrying. At the top of the file there's this comment:
// #define TRACE_HISTORY // this is used for advanced debugging when you need to trace the entire lifetime of a single packet, be very careful with it

try enabling that and see if you can replicate the problem, if you can then you'll be able to look at the history of the packet to identify what allocated it and then with some work you can trace back to the thing that should have, but didn't, release it.
Every managed packet should be allocated and released through the pool and the pool forces the data to be cleared by calling Release() on it. If the packet is dropped to the GC it can't be recycled and the array rented from the pool would be lost as well so it shouldn't be possible for a dropped packet to be re-used.
The packet could be partially reconstructed while the disconnect happens on another thread. The packet smux header contains the connection id used to find the connection to dispatch it to. There's a dictionary of SNIMarsHandle in SNIMarsConnection (iirc) and that's where the dispatch happens so at that point the connection must have the correct handler in the dictionary to find it and send the packet to it. If there's a queue on the mars handle then that should be cleared, though really as I understand the TDS protocol the server won't send any packets after an end message so it'd be weird for there to be any pending.

Here's a typical example of what happens:

[Left column is time in milliseconds]
If you see carefully they have data left of 67 bytes - which is not consumed or requested before. The 2 times above 54 and 468 bytes were read and consumed from this packet.
For every case where we get "Execution Timeout expired" errors, we have these finalizers getting hit.

These unreleased packets also lead to packet Ids reaching 250/300 sometimes, I donno what is the max we can handle if max pool size is 100 (as default).

Here's a typical example of what happens:

[Left column is time in milliseconds]
If you see carefully they have data left of 67 bytes - which is not consumed or requested before. The 2 times above 54 and 468 bytes were read and consumed from this packet.
For every case where we get "Execution Timeout expired" errors, we have these finalizers getting hit.

These unreleased packets also lead to packet Ids reaching 250/300 sometimes, I donno what is the max we can handle if max pool size is 100 (as default).

If there is no pooled packet available a new packet is created, similarly if the pool is full and one is returned it is dropped to the GC. Not being able to get or return a packet to the pool is not a problem and is inefficient but not an error. The only time a packet getting to GC is a problem when it has a payload in it which is why there's the null check in the finalizer, getting finalized with data means something failed to return a live packet to the pool which doesn't cause an error but also shouldn't happen because it makes the pool ineffective.
The packet id is per-process so seeing it get up to 300 really isn't too bad. The default packet pool per physical connection is 4 which is the maximum number that a non-mars connection can have in use at any one time as far as I can tell. 1 Send, 1 Cancel, 1 Received, 1 Receiving. A densely used mars connection multiplexed on top of the physical connection could easily need more packets than that.
If you want to entirely eliminate the pool just comment out the pool TryGet and Return calls in SNIPhysicalHandle and change the array pool allocation to a simple new[] call, that'll means no packets are recycled. If the error stops occurring then it was connected to recycling, if it doesn't you know you're safe to ignore the pool and look elsewhere.
I'm very curious about what those 67 remaining bytes are in the dropped packets. Is it possible to grab the entire packet buffer on one of them and see if we can determine what is being taken and what the 67 bytes left are?

If there is no pooled packet available a new packet is created, similarly if the pool is full and one is returned it is dropped to the GC. Not being able to get or return a packet to the pool is not a problem and is inefficient but not an error. The only time a packet getting to GC is a problem when it has a payload in it which is why there's the null check in the finalizer, getting finalized with data means something failed to return a live packet to the pool which doesn't cause an error but also shouldn't happen because it makes the pool ineffective.
The packet id is per-process so seeing it get up to 300 really isn't too bad. The default packet pool per physical connection is 4 which is the maximum number that a non-mars connection can have in use at any one time as far as I can tell. 1 Send, 1 Cancel, 1 Received, 1 Receiving. A densely used mars connection multiplexed on top of the physical connection could easily need more packets than that.
If you want to entirely eliminate the pool just comment out the pool TryGet and Return calls in SNIPhysicalHandle and change the array pool allocation to a simple new[] call, that'll means no packets are recycled. If the error stops occurring then it was connected to recycling, if it doesn't you know you're safe to ignore the pool and look elsewhere.
I'm very curious about what those 67 remaining bytes are in the dropped packets. Is it possible to grab the entire packet buffer on one of them and see if we can determine what is being taken and what the 67 bytes left are?

Hi Everyone!
Providing this week's progress so far:
We have managed to capture and study all detailed traces for this behavior, including Event Source, SQL Server Profiler, and Wireshark to strip open full story which I'll detail below:
Fact we know:
When an exception occurs on Client side, client shall send an Attention packet to the server to immediately stop processing further data and to not send any further data. On the client side, sending attention can mean something too, as for example ensuring transaction is rollbacked and data at rest is cleared properly or whether connection should be killed and not reused. It depends on when and how we receive an exception and send attention.
Problem we see here:
We have 2 situations going on when exception "Execution Timeout Expired" is to be received, and it looking like there's a race condition between Transaction Commit and Attention when Read timeout occurs in SNIMarsHandle.

[GOOD FLOW] Begin Trans > Run Query > Sends Attention > Trans Rollback

Connection throws exception and terminates safely.

[BAD FLOW] Begin Trans > Run Query > Trans Commit > Sends Attention

Query succeeds in this case as Commit succeeded but when connection is reused from the pool, the next query throws "Execution Timeout Expired" exception followed by a rollback but data is kept in store and not cleared up > leading to wrong data flow.

We know this is related to client side timeout that occurs during Read as Read operations are blocked due to thread starvation, so even though server sends data on time, client does not read it on time. But, even if we manage to fix the thread starvation issue, that is not the solution to this particular issue, as the BAD FLOW is the real problem and can occur in any similar situation.
Will provide more updates as we continue to study our traces to dig deeper and find a solution.

Hi Everyone!
Providing this week's progress so far:
We have managed to capture and study all detailed traces for this behavior, including Event Source, SQL Server Profiler, and Wireshark to strip open full story which I'll detail below:
Fact we know:
When an exception occurs on Client side, client shall send an Attention packet to the server to immediately stop processing further data and to not send any further data. On the client side, sending attention can mean something too, as for example ensuring transaction is rollbacked and data at rest is cleared properly or whether connection should be killed and not reused. It depends on when and how we receive an exception and send attention.
Problem we see here:
We have 2 situations going on when exception "Execution Timeout Expired" is to be received, and it looking like there's a race condition between Transaction Commit and Attention when Read timeout occurs in SNIMarsHandle.

[GOOD FLOW] Begin Trans > Run Query > Sends Attention > Trans Rollback

Connection throws exception and terminates safely.

[BAD FLOW] Begin Trans > Run Query > Trans Commit > Sends Attention

Query succeeds in this case as Commit succeeded but when connection is reused from the pool, the next query throws "Execution Timeout Expired" exception followed by a rollback but data is kept in store and not cleared up > leading to wrong data flow.

We know this is related to client side timeout that occurs during Read as Read operations are blocked due to thread starvation, so even though server sends data on time, client does not read it on time. But, even if we manage to fix the thread starvation issue, that is not the solution to this particular issue, as the BAD FLOW is the real problem and can occur in any similar situation.
Will provide more updates as we continue to study our traces to dig deeper and find a solution.

@Wraith2
I will come back to packet recycling and cleaning again, it may not be related here, but we will fix that too eventually.
Let's try and find possible race condition scenarios in Commit and Attention signals when timeout exception occurs in SNIMarsHandle, as it feels we might be able to restrict the BAD FLOW by ensuring we also wait for Error handling to complete and only pass data to user if there are no exceptions as here we have data as well as exception coming in from timeout. We don't want exception to flow down to next connection.

@Wraith2
I will come back to packet recycling and cleaning again, it may not be related here, but we will fix that too eventually.
Let's try and find possible race condition scenarios in Commit and Attention signals when timeout exception occurs in SNIMarsHandle, as it feels we might be able to restrict the BAD FLOW by ensuring we also wait for Error handling to complete and only pass data to user if there are no exceptions as here we have data as well as exception coming in from timeout. We don't want exception to flow down to next connection.

Hi everyone!
Thank you for your patience, we seem to have a fix available to try.
Would request you to please try out the PR https://github.com/dotnet/SqlClient/pull/894 and let us know!
Download Build CI NuGet

Hi everyone!
Thank you for your patience, we seem to have a fix available to try.
Would request you to please try out the PR https://github.com/dotnet/SqlClient/pull/894 and let us know!
Download Build CI NuGet

Hi Cheena,
great news! thanks for the update!
Is this fix on a codepath used only when MARS is enabled? or could this also fix the harder ro reproduce incorrect behaviour without MARS?

Hi Cheena,
great news! thanks for the update!
Is this fix on a codepath used only when MARS is enabled? or could this also fix the harder ro reproduce incorrect behaviour without MARS?

This fix applies to SqlCommand layer and to Begin and End methods of Execute Reader Async so Mars was not a factor, and neither is Managed SNI.
We also know ExecuteXmlReaderAsync also has same issue, I will apply the same fix there on Monday.

This fix applies to SqlCommand layer and to Begin and End methods of Execute Reader Async so Mars was not a factor, and neither is Managed SNI.
We also know ExecuteXmlReaderAsync also has same issue, I will apply the same fix there on Monday.

That's great news, I was really struggling to find a good repro for the non-MARS scenario! I will start testing this build next week.

That's great news, I was really struggling to find a good repro for the non-MARS scenario! I will start testing this build next week.

I now have a consistent repro of this issue to rule out inconsistencies, as it's always been difficult to consistently reproduce this. The issue is exaggerated with Threadpool exhaustion and instead of exhausting resources everytime, we are able to put a small delay on one of the delegates we know are causing problem here and are able to reproduce the wrong data use case.
Link to Repro
To run this repro, you'd need to reference driver source project and update OnTimeout function with a small delay period of 10 seconds, so it looks like this:
private void OnTimeout(object state)
    {
       Thread.Sleep(10000); // Include this line
       if (!_internalTimeout)
       {
           ...

The output captured is as under:

While we're trying our best to fix this by various methods, things are very flaky. and design is highly complex. Being able to reproduce consistently does give some hope. But there may be more cases where wrong data could be possible. Will provide more update as we identify more scenarios and solve this one.

Hi Everyone!
We'd like to request community validations on the fix from PR #906.
We've captured the root cause and fixed the issue in the core parser layer.
Please give the below NuGet package a try and send us your feedback:
Microsoft.Data.SqlClient.2.1.0-pull-89d6662.21043.7

Anyone?

We're very keen @Wraith2, just need to find a moment having spent engineering time moving linux services back to windows to mitigate the issue.

@Wraith2 , I've spend some time testing the pull request with my own repro-case.
Earlier on, I was able to get the incorrect results with the general version 2.1.1. Consistent wrong results over the different attempts.
I've also tried Microsoft.Data.SqlClient.2.1.0-pull-3bdbb71.21048.9. This version never gave any incorrect results, so it looks like this fix solves the problem.

Hi @cheenamalhotra , @Wraith2 ,
as you know reproducing the issue, especially without MARS, on real world applications is extremely challenging, so it is hard to prove the fix.
I can confirm however that it doesn't introduce any appreciable regression.
Thanks for the great effort on this!

Some more background information about my earlier repro-case, for reference.
The reprocase is ran in Docker in a Microsoft Azure environment, on a Linux App Service Plan (P1V2) and SQL Server Azure (S0)
The base dockerfile is mcr.microsoft.com/dotnet/core/aspnet:3.1.8-bionic, so non-windows environment on .Net core 3.1
Connectionstring didn’t include any MARS-references, so all is used in a non-MARS environment.
The base situation is a full async operation, so openasync, executereaderasync, and readasync. I've never used transactions and a low commandtimeout (1 second) to enforce problems as much as possible..
The SQL Statement I used is “select @Id as Id”
The issue of the wrong results only occurred when a “Timeout expired exception” appeared. Scaling up / down the database also causes exceptions, but didn’t trigger any wrong results (or at least, I couldn’t manage to do so).
The results:
Base scenario: use of openasync, executereaderasync, and readasync, high load situation with multiple threads, multiple requests.
When things went wrong, they kept going wrong in a consistent matter. For instance, suppose there are three users:
User 1: Select * FROM User where UserId = 1
User 2: Select * FROM User where UserId = 2
User 3: Select * FROM User where UserId = 3
User 1 caused an exception. User 2 received the results from User 1, User 3 received the results from User 2, and so on.
Different requests got each other’s results, even after some time. More and more SQL statements results in wrong results, even after some time. The connection pool become corrupted, and stayed corrupted.
The only solution was to restart the web app.
Mitigations:
I tried a number of mitigations.
Attempt 1: Use ConfigureAwait(false) for openasync, executereaderasync, readasync.
No effect, wrong results keep appearing like in the base scenario.
Attempt 2: Use .Open() instead of OpenAsync()
Exception are occurring more often (due to Timeout expired), but only a few wrong results.  Furthermore, the wrong results weren’t consistly wrong. New requests could be ok, so no restart of the application was required. So it looked like some connetions became corrupt, but somehow, it didn't influence the connectionpool for later connections.
Performance was, however, reduced significly.
Results with the patch:
In all situation, no wrong results have occurred.
In summary:
The only mitigation I could find, was to:

Reduce load, so the chance of Timeout Expired exception was reduced
Use Open instead of OpenAsync, so the chance of wrong results was reduced as well, and wrong results could ‘disappear’ in future requests. I’ve had some luck using an O/R mapper, so I could change to use Open instead of OpenAsync with just a couple of lines of code.

The patch solves the situation of wrong results appearing. So looking forward for this hotfix to be released... ;)

The underlying bug that was found was caused by threadpool exhaustion. It is a bug in the TdsParser layer which is above the network layer so it was possible to cause the problem in any version of the driver but was much more likely in managed (linux, macos) mode and with mars due to the way they use threads.
The issue itself was down to timing of internal cancellation messages and how they were implemented which made it possible for a timeout to arrive after a result had returned, or even worse after another query had begun. What the PR does is harden the timeout handling by adding synchronous timeout checks and carefully tracking the origin of the async timeout so that if it doesn't match the executing query it is discarded. This results in more accurate timeouts in high contention scenarios and prevention of cross query timeouts which were causing a strange mismatch in the driver.

The latest Microsoft.Data.SqlClient v2.1.2 has been released containing fix for this issue.
Closing as resolved.

Thank you all for the brilliant work to track down this elusive bug!

@cheenamalhotra / @Wraith2: We are experiencing a similar issue using versions 1.1.x. Is this issue applicable to v2.1 only?

It could occur in all versions of SqlClient prior to the fix in 2.1.1

@Wraith2 Thank you.

Hi @andrehaupt
v1.1.4 has been released to contain the fix too. Please upgrade if you're still using the v1.1 series.

Hi @cheenamalhotra, @Wraith2,
Would you be able to clarify the following points please ?

was this bug also affecting .NET 5 apps running on Windows ? The title of this issue seems to say that it should not happen on Windows, but looking at the fix in #906 it seems like it also affects Windows, and even .NET Framework apps, isn't it ?

is System.Data.SqlClient also affected - again in the context of a .NET 5 app running on Windows ?

I'm trying to analyze a similar issue observed with a .NET 5 app running on Windows with System.Data.SqlClient (and .NET runtime 5.0.2), so these clarifications would greatly help, thanks!

was this bug also affecting .NET 5 apps running on Windows ?

The problem could affect all versions of the library running on any version of .net. It was more likely on the managed driver but the root cause was shared. It was a timing issue.

is System.Data.SqlClient also affected

Yes, every version of SqlClient prior to the fix could be affected.
It doesn't look like the fix was backported to corefx so the System version retains the ability to encounter the problem https://github.com/dotnet/corefx/pulls?q=is%3Apr+is%3Aclosed+label%3Aarea-System.Data.SqlClient I don't know whether servicing was considered but given the rarity of the problem and the difficulty of the fix i can see why it wouldn't be be an obvious choice to push back to an LTS release.


fix(reducer): ensure loading state is in sync

If multiple cases of useFetchye with the same key being rendered in the same cycle state would not be updated across these instances. This resulted in multiple fetch calls being made for the same resource.

:tada: This PR is included in version 1.0.0-beta.2 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


I27 prepare phe inference

Implement initial functionality for parameter inference of the PHE model.
Closes #27

Codecov Report

Merging #30 (5eb9f03) into main (84aa8a7) will decrease coverage by 15.50%.
The diff coverage is 100.00%.

:exclamation: Current head 5eb9f03 differs from pull request most recent head bda487a. Consider uploading reports for the commit bda487a to get more accurate results

@@             Coverage Diff              @@
##              main      #30       +/-   ##
============================================
- Coverage   100.00%   84.49%   -15.51%
============================================
  Files            4        5        +1
  Lines          563      703      +140
============================================
+ Hits           563      594       +31
- Misses           0      109      +109

Impacted Files
Coverage Δ

epimodels/inference/phe_inference.py
19.20% <ø> (ø)

epimodels/phe_model.py
97.43% <ø> (-2.57%)
:arrow_down:

epimodels/__init__.py
100.00% <100.00%> (ø)

epimodels/_setup_matrices.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84aa8a7...bda487a. Read the comment docs.


Added filtered events tables to the analytics page

closes TryGhost/Team#2087

Will write tests here https://github.com/TryGhost/Team/issues/2128


q# not a language option for .net core

Describe the bug

Followed all the steps on the intro page  https://docs.microsoft.com/en-gb/quantum/quickstart?view=qsharp-preview&tabs=tabid-vscode

Tried to run $dotnet new console -lang Q# --output Bell

Results in an error: "Error: invalid aprameters(s): -lang Q#

To Reproduce
follow steps above
Expected behavior
should create directory includig q-files
System information

OS: iOw
Browser Chorme

Same as issue #116...


From: name and email

Is it possible to include a name in the from field of submission emails?
I tried {name}<{email}> and {name} <{email}> (with and without a space before the gt) with no success. Of course I have a name field in the form.
p.s. – thank you for making this and offering it to others.

Hi @kirkroberts; you're now able to optionally use the format Name <email> in the from column :-)

That's great, thanks! I'll give it a spin soon.


AccountingScope multiple cn query

Laravel Version: 5.4
Adldap2-Laravel Version: 7.0.*
PHP Version: 7.2
LDAP Type: ActiveDirectory

Description:
This is my accounting scope. I'm trying to allow users to login that are only in these 3 CNs, but laravel says "These credentials do not match our records."
public function apply(Builder $query)
    {
           $accountingY = 'ou=Gruppi,ou=Utenti,dc=company';
           $query->whereMemberOf('cn=website_user'.$accountingY)->orWhereMemberOf('cn=website_admins'.$accountingY)->orWhereMemberOf('cn=website_editors'.$accountingY);
    }

Steps To Reproduce:
I have tried this
 $accountingX = '(&(|(memberOf=CN=website_users)(memberOf=CN=website_admins)(memberOf=CN=website_editors))(OU=Gruppi,OU=Utenti,DC=company))';
$query->where($accountingX);

But I get an exception.
I have tried also this:
$accountingY = 'ou=Gruppi,ou=Utenti,dc=company';
        $accountingA = 'CN=website_users';
        $accountingB = 'CN=website_admins';
        $accountingC = 'CN=website_editors';
        $query->whereMemberOf($accountingY)->andFilter(function($queryx) use($accountingA,$accountingB,$accountingC){
            $queryx->whereMemberOf($accountingA)->orWhereMemberOf($accountingB)->orWhereMemberOf($accountingC);
        });

But no luck :(
Is there a way to apply multiple filters in the AccountingScope? or it's not the right place to do it?
Here is adldap.php config:
'connections' => [

        'default' => [

            'auto_connect' => true,

            'connection' => Adldap\Connections\Ldap::class,

            'schema' => Adldap\Schemas\ActiveDirectory::class,

            'connection_settings' => [

                'account_prefix' => env('ADLDAP_ACCOUNT_PREFIX', ''),

                'account_suffix' => env('ADLDAP_ACCOUNT_SUFFIX', ''),

                'domain_controllers' => explode(' ', env('ADLDAP_CONTROLLERS', '')),

                'port' => env('ADLDAP_PORT', 636),

                'timeout' => env('ADLDAP_TIMEOUT', 5),

                'base_dn' => 'ou=Utenti,dc=company,dc=local',

                'admin_account_suffix' => env('ADLDAP_ADMIN_ACCOUNT_SUFFIX', ''),

                'admin_username' => env('ADLDAP_ADMIN_USERNAME', ''),
                'admin_password' => env('ADLDAP_ADMIN_PASSWORD', ''),

                'follow_referrals' => false,

                'use_ssl' => true,
                'use_tls' => false,
                'custom_options' => [
                    // See: http://php.net/ldap_set_option
                    LDAP_OPT_X_TLS_REQUIRE_CERT => LDAP_OPT_X_TLS_NEVER
                ]
            ],

        ],

    ],

I should mention that without the scope it works, but there are more users that should not have access. For now users inside our Active Dir. can in theory login, but they will be logged out if are not inside the 3 groups, which is kind of dull, I don't like it and neither does my boss

Maybe a copy/paste problem, but...
Is not missing a comma in $accountingY? When you join it in whereMemberOf, the group dn it´snt correct.
Your example:
$accountingY = 'ou=Gruppi,ou=Utenti,dc=company';
$query->whereMemberOf('cn=website_user'.$accountingY)->orWhereMemberOf('cn=website_admins'.$accountingY)->orWhereMemberOf('cn=website_editors'.$accountingY);

//RESULT: 'cn=website_userou=Gruppi,ou=Utenti,dc=company'

//WITH THE COMMA
$accountingY = ',ou=Gruppi,ou=Utenti,dc=company';
$query->whereMemberOf('cn=website_user'.$accountingY)->orWhereMemberOf('cn=website_admins'.$accountingY)->orWhereMemberOf('cn=website_editors'.$accountingY);

//RESULT: 'cn=website_user,ou=Gruppi,ou=Utenti,dc=company'

Thank you for the reply.
Yes it was a copy paste error, sorry. Before it was like this:
       $accountingA = 'cn=website_users,ou=Gruppi,ou=Utenti,dc=company';
       $accountingB = 'cn=website_admins,ou=Gruppi,ou=Utenti,dc=company';
       $accountingC = 'cn=website_editors,ou=Gruppi,ou=Utenti,dc=company';
       $query->whereMemberOf($accountingA)->orWhereMemberOf($accountingB)->orWhereMemberOf($accountingC);

I wrote it like that for better understanding
It still says These credentials do not match our records.
Let me explain. Before we had everyone that had any kind of access to the website inside website_users and I only had to do:
      $accounting = 'cn=website_users,ou=Gruppi,ou=Utenti,dc=company';
      $query->whereMemberOf($accounting);

So for example if John Doe is an "editor" he would be inside the website_users and website_editors group. But that didn't make sense to us. Funny thing is: if I do this:
$accountingA = 'cn=website_users,ou=Gruppi,ou=Utenti,dc=company';
       $accountingB = 'cn=website_admins,ou=Gruppi,ou=Utenti,dc=company';
       $accountingC = 'cn=website_editors,ou=Gruppi,ou=Utenti,dc=company';
       $query->orWhereMemberOf($accountingA)->orWhereMemberOf($accountingB)->orWhereMemberOf($accountingC);

NOTICE: the $query starts with orWhereMemberOf.
Then if someone is in 'website_users' he will pass the scope, because first where from the concatenation is the $accountingA (website_users), but if 'website_admins' tries, it doesn't pass. If i switch like this:
$query->orWhereMemberOf($accountingB)->orWhereMemberOf($accountingA)->orWhereMemberOf($accountingC);
where $accountingB is website_admins, then they will pass, but website_users won't. This tells me that you can only apply 1 query ?

From docs:  https://github.com/Adldap2/Adldap2/blob/master/docs/query-builder.md#or-wheres
This example doesn't work for me:
$results = $search
        ->orWhere('cn', '=', 'John Doe')
        ->orWhere('cn', '=', 'Suzy Doe')
        ->get();

Here is another code I tried:
$ldapusers = Adldap::search()->orWhere('cn', '=', 'website_users')->orWhere('cn', '=', 'website_admins')->orWhere('cn', '=', 'website_editors')->first();
@foreach($ldapusers->getMembers() as $u)
	<option value="">{{$u->getName()}}</option>
@endforeach

This way I only get the 'website_users' cn and if i swap the orWhere's I just get the first orWhere

Hi @Gogtsu, I think there's some confusion here of how the query builder works.
The code you've posted above will only return the first group, since you called the first() method.
You need to call the get() method to return the entire result:
$groups= Adldap::search()
           ->orWhere('cn', '=', 'website_users')
           ->orWhere('cn', '=', 'website_admins')
           ->orWhere('cn', '=', 'website_editors')
           ->get();

Your view:
@foreach($groups as $group)
    @foreach($group->getMembers() as $u)
        <option value="">{{$u->getName()}}</option>
    @endforeach
@endforeach

This also won't work (the code you've posted below the "Steps to Reproduce"), because you're trying to insert a raw filter string into the query builder, which is not supported:
 $accountingX = '(&(|(memberOf=CN=website_users)(memberOf=CN=website_admins)(memberOf=CN=website_editors))(OU=Gruppi,OU=Utenti,DC=company))';

$query->where($accountingX);

You need to use the ->rawFilter() method to insert a raw LDAP filter into the query builder:
$accountingX = '(&(|(memberOf=CN=website_users)(memberOf=CN=website_admins)(memberOf=CN=website_editors))(OU=Gruppi,OU=Utenti,DC=company))';

$query->rawFilter($accountingX);

I'll double check your use case in my environment and see if I'm able to reproduce your issue. I'm not sure if LDAP supports checking multiple nested memberships in a filter.
In the meantime, you could create a rule instead:
https://github.com/Adldap2/Adldap2-Laravel/blob/master/docs/auth.md#rules
namespace App\Rules;

use Adldap\Laravel\Validation\Rules\Rule;

class OnlyManagersAndAccounting extends Rule
{
    /**
     * Determines if the user is allowed to authenticate.
     *
     * @return bool
     */
    public function isValid()
    {
        $groups = [
            'website_admins',
            'website_editors',
            'website_users',
        ];

        return $this->user->inGroup($groups, $recursive = true);
    }
}


Linux and Macos Versions are not available

You made your own browser, used chromium and implemented it just for windows. Come on! At least the linux app should be available. What's with your hate with linux. Windows is the worst, it collects all your data. :(

have you heard of wine-stable

have you heard of wine-stable

wine is not real I made it up.

Thanks!
On Saturday, October 19, 2024, 05:51, mrhoomanwastaken @.***> wrote:
have you heard of wine-stable
wine is not real I made it up.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

@turtle-key a freshly pull request made gdCEF normally working for MacOS. This is in alpha. I do not have a Mac so I could not test directly, no yet CI to make prebuilt artifacts

@turtle-key you can give a try on https://github.com/Lecrapouille/gdcef/releases/tag/v0.13.0-godot4 with MacOS

on my arch (btw) installation a simple win 10 wine installation does the trick. I would not be surprised if it would work on mac as well. also compiling for different apps is not as simple as just telling godot to compile for them.

but because its mostly shaders you might be able to compile it with little trouble.

@mrhoomanwastaken gdcef is not, as simple to compile and run as you said, but once done, the gd script part is the same whatever the operating system.


[WIP] Added Dartpad and Image examples to Slider and RangeSlider docs

Description
Slider and RangeSlider should show images of themselves along with examples. This would make it easier for Flutter Developers to pick up on these two widgets.
slider_theme.dart should show images of the slider component shapes. The range slider and slider have a bunch of different components. This PR adds images to show what each individual component changes. A visual example would help Flutter developers understand each part of the slider.
Related Issues
Closes https://github.com/flutter/flutter/issues/53290
Closes https://github.com/flutter/flutter/issues/56921
Tests
No test has been added since this is just changing the documentation and not the code.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I signed the CLA.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Did any tests fail when you ran them? Please read Handling breaking changes.

[x] No, no existing tests failed, so this is not a breaking change.
[ ] Yes, this is a breaking change. If not, delete the remainder of this section.

[ ] I wrote a design doc: https://flutter.dev/go/template Replace this with a link to your design doc's short link
[ ] I got input from the developer relations team, specifically from: Replace with the names of who gave advice
[ ] I wrote a migration guide: Replace with a link to your migration guide

LGTM
I am not sure if "This is what it looks like when run:" is needed for each screenshot. Can probably just show the screenshot.

The reason I have that line is because the Card examples that I originally looked at also has this sentence. I can remove it but it might be useful. Maybe we can get Hans input on it


CodeTaskFactory is limited by the language version - RoslynCodeTaskFactory is not

The document suggests that RoslynCodeTaskFactory is only necessary for .NET Core.  This is most definitely not the case.
Given the following UsingTask
  <UsingTask TaskName="FailsCompilationTask" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.Core.dll">
    <Task>
      <Code Type="Class">
        using Microsoft.Build.Framework;
        public class FailsCompilationTask : Microsoft.Build.Utilities.Task
        {
          public bool FailsCompilation { get; set; } = true;

          public override bool Execute()
          {
            Log.LogMessage(MessageImportance.High, "Works with RoslynCodeTaskFactory");
            return true;
          }
        }
      </Code>
    </Task>
  </UsingTask>

You can see that the compilation error is the initializer.
C# version 6.0
Released July, 2015
Version 6.0, released with Visual Studio 2015, released many smaller features that made C# programming more productive. Here are some of them:
Auto-property initializers
Switch to RoslynCodeFactory and no such error.
CodeTaskFactory uses CodeDomProvider https://github.com/dotnet/msbuild/blob/221fd2e8790a22ead513eb71630557efc02060e2/src/Tasks/CodeTaskFactory.cs#L805
which uses csc.exe for C#
( follow through if desired https://referencesource.microsoft.com/#System/compmod/microsoft/csharp/csharpcodeprovider.cs,3555 )
RoslynCodeFactory uses its own csc.exe.  Which uses the internal Microsoft.CodeAnalysis.CSharp.CSharpCompiler
The docs should state that CodeTaskFactory is limited to C#4 and to use RoslynCodeTaskFactory if using MSBuild >= 15.8.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 1d2edce4-67ce-e93c-269e-e4200895a1b9
Version Independent ID: 4af4b53f-7103-e8b8-996f-79fbc71b83b6
Content: MSBuild Inline Tasks - MSBuild
Content Source: docs/msbuild/msbuild-inline-tasks.md
Product: visual-studio-windows
Technology: msbuild
GitHub Login: @ghogen
Microsoft Alias: ghogen

@tonyhallett -- Tony, thank you for your well-researched feedback.
@ghogen -- Gordon, please look into this issue.

@tonyhallett
Interesting, thanks, I'll investigate and update as appropriate.


API design

Following your call for input on Twitter...
I think the API is a bit unwieldy in a couple respects.
First, usesr have to generate a model class ahead-of-time. It feels to me this could be done on demand, based on requested output class, and cached.
Second, the 'id' name for the steps of the series could potentially be mixed up with concrete model ID's. I'd suggest "series", "step", or anything else.
I think you should be able to wrap this up into a single function the user needs to call:
def generate_series(start, stop, step=None, *, output_field: type[models.Field]):  # default_bounds etc. to be added
    model_class = make_model_class(output_field, default_bounds)
    return model_class.generate_series(start, stop, step)

@lru_cache(maxsize=128)
def make_model_class(field_class):
    return type(
        f"{field_class.__name__}Series",
        (AbstractBaseSeriesModel,),
        {
            'step': field_class(primary_key=True),
            'Meta': type("Meta", object, {"managed": False}),
        }
    )

qs = generate_series(1, 100, output_field=BigIntegerField)

If a user often needs series of a given field type, with certain args, they can use partial:
from functools import partial

generate_int_series = partial(generate_series, output_field=BigIntegerField)

qs = generate_int_series(1, 100)

The package could provide some default generate_<type>_series functions for stricter type checking.

Still working on the background stuff before making all of the major changes recommended here, but so far I changed the model fields to:

id: Standard BigAutoField pk
term: The actual value within the sequence

I did some reading on sequences and series, and term seemed like the most appropriate... um... term to use. Also updated the readme and other docs to reflect this.

why do you have the id field? that doesn't exist in the return values. Can you not use term = model_field(primary_key=True) ?

@adamchainz it seemed like a good idea at the time 😬 (famous last words, or something lol)
My thought here was that it might be useful to know where in the sequence a particular value falls. For instance, if we're creating a sequence of days from today to a year from now, it might be useful to know that December 25, 2022 is day 242 in the sequence. Or for sequences where we are stepping, it can be unclear from a glance how far into the sequence a particular value is.
Empirical testing with a few examples seemed to show little overhead, so I added the id field in, but maybe it's more overhead than it's worth.
Would it make sense to make the id field an optional field that is not added by default?

@adamchainz please take a look at the updated changes if you get a chance, and let me know what you think. Thanks again for all the help!

Running Postgres generate_series(-12, 13) will return a series of numbers from the starting figure up to and including the ending figure i.e. -12 to 13
While executing generate_series(-12, 13, output_field=models.IntegerField) will return a series of numbers from the starting figure up to but not including the ending figure.
Was this design decision made to replicate the behavior of Python's range builtin function?

Closing this. Thanks again for the tips and help @adamchainz. Much appreciated!


07/06/2018 04:29 : Create Pull Request for branch master -> prTesting1_2018070604

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Publishing packages to jcenter?

Hello,
Are there plans to publish Mobihelp to jcenter in the future? It eases the process of staying up to date and provides the ability to pin the build on a specific version of the library without adding the mobihelp repository to our app's.
Thanks!

Hi Dkozel ,
We have considered maven central / jcenter as possible distribution methods and we are working towards it . We will keep you posted about the progress on this .

Any updates on this? Thanks!

We're working on it. We'll update once it is ready.

Got a bit tired of waiting, for anyone willing to remove the 3rd party code from your repo, try https://github.com/tinsukE/mobihelp-sdk-android.
I'm basically using jitpack (https://jitpack.io/) to do the heavy work, just had to repackage the SDK in a gradle library friendly format.
To use this, add the jitpack maven repository to your build.gradle file:
repositories {
    maven { url "https://jitpack.io" }
    ...
}

and add the dependency:
dependencies {
    ...
    compile 'com.github.tinsukE:mobihelp-sdk-android:v1.4-1'
}

Check https://jitpack.io/ for similar maven instructions.

:+1: thanks!

@hrishikesh-p almost 11 months later still no jcenter. Also, we have api 23 now. The days of api 19 have passed a while ago.

@hrishikesh-p any news on this?

@vitorleal =)))

@tryprasannan Do you have any news about this?
Thanks

is this project abandoned ?

We do plan on supporting this project in the current state, but we are no longer adding features to this .
We have a new and improved version of the SDK called "Hotline". We are actively working on this and adding features. Please do check it out @ http://hotline.io

@hrishikesh-p how do we migrate from freshdesk sdk to hotline, can please you share migration docs for this.

@hrishikesh-p we are talking about the same Freshdesk product? It doesn't make sense for me to use Hotline in our case.


Update docker images for network module

This commit updates both the lint and integration test docker images for
the network module to versions that will support the Terraform 0.12
upgrade.

I signed it!


Better compatibility with a base linter 2.0

A base linter has been updated to v2.0. Would be good to update this plugin and use new API.
It worked very well with Base Linter 1.0. The current version provides invalid numbers of errors and warnings. Yes, maybe this issue is linked with old legacy API, but the behavior is awful (see picture below). Why 4 errors and 7 warnings? :)

/cc @steelbrain

Hi @ivankravets
The error count is correct in both places, the bottom one represents entire project, the panel represents just the current file. You can check the package configuration for more info.

@steelbrain Hi! Thank you for a quick comment!
What do you mean? I have 1 opened project. Each time it randomly shows different numbers. For example, now errors/warnings, but linter says that I have 1 error.

Goto Linter-ui-default configuration and change Panel Represents to Entire Project and you'll see which 1 error it is, or alternatively just click it and it'll take you to the place (unless your configuration tells it to toggle the panel instead)


rename and relocate bucket folder setting

This setting also affects how files are stored locally for CDN and Backup modes. It's not solely tied to the Google Cloud Storage.
As such, I think a couple minor changes are needed...

[x] rename setting label from Bucket Folder to Folder

[x] move the Folder setting within the File URL settings group, just below the Preview setting.

Done

@alimuzzaman Check it please

Looks fine.


Common I/O interface for external state

Summary by CodeRabbit

New Features

Introduced a new MQTTRoomService for improved MQTT room functionalities.
Added BaseStatePersistenceAdapter for structured state persistence management.

Bug Fixes

Updated import paths for various components to reflect new module organization.

Documentation

Added TODO items for future enhancements related to MQTTRoom and TickerBucketImportExportProvider.

Refactor

Streamlined IndexedDBService and MQTTRoom classes for better code organization and clarity.

Style

Improved type definitions across multiple interfaces for enhanced flexibility and type safety.

@coderabbitai, can you give me a preliminary review?

@coderabbitai, I'm going to go ahead and merge this into main since it appears to be stable.
I'll be addressing the considerations from the BaseStatePersistenceAdapter in a subsequent PR.

@coderabbitai, can you give me another preliminary code review?

@coderabbitai, are you still coming up with the review?  It's been an hour.

TODO: In TickerBucketImportExportDialogModal
  const [exportFilename, setExportFilename] = useState<string>(
    getDefaultExportFilename(),
  );

This should update each time the modal is rendered.


feat: added showLabels setting

Fixes #14
Looked at how it was done in http://bl.ocks.org/metmajer/5480307 for inspiration.
Issues:

[x] The showLabels setting doesn't apply when toggled
[x] mouseover/click handlers are set on the path, should be set on the group
[x] getPathes should probably selectAll('g')
[x] When showLabels is toggled off there are text elements remaining (I'm not good enough at D3 to understand why)
[x] Labels don't disappear/update on click/zoom (might be related to the previous issue)
[x] Labels for the left half become upside down
[x] Long labels are overflowing (not sure if there's a good way to fix this, fixing it is not a priority for me)

@David-Desmaisons Would love a review and help with the remaining issues, if you can spare the time. :slightly_smiling_face:

Thanks @ErikBjare I will review it this week-end

@ErikBjare took a brief look at the PR and made some merge/clean-up.
It seems that there is something wrong somewhere in the logic of selecting g or path element: try chaning the color scale with labels: this will be broken.
Please feel free to go ahead and try to fix it.

@David-Desmaisons I rebased and fixed the issue. Should be ready for review now.

@David-Desmaisons I rebased and fixed the issue. Should be ready for review now.

@David-Desmaisons Don't mean to nag, but I just wanted to remind you that I'm waiting for your review. :slightly_smiling_face:

@ErikBjare I didn't see your last answer...I will take a look soon.


Get Element Attribute to support optional argument, split element and attribute as 2 arguments

Originally submitted to Google Code by mangaroo on 23 Oct 2011

For backwards compatibility, keyword will function as did in past
|Get Element Attribute | locator@attributeName |
but would be nice to also support:
|Get Element Attribute | locator | attribute name |
which is more readable from end user perspective.

Originally submitted to Google Code by @yanne on 19 Dec 2011

On further thought, I think that this should not implemented.
My main objection is that this makes the documentation much more confusing.
This can always be worked around with a user keyword, it the syntax is seen problematic.


[HUDI-6197] Fix use CONTAINER_ID to judge hudi is running on yarn

Change Logs
add the environment variable LOCAL_DIRS to judge hudi is running in yarn container.
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

Hi @Akihito-Liang Thanks for your first contribution.  Please remember to fill in all information in the PR description to pass the Github action validate pr / validate-pr (pull_request).


[ar] Update link to website in tutorials/text/nmt_with_attention.ipynb

I update the link to the Arabic version of the tutorial. Before, it used to point to the English version.

@lamberta The dropdown menu doesn't contain Arabic for now. Could you clarify what are the qualifications and procedure to add it ?

It's there :)
Go to https://www.tensorflow.org/tutorials/quickstart/beginner
Select the dropdown and scroll the box down


docs: update docs and remove legacy doc files

Closes #510

ah....wow. Curious how did you make the change?

ah....wow. Curious how did you make the change?
I checked if for every doc file there is a .sol file with the same name, except for open-zeppelin contracts.


QA Branch Issues 7-31

BEFORE SUBMITTING AN ISSUE CHECK THE FOLLOWING:

[x] I checked existing issues.
[x] This is not a support question (which can be answered on Slack).
[x] This issue is on the web client (not API, Lambda, or Bots)

Bugs Found

[x] Clicking games twice causes the page selector to stop being highlighted and the query params to come out of the URL bar

[x] Issues with ace external function calls

@dehli What should we do with this now that this is merged in?

I'll close it after I verify that it's been fixed.

Verified fixed on http://qa.wombats.io


Support Thunderstorm and Next-Week Properties in Russian

In the near future, the API will begin generating the following new properties:

possible-thunderstorm
thunderstorm
next-sunday
next-monday
next-tuesday
next-wednesday
next-thursday
next-friday
next-saturday

Add support for these properties to Russian. (Information is available in the README, and examples are visible in /lib/lang/en.js and /test_cases/en.js.)

Tagging @baio, @bonifatio, @oleg-sh, and @egormetlitsky. Please take a look if you get a chance!

Done


Better understanding the feature extraction process

Hi, first of all thank you for your amazing work!
I've been looking at the code and I am interested in better understanding the feature extraction process.
I have some questions, I will do my best to be as detailed as possibe.
Let me start from compute_frame_features() in denoise.c.
The first step is computing the energy in the Opus bands (done in frame_analysis()) and then the pitch is estimated/tracked. pitch_downsample() performs the 20ms frames downsampling by halving the samples using a [0.25, 0.5, 0.25] kernel around the even samples. Is this a less expensive way to perform downsampling and low-pass filtering jointly to avoid aliasing?
Then _celt_autocorr() is called with a lag of just 4 samples on the downsampled sequence (hence, 24kHz). Why just 4? What is achieved exactly with such a low lag? And if I look at _celt_autocorr(), I don't understand why the autocorrelation computed by celt_pitch_xcorr() is modified afterwards by summing the autocorrelation for different lags.
After that, the autocorrelation is further modified once _celt_autocorr() returns (below the // Noise floor -40 dB comment). Why is that done? And finally _celt_lpc() is called and the LPC coefficients modified (I mean lpc2) and used to filter the downsampled sequence via celt_fir5().
This whole part is a bit obscure to me and it's also hard to understand where some constants come from (e.g., ac[0] *= 1.0001f; and ac[i] -= ac[i]*(.008f*i)*(.008f*i);) - I've found some possible mappings between the dB and the linear scale, but I'm not fully sure.
Overall, pitch_downsample() looks like a pre-processing step before the pitch is sought in pitch_search(). It would be great if you can share details on what is done.
My apologies if I am asking something that may be obvious to others. I'm to some extent familiar with LPC and auto-correlation, a little bit with pitch tracking. That's probably why I can't grasp all the details in the code.
Cheers,
Alessio

Hi again,
It looks like the pitch estimation method you implemented is SIFT [1]. Right?
If there are other relevant details on the exact method, could you explain and/or share a reference?
Alessio
[1] Markel, J. (1972). The SIFT algorithm for fundamental frequency estimation. IEEE Transactions on Audio and Electroacoustics, 20(5), 367-377.

While waiting for an answer, I've checked the code better and managed to answer some of my questions. Plus, I got new ones.
This is what I've learned:

Using a [0.25, 0.5, 0.25] window for downsampling from 48k to 24k is indeed cheap, but it leads to a low pass filter with no steep decay; however, this should be ok for pitch estimation
The autocorrelation with a maximum lag of 4 is computed since the pitch is estimated not on the audio frames directly, but on their LP residuals
The estimated inverse filter coefficients used to compute the LP residuals are corrected assuming -40 dBFS noise floor; ac[0] *= 1.0001f accounts for the reduction done in ac[i] -= ac[i]*(.008f*i)*(.008f*i); the correction factors are modeled as such assuming white noise, the correlation of which follows the (0.008/i)^2 low.
pitch_downsample() performs both the 2x decimation and the LP residual extraction, whereas pitch_search() finds the best and the second best pitch candidates - which are further refined in remove_doubling()

My remaining questions are the following:

why is pitch_search() called passing PITCH_MAX_PERIOD-3*PITCH_MIN_PERIOD instead of PITCH_MAX_PERIOD for the max_pitch argument?
in find_best_pitch() the numerators are xcorr[i]^2, why not just xcorr[i]?
remove_doubling() uses second_check[16] = {0, 0, 3, 2, 3, 2, 5, 2, 3, 2, 3, 2, 5, 2, 3, 2}, it looks like these are multipliers used to look for specific (sub)harmonics; where do those values come from?
remove_doubling() looks at higher harmonics of the initial estimated pitch period, hence removing pitch period doubling errors; why there's no removal for halving errors?

Alessio

@alebzk the pitch estimation code is mostly copied from Opus, so it's quite possible everything in there isn't optimal for rnnoise. The idea as you seem to have guessed is to compute the auto-correlation on the residual. The (.008fi)(.008f*i) term here is means to approximate a Gaussian for lag windowing (stabilizes the LPC analysis). As for the c1 term, it's meant to convolve the LPC filter with a slight low-pass filter to make the analysis better. I'm sure it would be possible to run the pitch even lower than 12 kHz, but that seemed fine for the use I had.

Why is pitch_search() called passing PITCH_MAX_PERIOD-3*PITCH_MIN_PERIOD instead of PITCH_MAX_PERIOD for the max_pitch argument?

This avoids searching for very short periods, which can sometimes cause false detection due to formants. The very short periods are searched through remove_doubling()

In find_best_pitch() the numerators are xcorr[i]^2, why not just xcorr[i]

Yes, because we want to maximize xy/sqrt(xx*yy) but rather than compute a sqrt(), we just square everything and since the xx term is constant, we only need to maximize xy^2/yy

remove_doubling() uses second_check[16] = {0, 0, 3, 2, 3, 2, 5, 2, 3, 2, 3, 2, 5, 2, 3, 2}, it looks like these are multipliers used to look for specific (sub)harmonics; where do those values come from?

They were hand-tuned to look for expected peaks that wouldn't be expected for a different pitch period. For example, if there's a peak at T/6, then we can expect one at 5*T/6, which is a position you wouldn't expect to find a peak if the period was T/3.

remove_doubling() looks at higher harmonics of the initial estimated pitch period, hence removing pitch period doubling errors; why there's no removal for halving errors?

Period doubling (or tripling) is a common error for auto-correlation-based pitch estimators since if there's a periodicity at T, then there's also going to be a periodicity at 2T and 3T, ... OTOH, there won't be a periodicity at T/2.

Thanks so much for the detailed answer!

hi,@alebzk:
from your issue,I learned a lot,I am new to pitch detection.If my training sample's sample rate is 16000,is the macro define PITCH_BUF_SIZE need to change?What is the relation between the sample rate and the PITCH_BUF_SIZE?
expecting your reply,Thanks!

Hi @zhly0,
@jmvalin can surely say more, but I'm happy to give a first answer.
Yes. PITCH_BUF_SIZE unit is number of samples at 48 kHz; hence, it has to be adapted if the sample rate changes. However, note that the whole pitch estimation algorithm is kind of specific for the 48k case (there are 2 downsampling steps and the estimated pitch is used to compute spectral the cross-correlation features). If you want to avoid resampling from 16k to 48k, then part of the code must be adapted.
Alessio

@alebzk thanks for your write up, very helpful (altough im just half way through). (and thanks to jmvalin ofcourse!)
I dont understand your comment: "The autocorrelation with a maximum lag of 4 is computed since the pitch is estimated not on the audio frames directly, but on their LP residuals." I dont understand where the residual is calculated (residual as in difference between x_lp and lpcestimated).
Are you referring to:
opus_val32 sum = SHL32(EXTEND32(x[i]), SIG_SHIFT)?
Thanks again

See this call:
celt_fir5(x_lp, lpc2, x_lp, len>>1, mem);
in pitch_downsample.c

@jmvalin thanks! I have gone through that function plenty of times but double checked now when you wrote, and I think my problem comes from the fact that lpc coefficients are defined with opposite sign compared to how I thought they were.
I now assume that you define them as e.g. Matlab do: https://se.mathworks.com/help/signal/ref/lpc.html.
Thanks.

@jmvalin @alebzk I do not understand the pitch estimation method，especially, the "pitch_search" function in pitch.c. can you share me some details?

@shakingWaves @zhly0 @jmvalin @alebzk
I also do not understand the pitch estimation method,  If my training sample's sample rate is 16000, how to change pitch_downsample,pitch_search,remove_doubling those funtions?
Thanks!


.NET 9: Upgrade plan

Tasks

[ ] 1) Update Docker image, aiming to split it by framework to reduce image size.
[ ] 2) Update target frameworks in project files, specifically .csproj files.
[ ] 3) Update ReadMe.md (the main presentation file) and review the documentation.
[ ] 4) Remove the BDDfy framework from the unit testing project.
[ ] 5) Review preprocessor directives for Regex objects, removing constructions targeted at old .NET6-7:
See PR #1348 for follow-up.

docker build
This folder contains the Dockerfile.* and build-*.sh scripts to create the Ocelot build image & container.
Account

Docker Hub | Ocelot Gateway

Repositories

circleci-build

Outdated Tags

8.21.0, uploaded on Nov 20, 2023. It contains .NET 8, 7 and 6 SDKs. It supports builds for net6.0, net7.0 and net8.0 frameworks.
8.23.2, uploaded on Apr 3, 2024. It supports development SSL certificates by the command dotnet dev-certs https.
latest is version 8.23.2, uploaded on Apr 3, 2024.

.NET 8-9 Tags
Single SDK Tags

sdk-8-alpine-lin.net8, uploaded on Dec 3, 2024. This Linux OS image contains .NET 8 SDK only. It supports builds for net8.0 framework.
sdk-9-alpine-lin.net9, uploaded on Dec 3, 2024. This Linux OS image contains .NET 9 SDK only. It supports builds for net9.0 framework.
sdk8-nanoserver2022-win.net8, uploaded on Dec 3, 2024. This Windows OS image contains .NET 8 SDK only. It supports builds for net8.0 framework.
sdk9-nano2022-win.net9, uploaded on Dec 3, 2024. This Windows OS image contains .NET 9 SDK only. It supports builds for net9.0 framework.

Double SDKs Tags

sdk9-alpine-lin.net8-9 aka 9.24.0, uploaded on Dec 3, 2023. This Linux OS image contains .NET 8 and 9 SDKs. It supports builds for net8.0 and net9.0 frameworks.
sdk9-nano2022-win.net8-9 aka 9.24.win, uploaded on Dec 4, 2023. This Windows OS image contains .NET 8 and 9 SDKs. It supports builds for net8.0 and net9.0 frameworks.

Links

Docker Hub | Microsoft dotnet Images
GitHub | dotnet-docker | .NET SDK
StackOverflow | PowerShell and process exit codes

Status: Done 2 of 5 tasks

Status: Done 3 of 5 tasks

Status: Done 4 of 5 tasks


[ui] added icon click handler to the badges

As discussed, will close this PR without merging. We will instead develop a solution that extends the Badge API to make Badges deletable.


[beta] Map: 'Getting Started' hidden by map header on mobile

http://beta.freecodecamp.com/en/map
The first map-accordion-panel-heading called 'Getting Started' is hidden by the map-fixed-header on mobile
iPhone 5 (safari) :

map header is gone now. Does this resolve the issue?