Installation fail -- requirements are invalid

The current setup.py lists:
    install_requires=["numpy>=1.20.0"],
    tests_require=["scipy>=1.0.0"],
    setup_requires=["numpy>=1.20.0"],

This means pip will try to install the current version on lapjv even if numpy is older than 1.20. It will build against numpy 1.20, install it in your environment with older numpy, and import lapjv will fail with:
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: numpy.core.multiarray failed to import

Please sync install_requires with setup_requires as building with one version and running with another does NOT work.

It appears this has already been fixed by 32c7108d40c1ea30dfa9d9115642fee901e8afbf in 1.3.14 but pip will just grab 1.3.13 because it will think that version is compatible with any numpy.

This is the problem that I caught in my own production today, let's fix this.

NP with yanking 1.3.13, but won't the story repeat with 1.3.12, 1.3.11, etc.?

I don't think so, the problem was introduce by #21 right before 1.3.13. Previous versions used build requirement numpy>=1.0.0 and install requirement numpy>=1.0.0, so they matched.

The metadata is out of sync again, setup.py says install_requires=["numpy>=1.20.0"] and pyproject.toml says requires = ["numpy>=1.21.0"]


Add containerd

Created containerd support in turbinia using container-explorer.

Looks like the tests failed, but I'm re-running them and maybe they'll pass now after some of the other fixes that went in this week.

FYI, it looks like this is the current error the e2e test is failing with:
# Failed Tasks
* ContainerdEnumerationTask (/evidence/artifact_disk.dd:/p1): Error enumerating containerd containers: Error decoding container-explorer output: Expecting value: line 1 column 1 (char 0)
    * `/evidence/123456789/1663118926-db24cc9c96f14ddb8b8cdbc12bedf5cc-ContainerdEnumerationTask/worker-log.txt`


Multiple IdSrvs mapped on one website

I am starting to port our code to v4 to see if we run into any issues so we can provide feedback / questions per this comment
Using IdSrv 3 we have 2 identity servers mapped to the same host (host.com/users, host.com/clients - the main reasons for this are completely different user and view services between the two).  After reviewing the sample, pouring over documentation and looking at the v4 codebase itself, I cannot figure out how to set up the same type of configuration in v4.  I'm assuming I'm missing something simple, but in case I'm not, figured I'd ask if this type of configuration is still possible?

This is a fundamental design flaw of ASP.NET Core's DI system -- there is no official way to have distinct DI systems for different branches in the pipeline. We (and many others) have provided this feedback to Microsoft. They say they're aware of it and will be doing something at some point to allow for this. BTW, they call this "multi-tenancy" support, which is just... an awful.. awful name.

I am also facing this issue. I had a similar issue with asp.net identity, but the solution I used for that doesn't seem possible with IdentityServer4.
The difference is that asp.net identity uses IOptions to get options used when setting the cookie options and auth scheme from IApplicationBuilder and they use
IOptions<IdentityOptions>

which allowed me to implement a custom implementation of the same so that when the actual options are accessed from the IOptions.Value I could resolve the correct settings based on the request url. even though the IOptions itself is a singleton the Value when accessed can return a different result based on  the request.
The way IdentityServerOptions is a singleton and not wrapped in IOptions poses a challenge.
I'm using sasskit for multi-tenancy which enables me to support tenants resolved from the first folder segment of the url (as one possibility), so in startup I wire up the per folder tenant authentication like this:
        app.UseMultitenancy<cloudscribe.Core.Models.SiteSettings>();

        app.UsePerTenant<cloudscribe.Core.Models.SiteSettings>((ctx, builder) =>
        {
            builder.UseCloudscribeCoreDefaultAuthentication(
                loggerFactory,
                multiTenantOptions,
                ctx.Tenant);

        });

in that extension I'm setting up the the auth middleware using the correctly resolved IdentityOptions to set the cookie auth middleware cookieoptions. I wish there was a similar way for me to pass the IdentityServerOptions with app.UseIdentityServer so they could be resolved per request or configured per middleware branch
I'm struggling a bit to figure out how to do something similar with IdentityServer so if anyone has any suggestions please let me know, otherwise if I do figure out a solution I will post it here.

I don't think you can use IOptions<T> in any middleware, since there could be many cookie middlewares in the pipeline, thus many options with different settings. This is what I meant above by the design just being flawed. This is not an IdSvr issue, it's a pipeline architecture flaw. IOW, the right solution is to combine the DI system with the pipeline itself -- in a sense have the DI configured at different levels in the pipeline (e.g. different/additional config in a Map). Each level would inherit the parent DI config. But, no one at Microsoft asked my feedback (or by the time they did, it was too late).
As for IdentityServer -- what's the specific thing you're struggling with? I can't decipher what it is from the above text. If you need different implementations of the various interfaces based on the tenant (via the host name for example) then you'd need to do this by registering the implementations in DI via the callback/factory APIs. So in short something like this pseudo-code:
services.AddTransient<IFoo>(container => {
    var tenant = container.Resolve<IHttpContextAccessor>().HttpContext.GetCurrentRequesrtTenant();
    return CreateFooFromTenant(tenant);
});

Again, you can thanks Microsoft for this.

Thanks for the prompt reply, on a Saturday morning, that is more than I expected!
I'm trying to build/provide integration between my cloudscribe Core project and IdentityServer4. My project would be the user store similar as you have provided for ASP.NET Identity in your quick starts.
My project is a mutli-tenant implementation of ASP.NET Identity, basically implemented with tenant aware UserStore and RoleStore so it has the tenant resolved from the request and it looks up other things like users and roles with the tenantid as a parameter.
My project supports 2 mutually exclusive configuration options for mutli-tenancy, it can be configured to use host name to differentiate tenants or alternatively it can have one root level tenant and other tenants demarcated by the first folder segment of the url. These additional tenants can be added from the UI and it does not require restarting the app for the new tenants to become active because of the way we can branch the middleware with saaskit. But in this case since we do not get cookie isolation from the browser it is important to be able to set a different authscheme and cookie name per tenant, and I have been able to accomplish that. In general my solution does not use different dependencies per tenant, if I really needed different dependencies per tenant in general I probably would just host the sites separately with different startup code. But all the tenants in my scenario are using the same software components.
With the host name tenant scenario I don't think there will be any integration problem because it doesn't matter if the authscheme and cookie name are the same for each tenant since we get cookie isolation in the browser per host.
With the folder tenant scenario is where I face a challenge, because I need different auth scheme and cookie names per folder tenant and I need additional endpoints setup for IdentityServer prefixed with the folder name corresponding to the folder tenants.
I "could" just not support IdentityServer integration with the folder tenant scenario, but I would like to be able to if I can find a solution.
Are you saying that pseudo code callback syntax for DI works? Can I do that to setup multiple singletons of the IdentityServerOptions?

My project is a mutli-tenant implementation of ASP.NET Identity, basically implemented with tenant aware UserStore and RoleStore so it has the tenant resolved from the request and it looks up other things like users and roles with the tenantid as a parameter.

So how are you handling this, given that AspId doesn't support multi-tenancy? Is this what saaskit does for you (I'm not familiar with it).
If you need multiple cookie names, and sasskit is forking the pipeline for you then why not put the cookie middleware into the forked pipeline with different cookie names?

yes saaskit provides helpful functionality to branch the middleware per tenant resolved from the request, and I am configuring the cookie middleware per tenant that way. But asp.net identity SignInManager invokes the authentication using auth scheme from IOptions of IdentityOptions so I had to override that per tenant with a custom implementation so that any place where it is used gets the appropriate tenant specific settings.
that is the part I'm trying to figure out with IdentityServer middleware, how to make it use the same settings. maybe I can fork and modify the extension methods for app.AddUseIdentityServer so that I can pass in the needed settings when setting up tenant branches inside the app.UsePerTenant section shown above in the code of my first post

Why not give each branched cookie MW the same scheme, but assign each one different cookie names (based on the tenant)?
with IdSvr, if you register the cookie MW yourself we can use your cookie and won't register our own. just set the AuthenticationScheme on our authentication options and we'll use that.

with folder tenants I'm thinking I need different auth scheme per tenant since cookies are shared within the same host I don't want someone to be able to just rename a cookie and be authenticated in another tenant.
I will do some experimenting over the next few days and report back here how it goes. so far I don't see how to set a per tenant auth scheme on Identity Server middleware unless I make my own extension methods. I can only set it to one value from DI

if all else fails, the author of saaskit, Ben Foster, also wrote blog post about per tenant dependencies and shows how to do it with structure map. I would prefer a DI framework agnostic solution so I will save that approach as something to try if I can't get it working by other means

actually I already have a tenant claim in the cookie that must match the tenant id or the claims principal will be rejected, so maybe I don't need to have per tenant auth scheme and could get away with just having per tenant cookie names for the folder tenant scenario.

cookie issues seem to be solved by using the same authscheme name across tenants and just varying the cookie name for folder tenants.
next up is trying to figure out how to add additional endpoints for folder tenants. it looks like I could possibly implement a custom IEndpointRouter to match the endpoint when the tenant folder is part of the url. Does that sound like the right idea?
to do that looks like I'll have to copy/modify your IdentityServerServiceCollectionExtensions and IdentityServerBuilderExtensions. If your method used Services.TryAddSingleton instead of .AddSingleton when adding IEndpointRouter then I wouldn't need to do that I could just add a custom one first before calling services.AddIdentityServer

making progress, turned out I did not need to copy/modify IdentityServiceCollectionextensions and IdentityBuilderExtensions, simply registering my custom IEndpointRouter after calling .AddIdentityServer did the trick. That was just a little DI confusion on my part making me think I would need to do that.  In order to get the discovery url in the folder tenant to return the urls with the folder tenant segment included, I just had to call .SetBasePath on the HttpContext from within my IEndpointRouter if the current request matches a folder tenant.
Now I think the final piece of the puzzle is that I need to implement tenant aware operational stores so I can segregate and filter the data with a tenantid.
Does that sound about right or are there any other pieces I need to think about?

¯_(ツ)_/¯
in all seriousness, you're far deeper into what you're doing and i've lost track -- i'm just too focused on the internals of idsvr yet to know if all these internal things are right for replacing.

I got the discovery url working for both root tenants and folder tenants by implementing a custom IEndpointRouter and I implemented tenant aware storage for configuration and operational data. The root level tenant is working and I thought things were looking good but I just hit a brick wall trying to get folder tenants working.
The problem for me is in the AuthorizeEndpoint. Even though my IEndpointRouter is returning the AuthorizeEndpoint, the endpoint itself is responding with a 404 because of its internal logic which compares the request path against constants.
So I thought maybe I need to implement a custom AuthorizeEndpoint in order to adjust the patch matching logic to take into account my folder tenant segment. But I can't do that because ir depends on a lot of things inaccessible from my code. Specifically these things are inaccessible:
** IAuthorizeRequestValidator
** IAuthorizeInteractionResponseGenerator
** IAuthorizeResponseGenerator
** context.GetIdentityServerUserAsync()
** AuthorizeResult
** AuthorizeRequestValidationLog
** AuthorizeResponse
** RaiseSuccessfulEndpointEventAsync
** RaiseFailureEndpointEventAsync
I know I cannot ask you to make those things accessible and actually I would rather not go down the path of implementing a custom AuthorizeEndpoint. However I do have an idea I would like to suggest that would solve the problem for me and I wonder if you would consider a pull request?
My idea would be to make an interface with methods to match the path, with a default implementation that has the current logic, which would allow me to plugin custom logic via DI.
    public interface IMatchAuthorizeProtocolRoutePaths
    {
        bool IsAuthorizePath(string requestPath);

        bool IsAuthorizeAfterLoginPath(string requestPath);

        bool IsAuthorizeAfterConsentPath(string requestPath);
    }

    public class DefaultAuthorizeProtocolRouteMatcher : IMatchAuthorizeProtocolRoutePaths
    {
        public bool IsAuthorizePath(string requestPath)
        {
            return (requestPath == Constants.ProtocolRoutePaths.Authorize.EnsureLeadingSlash()));
        }

        public bool IsAuthorizeAfterLoginPath(string requestPath)
        {
            return (requestPath == Constants.ProtocolRoutePaths.AuthorizeAfterLogin.EnsureLeadingSlash()));
        }

        public bool IsAuthorizeAfterConsentPath(string requestPath)
        {
            return (requestPath == Constants.ProtocolRoutePaths.AuthorizeAfterConsent.EnsureLeadingSlash()));
        }
    }

the the AuthorizeEndpoint would take a constructor dependency on IMatchAuthorizeProtocolRoutePaths and its internal logic would change like this:
public async Task<IEndpointResult> ProcessAsync(HttpContext context)
{
	if (context.Request.Method != "GET")
	{
		_logger.LogWarning("Invalid HTTP method for authorize endpoint.");
		return new StatusCodeResult(HttpStatusCode.MethodNotAllowed);
	}

	if (_matcher.IsAuthorizePath(context.Request.Path))
	{
		return await ProcessAuthorizeAsync(context);
	}

	if (_matcher.IsAuthorizeAfterLoginPath(context.Request.Path))
	{
		return await ProcessAuthorizeAfterLoginAsync(context);
	}

	if (_matcher.IsAuthorizeAfterConsentPath(context.Request.Path))
	{
		return await ProcessAuthorizeAfterConsentAsync(context);
	}

	return new StatusCodeResult(HttpStatusCode.NotFound);
}

would you be open to a pull request or making changes like this?

I don't have time right now to read all of this, but I'm going to reopen so I don't lose track of this thread.

thanks for re-opening. I'm going to submit a pull request with the minimal changes that enabled me to get things working correctly in the folder based multi tenant scenario of my cloudscribe Core project. I want to say right up front that I have no expectation that the pull request will be merged just hoping it could start a conversation to get something merged that would enable my scenario. The changes in this pull request do enable me to accomplish my goal and I did manage to get things integrated and working. I really wasn't sure what namespace to put the files in and am sure I chose wrong on that but again this is just to get a conversation going.
I've published a nuget of my fork named as cloudscribe.IDS4Fork that I can use for now to be unblocked, but obviously I do not want to have to maintain any fork, this is only intended to unblock me and allow time for you guys to review the issue whenever you have the time to do so without any pressure.
I'm going to move forward now working to add UI to my integration to manage clients and scopes. I think my cloudscribe project combined with IdentityServer4 integration provides a pretty compelling solution to provision and manage IdentityServer4 authority endpoints with user management and soon client and scope management. It is especially easy to provision new folder tenants that provide both the endpoint and the management since no additional dns is needed for each new tenant. Once I get the management of scopes and clients implemented I'll setup a sample app for anyone who is interested in seeing how this combination could be useful to them. Specifically it would solve the request of @edwinf  in that tenants could easily be provisioned for host.com/users, host.com/clients, etc with completely separate users, clients, and scopes.
My only hope is that eventually something will make its way into the main IdentityServer4 package to enable this scenario so that I can drop my temporary fork.
Anyway I appreciate you taking the time to read and consider this stuff whenever you can get to it and I will go ahead and submit my pull request next referencing this issue.

thought my pull request would auto link, but for reference https://github.com/IdentityServer/IdentityServer4/pull/382

To get my folder tenant scenario working I needed another small change to make cors policy work.
For anyone interested I have a working sample here, pre-populated with 2 folder tenants along with working apis and javascript clients for each tenant. This sample provides a complete solution for standing up a multi-tenant OP Server with full management for tenants, users, roles, claims, clients, and scopes. This sample uses NoDb file system storage which made it easy to pre-populate data to demo. I also have another sample here that uses EFCore but is not pre-populated with any data and doesn't have any clients or apis in the sample solution but is a ready to run multi-tenant OPServer with the same management features.
Again, to get these samples working I had to fork IdentityServer4 which I would rather not do, so hoping someday some changes can be made to facilitate the use of custom folder segments in the endpoint urls. The primary benefit of folder tenants vs host name tenants (which is also supported in my solution) is that with folder tenants one can easily provision new OP Server tenants, each with their own users, roles, claims, clients, and scopes, without need of any new DNS records or additional SSL certificates. Any feedback on this work would be greatly appreciated.

I would like to revive this discussion. Previously I made a pull request which is now old and has been closed with the comment:
“Given that this PR is so old, I think it makes sense to close it and follow up on the open issue so we can understand the real ask here.”
So I want to clarify what the ask is. The ask is to make it possible to support custom urls for the endpoints and cors policy which is not possible in the current code because paths are stored as constants that are checked in endpoints such as AuthorizeEndpoint, ie with code like this:
if (context.Request.Path == Constants.ProtocolRoutePaths.Authorize.EnsureLeadingSlash())
{
       return await ProcessAuthorizeAsync(context);
}

Specifically I am supporting a multi-tenant scenario with a root tenant at the root url and additional tenants based on the first folder segment of the url. Since the presence or not of the folder segment is used to resolve the current tenant in my scenario, all of the urls for the given tenant must start with a unique folder segment. So the problem for me is that IDS4 will only accept those hard coded url paths:
public const string Authorize              = "connect/authorize";
public const string AuthorizeAfterConsent  = Authorize + "/consent";
public const string AuthorizeAfterLogin    = Authorize + "/login";
public const string DiscoveryConfiguration = ".well-known/openid-configuration";
public const string DiscoveryWebKeys       = DiscoveryConfiguration + "/jwks";
public const string Token                  = "connect/token";
public const string Revocation             = "connect/revocation";
public const string UserInfo               = "connect/userinfo";
public const string Introspection          = "connect/introspect";
public const string EndSession             = "connect/endsession";
public const string EndSessionCallback     = EndSession + "/callback";
public const string CheckSession           = "connect/checksession";

Whereas I need to be able to prefix those paths {tenant}/connect/authorize, and so on.
In my previous pull request I abstracted that url checking logic into an interface with a default implementation that kept the original logic, but this allowed me to inject a custom implementation to support custom endpoint urls.
public interface IMatchAuthorizeProtocolRoutePaths
{
    bool IsAuthorizePath(string requestPath);

    bool IsAuthorizeAfterLoginPath(string requestPath);

    bool IsAuthorizeAfterConsentPath(string requestPath);
}

public class DefaultAuthorizeProtocolRouteMatcher : IMatchAuthorizeProtocolRoutePaths
{
    public bool IsAuthorizePath(string requestPath)
    {
        return (requestPath == Constants.ProtocolRoutePaths.Authorize.EnsureLeadingSlash());
    }

    public bool IsAuthorizeAfterLoginPath(string requestPath)
    {
        return (requestPath == Constants.ProtocolRoutePaths.AuthorizeAfterLogin.EnsureLeadingSlash());
    }

    public bool IsAuthorizeAfterConsentPath(string requestPath)
    {
        return (requestPath == Constants.ProtocolRoutePaths.AuthorizeAfterConsent.EnsureLeadingSlash());
    }

EndSessionEndpoint also had similar hard coded path validation, which again, I wrapped in an interface with a default implementation keeping the current logic.
public interface IMatchEndSessionProtocolRoutePaths
{
    bool IsEndSessionPath(string requestPath);

    bool IsEndSessionCallbackPath(string requestPath);
}

public class DefaultEndSessionProtocolRouteMatcher : IMatchEndSessionProtocolRoutePaths
{
    public bool IsEndSessionPath(string requestPath)
    {
        return (requestPath == Constants.ProtocolRoutePaths.EndSession.EnsureLeadingSlash());
    }

    public bool IsEndSessionCallbackPath(string requestPath)
    {
        return (requestPath == Constants.ProtocolRoutePaths.EndSessionCallback.EnsureLeadingSlash());
    }
}

A similar problem exists also in the CorsPolicyProvider where again the allowed paths come from hard coded constants. There is some possibility to add allowed paths in the IdentityServerOptions but it would mean knowing at startup time all the allowed paths, but basically what I need is to support those same paths with a prefix segment and since I can create new tenants at runtime, startup configuration is not a good solution for me. So in my previous pull request I created an interface and a default implementation of the interface to keep the current logic but allowing me to plugin custom logic.
public interface ICorsPathValidator
{
    bool IsPathAllowed(PathString path, CorsOptions options);
}

public class DefaultCorsPathValidator : ICorsPathValidator
{
    public bool IsPathAllowed(PathString path, CorsOptions options)
    {
        return options.CorsPaths.Any(x => path == x);
    }
}

With these changes I was able to achieve my goal but now I am stuck maintaining a fork of IdentityServer4 which has been challenging given the frequent changes and releases. I will keep maintaining it if I must but would really really appreciate it if these changes or something similar could make it upstream so that I can support my scenario without a fork. Note that the changes identified in this post are all the changes I need, those interface just need to be injected into the places where the current code is, ie IMatchAuthorizeProtocolRoutePaths is injected into AuthorizeEndpoint and the logic changes to use the interface like this:
if (_pathMatcher.IsAuthorizePath(context.Request.Path))
 {
     return await ProcessAuthorizeAsync(context);
 }

It is a total of 2 new files and 3 modified files to achieve my goal. I am happy to submit a new pull request against the latest code if there is any indication from you that you would accept it.

Things have changed in 2.0. Please open a more specific issue if the problem still exists


404 resource not found

i installed myrtille by downlinf .msi file from https://cedrozor.github.io/myrtille/ and installed it , but when accessing the URL it shows 404
http://myserver:8080/myrtille.
also
http://x.x.x.x:8080/Myrtille/?__EVENTTARGET=connect&__EVENTARGUMENT=&server=x.x.x.x&user=username&password=password&connect=Connect!
and
http://x.x.x.x:8080/Myrtille/?__EVENTTARGET&__EVENTARGUMENT=&server=x.x.x.x:8080&user=username&password=password&connect=Connect!

A simple test to do is try to connect http://myserver. If you don't see the IIS page, then either myserver is not accessible or IIS is not correctly installed. From the machine itself, also try http://localhost/myrtille; if it works, it may also be a DNS problem resolving "myserver".
Can you check with the IIS management console that Myrtille (web application) is correctly installed under the default website? Do you use the standard 80 (http) and 443 (https) ports? Did you tried https://myserver/myrtille? (just in case IIS is configured to reject http connections).  In any case, Myrtille is not accessible on port 8080.
For further details, have a look to the install logs (myrtille\log\Myrtille.Setup.log and Myrtille.Web.Install.log among others logs).

thanks for the response
, i was mistaken, I did not need to use the 8080 port, it was accessible on the 80 port, in the installation is mentioned that to access the service use the port 8080,

8080 is the port for the Myrtille backend (services) and is accessed internally by the Myrtille frontend (IIS webapp), from the Myrtille machine itself.


Add Cross WOC Banner or Poster

Is your feature request related to a problem? Please describe.
Add Cross WOC Banner or Poster
Describe the solution you'd like
Add Cross WOC to  Banner or Poster

I would like to work on this issue.


[1pt] Create basic structure for Rockets - Setup

Create a route and a view component. Use <NavLink /> for the page navigation links and style active class to indicate which section/page the user is currently on (underline active navigation link).
Create a directory for all Redux state slice files.

structure of rocket setup has been completed in the this Pull Request


How to connect "db.execute(query);" with moor_isolates?

Hi @simolus3, hope you are well! As I was working on my project, I have successfully used moor_isolates for generated moor tables. But I'm wondering how can we use moor_isolates with non-generated tables which use db.execute(queryStr);
MoorIsolate isolate = await MoorIsolate.spawn(backgroundConnection);
DatabaseConnection connection = await isolate.connect();

with
String query = r'''
    CREATE TABLE ''' + tableName + ''' (
      ''' + columnString.toString() + '''
    );
  ''';
final db = Database.openFile(File(resolver.dbFilename));
db.execute(query);

If you only need to run custom queries, I would still recommend to let moor generate an empty database class for you (just set tables: [] in your @UseMoor annotation). Inside that generated class, you can use customStatement to run queries. Moor will take care of opening the underlying connection and so on.
If you want to create custom tables, you could use customStatement in MigrationStrategy.onCreate or onUpgrade. Moor will automatically use the right database connection under the hood, even if it's connected to another isolate.

@simolus3, Thanks for the reply. Can this approach generate a table dynamically/on runtime?

You can use customStatement to run a CREATE TABLE statement, if that's what you need.

Awesome, I will give it a try and let you know.

It worked, thanks!


fix(add express file upload middleware as dependency): add express fi…

…le upload middleware as dep
Add express file upload middleware as dependency

:tada: This PR is included in version 1.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Fix typos in docs and comments

Change Summary
fix typos in docs and comments:
packages/anon-aadhaar-contracts/README.md
packages/anon-aadhaar-pcd/script/setup-dev.sh
packages/anon-aadhaar-pcd/test/utils.test.ts

We are working on the new protocol. A lot of docs will change so we don't merge your PR yet!
Thank you so much for you contribute!


potential false negative in SC2 sequencing

I am using the  artic pipeline to call variants on corona virus sequencing. I am afraid nanopolish missed something that should have been called, as shown below:
case 1

The two tracks represent sequencing of the same sample. Even more puzzling is the fact that the SNP was called on the top track i.e. that of lower depth in that region.
This does not sound too intuitive to me. If this is the expected behaviour, though, what would be the explanation?
Below is the vcf entry, where the SNP was called
MN908947.3      21638   .       C       T       663.8   PASS    TotalReads=197;SupportFraction=0.8005;SupportFractionByStrand=0.990136,0.744358;BaseCalledReadsWithVariant=187;BaseCalledFraction=0.903382;AlleleCo
unt=1;StrandSupport=0,39,45,113;StrandFisherTest=47;SOR=0.00820477;RefContext=TACCCCCTGCA;Pool=nCoV-2019_2   GT      1

case 2
Another sample was sequenced twice. The SNP was called on the sequencing with higher depth only:

Although it sounds intuitive to me that the SNP should also have been called in the bottom track (depth is 156x, supported by 94% of the reads in that position).
Below is the vcf entry of the sample in which the SNP was called
MN908947.3      15096   .       T       C       2234.3  PASS    TotalReads=508;SupportFraction=0.843625;SupportFractionByStrand=0.925903,0.747281;BaseCalledReadsWithVariant=460;BaseCalledFraction=0.894942;Allele
Count=1;StrandSupport=20,59,254,175;StrandFisherTest=75;SOR=0.798838;RefContext=AAGAATAGAGC;Pool=nCoV-2019_1 GT      1

Please let me know if I should upload any other file for a better assessment.

Hi,
Is this mutation in the *.fail.vcf file for the bottom sample?
Jared

On Nov 29, 2021, at 5:35 AM, Bruno Hinckel @.***> wrote:

I am using the artic pipeline to call variants on corona virus sequencing. I am afraid nanopolish missed something that should have been called, as shown below:
The two tracks represent sequencing of the same sample. Even more puzzling is the fact that the SNP was called on the top track i.e. that of lower coverage in that region.
This does not sound too intuitive to me. If this is the expected behaviour, though, what would be the explanation?
Please let me know if I should upload any other file for a better assessment.
Below is the vcf entry, where the SNP was called
MN908947.3      21638   .       C       T       663.8   PASS    TotalReads=197;SupportFraction=0.8005;SupportFractionByStrand=0.990136,0.744358;BaseCalledReadsWithVariant=187;BaseCalledFraction=0.903382;AlleleCo
unt=1;StrandSupport=0,39,45,113;StrandFisherTest=47;SOR=0.00820477;RefContext=TACCCCCTGCA;Pool=nCoV-2019_2   GT      1
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.

Thanks. Nanopolish has called variants here but they have narrowly failed an ARTIC pipeline QC check (QUAL / TotalReads > 3 - these variants have values of 2.95 and 2.75, respectively). These positions will be masked with N in the consensus sequence to avoid false negatives/false positives causing issues in downstream analysis. Here's the relevant code from the artic pipeline:
https://github.com/artic-network/fieldbioinformatics/blob/master/artic/vcf_filter.py#L30

Hi Jared, thanks for the reply, that clarifies.
Not sure whether or not you were involved in this, but do you know to which sequencing depth this QC check was adapted to? We have a median sequencing depth of 428x and I was wondering if I should lower the threshold (3) a bit to (2.75 max) to accept this kind of SNPs.

I was involved in developing these thresholds. The reason we divide by TotalReads is to make a metric that is not influenced by depth (I interpret this metric as how strongly the model supports the variant over the reference base, with higher numbers being better). Its up to you whether you change the threshold, dropping it to 2.75 is probably fairly safe.


DocsArchive-<IP_ADDRESS>Auto archive content

Auto archive content from https://github.com/OPS-E2E-PPE/docs-archive-test

Docs Build status updates of commit 285e9b0:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error]

Cannot sync git repo to specified commit because branch Archive_master_2019-09-29-04-03-42 has been deleted or has been force pushed: fatal: Couldn't find remote ref Archive_master_2019-09-29-04-03-42

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Update to Android 10 (API 29)

Add Google maven repo to all projects
Gradle 4.1 -> 5.4.1
Build tools 3.0.1 -> 3.5.1
Android sdk -> 29
Android build tools -> 29.0.2
Remove obsolete  tag in AndroidManifest.xml - generated by gradle instead
Bump gson to 2.8.5

Per @npmccallum going to red-button merge.


Improve documentation for followed topics

Fixes part of #27510, but I'd like to keep it open for now.

Updates Inbox view documentation for #27552.

Because the dropdown is unlabeled, I think it's worth listing out the options.

Screenshot

Current: https://chat.zulip.org/help/inbox#include-muted-conversations

Adds instructions for going though your followed topics.

Screenshot

Current page (this section is new): https://chat.zulip.org/help/follow-a-topic

Documents some workflows with followed topics on https://chat.zulip.org/help/follow-a-topic and https://chat.zulip.org/help/reading-strategies.

Screenshots

@drrosa Could you please review this one?

Test failure is this: Fragment #configure-notifications-for-followed-topics is not found on page http://localhost:9981/help/reading-strategies#configure-notifications-for-followed-topics

Looks great, fixed the CI failure and marked to merge once CI passes so we can keep building on top of this. (But @drrosa you should still review! You're more thorough than I am on these)


Entity IO fix for extra comma-separated values

Fix for this issue: https://github.com/momentum-mod/lumper/issues/64

Your branch seems to be behind the last refactor/cleanup, I changed this code quite significantly during review, a week or so ago.
Sorry this is so annoying, we're fairly close to getting refactor/cleanup merged, just a lot of review for BIRD to do.

Oh sorry, didn't realise I was behind. I've rebased this into the most recent commit.

Okay, quite busy with other areas of Momentum atm so will check this and all other PRs out once refactor/cleanup review is finished and I have some more time for Lumper. Feel free to PR more stuff before then, sorry for any slow reviews!

This was really confusing me so I took a look at the engine code. Parsing the delay and timesToFire use atof/atoi respectively, so so they'll ignore any subsequent non-digit chars. Loading surf_deity_ksf in debug results in a "times to fire" value of -1.
What you suggested about a tool not understanding ESC makes a lot of sense. I've tweaked it very slightly to throw an error if neither parse succeeds, and added some more info to the comment.


Faster Union Write

Hi there,
I'm running into a problem where it takes a long time to output my data.
I've done some digging, and a large portion of the time is a result of my schema having a large union of records in it (24 different records, each of which are fairly complex).
This causes validation to be run on each potential candidate for the union until one succeeds.
Is there something that could be done to improve this performance without having to change the schema?
My code knows ahead of time which candidate needs to be used from the union, so if that information could be passed into write_union somehow that would improve the performance significantly.
For example, if instead of simply providing a dictionary for the record, I provided a tuple of the name and the dictionary, then write_union could simply find the desired record based on the name instead of having to run verification.
What are your thoughts on this?
Thanks,
Nick

Hi Nick,
Thanks for using fastavro. I'll try to help you but it's been a while I've worked on it (and on Avro in general).

Did you profile to find out where the bottleneck is?
Did you try using an empty validator to writer?

HTH,
Miki

Hi Miki,
I'll answer # 2 first because it's more straightforward:
I'm not currently passing a validator to writer.
The slowness I'm seeing is a result of the call to validate() defined in writer.py.
Specifically, where it's called by write_union
As far as profiling goes, here is what I'm seeing:
(I'm not sure how to profile cython code, so I had to delete the .so files in the lib folder to get these numbers)
I created a simple test that outputs a number of avro records with a schema containing the complex union I mentioned above.
The profile looks like this:
         1628870 function calls (1434433 primitive calls) in 1.774 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
161200/39956    0.571    0.000    1.177    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:147(validate)
   509136    0.192    0.000    0.236    0.000 {isinstance}
   102739    0.189    0.000    0.816    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:206(<genexpr>)
   226247    0.173    0.000    0.322    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/schema.py:35(extract_record_type)
2259/1130    0.103    0.000    1.764    0.002 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:235(write_record)
   253278    0.089    0.000    0.089    0.000 {method 'get' of 'dict' objects}
19193/12419    0.088    0.000    1.647    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:217(write_union)
65047/1130    0.088    0.000    1.767    0.002 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:280(write_data)
    40686    0.074    0.000    0.150    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:45(write_int)
    60160    0.052    0.000    0.052    0.000 {method 'write' of 'cStringIO.StringO' objects}
    60142    0.034    0.000    0.034    0.000 {_struct.pack}
    15806    0.024    0.000    0.834    0.000 {all}
    15806    0.024    0.000    0.044    0.000 <redacted>/env/lib64/python2.7/abc.py:128(__instancecheck__)
    25279    0.017    0.000    0.158    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:183(<genexpr>)
    15808    0.015    0.000    0.015    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:68(__contains__)
     5204    0.008    0.000    0.030    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:96(write_enum)
     9032    0.007    0.000    0.163    0.000 {any}
    15808    0.005    0.000    0.005    0.000 {getattr}
     5204    0.004    0.000    0.004    0.000 {method 'index' of 'list' objects}
   1372/2    0.003    0.000    0.004    0.002 <redacted>/env/lib/python2.7/site-packages/fastavro/schema.py:57(extract_named_schemas_into_repo)
     1129    0.003    0.000    1.771    0.002 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:393(write)
     2258    0.002    0.000    0.005    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:39(write_boolean)
     2258    0.002    0.000    0.005    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:65(write_double)
    10161    0.002    0.000    0.002    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:34(write_null)
     2234    0.002    0.000    0.005    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:58(write_float)
        1    0.001    0.001    0.001    0.001 /usr/lib64/python2.7/json/encoder.py:212(iterencode)
        1    0.001    0.001    1.778    1.778 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:407(writer)
        4    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
       94    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/schema.py:45(schema_name)
     1130    0.000    0.000    0.000    0.000 {method 'tell' of 'cStringIO.StringO' objects}
        1    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
      2/1    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/abc.py:148(__subclasscheck__)
        6    0.000    0.000    0.001    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:385(dump)
        1    0.000    0.000    0.000    0.000 {posix.urandom}
        6    0.000    0.000    0.000    0.000 {method 'getvalue' of 'cStringIO.StringO' objects}
        1    0.000    0.000    0.006    0.006 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:359(__init__)
        1    0.000    0.000    0.001    0.001 /usr/lib64/python2.7/json/__init__.py:193(dumps)
        1    0.000    0.000    0.001    0.001 /usr/lib64/python2.7/json/encoder.py:186(encode)
        4    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:58(__iter__)
        1    0.000    0.000    0.001    0.001 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:295(write_header)
       47    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:349(<lambda>)
        6    0.000    0.000    0.001    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:305(null_write_block)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:36(__init__)
        1    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:122(write_map)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:26(__exit__)
       12    0.000    0.000    0.000    0.000 {len}
      3/1    0.000    0.000    0.000    0.000 {issubclass}
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:81(add)
        1    0.000    0.000    0.004    0.004 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:343(acquaint_schema)
        1    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:401(flush)
        6    0.000    0.000    0.000    0.000 {method 'truncate' of 'cStringIO.StringO' objects}
       47    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:354(<lambda>)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:20(__enter__)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/six.py:47(py2_iteritems)
        4    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:72(write_bytes)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_abcoll.py:98(__subclasshook__)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:90(write_fixed)
        6    0.000    0.000    0.000    0.000 {method 'seek' of 'cStringIO.StringO' objects}
        4    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/six.py:39(py2_utob)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:78(write_utf8)
        4    0.000    0.000    0.000    0.000 {method 'add' of 'set' objects}
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:52(_commit_removals)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib64/python2.7/_weakrefset.py:16(__init__)
        2    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
        2    0.000    0.000    0.000    0.000 {method 'remove' of 'set' objects}
        1    0.000    0.000    0.000    0.000 {method '__subclasses__' of 'type' objects}
        1    0.000    0.000    0.000    0.000 {cStringIO.StringIO}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {method 'flush' of 'cStringIO.StringO' objects}

I then created a schema identical to the one I used above, except I switched the union of 24 records to a record of the one type all of my test objects were.
When I ran the same code with this new schema, I got the following profile:
         626670 function calls (563598 primitive calls) in 0.666 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2259/1130    0.103    0.000    0.657    0.001 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:235(write_record)
27537/24150    0.090    0.000    0.130    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:147(validate)
59402/1130    0.078    0.000    0.660    0.001 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:280(write_data)
    39557    0.068    0.000    0.136    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:45(write_int)
    18064    0.065    0.000    0.340    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:217(write_union)
   165726    0.065    0.000    0.065    0.000 {isinstance}
    86939    0.064    0.000    0.126    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/schema.py:35(extract_record_type)
    59030    0.049    0.000    0.049    0.000 {method 'write' of 'cStringIO.StringO' objects}
    79148    0.030    0.000    0.030    0.000 {method 'get' of 'dict' objects}
    59012    0.029    0.000    0.029    0.000 {_struct.pack}
     5204    0.008    0.000    0.026    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:96(write_enum)
     5204    0.004    0.000    0.004    0.000 {method 'index' of 'list' objects}
     1129    0.003    0.000    0.664    0.001 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:393(write)
     2258    0.002    0.000    0.005    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:65(write_double)
     2258    0.002    0.000    0.005    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:39(write_boolean)
     2234    0.002    0.000    0.005    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:58(write_float)
    10161    0.002    0.000    0.002    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:34(write_null)
        1    0.001    0.001    0.667    0.667 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:407(writer)
    286/2    0.001    0.000    0.001    0.001 <redacted>/env/lib/python2.7/site-packages/fastavro/schema.py:57(extract_named_schemas_into_repo)
       14    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:349(<lambda>)
     1130    0.000    0.000    0.000    0.000 {method 'tell' of 'cStringIO.StringO' objects}
        1    0.000    0.000    0.000    0.000 /usr/lib64/python2.7/json/encoder.py:212(iterencode)
       28    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/schema.py:45(schema_name)
        6    0.000    0.000    0.000    0.000 {method 'getvalue' of 'cStringIO.StringO' objects}
        4    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {posix.urandom}
        6    0.000    0.000    0.001    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:385(dump)
        1    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
        1    0.000    0.000    0.002    0.002 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:359(__init__)
        1    0.000    0.000    0.000    0.000 /usr/lib64/python2.7/json/__init__.py:193(dumps)
        1    0.000    0.000    0.000    0.000 /usr/lib64/python2.7/json/encoder.py:186(encode)
        1    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:295(write_header)
        6    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:305(null_write_block)
        1    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:122(write_map)
        1    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:401(flush)
        6    0.000    0.000    0.000    0.000 {method 'truncate' of 'cStringIO.StringO' objects}
       12    0.000    0.000    0.000    0.000 {len}
        4    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/six.py:39(py2_utob)
        1    0.000    0.000    0.001    0.001 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:343(acquaint_schema)
        4    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:72(write_bytes)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/six.py:47(py2_iteritems)
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:90(write_fixed)
        6    0.000    0.000    0.000    0.000 {method 'seek' of 'cStringIO.StringO' objects}
        2    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:78(write_utf8)
       14    0.000    0.000    0.000    0.000 <redacted>/env/lib/python2.7/site-packages/fastavro/writer.py:354(<lambda>)
        1    0.000    0.000    0.000    0.000 {cStringIO.StringIO}
        2    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'flush' of 'cStringIO.StringO' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

The things I find interesting here are:

The overall time was reduced from 1.774 to 0.666.
The internal time of validate was reduced from 0.571 to 0.090 and the cumulative time was reduced from 1.177 to 0.130

Note that I do still have (simpler) unions in this schema, so the validate calls don't disappear entirely.

I think it would be hard to do what you want and still have record dictionaries that can be processed using both fastavro as well as the normal avro library. I don't know if the author feels that that is important, but I found it useful when first using fastavro and being able to detect some bugs by giving both avro and fastavro the same dictionary input.

I can understand the desire to keep things consistent with the normal Avro library.
One possibility would be to write in such a way that it is a superset of the Avro library.
I think you would want this regardless in order to maintain backwards compatibility, and keep things simple for smaller unions.
Going back to my original idea of passing in a tuple this would mean something like:
def write_union(fo, datum, schema):
    ...
    if isinstance(datum, tuple):
        name, datum = datum
        <perform lookup by name>
    else:
        <fall back to default behaviour>

This would allow for the record dictionaries to be passed in to the normal Avro library as long as they were not designed to use this functionality.
Alternatively, the information could be passed in through some other method, a tuple was just a candidate.
Perhaps you could pass in a mapping from union to type along with the record dictionary to the write call, which could then be used in the write_union function.

I prefer tuples since basic Python types are always faster. If you'd like an contribute a patch that will be awesome. If you can include some timing information with the patch it'll be even better.
I'm on the road next 3 weeks so might not be that responsive.


Camera preview is upside down on NEXUS 5X running Android 7.1.2

This sample has been migrated to a new location (check README for more information).
As recommended by GitHub, we are closing all issues and pull requests now that this older repo will be archived.
If you still see this issue in the updated repo, please reopen the issue/PR there. Thank you!


Team update - Jul 19, 2023

This is a @Quansight-Labs/czi-a11y-grant team sync 🎉🎉🎉! This is a way for the Team Members to provide status reports on what they've been up to this week and request help and attention for things they are working on. This issue will be closed at the end of the day.
Copy and paste the template below, and answer questions as you wish!

Response Template
**Thanks I'd like to give 🙌**
- So-and-so helped me out a lot with foo...

**Updates from last week :heavy_check_mark:**
- I worked towards deliverable: <link-to-deliverable>

**Challenges I faced and things I'd like assistance with 🙏**
- I had a hard time figuring out ...

**My availability for next week**
- I'll be off on foo day...

**Important items for discussion 💬**
- I have a question about goal <link-to-goal>
- Can @foo give a comment on issue #NN ?
- I opened #NN for discussion, please chime in
- I need to discuss foo

:mag: Needs Triage
The Needs Triage issues require an initial assessment and labeling.
No issues need triage! 🎉
:dart: Project boards
Please make sure the boards reflect the current status of the project.
Jupyter a11y grant project

Thanks I'd like to give 🙌
Updates from last week :heavy_check_mark:

#184 I replied to more review. There are still a few comments I need to address I think, but I'm not sure how I want to do it at the time of writing.
#178 JupyterLab 3.4.4 accessibility statement has a PR to the JupyterLab docs. Thank you to @steff456 for giving this a look since I haven't had to add to JupyterLab docs so completely before.
I presented on notebook authoring best practices. Happy to answer any questions about that if you have them.

Challenges I faced and things I'd like assistance with 🙏

I don't have anything right now.

My availability for next week

I'll be here all week.

Important items for discussion 💬

Are we planning to have a post-grant discussion to talk about what we think worked and what we might want to do differently in the future?

Are we planning to have a post-grant discussion to talk about what we think worked and what we might want to do differently in the future?

Yes we should, will only happen in August as we have now 2 weeks left to wrap up stuff
Things that would be good to discuss while I am missing this:

How are we doing for deliverables?
What help do you need from me?

From top of my head - @steff456  only straggling deliverable is getting the GH dark PR merged which is top of my TODO for tomorrow

Thanks I'd like to give 🙌

@steff456 for agreeing to cross-check PR https://github.com/Quansight/peri-store-frontend/pull/62 with https://github.com/Quansight-Labs/jupyterlab-accessible-themes/issues/59

Updates from last week :heavy_check_mark:

Responding to review on https://github.com/jupyterlab/jupyterlab/pull/14115

Challenges I faced and things I'd like assistance with 🙏

I had a hard time figuring out why sidebars can be tabbed when installing Jupyter Lab (v4.0.3) via Conda but not when building source from latest main branch.

My availability for next week

I'll be off on Friday, July 28 through Monday, July 31 (four-day weekend)

Updates from last week :heavy_check_mark:

Wrapped up the theme extension work:

Made changes to GH dark https://github.com/Quansight-Labs/jupyterlab-accessible-themes/pull/53
Fix some issues in the pyproject.toml and readme - https://github.com/Quansight-Labs/jupyterlab-accessible-themes/pull/67
Made changes to Pitaya Smoothie and GH light themes - https://github.com/Quansight-Labs/jupyterlab-accessible-themes/pull/62

Check build of Isabela's documentation PR

Challenges I faced and things I'd like assistance with 🙏

I'm ok

My availability for next week

I'll be here

Important items for discussion 💬

what will be our time allocation for the following week?

To answer

What help do you need from me?

@steff456 needs review on GitHub dark PR
@steff456 needs review (and merge?) on audit PR
@gabalafou help figuring out about the aforementioned "I had a hard time figuring out why sidebars can be tabbed when installing Jupyter Lab (v4.0.3) via Conda but not when building source from latest main branch." (optional).
@isabela-pf doesn't need anything at the moment because she needs to reply to some existing review.

For

How are we doing for deliverables?

We feel like most of the answers are in the updates. Tasks are still moving rather than blocked. I feel like all the things I'm seeing in the to do section of the project board has been there for a bit, and they are not ones we've recently discussed. This makes me think everything we have discussed for the end of grant is at least in progress.
Is that a helpful answer?


[SPARK-46921][BUILD] Move ProblemFilters that do not belong to defaultExcludes to v40excludes

What changes were proposed in this pull request?
This pr just move ProblemFilters that do not belong to defaultExcludes to v40excludes.
Why are the changes needed?
We should not arbitrarily add entries to defaultExcludes, as it represents never participating in the mima check
Does this PR introduce any user-facing change?
No
How was this patch tested?
Mima check passed
Was this patch authored or co-authored using generative AI tooling?
No

Merged to master.

Thanks @dongjoon-hyun @srowen


Support for whitecube/nova-flexible-content

Love the simplicity of the package and wish to use it on one of our projects.
We're running into an issue with https://github.com/whitecube/nova-flexible-content; when the content is reordered (i.e. rerendered), the tinyMCE field breaks. The content entered goes empty, and no content can be entered after. The toolbar does remain intact.
I've done some initial research but could not quickly find a solution.

Tested and working with latest Flexible Content

Hi there, since moving from flexible content to repeater fields in nova4, upon creating new models (it works for update) the second repeated element does not show the editor anymore (it's just a textarea). Should I open a new ticket and provide more information (it's pretty similar to this one)

Hi there, since moving from flexible content to repeater fields in nova4, upon creating new models (it works for update) the second repeated element does not show the editor anymore (it's just a textarea). Should I open a new ticket and provide more information (it's pretty similar to this one)

Thank you,
I honestly hadn't seen this field in the new updates yet. Yes, if you create a new ticket it is more convenient for everyone to see the problem and propose a fix (I'll see if I can do something in the next few days).


(Backend) Create comment system for posts

Make a new comment object that has a foreign key to a post object. Comments should have the following functions attached to them so the frontend can make proper interactions.

CreateComment() [create a comment from a postid and a text body. Should auto-fill user from token and requires login]
DeleteComment() [delete a comment from comment id. requires login]
EditComment() [edit a comment from comment id. requires login]

Implemented.


Misc doc changes

Besides other documentation changes, this commit ensures the generated
HTML doc for HexDocs.pm will become the main source doc for this Elixir
library which leverage on latest ExDoc features.

Great stuff, thank you!

Ah, we should add the format check to the CI. Will fix in master.

Well, it is already done :) https://github.com/absinthe-graphql/absinthe_relay/pull/181


Model and dataset download and usage

Hello,
I want to download the trained models and dataset, however, they were put in Google Drive which is not easy to transport to HPCs or not accessible in regions without Google services. Can you put them on ftp servers for easy download? CyVerse can store user uploaded data and generate ftp links, for example.
The link to ClassifyTE_dataset is provided wrong. I clicked open it and it's the ClassifyTE.zip file.
Do you have guidelines to use the trained models to classify unknown sequences? Thank you very much!
Best,
Shujun

Hi Shujun,
I have updated the links and readme file in github. Right now the datasets
and models are not that huge that is the reason why we have uploaded them
to google drive. I can upload it to dropbox if dropbox would be easier for
you.
If you follow "demo" and "deployment" section from readme file, you should
be able to use trained models. Let me know if you have any questions.
Best,
Manisha
On Wed, Sep 23, 2020 at 9:18 PM Shujun Ou<EMAIL_ADDRESS>wrote:

Hello,
I want to download the trained models and dataset, however, they were put
in Google Drive which is not easy to transport to HPCs or not accessible in
regions without Google services. Can you put them on ftp servers for easy
download? CyVerse can store user uploaded data and generate ftp links, for
example.
The link to ClassifyTE_dataset is provided wrong. I clicked open it and
it's the ClassifyTE.zip file.
Do you have guidelines to use the trained models to classify unknown
sequences? Thank you very much!
Best,
Shujun
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/manisa/ClassifyTE/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABAQQGGOXUGPYQQHPNLQ5G3SHKT5ZANCNFSM4RXXMPMQ
.

--
https://about.me/manishapanta?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb
Manisha Panta
about.me/manishapanta
https://about.me/manishapanta?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb

Hi Manisha,
Thank you for your prompt response. I can see the updated dataset download link now. Could you help to upload them to the dropbox? I can test if the dropbox works better on my side. Thanks!
Shujun


Add this to the System.Diagnostics.Eventing.Reader.EventLogInformation.Attributes documentation

Please consider adding the following to the System.Diagnostics.Eventing.Reader.EventLogInformation.Attributes documentation.
`
System.Diagnostics.Eventing.Reader.EventLogInformation eli;
System.IO.FileAttributes fa = (System.IO.FileAttributes)Enum.ToObject(typeof(System.IO.FileAttributes), eli.Attributes.Value);
`

As far as I can tell by looking at the Reference Source and the documentation, you're right that EventLogInformation.Attributes contains FileAttributes, even though it's typed as int.
But I don't think just code like the one you're showing is enough, an explanation about what the code does and why is also necessary.
Also, your code is unnecessarily complicated and won't work correctly if Attributes is null.
What do you say about the following?

The values of this property are the same as those of the FileAttributes type. To make working with this property easier, you can convert the value to that type using the following code:
EventLogInformation eventLogInformation;
FileAttributes? fileAttributes = (FileAttributes?)eventLogInformation.Attributes;

We agree with the suggestion to add this considering @svick's comments that it should consider the Nullable case. Putting in our backlog for now, but we'll gladly accept a PR.


AttributeError: module 'numpy' has no attribute '_no_nep50_warning'

Tried to ran:
from textblob import TextBlob
s = TextBlob('Test Text.')
s

And this error came out.
Potentially about the versions of numpy?

I solved it by reinstalling numpy in conda prompt, but not sure what the exact problem was.


Update find_audio_files to look into subfolders as well

I just downloaded EchoThief RIRs and naively thought the find_audio_files would look into subfolders. It did not and I got a nice error message telling me it could find any audio file...
The reason is that EchoThief RIRs are stored in subfolders by category.
We should update find_audio_files to look into subfolders.

Let's copy this one:
https://github.com/iver56/audiomentations/blob/master/audiomentations/core/utils.py#L9


RM naming - delimiter vs filter

I always assumed that the RM sample name was defined by the delimiter (just like all the other samples), not the filter used to assign it.  However this is not the case, which means when you go to the visualization (concordia or wt mean) the name of the RM is whatever you used to type into the filter.  If the RM has a really distinctive name, one might only filter using the first few letters:

This is less than ideal, but is mostly an aesthetic/clarity of labelling problem on the plots.  Can obviously be overcome by typing in the full name of the RM into the filter, but if it is easy to transition the RM name from the filter to the delimiter that would be better.

So if you use the hyphen as the delimiter and the dropdown box of sample names to "type" your filter, you get TEMORA.  It appears you are using a count of 3 characters as the delimiter?

I have set the hyphen as the delimiter, not character count.  When I use the filter to define the RM (no dropdown box, just start typing in the the Manage Sample field) I might only type a couple of characters because that is enough to identify the RM (we commonly use one that starts with 6_ _ _ , most of our unknown samples start with 1_ _ _ _).  So then whatever I put in the filter (1 character, 3 characters) becomes the label that appears on plots etc (lower image above).

SQUID "knows" that the name is 6266 because that is what I have set in the delimiter (it shows up properly in the dropdown), it just uses the filter as the ID moving forward, not the name.

My point is don't type anything, use the dropdown box to get the full set of sample names and pick one.

Yes - there are many ways to solve this problem (type it fully, or use the dropdown).  Maybe we don't need the filter option, it provides 2 ways to do the same thing and one of those ways is prone to funny labelling further downstream.

I agree - I added the dropdown a while back but left the filter typing in ... it really serves no purpose since you are going to choose a sample by name ultimately.  OK if we remove it?

@NicoleRayner and @sbodorkos  - thoughts?

OK by my to remove the filter option.


[Bug]: crafting planks with axe consumes it

General Info

[X] I didn't added any mods
[X] I get this issue consistently in single-player
[x] I get this issue consistently in multi-player
[ ] I am crashing and can provide my crash report(s)
[X] I am using the latest version of the modpack

Your launcher
ATlauncher
Modpack version
1.0.2
Describe your issue
when you use axe to make planks, it consumes axe. its pretty annoying at start of game when you don't have any saw
at previous version 1.0.1 i didn't saw that bug
Steps to reproduce the issue

get logs and any axe
try craft planks
your axe is gone

Additional Information
https://gist.github.com/joinmod/fe2b60534e2b8cf4270eb8933988ec51

The Fix is posted on my Discord


Fix hasMessageAvailable might return true after seeking to latest

Motivation
See https://github.com/apache/pulsar-client-python/issues/199
There is a race condition when hasMessageAvailable is called after seek if the start message ID of Reader is earliest.
In ConsumerImpl::hasMessageAvailableAsync, if the connection is not established at the moment, lastDequedMessageId_ will be earliest because no message is received. Since lastMessageIdInBroker_ is also earliest, getLastMessageIdAsync will be called and then it comes at
https://github.com/apache/pulsar-client-cpp/blob/e2cacb7dfb57b6d059b49fead2e1611548ff89b0/lib/ConsumerImpl.cc#L1554
However, before getLastMessageIdAsync is called, messageId was earliest because lastDequedMessageId_ and startMessageId_ were both earliest. However, when the callback is called, the startMessageId_ has already been updated to latest in connectionOpened, so we should compare to latest.
Modifications
In the callback of getLastMessageIdAsync, retrieve the latest value of startMessageId_ to compare rather then reusing the old value.
Refactor the seek flow to reset the seek states and trigger the callback after updating the startMessageId_.
ReaderTest.testHasMessageAvailableAfterSeekToEnd is added to cover the changes.

@merlimat @shibd @RobertIndie @Demogorgon314 Could you take a look?

@RobertIndie @shibd Comments are addressed, PTAL again.


Settings/Storage shows again wrong partitions

Should be fixable by custom settings

Fixed by using Tama https://github.com/sailfishos-sony-tama/nemo-qml-plugin-systemsettings


Setting rwsetV1ProtoBytes as binary using local .gitattributes

Description

Motivation and Context

Fixes #
How Has This Been Tested?

Checklist:

[] I have added a Signed-off-by.
[] I have either added documentation to cover my changes or this change requires no new documentation.
[] I have either added unit tests to cover my changes or this change requires no new tests.
[] I have run golint and have fixed valid warnings in code I have added or modified. This tool generates false positives so you may choose to ignore some warnings. The goal is clean, consistent, and readable code.

Signed-off-by:

Senthil, I think that you pushed your changes from your existing branch to this new branch. It shows me as if you want to merge 9 commits (which are from the previous branch). I guess, that you would need to get a new copy of this branch in your local and then shift your changes from your  existing local to new local before pushing a PR.


Having an issue using Observer from Rails Engine

I keep getting this error when I start a Rails 4.2.4 application consuming a Rails Engine with a Mailer Observer configured.
engine.rb configuration:
config.autoload_paths += %W(#{config.root}/app/observers)
config.active_record.observers = 'NameOfModule::NameOfObserver'

Error:
$ rails s
=> Booting Thin
=> Rails 4.2.4 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
=> Ctrl-C to shutdown server
Exiting
/Users/User/.rvm/gems/ruby-2.3.0@code/gems/attr_encrypted-1.3.5/lib/attr_encrypted.rb:259:in `method_missing': undefined method `add_observer' for Buccaneer::EntryMailer:Class (NoMethodError)
	from /Users/User/.rvm/gems/ruby-2.3.0@code/gems/actionmailer-4.2.4/lib/action_mailer/base.rb:569:in `method_missing'
	from /Users/User/.rvm/gems/ruby-2.3.0@code/gems/rails-observers-0.1.2/lib/rails/observers/active_model/observing.rb:364:in `add_observer!'
	from /Users/User/.rvm/gems/ruby-2.3.0@code/gems/rails-observers-0.1.2/lib/rails/observers/activerecord/observer.rb:105:in `add_observer!'
	from /Users/User/.rvm/gems/ruby-2.3.0@code/gems/rails-observers-0.1.2/lib/rails/observers/active_model/observing.rb:341:in `block in initialize'
	from /Users/User/.rvm/gems/ruby-2.3.0@code/gems/rails-observers-0.1.2/lib/rails/observers/active_model/observing.rb:341:in `each'
	from /Users/User/.rvm/gems/ruby-2.3.0@code/gems/rails-observers-0.1.2/lib/rails/observers/active_model/observing.rb:341:in `initialize'

I will add any further details if I discover any.

OK, I fixed by not setting config.active_record.observers= in engine.rb nor having an observe class method call in Observer, yet instead just using register_observer class method call in Mailer itself. Must be a special issue with Rails Mailer Observers


Needs for EPOLLET

I am trying poll for simple_httpd and failed.
One of the reason is I have several OCaml domains polling on the same socket to accept connection (which works very well with select)
For this to work, I need the EPOLLET flag. For other sockets, I want level trigerred poll because I do not want to read all available data untile read/write anser AGAIN.
So I would need access to the EPOLLET flag in Poll.set
Clearly the question is : is there an equivalent flag on the other platform. But the other platform could ignore the flag ?

Looking at the code, I also see you use ONESHOT by default. I could manage that on read/write socket, but better not
on socket where I wait for accept. But taking this in acount does not solve my problem.
So the best would be to have access to both flags for epoll.

I managed to have simple_httpd working with polly (which is a direct interface only supporting linux epoll) but not with your software. From what I tried, I think there is something I don't understand, because even without EPOLLET and with
ONSHOT everywhere it should work.
If you want to have a look, simple_httpd has 3 branches :

master using select
poll using your poll sofware
polly using polly.

The code using select/poll/polly is in src Simple_httpd_domain.ml in only one function.
Doing make, does some test that never ends with poll while working with the 2 others.

Clearly the question is : is there an equivalent flag on the other platform. But the other platform could ignore the flag ?

Kqueue supports edge triggered events as well so at-least on BSD like systems an edge triggered behavior could be supported. At the moment, I pick one shot as the lowest common denominator as on windows I don't have access to the EPOLLET flag in the wepoll library.

From what I tried, I think there is something I don't understand, because even without EPOLLET and with ONSHOT everywhere it should work.

I'd expect something workable with ONESHOT events as well. I'll try and take a look at your project soon to see what's going on with poll. Thanks for preparing a reproduction case!

I need to spend more time on this to confirm, but I wouldn't be surprised if the problem is related to the epoll bindings in this library trying to be "smart" by keeping track of the existing state of each file descriptor's flags that are registered with it.
To do this the library maintains internal state in a hash table (https://github.com/anuragsoni/poll/blob/main/src/epoll_poll.ml#L66), and if a fd isn't unregistered via Poll.set poll fd Poll.Event.none it will lead to issues because this internal hash table will never remove the entry recorded for the fd. Similarly sharing the poll descriptor among multiple threads is most likely going to cause issues without edge triggered modes, so in my multicore explorations I've ended up with a workflow of assigning a unique poll descriptor per domain (https://github.com/anuragsoni/sandbox/blob/c8e3c122510a1f826f9786a7aca3027ab6d4ef4a/ocaml/async_io/src/local_run_queue.ml). None of this is to say that this library can't be updated to help resolve this paper cuts though! I am not tied to having wepoll support in the library so I'd be open to removing that and focus on kqueue and epoll only if that ends up allowing a better api for bad/linux. Similarly, the internal state used to support set style use so the end-users don't need to maintain their own state to figure out whether to call add vs update can be revisited. This probably shows my bias towards kqueue as there is no need for a different call for add vs modify vs delete when it comes to updating the kernel state for the readiness interest, but it ends up being less confusing of an api for the end-users we could adapt the interface to better match what epoll does and have the kqueue implementation match that.

Remark: I use personnal branch for http-cookie and ocaml-ssl. PR's are submitted, but if you want to try now,
have a look at .github/worflow to see the correct opam pin command.

I need to spend more time on this to confirm, but I wouldn't be surprised if the problem is related to the epoll bindings in this library trying to be "smart" by keeping track of the existing state of each file descriptor's flags that are registered with it.

That is something which is actually recommended on the man of epoll to solve some frequent problem. So this is not chocking.

To do this the library maintains internal state in a hash table (https://github.com/anuragsoni/poll/blob/main/src/epoll_poll.ml#L66), and if a fd isn't unregistered via Poll.set poll fd Poll.Event.none it will lead to issues because this internal hash table will never remove the entry recorded for the fd.

Similarly sharing the poll descriptor among multiple threads is most likely going to cause issues without edge triggered modes, so in my multicore explorations I've ended up with a workflow of assigning a unique poll descriptor per domain (https://github.com/anuragsoni/sandbox/blob/c8e3c122510a1f826f9786a7aca3027ab6d4ef4a/ocaml/async_io/src/local_run_queue.ml).

I don't do that and do not remember the use case for this that I saw somewhere. What I do is a socket on which I listen
to accept connection is set in one distinct epoll on each domain. The problem is to avoid the risk of accepting the socket twice in two different domain. What I understood from the documentation is that this require EPOLLET, but I am not sure.

None of this is to say that this library can't be updated to help resolve this paper cuts though! I am not tied to having wepoll support in the library so I'd be open to removing that and focus on kqueue and epoll only if that ends up allowing a better api for bad/linux.

I think using GADT you could easily have option specific to each backend, without any possible confusion. This way you can have the best interface on each backend and keep a common denominator. Moreover, the user can write code
specific for each backend. I would do in backend.ml
type _ t = ..
type epoll = [ `Epoll ]
type wepoll = [ `Wepoll ]
type kqueue = [ `Kquueu ]
type _ t += Kqueue : kqueue t | Epoll : epoll t | Wepoll : wepoll t

Then you could have flag of type _ Event.t (with the correct variance ...) for individual flag with some flag of type
[Epoll | Kqueue] Event.t if there are common to two backend.
Then epoll would also be parametrised by the backend and set type would only allow
the option for a specific backend, that the user can build by pattern matching on the backend type.
The only difficulty is to get variance and subtyping right. It you want I can give it a try.
But I will also try to monitor your hashtbl to see if the problem is related to that.

I did ocamlformat.

mdx 2.1.0 with ocaml 4.13.1 reports no error
mdx 2.1.0 with ocaml 5.0.0 reports the above error (on master branch too, so I do a sperate issue)

A note on epoll: I think add is O(1) amortized (hashtbl ?) while upd is O(1) in all cases. This is why
I don't think remove should be automatic. A good way to manage connection to a server is to add at accept,
remove at close and upd during the treatment.
I think poll should allow this scheme when the backend is epoll. A way to do it is to have a flag "keep".
I looked at libevent, but the fact I can not write the mainloop myself rules it out for me. Your solution is probably
the best lightweight solution but it needs more tests.

When you tell me you view point, about the above messages, I will try to create clear separate issue and start
to do pull request. I already added on for tests.

This is why I don't think remove should be automatic. A good way to manage connection to a server is to add at accept, remove at close and upd during the treatment.

Remove isn't automatic even in the current state. Once an event is fired, the internal state for a fd for the epoll backend isn't removed till Poll.set poll fd Poll.Event.none is called. The problem arises if this operation doesn't take place for closed fds as then this state is never cleared. The intention (should be documented) is that since events are registered as one shot, the user on seeing a file descriptor in a poll event list should notice that its ready for I/O, and then keep using it till seeing EAGAIN or EOF. If they encounter EAGAIN they register interest again, and on EOF/closing they need to remember to set none so epoll's internal state is cleared. The current poll interface means that the  Kqueue backend doesn't require this internal book-keeping, and whenever a FD is closed the fd is also removed from the kqueue descriptor by the kernel. For Epoll the fact that poll is managing an internal state can lead to some surprises since there are no ways to guarantee that the internal state gets cleared so its all based on users remembering to do that.

I looked at libevent, but the fact I can not write the mainloop myself rules it out for me. Your solution is probably
the best lightweight solution but it needs more tests.

I'm not 100% sure about your use-case, and I'm not sure you've seen these yet, but there are pretty nice OCaml bindings for libev available as well https://github.com/rgrinberg/lev which also cover more use-cases like timers/signals etc. It could also be of interest for your use-case!

Remove isn't automatic even in the current state. Once an event is fired, the internal state for a fd for the epoll backend isn't removed till Poll.set poll fd Poll.Event.none is called.

I understood this, but I don't think it is the best choice. I suggest

Poll.set poll fd Poll.Event.none remove immediatly (del on epoll)
Poll.set poll fd Poll.Event.keep remove all event but keep in the epoll list (upd on epoll)
both may do the same thing on other backend (meaning the option could be available on all backend)

The problem arises if this operation doesn't take place for closed fds as then this state is never cleared.

This is always a problem, you have to clear information about individual connection in many application. I don't think
this library should care. The user should just need a way to remove a socket from the hashtbl even after the socket was closed.
Maybe some exception should trigger a check on all sockets and removal (if wait raises a Unix_error check all sockets with fstat, do a try Unix.close s; None with e -> Some e on all wrong sockets an reraise an exception containing the original exception and the list of closed socket with the optional exception generated by close. This way the library helps the user but no information is lost for the user to debug.

The intention (should be documented) is that since events are registered as one shot, the user on seeing a file descriptor in a poll event list should notice that its ready for I/O, and then keep using it till seeing EAGAIN or EOF. If they encounter EAGAIN they register interest again, and on EOF/closing they need to remember to set none so epoll's internal state is cleared.

You don't want oneshot with non blocking io as you may even want to go to wait before reading/writing all (if you want a fair scheduler).

You want oneshot + epollet for listenning for connection on the same socket in different thread/domain.
This way the linux scheduler will work for you and only one thread will accept the connection.

This means you can not use epoll if you can't set oneshot/epollet manually for all (most?) application. See my suggestion for GADT.

The current poll interface means that the Kqueue backend doesn't require this internal book-keeping, and whenever a FD is closed the fd is also removed from the kqueue descriptor by the kernel. For Epoll the fact that poll is managing an internal state can lead to some surprises since there are no ways to guarantee that the internal state gets cleared so its all based on users remembering to do that.

With a careful implementation, we can prove that both state are always in sync. This is hard but possible, especially
that epoll man page describe situations where it recommends keeping your own state.

I'm not 100% sure about your use-case, and I'm not sure you've seen these yet, but there are pretty nice OCaml bindings for libev available as well https://github.com/rgrinberg/lev which also cover more use-cases like timers/signals etc. It could also be of interest for your use-case!

Thanks for the pointer... opam search libevent does not report libev! (only conf-libev). And indeed could work for me.

Another problem for me (more general) is a lack of solution that use what is available to build the best possible
mutex/semaphore/condition on which you can wait together with sockets. It is a pity that libevent does no provide this, even if you can implement it yourself .... with the high risk of not getting the semantics you want! Currently I wait with a
low timeout (around 30ms) and then check all mutexes.
Typically in my system I have a mutex around session information available on all thread/domain to treat the case of several connections for the same http client. Ideally I need to wait on this mutex (if it is locked) in the same time as I wait on sockets.  And for instance, I think implementing mutex/condition above epoll is clearly possible, but tricky to make sure all corner case work and we get the best possible efficiency. So it should be provided by the library. I did not find this  (may be I missed it) on libev/libevent/poll for ocaml nor libevent for C.

Poll.set poll fd Poll.Event.keep remove all event but keep in the epoll list (upd on epoll)

I don't really see what this adds over the current interface. For both one shot and edge trigger events once an event has surfaced via wait you won't see it again. In the case of one shot you won't see it till you register it again, in the case of edge trigger you won't see it again till the fd state changes. In all possible cases if you want the behavior of keep you simply avoid called none.

The user should just need a way to remove a socket from the hashtbl even after the socket was closed.

Closing a fd before de-registering it with epoll is a bad idea, so I'd rather only keep an interface that encourages at-least using EPOLL_CTL_DEL before closing a fd. https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-broken-22/ has more details.

You often don't want oneshot nor epollet with non blocking io as you may even want to go to wait before reading/writing all (if your application is a "try to be fair" scheduler).

This should work with one shot without any issues. If you re-arm an even for a fd that initially surfaced via epoll_wait but without actually reading/writing from it, it should still show up in the next epoll wait.

Poll.set poll fd Poll.Event.keep remove all event but keep in the epoll list (upd on epoll)

I don't really see what this adds over the current interface.

It replaces in many cases (like mine) one add + del by one upd. And I think upd is more efficient than both add and del sepatly.

The user should just need a way to remove a socket from the hashtbl even after the socket was closed.

Closing a fd before de-registering it with epoll is a bad idea

Sure, this is always a bug. But you want the user to easily diagnose the bug. This is why I want to report the exception
generated by close to the user, If automatic removal is a good idea.

You often don't want oneshot nor epollet with non blocking io as you may even want to go to wait before reading/writing all (if your application is a "try to be fair" scheduler).

This should work with one shot without any issues. If you re-arm an even for a fd that initially surfaced via epoll_wait but without actually reading/writing from it, it should still show up in the next epoll wait.

I don't want to pay the price of rearm.  I at least want to be able to time and experiment both solutions.
In that case, you replace one add + del, with nothing.
There are also all the other flags of epoll (hup, rdhup, rdband, ...) which are sometimes needed.
Again, GADT would allow fine control on each platform of all options and you still did not say if you like the idea?
I am proposing to do a PR for this at least to support all epoll flags. And also I would look how to warant sync between epoll internal state and your state. But I won't do a PR if there is no chance you would accept it ;-)

And by the way, as the user may want to attach information to sockets, it would be nice to propose user data in your internal hashtable.
Do you want me to create 2-3 issues with specific goals, we discuss on each issue... and then I do a PR ?

There are also all the other flags of epoll (hup, rdhup, rdband, ...) which are sometimes needed.
Again, GADT would allow fine control on each platform of all options and you still did not say if you like the idea?
I am proposing to do a PR for this at least to support all epoll flags. And also I would look how to warant sync between epoll internal state and your state. But I won't do a PR if there is no chance you would accept it ;-)

I'll admit that feature parity with the entire set of options available within epoll, kqueue or windows/IOCP/wepoll was not my intention. The current state of the library is a minimal set of features that are available on all platforms that can feasibly provide an I/o event notification view on the various platforms, and nothing more than that. Most of this started as experiements in using effects for writing direct style single threaded event loops  (https://github.com/anuragsoni/sandbox/tree/main/ocaml/effect_poll), or use it as one small piece within a multi-threaded task system (https://github.com/anuragsoni/sandbox/tree/main/ocaml/async_io) and I admit this is a very opinionated set of interface that won't cover very general use-cases. I'd wager this might be a roadblock with most abstractions over various platform libraries as for better or worse most of these platforms have very different feature-sets when it comes to the I/o event notifications and its hard to come up with a consistent api that works well everywhere but still allows access to the platform exclusive interfaces/flags.
I'd suggest that if you know for sure that linux is the only platform you'd like to support then go directly to epoll and use all the features a specific epoll interface (like polly or other direct c bindings to epoll) will provide as your application will be able to benefit from it instead of having to be constrained by a generic interface like this library. If you'd like to explore targeting more than linux (say macOS or BSD) but with a different api that exposes more of the internals, I also have kqueue bindings available (https://github.com/anuragsoni/kqueue-ml) that are very low level and expose the underlying kqueue sys calls as-is, which you can then wrap yourself in a layer that fits your application best.

I'll also mention that if you are writing multi-threaded servers, you might also benefit from the abstractions provided by https://github.com/ocaml-multicore/eio since it also comes with a lot of other concurrency primitives one might need in such applications. They also have a more modern linux I/o stack via io-uring that can be of interest if you know you are exclusively going to target linux.

I am aware of eio and decided not to use for two reasons:

it does not work yet and I want to work now (see my issue on cohttp-eio, which is still unsolved). Guess they have problem with polling!
I believe a scheduler adapted to an http server is better than a general scheduler. I do not need
a full set of primitive, like eio would propose. Have a look at my project simple_httpd

Still I would like to support linux and bsd and why not windows.
I am looking for simple_httpd to find the best compromise in

simplicity to write an httpd server
good set of features (I support cookies, session (work in progress), ssl, zip)
good performance (using ocaml 5.0 and effect)
minimal set of dependencies (which rules out for instance anything from janastreet which almost always bring 100 dependencies). I currently depend on http-cookie, ssl, zip (optional) and result and I would like to have  ssl optional.

So I found you code quite nice, I at minimum need the epollet/onshot combination, but probably a few other thing.
And I add rather have access to all feature of epoll and kqueue to experiment.
I currently have

a select implementation working very well, but limited to 1024 sockets, apparently even if one increase ulimit
and an epoll/polly that nearly works (still some error on github actions that are not systematic and that I can not   reproduce on my machine).

So I would be happy to collaborate with you on poll, but I need to fix at least the missing epollet. I also think
that if you maintain a socket hashtbl it is better to open it to the user for its own data and keep the socket in it
until they are closed (which is an easy invariant to maintain ?).

I'd probably recommend taking the codebase in poll as a starting point then and adapting it your needs by removing things you don't need and modifying it specifically for what's needed for your use-case. I mostly say this as I'm not sure about how much time I'll be able to allocate to evolve the library beyond my intended goal for it which is the current state where it fills the role I need it for as a small piece in a different exploration where the current api suffices. I only intend for this library to be used in more generic I/o libraries where fine-grained control over the exact flags used for epoll vs kqueue (vs something) else isn't as important.
I no longer use OCaml at my day-job, and at the moment most of the time I do get to spend on OCaml is spent on other projects, and almost exclusively with janestreet libraries since they help me make progress faster by being able to leverage their well tested high quality libraries. I mostly say this to level-set expectations about my own time commitment, and I think you'll be able to make progress in a better manner and definitely a lot faster without being bottle-necked on poll. The library is fairly small and should be easy-ish to vendor/port to live within simple_httpd. Most of the complication lies in the build setup that uses janestreet's ppx_optcomp for conditional compilation, but there could be simpler ways of achieving that if you are writing a more targeted wrapper around kqueue/epoll.

Thanks for your comment ... I am still thinking, but I might fork your library then. If I can keep it separated from simple_httpd I will be happy.
I think I will got for an oneshot/epollet strategy, maintening a queue of socket with available data. This way,
I get O(1) scheduler I (this is the only way, and bench from libevent show that this is possible).
This will give

socket are encapsulated in a small record
a hashtbl with all available sockets (not closed)
a queue with socket with available data
a poll list with all other sockets + socket listenning for connection.
invariant: the hashtbl is the union of the queue and the poll socket which are not listening socket
the hashtbl is available to the user to store its own data.

But before doing that I have to think about the implementation of mutex on top of file descriptor,
(writing one byte when I unlock, reading that by when I lock) and compare if the benefit of
being able to wait on a mutex with select/poll is not counterbalanced by using socket instead of atomic memory operation.
Cheers,
Christophe
PS: feel free to close this issue as you please.

Thanks for your comment ... I am still thinking, but I might fork your library then. If I can keep it separated from simple_httpd I will be happy.

That sounds like a solid plan! I'm going to close this issue for now since I don't anticipate making changes directly within poll. Goodluck with your explorations!


More secure key storage: It’s not practical to use a simple encrypted mnemonic string in localStorage as the ‘vault'. Please refer to MetaMask’s implementation on how to persist the Keystore.

Use https://github.com/MetaMask/KeyringController

Fixed above by:

Switched password encryptor to the one used by Metamask  https://github.com/0xkarl/vite-extension/commit/bb2cb23c088390f3bed4edf47ceb40391ad77574
Switching from localStorage to extension.storage.local api local-store.js#L51 cache.js


How to utilize the Sentry API?

I am trying to figure out how to addUser context within the error handler - how would you do this with this package, or is this handled by something like toucan-js?

Thanks for the feedback.
There is currently no support for the Sentry API other than error capture.
This is a temporary workaround for @sentry/javascript #8291, as we are focusing on being able to hide the details of things and deploy them immediately in your application.
However, since @sentry/javascript #8291 support does not seem to be progressing, we will consider supporting it if we receive many similar requests.