Stacksets deployment of Discover

Updated version of https://github.com/jhseceng/discover-org-registration.
The solution creates a stackset which can be used to deploy Discover for AWS to all accounts within an AWS organization.

Can someone merge :)


Map Tameable.getWorld()

Marked as draft, dont merge untill after 1.21 I dont want to risk breaking anything before the release.


Create get-enumeration-function.yml

https://github.com/fireeye/capa-rules/issues/31

Thanks for the feedback.
I made the necessary changes.

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

I'd vote to remove the nursery rule instead and keep this one with EnumWindows.

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

I'd vote to remove the nursery rule instead and keep this one with EnumWindows.

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

Ok.

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

Ok.

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

I'd vote to remove the nursery rule instead and keep this one with EnumWindows.

Removed.

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

I'd vote to remove the nursery rule instead and keep this one with EnumWindows.

Removed.

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

I'd vote to remove the nursery rule instead and keep this one with EnumWindows.

@johnk3r @mr-tz I apologize if I wasn't clear I originally meant to remove the nursery rule 😅

Awesome, thank you for your work! It looks like an existing rule in the nursery contains a match for EnumWindows:
https://github.com/fireeye/capa-rules/blob/1ff994f7916d66e39b4b5b8dbb310d0e0b051f7f/nursery/enumerate-graphical-windows.yml#L11
Can you remove the above rule as part of this PR? Otherwise, everything looks great! 🚀

I'd vote to remove the nursery rule instead and keep this one with EnumWindows.

@johnk3r @mr-tz I apologize if I wasn't clear I originally meant to remove the nursery rule 😅


Add shortcuts for common, but large search queries

These could be specified in the config file, but being able to apply some other set of search rules, either with another option, or a special syntax in the query would be useful.
If added to search query, you could use maybe $FilterAliasName, as I don't think dollar sign works in gmail search.

This is effectively provided by aliases (issue #16), particularly since it allows you to insert other terms into the query. closing


Link to user from book list

Now from a list of books, we can link to the user's profile.

oh hai


Compile reporting refactoring signatures better ignores

This is a grab-bag of features I was developing while working crazy hours this week. It supports:

Inline error reports on compile fail
Method signature help
Improved goto definition
Refactors how the extension handles the unhandled texts, and introduces the notion of state when initializing.

In addition, also refactors the implementation to better separate classes for future dev.
Please note. All checks pass; but I've been doing this like a mad-man as I needed the features : there's no extra tests at this point.
Saying that - I've got some issues which I'm unsure come from this build or not:
Sometimes there's a Run Time Error; but the debugger doesn't enter debugging mode
I want to make the app exit run state on compile error but don't know how.

Pull Request Test Coverage Report for Build 76

60 of 700 (8.57%)  changed or added relevant lines in 2 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.8%) to 10.081%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/brightScriptDebugSession.ts
44
320
13.75%

src/RokuAdapter.ts
16
380
4.21%

Files with Coverage Reduction
New Missed Lines
%

src/RokuAdapter.ts
1
6.1%

Totals

Change from base Build 67:
-0.8%

Covered Lines:
79

Relevant Lines:
720

💛  - Coveralls

The code coverage percent has decreased. Can you add some unit tests to cover and validate the new functionality that you have added in this PR?

I can add some tests; but we’re gonna have to horse-trade about coverage :
There’s no way I can get coverage for all of this any time soon, and the
features work and are useful, other’s are already using my branch, such is
their utility. Also, I’m shortly gonna rip out a lot this code and replace
the regex’s with your Brightscript parser so spending time writing tests
now to attain an arbitrary coverage number doesn’t seem like a worthwhile
pursuit imho, and with all due respect. I usually develop with TDD; but I
repurposed code from several places while learning how to write all of
this, so I didn’t benefit from using tests as struts/rigs in this case.
I do concur that some testing needs to happen, especially with the changes
I made to your debugger code to introduce states. The question is: what’s
going to be acceptable to you?
On November 28, 2018 at 7:13:53 AM, Bronley Plumb<EMAIL_ADDRESS>wrote:
The code coverage percent has decreased. Can you add some unit tests to
cover and validate the new functionality that you have added in this PR?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TwitchBronBron/vscode-brightscript-language/pull/37#issuecomment-442426266,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKIUc3QE4mVZRquvbGgl57bWLdMTmMpks5uzn4BgaJpZM4YxKt9
.

I definitely am not expecting you to hit 100% code coverage for your changes. For now, just shoot for writing a test or two that validates the major pieces of functionality, so that we can try to guard against regression issues in the future.

I'm resubmitting a new PR - it's simply too much work to rebase all of this.


Projects go missing when there is no image

When the image tag is removed and the screen width > 768 px, project cards are missing.

Hi,
This was addressed in release 1.2.1. For more information on how to add projects without images, see here.
Hope that helps! If you some reason this isn't what you're referring to, let me know and I'll re-open.
Ryan

Sorry about the duplicate issue. Didn't see that it was solved. Thanks a lot!

Not a problem, I appreciate you creating the issue regardless!


Component: Clause Template Editor

Is your feature request related to a problem? Please describe.
Provide a specific editor to handle the clause templates
Describe the solution you'd like
This likely will happen within the ContractEditor and will edit a clause template with rich text formatting
Describe alternatives you've considered
N/A
Additional context
N/A

Can you describe more about this issue?

@Innovator007 I think following along in our process with cicero-ui and template-studio-v2, as well as joining a couple working group calls, would be the best bet on getting up to speed on this project.


migrating from django 1.2 -> 1.6

I got this on taggit 0.9.3:
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/taggit/managers.py", line 47, in init
self.unique = False
AttributeError: can't set attribute
but I want to preserve ability to set non unique tag names, is there a way?

The latest version of taggit is 0.18.2. I strongly recommend upgrading, it might fix your issue.

I just tried: on .0.18.2 my e2e are failing because 2 separate users fail on adding the same tag "abc1" to 2 different objects, my models bellow:
class UserTag(TagBase):
user = models.ForeignKey('MyUser', null=True)
class TaggsIntermediate(GenericTaggedItemBase):
tag = models.ForeignKey(UserTag, related_name="tagged_items")
that was working on 0.9.3

What's the exact operation you're performing and what error are you receiving?

WHEN:
tag, status = UserTag.objects.get_or_create(name="abc1", user = User.objects.get(id=request.user.id))
THEN:
ERROR|2016-05-22 08:15:14,461|django.request>> Internal Server Error: /hdesk/tickets/2/add_tag/
Traceback (most recent call last):
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/home/pepto/playground/test_hdesk/hdesk/views/ajax.py", line 66, in add_user_tag
user = User.objects.get(id=request.user.id) )
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
return self.get_queryset().get_or_create(**kwargs)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/query.py", line 391, in get_or_create
six.reraise(*exc_info)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/taggit/models.py", line 86, in save
return super(TagBase, self).save(*args, **kwargs)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/base.py", line 545, in save
force_update=force_update, update_fields=update_fields)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/base.py", line 573, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/base.py", line 654, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/base.py", line 687, in _do_insert
using=using, raw=raw)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/manager.py", line 232, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/query.py", line 1514, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql
cursor.execute(sql, params)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/backends/util.py", line 69, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/pepto/.virtualenvs/hdesk/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 452, in execute
return Database.Cursor.execute(self, query, params)
IntegrityError: UNIQUE constraint failed: hdesk_usertag.name
on pip:
django-taggit (0.18.3)
Django (1.6.11)

can I provide any more helpful info here?

I'm not sure. I'm pretty busy so I unfortunately can't look into it.

try update your django to 1.8

this is not related to django-taggit itself hence should be closed


Remove Blockstream Green Wallets

Closes #290
Removes Android & iOS apps.
Checked the site for Green Address desktop wallet. Couldn't find any mention of tokens so left as-is.

Thanks


Query performance issues between 3.9.0 and 3.17.0

Describe the bug
We're upgrading from 3.9.0 to 3.17.0-preview on a performance intensive product. We've noticed a 23 ms per call increase in query times between the 2 SDKs.
To Reproduce
Upgraded from 3.9.0 to 3.17.0-preview. Read the release notes on all releases to follow the upgrade notes. Added the following client performance items:
`
.WithConnectionModeDirect()
options.PortReuseMode = PortReuseMode.PrivatePortPool;
options.IdleTcpConnectionTimeout = TimeSpan.FromHours(1);
options.AllowBulkExecution = false;
`
Also converted query code from:
`
query.ToStreamIterator();
to
container.GetItemQueryStreamIterator(query.ToQueryDefinition(), null, new QueryRequestOptions { MaxConcurrency = -1, MaxBufferedItemCount = -1 }))
`
Expected behavior
Performance difference between the two versions to be nominal.
Actual behavior
Ran load tests and observed query durations take 20 ms longer with newer APIs and changes.
Environment summary
SDK Version: 3.17.0-preview
OS Version Linux containers
Additional context
Below is a graph of performance comparisons:

Yellow box is the 3.17.0-preview library upgrade with no code changes,
Green box is 3.9.0 pre-upgrade
Red box is 3.17.0-preview with performance related code changes made.

@sboshra and/or @bchong95 can you please take a look?

Thanks for creating this issue @dpiessens! Could you include the diagnostics for a slow query for both 3.8 and 3.17-preview?

@dpiessens if you can also provide a way to reproduce the issue it would be very helpful to root cause the issue.

Apologies it was 3.9.0 not 3.8.0. Here's an example of the baseline diagnostics, I did redact URLs and container names.
Here's a trace from 3.9.0
{     "DiagnosticVersion": "2",     "Summary": {         "StartUtc": "2021-03-02T14:06:20.4003215Z",         "TotalElapsedTimeInMs": 11.7576,         "UserAgent": "cosmos-netstandard-sdk/3.9.0|3.9.0|30681|X64|Linux 5.4.0-1031-azure 32 18.|.NET 5.0.3|",         "TotalRequestCount": 2,         "FailedRequestCount": 0     },     "Context": [{         "Id": "CreateQueryPipeline",         "ElapsedTimeInMs": 2.0086     }, {         "Id": "Microsoft.Azure.Cosmos.Handlers.RetryHandler",         "HandlerElapsedTimeInMs": 1.8598     }, {         "Id": "Microsoft.Azure.Cosmos.Handlers.RouterHandler",         "HandlerElapsedTimeInMs": 1.8553     }, {         "Id": "Microsoft.Azure.Cosmos.Handlers.TransportHandler",         "HandlerElapsedTimeInMs": 0.2206     }, {         "Id": "AggregatedClientSideRequestStatistics",         "ContactedReplicas": [],         "RegionsContacted": [],         "FailedReplicas": []     }, {         "Id": "Microsoft.Azure.Cosmos.GatewayStoreModel",         "ElapsedTimeInMs": 1.7965     }, {         "Id": "GatewayRequestTime",         "ElapsedTimeInMs": 1.6982     }, {         "Id": "PointOperationStatistics",         "ActivityId": null,         "ResponseTimeUtc": "2021-03-02T14:06:20.4022321Z",         "StatusCode": 200,         "SubStatusCode": 0,         "RequestCharge": 0.0,         "RequestUri": "db/reveal/colls/*******",         "RequestSessionToken": null,         "ResponseSessionToken": null     }, {         "PKRangeId": "94",         "StartUtc": "2021-03-02T14:06:20.4023608Z",         "QueryMetric": "totalExecutionTimeInMs=3.33;queryCompileTimeInMs=0.39;queryLogicalPlanBuildTimeInMs=0.11;queryPhysicalPlanBuildTimeInMs=0.37;queryOptimizationTimeInMs=0.02;VMExecutionTimeInMs=2.09;indexLookupTimeInMs=0.89;documentLoadTimeInMs=0.65;systemFunctionExecuteTimeInMs=0.00;userFunctionExecuteTimeInMs=0.00;retrievedDocumentCount=31;retrievedDocumentSize=245648;outputDocumentCount=31;outputDocumentSize=15401;writeOutputTimeInMs=0.10;indexUtilizationRatio=1.00",         "IndexUtilization": "",         "ClientCorrelationId": "02b10690-70c9-4485-b222-81b5c570298c",         "Context": [{             "Id": "AggregatedClientSideRequestStatistics",             "ContactedReplicas": [],             "RegionsContacted": ["https://**********.documents.azure.com:443/"],             "FailedReplicas": []         }, {             "Id": "Microsoft.Azure.Cosmos.Handlers.RetryHandler",             "HandlerElapsedTimeInMs": 9.1108         }, {             "Id": "Microsoft.Azure.Cosmos.Handlers.RouterHandler",             "HandlerElapsedTimeInMs": 9.1067         }, {             "Id": "Microsoft.Azure.Cosmos.Handlers.TransportHandler",             "HandlerElapsedTimeInMs": 0.201         }, {             "Id": "Microsoft.Azure.Documents.ServerStoreModel",             "ElapsedTimeInMs": 9.0283         }, {             "Id": "StoreResponseStatistics",             "StartTimeUtc": "2021-03-02T14:06:20.4024462Z",             "ResponseTimeUtc": "2021-03-02T14:06:20.4113962Z",             "ElapsedTimeInMs": 8.95,             "ResourceType": "Document",             "OperationType": "Query",             "LocationEndpoint": "https://**********.documents.azure.com:443/",             "StoreResult": "StorePhysicalAddress: rntbd:/***********.documents.azure.com:14071/apps/1b020ad4-35b5-405c-817f-54701bf5cd52/services/acd83627-ccde-40f9-8ca4-261837813e6c/partitions/3c3faf19-dcc9-4f08-b65f-41625e86bc59/replicas/132587311508741895s/, LSN: 3711341, GlobalCommittedLsn: 3711340, PartitionKeyRangeId: 94, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 7.41, ItemLSN: -1, SessionToken: -1#3711341, UsingLocalLSN: True, TransportException: null"         }]     }] }

I was still able to get overall client elapsed time and it's between 22 and 27 ms for 3.17.0

@dpiessens if you can also provide a way to reproduce the issue it would be very helpful to root cause the issue.

I'd like to try to get traces from both first. This code is at the core of a lot of moving parts and would be a multi-day project to create an example that replicates what it does. If the traces don't point to the issue then I'll be happy to create one.

I'm working on a 3.17 release which should be done today. This new releases will contain the fix for the ArgumentNullException.

@dpiessens 3.17.0-preview1 is released.

Thanks @j82w, I updated to preview 1 and got a trace. This trace executed the same query, but on a different ID so it hit a different server side partition.  Let me know if anything obvious stands out that I should tweak:
{
    "name": "FeedIterator Read Next Async",
    "id": "ced01b61-0762-47b2-9f96-dc715c0e6ee0",
    "component": "Unknown",
    "caller info": {
        "member": "ReadNextWithRootTraceAsync",
        "file": "FeedIteratorInlineCore.cs",
        "line": 48
    },
    "start time": "09:46:22:240",
    "duration in milliseconds": 21.0619,
    "data": {},
    "children": [{
        "name": "Create Query Pipeline",
        "id": "a129d9ed-a372-457f-8b70-4e88b4aa1e68",
        "component": "Query",
        "caller info": {
            "member": "TryCreateCoreContextAsync",
            "file": "CosmosQueryExecutionContextFactory.cs",
            "line": 85
        },
        "start time": "09:46:22:240",
        "duration in milliseconds": 2.5604,
        "data": {},
        "children": [{
            "name": "Get Container Properties",
            "id": "efc67f67-3c40-4b18-8df0-35b591d6545e",
            "component": "Transport",
            "caller info": {
                "member": "GetCachedContainerPropertiesAsync",
                "file": "ClientContextCore.cs",
                "line": 349
            },
            "start time": "09:46:22:240",
            "duration in milliseconds": 0.0119,
            "data": {},
            "children": [{
                "name": "Get Collection Cache",
                "id": "4a5fc555-d8c6-4ea5-8da8-0f0f7d24f3ad",
                "component": "Routing",
                "caller info": {
                    "member": "GetCollectionCacheAsync",
                    "file": "DocumentClient.cs",
                    "line": 542
                },
                "start time": "09:46:22:240",
                "duration in milliseconds": 0.0016,
                "data": {},
                "children": []
            }]
        }, {
            "name": "Gateway QueryPlan",
            "id": "e128ce0f-004f-4377-9530-f0a9771f402f",
            "component": "Query",
            "caller info": {
                "member": "GetQueryPlanThroughGatewayAsync",
                "file": "QueryPlanRetriever.cs",
                "line": 112
            },
            "start time": "09:46:22:240",
            "duration in milliseconds": 0.4017,
            "data": {},
            "children": []
        }, {
            "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
            "id": "4671a334-115e-4af8-9c9c-410717d163cc",
            "component": "RequestHandler",
            "caller info": {
                "member": "SendAsync",
                "file": "RequestInvokerHandler.cs",
                "line": 148
            },
            "start time": "09:46:22:240",
            "duration in milliseconds": 2.3294,
            "data": {},
            "children": [{
                "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
                "id": "88de0897-25ec-4b45-b0a3-91905abd84e6",
                "component": "RequestHandler",
                "caller info": {
                    "member": "SendAsync",
                    "file": "RequestInvokerHandler.cs",
                    "line": 51
                },
                "start time": "09:46:22:240",
                "duration in milliseconds": 2.3144,
                "data": {},
                "children": [{
                    "name": "Send Async",
                    "id": "60ae3d73-4d90-4902-92f1-3d99dda58a0f",
                    "component": "RequestHandler",
                    "caller info": {
                        "member": "SendAsync",
                        "file": "RequestHandler.cs",
                        "line": 50
                    },
                    "start time": "09:46:22:240",
                    "duration in milliseconds": 2.2835,
                    "data": {
                        "CPU Load History": {
                            "CPU History": "(2021-03-02T21:45:05.7947927Z 100.000), (2021-03-02T21:45:15.6891312Z 100.000), (2021-03-02T21:45:25.6716779Z 97.755), (2021-03-02T21:45:45.6840788Z 100.000), (2021-03-02T21:45:55.7350774Z 95.455), (2021-03-02T21:46:15.7287876Z 100.000)"
                        }
                    },
                    "children": [{
                        "name": "Send Async",
                        "id": "f67e1e06-c66e-436d-9a28-9ecca557415b",
                        "component": "RequestHandler",
                        "caller info": {
                            "member": "SendAsync",
                            "file": "RequestHandler.cs",
                            "line": 50
                        },
                        "start time": "09:46:22:240",
                        "duration in milliseconds": 2.2773,
                        "data": {},
                        "children": [{
                            "name": "Send Async",
                            "id": "43c0f323-f2fd-42d1-9116-981b905c4b43",
                            "component": "RequestHandler",
                            "caller info": {
                                "member": "SendAsync",
                                "file": "AbstractRetryHandler.cs",
                                "line": 23
                            },
                            "start time": "09:46:22:240",
                            "duration in milliseconds": 2.2742,
                            "data": {},
                            "children": [{
                                "name": "Send Async",
                                "id": "5784f620-b9b3-44f5-a925-874f6eb87d27",
                                "component": "RequestHandler",
                                "caller info": {
                                    "member": "SendAsync",
                                    "file": "RequestHandler.cs",
                                    "line": 50
                                },
                                "start time": "09:46:22:240",
                                "duration in milliseconds": 2.2653,
                                "data": {},
                                "children": [{
                                    "name": "Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request",
                                    "id": "cc8582f6-21b3-426d-8717-204ba853c431",
                                    "component": "Transport",
                                    "caller info": {
                                        "member": "ProcessMessageAsync",
                                        "file": "TransportHandler.cs",
                                        "line": 87
                                    },
                                    "start time": "09:46:22:240",
                                    "duration in milliseconds": 2.2059,
                                    "data": {
                                        "User Agent": "cosmos-netstandard-sdk/3.17.0|3.17.1|01|X64|Linux 5.4.0-1031-azure 32 18.|.NET 5.0.3|",
                                        "Client Side Request Stats": {
                                            "Id": "AggregatedClientSideRequestStatistics",
                                            "ContactedReplicas": [],
                                            "RegionsContacted": [],
                                            "FailedReplicas": [],
                                            "AddressResolutionStatistics": [],
                                            "StoreResponseStatistics": []
                                        },
                                        "PointOperationStatisticsTraceDatum": {
                                            "Id": "PointOperationStatistics",
                                            "ActivityId": null,
                                            "ResponseTimeUtc": "2021-03-02T21:46:22.2427593Z",
                                            "StatusCode": 200,
                                            "SubStatusCode": 0,
                                            "RequestCharge": 0,
                                            "RequestUri": "dbs/reveal/colls/*******",
                                            "ErrorMessage": null,
                                            "RequestSessionToken": null,
                                            "ResponseSessionToken": null
                                        }
                                    },
                                    "children": []
                                }]
                            }]
                        }]
                    }]
                }]
            }]
        }, {
            "name": "Get Partition Key Ranges",
            "id": "561ec0c8-122f-48df-a3dc-591f5c3ad5ad",
            "component": "Routing",
            "caller info": {
                "member": "GetTargetPartitionKeyRangesAsync",
                "file": "CosmosQueryClientCore.cs",
                "line": 240
            },
            "start time": "09:46:22:242",
            "duration in milliseconds": 0.032,
            "data": {},
            "children": []
        }]
    }, {
        "name": "MoveNextAsync",
        "id": "03a8f150-9a88-47bc-9a75-f16f1b6c5d64",
        "component": "Pagination",
        "caller info": {
            "member": "MoveNextAsync",
            "file": "CrossPartitionRangePageAsyncEnumerator.cs",
            "line": 114
        },
        "start time": "09:46:22:242",
        "duration in milliseconds": 18.4254,
        "data": {},
        "children": [{
            "name": "Prefetching",
            "id": "b9a9499a-3234-43ac-a053-fe0714e4f64b",
            "component": "Pagination",
            "caller info": {
                "member": "PrefetchInParallelAsync",
                "file": "ParallelPrefetch.cs",
                "line": 31
            },
            "start time": "09:46:22:242",
            "duration in milliseconds": 18.3498,
            "data": {},
            "children": [{
                "name": "Prefetch",
                "id": "1660d23a-ab1f-4c64-88aa-704c97b92904",
                "component": "Pagination",
                "caller info": {
                    "member": "PrefetchAsync",
                    "file": "BufferedPartitionRangePageAsyncEnumerator.cs",
                    "line": 54
                },
                "start time": "09:46:22:242",
                "duration in milliseconds": 18.3004,
                "data": {},
                "children": [{
                    "name": "[05C1E7FFFFFFFE,05C1E93F5D0DD0) move next",
                    "id": "39837de7-a964-47d5-b13e-872b81afb6b4",
                    "component": "Pagination",
                    "caller info": {
                        "member": "MoveNextAsync",
                        "file": "PartitionRangePageAsyncEnumerator.cs",
                        "line": 49
                    },
                    "start time": "09:46:22:242",
                    "duration in milliseconds": 18.2933,
                    "data": {
                        "Query Metrics": "Retrieved Document Count : 9 Retrieved Document Size : 72,156 bytes Output Document Count : 9 Output Document Size : 5,767 bytes Index Utilization : 100.00 % Total Query Execution Time : 11.24 milliseconds Query Preparation Time : 1.03 milliseconds Index Lookup Time : 1.90 milliseconds Document Load Time : 7.65 milliseconds Runtime Execution Times : 0.00 milliseconds Document Write Time : 0.05 milliseconds  Index Utilization Information  Utilized Single Indexes  Potential Single Indexes  Utilized Composite Indexes  Potential Composite Indexes"
                    },
                    "children": [{
                        "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
                        "id": "440227a4-d376-4da0-b8b4-aec6900df583",
                        "component": "RequestHandler",
                        "caller info": {
                            "member": "SendAsync",
                            "file": "RequestInvokerHandler.cs",
                            "line": 148
                        },
                        "start time": "09:46:22:243",
                        "duration in milliseconds": 17.9901,
                        "data": {},
                        "children": [{
                            "name": "Get Collection Cache",
                            "id": "077d3811-e1b1-4ae3-aab3-e393d6517fb2",
                            "component": "Routing",
                            "caller info": {
                                "member": "GetCollectionCacheAsync",
                                "file": "DocumentClient.cs",
                                "line": 542
                            },
                            "start time": "09:46:22:243",
                            "duration in milliseconds": 0.0009,
                            "data": {},
                            "children": []
                        }, {
                            "name": "Try Get Overlapping Ranges",
                            "id": "fcc9baac-7078-49fa-9cde-2af2c3e23ffb",
                            "component": "Routing",
                            "caller info": {
                                "member": "TryGetOverlappingRangesAsync",
                                "file": "PartitionKeyRangeCache.cs",
                                "line": 49
                            },
                            "start time": "09:46:22:243",
                            "duration in milliseconds": 0.0098,
                            "data": {},
                            "children": []
                        }, {
                            "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
                            "id": "5d2d935a-a3e0-49e0-b52f-fb114cd80322",
                            "component": "RequestHandler",
                            "caller info": {
                                "member": "SendAsync",
                                "file": "RequestInvokerHandler.cs",
                                "line": 51
                            },
                            "start time": "09:46:22:243",
                            "duration in milliseconds": 17.9389,
                            "data": {},
                            "children": [{
                                "name": "Send Async",
                                "id": "b3fedb6d-806e-407f-9d13-169fec787e28",
                                "component": "RequestHandler",
                                "caller info": {
                                    "member": "SendAsync",
                                    "file": "RequestHandler.cs",
                                    "line": 50
                                },
                                "start time": "09:46:22:243",
                                "duration in milliseconds": 17.9299,
                                "data": {
                                    "CPU Load History": {
                                        "CPU History": "(2021-03-02T21:45:05.7947927Z 100.000), (2021-03-02T21:45:15.6891312Z 100.000), (2021-03-02T21:45:25.6716779Z 97.755), (2021-03-02T21:45:45.6840788Z 100.000), (2021-03-02T21:45:55.7350774Z 95.455), (2021-03-02T21:46:15.7287876Z 100.000)"
                                    }
                                },
                                "children": [{
                                    "name": "Send Async",
                                    "id": "6e2b26e0-8576-476f-8f94-8871bd7e2d8d",
                                    "component": "RequestHandler",
                                    "caller info": {
                                        "member": "SendAsync",
                                        "file": "RequestHandler.cs",
                                        "line": 50
                                    },
                                    "start time": "09:46:22:243",
                                    "duration in milliseconds": 17.926,
                                    "data": {},
                                    "children": [{
                                        "name": "Send Async",
                                        "id": "3f7b1211-c41d-466b-8056-068630dcb85a",
                                        "component": "RequestHandler",
                                        "caller info": {
                                            "member": "SendAsync",
                                            "file": "AbstractRetryHandler.cs",
                                            "line": 23
                                        },
                                        "start time": "09:46:22:243",
                                        "duration in milliseconds": 17.9233,
                                        "data": {},
                                        "children": [{
                                            "name": "Send Async",
                                            "id": "9ef57489-b1cd-4a7f-b3cc-13c2534891a0",
                                            "component": "RequestHandler",
                                            "caller info": {
                                                "member": "SendAsync",
                                                "file": "RequestHandler.cs",
                                                "line": 50
                                            },
                                            "start time": "09:46:22:243",
                                            "duration in milliseconds": 17.9173,
                                            "data": {},
                                            "children": [{
                                                "name": "Microsoft.Azure.Documents.ServerStoreModel Transport Request",
                                                "id": "decc7268-c234-46f0-b0da-54f59aa38ff7",
                                                "component": "Transport",
                                                "caller info": {
                                                    "member": "ProcessMessageAsync",
                                                    "file": "TransportHandler.cs",
                                                    "line": 87
                                                },
                                                "start time": "09:46:22:243",
                                                "duration in milliseconds": 17.8771,
                                                "data": {
                                                    "User Agent": "cosmos-netstandard-sdk/3.17.0|3.17.1|01|X64|Linux 5.4.0-1031-azure 32 18.|.NET 5.0.3|",
                                                    "Client Side Request Stats": {
                                                        "Id": "AggregatedClientSideRequestStatistics",
                                                        "ContactedReplicas": [],
                                                        "RegionsContacted": ["https://**********.documents.azure.com/"],
                                                        "FailedReplicas": [],
                                                        "AddressResolutionStatistics": [],
                                                        "StoreResponseStatistics": [{
                                                            "ResponseTimeUTC": "2021-03-02T21:46:22.2609143Z",
                                                            "ResourceType": "Document",
                                                            "OperationType": "Query",
                                                            "LocationEndpoint": "https://**********.documents.azure.com/",
                                                            "StoreResult": "StorePhysicalAddress: rntbd://**********.documents.azure.com:14198/apps/1072af28-7265-4a2c-9208-a3ae5332dec2/services/fa4dd569-7f60-4935-9ade-412d6ff2de42/partitions/01123852-8264-4de5-8c62-07fa08e93cb4/replicas/132460929699733530s/, LSN: 3697338, GlobalCommittedLsn: 3697337, PartitionKeyRangeId: 117, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 5.76, ItemLSN: -1, SessionToken: -1#3697338, UsingLocalLSN: True, TransportException: null"
                                                        }]
                                                    }
                                                },
                                                "children": []
                                            }]
                                        }]
                                    }]
                                }]
                            }]
                        }]
                    }, {
                        "name": "Get Cosmos Element Response",
                        "id": "62a0895c-100c-4bc1-ad5e-dd91c8b9051a",
                        "component": "Json",
                        "caller info": {
                            "member": "GetCosmosElementResponse",
                            "file": "CosmosQueryClientCore.cs",
                            "line": 282
                        },
                        "start time": "09:46:22:261",
                        "duration in milliseconds": 0.2346,
                        "data": {},
                        "children": []
                    }]
                }]
            }]
        }, {
            "name": "[05C1E7FFFFFFFE,05C1E93F5D0DD0) move next",
            "id": "c9eb1392-ee93-4211-8b36-93fa4cfa50c3",
            "component": "Pagination",
            "caller info": {
                "member": "MoveNextAsync",
                "file": "PartitionRangePageAsyncEnumerator.cs",
                "line": 49
            },
            "start time": "09:46:22:261",
            "duration in milliseconds": 0.007,
            "data": {},
            "children": [{
                "name": "Prefetch",
                "id": "4c087252-3717-4a2a-adcf-aa17f6aedd59",
                "component": "Pagination",
                "caller info": {
                    "member": "PrefetchAsync",
                    "file": "BufferedPartitionRangePageAsyncEnumerator.cs",
                    "line": 54
                },
                "start time": "09:46:22:261",
                "duration in milliseconds": 0.0008,
                "data": {},
                "children": []
            }]
        }]
    }]
}

@timsander1 can you please take a look? Just taking a quick look it seems like 3.17 is doing 2 page read requests while 3.9 is only doing 1 page read request.
@dpiessens are you doing a paging API? Can you share the code on how you are draining the query.

Sure here's the code we're using:
public virtual async Task<(IEnumerable<T> results, string sessionToken)> ExecuteQueryAsync<T>(IQueryable<T> query, CancellationToken cancellationToken = default(CancellationToken))
{
    using (LogContext.PushProperty("operationName", "Query"))
    using (LogContext.PushProperty("query", query.ToString()))
    {
        var results = new List<T>();
        string sessionToken = null;
        var options = new QueryRequestOptions { MaxConcurrency = -1, MaxBufferedItemCount = -1 };
        using (var iterator = this.container.GetItemQueryStreamIterator(query.ToQueryDefinition(), null, options))
        {
            while (iterator.HasMoreResults)
            {
                Func<Task<ResponseMessage>> action = () => iterator.ReadNextAsync(cancellationToken);
                using var response = await Execute(action, "Query");
                using var jsondoc = await JsonDocument.ParseAsync(response.Content, cancellationToken: cancellationToken);
                var queryResults = JsonConverter.Deserialize<T>(jsondoc.RootElement.GetProperty("Documents").EnumerateArray());

                results.AddRange(queryResults);

                sessionToken = response.Headers.Session;
            }
        }

        return (results, sessionToken);
    }
}

One note on this the await Execute line simply allows us to wrap the call in our metrics and extract your client metadata consistently for all operations. Also, we're using a direct serializer to date since we always have lists of results for queries and to date serialize methods have done objects one at a time which is slower.
We didn't change that part of the code yet so there to limit perf change areas, but we're happy to move that over if you feel the newer query methods have better performance.

Hi @dpiessens
Would it be possible to send two traces for the same query with the same filter? The query metrics are different enough that it's hard to rule this out as a possible reason for the difference.
Thanks!

Yup, will get that to you later today.
On Wed, Mar 3, 2021, 7:02 PM Tim Sander<EMAIL_ADDRESS>wrote:

Hi @dpiessens https://github.com/dpiessens
Would it be possible to send two traces for the same query with the same
filter? The query metrics are different enough that it's hard to rule this
out as a possible reason for the difference.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-cosmos-dotnet-v3/issues/2270#issuecomment-790200692,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAP3GMFJUUTJTXBUVWDF2DTB3L35ANCNFSM4YMZ3SPA
.

OK so I ran two full runs and wrote something to compare items down to the query. The result is that query calls on average increased 15 ms. This query was at the average:
Baseline
Elapsed Time: 29.3056 ms
Client Elapsed Time: 29.2932 ms
Diagnostics:
{
    "DiagnosticVersion": "2",
    "Summary": {
        "StartUtc": "2021-03-05T03:33:01.2618255Z",
        "TotalElapsedTimeInMs": 29.2932,
        "UserAgent": "cosmos-netstandard-sdk|3.9.0|3.9.0|61579|X64|Linux 5.4.0-1031-azure 32 18.|.NET 5.0.3|",
        "TotalRequestCount": 2,
        "FailedRequestCount": 0
    },
    "Context": [{
        "Id": "CreateQueryPipeline",
        "ElapsedTimeInMs": 2.095
    }, {
        "Id": "Microsoft.Azure.Cosmos.Handlers.RetryHandler",
        "HandlerElapsedTimeInMs": 1.8682
    }, {
        "Id": "Microsoft.Azure.Cosmos.Handlers.RouterHandler",
        "HandlerElapsedTimeInMs": 1.8624
    }, {
        "Id": "Microsoft.Azure.Cosmos.Handlers.TransportHandler",
        "HandlerElapsedTimeInMs": 0.2576
    }, {
        "Id": "AggregatedClientSideRequestStatistics",
        "ContactedReplicas": [],
        "RegionsContacted": [],
        "FailedReplicas": []
    }, {
        "Id": "Microsoft.Azure.Cosmos.GatewayStoreModel",
        "ElapsedTimeInMs": 1.7582
    }, {
        "Id": "GatewayRequestTime",
        "ElapsedTimeInMs": 1.6436
    }, {
        "Id": "PointOperationStatistics",
        "ActivityId": null,
        "ResponseTimeUtc": "2021-03-05T03:33:01.2637739Z",
        "StatusCode": 200,
        "SubStatusCode": 0,
        "RequestCharge": 0.0,
        "RequestUri": "dbs/*****/colls/****",
        "RequestSessionToken": null,
        "ResponseSessionToken": null
    }, {
        "PKRangeId": "126",
        "StartUtc": "2021-03-05T03:33:01.2640741Z",
        "QueryMetric": "totalExecutionTimeInMs=20.91;queryCompileTimeInMs=0.43;queryLogicalPlanBuildTimeInMs=0.13;queryPhysicalPlanBuildTimeInMs=0.45;queryOptimizationTimeInMs=0.03;VMExecutionTimeInMs=19.56;indexLookupTimeInMs=1.57;documentLoadTimeInMs=17.29;systemFunctionExecuteTimeInMs=0.00;userFunctionExecuteTimeInMs=0.00;retrievedDocumentCount=18;retrievedDocumentSize=151748;outputDocumentCount=18;outputDocumentSize=11511;writeOutputTimeInMs=0.24;indexUtilizationRatio=1.00",
        "IndexUtilization": "",
        "ClientCorrelationId": "2e835ccf-4585-4006-9298-1f453b6b1129",
        "Context": [{
            "Id": "AggregatedClientSideRequestStatistics",
            "ContactedReplicas": [],
            "RegionsContacted": ["https://**********.documents.azure.com:443/"],
            "FailedReplicas": []
        }, {
            "Id": "Microsoft.Azure.Cosmos.Handlers.RetryHandler",
            "HandlerElapsedTimeInMs": 26.7368
        }, {
            "Id": "Microsoft.Azure.Cosmos.Handlers.RouterHandler",
            "HandlerElapsedTimeInMs": 26.7322
        }, {
            "Id": "Microsoft.Azure.Cosmos.Handlers.TransportHandler",
            "HandlerElapsedTimeInMs": 0.2232
        }, {
            "Id": "Microsoft.Azure.Documents.ServerStoreModel",
            "ElapsedTimeInMs": 26.6797
        }, {
            "Id": "StoreResponseStatistics",
            "StartTimeUtc": "2021-03-05T03:33:01.2642095Z",
            "ResponseTimeUtc": "2021-03-05T03:33:01.2908277Z",
            "ElapsedTimeInMs": 26.6182,
            "ResourceType": "Document",
            "OperationType": "Query",
            "LocationEndpoint": "https://**********.documents.azure.com:443/",
            "StoreResult": "StorePhysicalAddress: rntbd://cdb-ms-prod-eastus2-***.documents.azure.com:14440/apps/796f6a08-ae9e-4a5f-8014-6b8972ffefdc/services/b0c7544f-49fc-49d3-ae79-ef9df71474e3/partitions/5b8fb755-4144-4fda-8245-3e6f180ed766/replicas/132588272726853466s/, LSN: 3777274, GlobalCommittedLsn: 3777273, PartitionKeyRangeId: 126, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 6.56, ItemLSN: -1, SessionToken: -1#3777274, UsingLocalLSN: True, TransportException: null"
        }]
    }]
}

Latest
Elapsed Time: 44.9036 ms
Client Elapsed Time: 44.3216 ms
Diagnostics:
{
    "name": "FeedIterator Read Next Async",
    "id": "4baf1d60-acee-4beb-aa3c-6fd37f277eab",
    "component": "Unknown",
    "caller info": {
        "member": "ReadNextWithRootTraceAsync",
        "file": "FeedIteratorInlineCore.cs",
        "line": 48
    },
    "start time": "10:38:54:058",
    "duration in milliseconds": 44.3216,
    "data": {},
    "children": [{
        "name": "Create Query Pipeline",
        "id": "0279b6ab-cd23-455a-b7d1-837047ed52ed",
        "component": "Query",
        "caller info": {
            "member": "TryCreateCoreContextAsync",
            "file": "CosmosQueryExecutionContextFactory.cs",
            "line": 85
        },
        "start time": "10:38:54:058",
        "duration in milliseconds": 3.1787,
        "data": {},
        "children": [{
            "name": "Get Container Properties",
            "id": "7cc3492a-f7eb-4029-88fa-329be4981076",
            "component": "Transport",
            "caller info": {
                "member": "GetCachedContainerPropertiesAsync",
                "file": "ClientContextCore.cs",
                "line": 349
            },
            "start time": "10:38:54:058",
            "duration in milliseconds": 0.0092,
            "data": {},
            "children": [{
                "name": "Get Collection Cache",
                "id": "c07c8bd1-bcc0-41cd-899d-fdd9e6cfd63d",
                "component": "Routing",
                "caller info": {
                    "member": "GetCollectionCacheAsync",
                    "file": "DocumentClient.cs",
                    "line": 542
                },
                "start time": "10:38:54:058",
                "duration in milliseconds": 0.001,
                "data": {},
                "children": []
            }]
        }, {
            "name": "Gateway QueryPlan",
            "id": "bce69c10-abc3-44ed-be52-6733d9446317",
            "component": "Query",
            "caller info": {
                "member": "GetQueryPlanThroughGatewayAsync",
                "file": "QueryPlanRetriever.cs",
                "line": 112
            },
            "start time": "10:38:54:058",
            "duration in milliseconds": 0.2977,
            "data": {},
            "children": []
        }, {
            "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
            "id": "f414fc24-a0c2-4f9b-8ab2-8a8594951503",
            "component": "RequestHandler",
            "caller info": {
                "member": "SendAsync",
                "file": "RequestInvokerHandler.cs",
                "line": 148
            },
            "start time": "10:38:54:058",
            "duration in milliseconds": 3.004,
            "data": {},
            "children": [{
                "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
                "id": "8ec8c65c-f356-4337-a84b-26bd0c252023",
                "component": "RequestHandler",
                "caller info": {
                    "member": "SendAsync",
                    "file": "RequestInvokerHandler.cs",
                    "line": 51
                },
                "start time": "10:38:54:058",
                "duration in milliseconds": 2.9855,
                "data": {},
                "children": [{
                    "name": "Send Async",
                    "id": "1fc064a3-614d-4b71-afd4-350d32648bc4",
                    "component": "RequestHandler",
                    "caller info": {
                        "member": "SendAsync",
                        "file": "RequestHandler.cs",
                        "line": 50
                    },
                    "start time": "10:38:54:058",
                    "duration in milliseconds": 2.9796,
                    "data": {
                        "CPU Load History": {
                            "CPU History": "(2021-03-03T11:12:09.7821310Z 0.000), (2021-03-03T11:12:19.7807023Z 100.000), (2021-03-03T11:12:29.7818508Z 33.333), (2021-03-03T11:12:39.7828943Z 33.333), (2021-03-03T11:12:49.7805599Z 37.220), (2021-03-03T11:12:59.7806116Z 50.319)"
                        }
                    },
                    "children": [{
                        "name": "Send Async",
                        "id": "fc912e4a-fd49-418f-b283-9268b9b5c45c",
                        "component": "RequestHandler",
                        "caller info": {
                            "member": "SendAsync",
                            "file": "RequestHandler.cs",
                            "line": 50
                        },
                        "start time": "10:38:54:058",
                        "duration in milliseconds": 2.9748,
                        "data": {},
                        "children": [{
                            "name": "Send Async",
                            "id": "94cb9950-5127-47f8-afc6-184ed4f8b093",
                            "component": "RequestHandler",
                            "caller info": {
                                "member": "SendAsync",
                                "file": "AbstractRetryHandler.cs",
                                "line": 23
                            },
                            "start time": "10:38:54:058",
                            "duration in milliseconds": 2.9717,
                            "data": {},
                            "children": [{
                                "name": "Send Async",
                                "id": "40709019-a2b2-4bce-981c-393ce763a50b",
                                "component": "RequestHandler",
                                "caller info": {
                                    "member": "SendAsync",
                                    "file": "RequestHandler.cs",
                                    "line": 50
                                },
                                "start time": "10:38:54:058",
                                "duration in milliseconds": 2.9651,
                                "data": {},
                                "children": [{
                                    "name": "Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request",
                                    "id": "09e1b5b5-adc9-4533-97e2-99d597ea49b4",
                                    "component": "Transport",
                                    "caller info": {
                                        "member": "ProcessMessageAsync",
                                        "file": "TransportHandler.cs",
                                        "line": 87
                                    },
                                    "start time": "10:38:54:058",
                                    "duration in milliseconds": 2.9294,
                                    "data": {
                                        "User Agent": "cosmos-netstandard-sdk/3.17.0|3.17.1|01|X64|Linux 5.4.0-1031-azure 32 18.|.NET 5.0.3|",
                                        "Client Side Request Stats": {
                                            "Id": "AggregatedClientSideRequestStatistics",
                                            "ContactedReplicas": [],
                                            "RegionsContacted": [],
                                            "FailedReplicas": [],
                                            "AddressResolutionStatistics": [],
                                            "StoreResponseStatistics": []
                                        },
                                        "PointOperationStatisticsTraceDatum": {
                                            "Id": "PointOperationStatistics",
                                            "ActivityId": null,
                                            "ResponseTimeUtc": "2021-03-04T22:38:54.0611779Z",
                                            "StatusCode": 200,
                                            "SubStatusCode": 0,
                                            "RequestCharge": 0,
                                            "RequestUri": "dbs/*****/colls/*****",
                                            "ErrorMessage": null,
                                            "RequestSessionToken": null,
                                            "ResponseSessionToken": null
                                        }
                                    },
                                    "children": []
                                }]
                            }]
                        }]
                    }]
                }]
            }]
        }, {
            "name": "Get Partition Key Ranges",
            "id": "c4b31db9-7ca4-4489-839c-309187ec6c16",
            "component": "Routing",
            "caller info": {
                "member": "GetTargetPartitionKeyRangesAsync",
                "file": "CosmosQueryClientCore.cs",
                "line": 240
            },
            "start time": "10:38:54:061",
            "duration in milliseconds": 0.0212,
            "data": {},
            "children": []
        }]
    }, {
        "name": "MoveNextAsync",
        "id": "540549f1-6217-4d5a-85e9-9f68ced9a622",
        "component": "Pagination",
        "caller info": {
            "member": "MoveNextAsync",
            "file": "CrossPartitionRangePageAsyncEnumerator.cs",
            "line": 114
        },
        "start time": "10:38:54:061",
        "duration in milliseconds": 41.0726,
        "data": {},
        "children": [{
            "name": "Prefetching",
            "id": "8796703e-32e5-4d44-a287-f2c40539c435",
            "component": "Pagination",
            "caller info": {
                "member": "PrefetchInParallelAsync",
                "file": "ParallelPrefetch.cs",
                "line": 31
            },
            "start time": "10:38:54:061",
            "duration in milliseconds": 41.0125,
            "data": {},
            "children": [{
                "name": "Prefetch",
                "id": "b18990d6-8eb7-4736-9057-aa3dee27588a",
                "component": "Pagination",
                "caller info": {
                    "member": "PrefetchAsync",
                    "file": "BufferedPartitionRangePageAsyncEnumerator.cs",
                    "line": 54
                },
                "start time": "10:38:54:061",
                "duration in milliseconds": 40.9627,
                "data": {},
                "children": [{
                    "name": "[05C1E5C373C764,05C1E5FFFFFFFE) move next",
                    "id": "26a19e66-b841-4ca2-a8e4-928ee3df2aeb",
                    "component": "Pagination",
                    "caller info": {
                        "member": "MoveNextAsync",
                        "file": "PartitionRangePageAsyncEnumerator.cs",
                        "line": 49
                    },
                    "start time": "10:38:54:061",
                    "duration in milliseconds": 40.9557,
                    "data": {
                        "Query Metrics": "Retrieved Document Count: 16, Retrieved Document Size: 132, 449 bytes Output Document Count: 16, Output Document Size: 10, 238 bytes, Index Utilization 100.00 % Total Query Execution Time: 21.27 milliseconds, Query Preparation Time: 1.06 milliseconds Index Lookup Time: 1.58 milliseconds  Document Load Time: 17.49 milliseconds  Runtime Execution Times: 0.00 milliseconds   Document Write Time: 0.25 milliseconds Index Utilization Information   Utilized Single Indexes   Potential Single Indexes   Utilized Composite Indexes   Potential Composite Indexes"
                    },
                    "children": [{
                        "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
                        "id": "65fab615-615d-443f-9d54-a2e0ad983e92",
                        "component": "RequestHandler",
                        "caller info": {
                            "member": "SendAsync",
                            "file": "RequestInvokerHandler.cs",
                            "line": 148
                        },
                        "start time": "10:38:54:061",
                        "duration in milliseconds": 40.4332,
                        "data": {},
                        "children": [{
                            "name": "Get Collection Cache",
                            "id": "cce8562a-e318-489e-bfb8-ddbbb6b7585c",
                            "component": "Routing",
                            "caller info": {
                                "member": "GetCollectionCacheAsync",
                                "file": "DocumentClient.cs",
                                "line": 542
                            },
                            "start time": "10:38:54:061",
                            "duration in milliseconds": 0.0007,
                            "data": {},
                            "children": []
                        }, {
                            "name": "Try Get Overlapping Ranges",
                            "id": "affdc18e-4717-4564-af95-b56d9ba3b5d6",
                            "component": "Routing",
                            "caller info": {
                                "member": "TryGetOverlappingRangesAsync",
                                "file": "PartitionKeyRangeCache.cs",
                                "line": 49
                            },
                            "start time": "10:38:54:061",
                            "duration in milliseconds": 0.008,
                            "data": {},
                            "children": []
                        }, {
                            "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
                            "id": "1f460122-e5db-4354-9473-f96848cf434f",
                            "component": "RequestHandler",
                            "caller info": {
                                "member": "SendAsync",
                                "file": "RequestInvokerHandler.cs",
                                "line": 51
                            },
                            "start time": "10:38:54:061",
                            "duration in milliseconds": 40.3982,
                            "data": {},
                            "children": [{
                                "name": "Send Async",
                                "id": "d5487e60-b44e-4538-bdd0-38f2d0c75151",
                                "component": "RequestHandler",
                                "caller info": {
                                    "member": "SendAsync",
                                    "file": "RequestHandler.cs",
                                    "line": 50
                                },
                                "start time": "10:38:54:061",
                                "duration in milliseconds": 40.3917,
                                "data": {
                                    "CPU Load History": {
                                        "CPU History": "(2021-03-03T11:12:09.7821310Z 0.000), (2021-03-03T11:12:19.7807023Z 100.000), (2021-03-03T11:12:29.7818508Z 33.333), (2021-03-03T11:12:39.7828943Z 33.333), (2021-03-03T11:12:49.7805599Z 37.220), (2021-03-03T11:12:59.7806116Z 50.319)"
                                    }
                                },
                                "children": [{
                                    "name": "Send Async",
                                    "id": "4bee8637-2b50-4464-b360-aeaf15c49ae7",
                                    "component": "RequestHandler",
                                    "caller info": {
                                        "member": "SendAsync",
                                        "file": "RequestHandler.cs",
                                        "line": 50
                                    },
                                    "start time": "10:38:54:061",
                                    "duration in milliseconds": 40.388,
                                    "data": {},
                                    "children": [{
                                        "name": "Send Async",
                                        "id": "7287901b-9dbc-40bc-8602-fd354caf1bfe",
                                        "component": "RequestHandler",
                                        "caller info": {
                                            "member": "SendAsync",
                                            "file": "AbstractRetryHandler.cs",
                                            "line": 23
                                        },
                                        "start time": "10:38:54:061",
                                        "duration in milliseconds": 40.3856,
                                        "data": {},
                                        "children": [{
                                            "name": "Send Async",
                                            "id": "533445a3-6377-4fc1-b0b2-de92f9e5ac61",
                                            "component": "RequestHandler",
                                            "caller info": {
                                                "member": "SendAsync",
                                                "file": "RequestHandler.cs",
                                                "line": 50
                                            },
                                            "start time": "10:38:54:061",
                                            "duration in milliseconds": 40.3797,
                                            "data": {},
                                            "children": [{
                                                "name": "Microsoft.Azure.Documents.ServerStoreModel Transport Request",
                                                "id": "67a59ec8-2a43-468e-b3e3-3d7a8f338aa7",
                                                "component": "Transport",
                                                "caller info": {
                                                    "member": "ProcessMessageAsync",
                                                    "file": "TransportHandler.cs",
                                                    "line": 87
                                                },
                                                "start time": "10:38:54:061",
                                                "duration in milliseconds": 40.3399,
                                                "data": {
                                                    "User Agent": "cosmos-netstandard-sdk/3.17.0|3.17.1|01|X64|Linux 5.4.0-1031-azure 32 18.|.NET 5.0.3|",
                                                    "Client Side Request Stats": {
                                                        "Id": "AggregatedClientSideRequestStatistics",
                                                        "ContactedReplicas": [],
                                                        "RegionsContacted": ["https://*********.documents.azure.com/"],
                                                        "FailedReplicas": [],
                                                        "AddressResolutionStatistics": [],
                                                        "StoreResponseStatistics": [{
                                                            "ResponseTimeUTC": "2021-03-04T22:38:54.1017619Z",
                                                            "ResourceType": "Document",
                                                            "OperationType": "Query",
                                                            "LocationEndpoint": "https://**********.documents.azure.com/",
                                                            "StoreResult": "StorePhysicalAddress: rntbd://cdb-ms-prod-eastus2-***.documents.azure.com:****/apps/796f6a08-ae9e-4a5f-8014-6b8972ffefdc/services/b0c7544f-49fc-49d3-ae79-ef9df71474e3/partitions/5b8fb755-4144-4fda-8245-3e6f180ed766/replicas/132587240566165890s/, LSN: 3763194, GlobalCommittedLsn: 3763193, PartitionKeyRangeId: 126, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 6.33, ItemLSN: -1, SessionToken: -1 #3763194, UsingLocalLSN: True, TransportException: null"
                                                        }]
                                                    }
                                                },
                                                "children": []
                                            }]
                                        }]
                                    }]
                                }]
                            }]
                        }]
                    }, {
                        "name": "Get Cosmos Element Response",
                        "id": "7fa43540-943a-4c62-9a4b-bdfb1ca5c2ba",
                        "component": "Json",
                        "caller info": {
                            "member": "GetCosmosElementResponse",
                            "file": "CosmosQueryClientCore.cs",
                            "line": 282
                        },
                        "start time": "10:38:54:101",
                        "duration in milliseconds": 0.4712,
                        "data": {},
                        "children": []
                    }]
                }]
            }]
        }, {
            "name": "[05C1E5C373C764,05C1E5FFFFFFFE) move next",
            "id": "f08a1c5a-35a8-4bb5-88e4-ef5cbd6b3046",
            "component": "Pagination",
            "caller info": {
                "member": "MoveNextAsync",
                "file": "PartitionRangePageAsyncEnumerator.cs",
                "line": 49
            },
            "start time": "10:38:54:102",
            "duration in milliseconds": 0.0068,
            "data": {},
            "children": [{
                "name": "Prefetch",
                "id": "49b8698e-dd60-49a7-bcb4-7daf2d284a8a",
                "component": "Pagination",
                "caller info": {
                    "member": "PrefetchAsync",
                    "file": "BufferedPartitionRangePageAsyncEnumerator.cs",
                    "line": 54
                },
                "start time": "10:38:54:102",
                "duration in milliseconds": 0.0006,
                "data": {},
                "children": []
            }]
        }]
    }]
}

It does appear the new version is doing a double read.

@timsander1 here is a formatted version:
root|root
└── FeedIterator Read Next Async|44.3216
    ├── Create Query Pipeline|3.1787
    │   ├── Get Container Properties|0.0092
    │   │   └── Get Collection Cache|0.001
    │   ├── Gateway QueryPlan|0.2977
    │   ├── Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler|3.004
    │   │   └── Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler|2.9855
    │   │       └── Send Async|2.9796
    │   │           └── Send Async|2.9748
    │   │               └── Send Async|2.9717
    │   │                   └── Send Async|2.9651
    │   │                       └── Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request|2.9294
    │   └── Get Partition Key Ranges|0.0212
    └── MoveNextAsync|41.0726
        ├── Prefetching|41.0125
        │   └── Prefetch|40.9627
        │       └── [05C1E5C373C764,05C1E5FFFFFFFE) move next|40.9557 | Retrieved Document Count: 16, Retrieved Document Size: 132, 449 bytes Output Document Count: 16, Output Document Size: 10, 238 bytes, Index Utilization 100.00 % Total Query Execution Time: 21.27 milliseconds, Query Preparation Time: 1.06 milliseconds Index Lookup Time: 1.58 milliseconds  Document Load Time: 17.49 milliseconds  Runtime Execution Times: 0.00 milliseconds   Document Write Time: 0.25 milliseconds Index Utilization Information   Utilized Single Indexes   Potential Single Indexes   Utilized Composite Indexes   Potential Composite Indexes
        │           ├── Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler|40.4332
        │           │   ├── Get Collection Cache|0.0007
        │           │   ├── Try Get Overlapping Ranges|0.008
        │           │   └── Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler|40.3982
        │           │       └── Send Async|40.3917
        │           │           └── Send Async|40.388
        │           │               └── Send Async|40.3856
        │           │                   └── Send Async|40.3797
        │           │                       └── Microsoft.Azure.Documents.ServerStoreModel Transport Request|40.3399
        │           └── Get Cosmos Element Response|0.4712
        └── [05C1E5C373C764,05C1E5FFFFFFFE) move next|0.0068
            └── Prefetch|0.0006

Looks like 90 percent of the time is spent over the network:
Microsoft.Azure.Documents.ServerStoreModel Transport Request|40.3399
So moving to the transport team to investigate.

@bchong95 please verify that the queries are the same and that same amount of data is being sent over the wire.

For background we are running through AKS with a virtual endpoint on that VNET to cosmos. The environment, files etc. between the calls were as identical as networking can be, The only difference between the two calls is the client library used.

The queries are different but the output doc size is about the same:
3.9:  Query had OutputDocumentCount=18;outputDocumentSize=11511
3.17: Query had Output Document Count =16, Output Document Size: 10, 238 bytes

@dpiessens just to confirm the following questions

Are both using the exact same AKS container? We have seen some latency issues based on the AKS container that was being used.
Are both running in the same region as the cosmos container?
Is there any way to provide multiple diagnostics?

@timsander1

The queries are different but the output doc size is about the same:
3.9: Query had OutputDocumentCount=18;outputDocumentSize=11511
3.17: Query had Output Document Count =16, Output Document Size: 10, 238 bytes

The query is actually the same, the results have to do with the process it's embedded in and the comparison. 3.17 ran first so it was loading the data. When it ran the query the record it was working on wasn't in the DB yet so not in the results. The target being loaded also had another record after this one so that was loaded.  3.9 was then run so it was re-loading the results. This affects the upsert and replace operations but those had no performance changes. Queries for both loads are identical aside from the 2 additional records, but I don't this this can account for the difference.
Our process tracks the load, origin record, method name and the query. I made all of these items the key when I lined up comparisons so I'm completely confident the results are from the same query with the same parameters.
@j82w

Yes we use a CI process to construct each container image, and both tests were deployed to the same AKS cluster. That cluster had no other work on it during the tests and the container has an anti-affinity so they are distributed on the same node set.
Yes both the cluster an DB are in East US 2
Yes I can provide additional traces.

Just to update the issue based on the email thread. I’ve looked through a several of the diagnostics, and noticed that the call to get the query plans from gateway was taking longer on average on 3.17. Can you try making the following changes and let us know if you see any improvement?

Set the CosmosClientOptions.GatewayModeMaxConnectionLimit to int.MaxValue https://github.com/Azure/azure-cosmos-dotnet-v3/blob/65e22f8ad15bae157bd8f251f066c12a3ba0c832/Microsoft.Azure.Cosmos/src/CosmosClientOptions.cs#L131
a.	There was a fix in 3.12 to make it where the connection limit is actually honored on .NET core. It’s possible that the connection limit is causing the latency of the gateway requests to be higher: Connection limits: Fixes .NET core to honor gateway connection limit by j82w · Pull Request #1740 · Azure/azure-cosmos-dotnet-v3 (github.com)
Set the PartitionKey in the QueryRequestOptions when possible.
a.	In 3.13 an optimization was added to skip the get query plan network call if it’s a single partition query with no aggregate operations like group by. https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/changelog.md#-3130---2020-09-21

So I updated our code with the two performance items you listed, and the lag is reduced. The average delta went from 15ms to 10ms so that is an improvement. I'm sending over the new traces so you can see the difference. This did leave to one question:
Given that many of the queries access a single partition and specify that in the query, is there a way for you to add that into the Cosmos methods? Right now we're creating an IQueryable via GetItemLinqQueryable<> and then calling the extension method ToStreamIterator().

The latest diagnostics provided shows a 2 to 3x increase in 3.17 SDK networking time vs 3.9 SDK time. Testing the SDK versions in an Azure VM showed they were roughly the same networking latency. Can you try running both SDK versions in the same container against the same Cosmos DB endpoint? At this point it seems likely the difference is caused by something in the environment.

I attempted to put both version in the same container, but while possible it proved to be more of a challenge than the outcome is worth. At this point we will accept the timing for what it is and scale out to return to our previous throughput.


bazel coverage not working for 0.0.9

Description of the problem / feature request:
Bazel Coverage failing with rules_cc 0.0.9
Feature requests: what underlying problem are you trying to solve with this feature?
We're trying to uprev our rules_cc from ancient 0.0.1 to latest version as we prepare to migrate to bazel 7.0.0
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
test.cpp
#include <iostream>

int main() {
    std::cout << "Hello, World!" << std::endl;
    return 0;
}

BUILD
cc_library(
    name = "test_lib",
    srcs = ["test.cpp"],
)
cc_test(
    name = "test",
    srcs = [":test_lib"],
)

What operating system are you running Bazel on?
ubuntu 20.04
What's the output of bazel info release?
release 6.3.2
What version of rules_cc do you use? Can you paste the workspace rule used to fetch rules_cc? What other relevant dependencies does your project have?
0.0.9
What Bazel options do you use to trigger the issue? What C++ toolchain do you use?
I ran this with bazel 6.3.2 in the rules cc example section by adding the following section above. I'm using the cc toolchain configured by rules_cc.
System gcc is 9.0.0
Have you found anything relevant by searching the web?
https://groups.google.com/d/topic/bazel-discuss/dtduJBlb-nw (doesn't seem relevant)
Any other information, logs, or outputs that you want to share?
bazelisk coverage //examples/my_c_archive/...

ERROR: /home/ryang/repos/rules_cc/examples/my_c_archive/BUILD:36:13: in my_c_compile rule //examples/my_c_archive:object:
Traceback (most recent call last):
        File "/home/ryang/repos/rules_cc/examples/my_c_compile/my_c_compile.bzl", line 30, column 57, in _my_c_compile_impl
                feature_configuration = cc_common.configure_features(
Error in configure_features: Symbol profile is provided by all of the following features: coverage gcc_coverage_map_format
ERROR: /home/ryang/repos/rules_cc/examples/my_c_archive/BUILD:36:13: Analysis of target '//examples/my_c_archive:object' failed
ERROR: Analysis of target '//examples/my_c_archive:object' failed; build aborted:
INFO: Elapsed time: 0.090s

@c-mita for triage
possibly similar to https://github.com/bazelbuild/bazel/issues/20993

@c-mita friendly ping

I can't even build the example in this repo with Bazel 6.0.0 or later...
Regardless, I suspect the issue is here: https://github.com/bazelbuild/rules_cc/blob/35fe45e91bdca3128c873ebc7dd18f52063524c5/cc/private/toolchain/unix_cc_toolchain_config.bzl#L997
The "coverage" feature is a duplicate of the "gcc_coverage_map_format" one, but the latter "requires" the former and there's a conflict since they provide the same "profile" symbol.

I got that issue too on Debian 12 with GCC 12. If you enable rool_cc toolchain coverage fails (because symbol profile is provided by several features.  Using the built-in toolchains the issue is gone.  To reproduce using the OP's code and BUILD file, use the following WORKSPACE.bazel.
workspace(name = "cov_issue")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# ----------------- rules_cc -----------------------------------
http_archive(
    name = "rules_cc",
    urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.10-rc1/rules_cc-0.0.10-rc1.tar.gz"],
    sha256 = "d75a040c32954da0d308d3f2ea2ba735490f49b3a7aa3e4b40259ca4b814f825",
)

load("@rules_cc//cc:repositories.bzl", "rules_cc_dependencies", "rules_cc_toolchains")
# ----------------------------------------------------

rules_cc_dependencies()
rules_cc_toolchains()

Running bazel coverage --noenable_bzlmod --combined_report=lcov //:test produces the error.  Commenting the last two lines in the workspace file and running the bazel command again, succeeds.


Conan package for glog

Hello,
Do you know about Conan package manager? Can you add Conan package for the library and maintain it? Here you can find example of my conan recipe for libpng: https://github.com/ZaMaZaN4iK/conan-libpng

Never heard of it. You're welcome to propose a patch though.

Hi,
I've go a recipe for glog conan package, just take a look at my repositories.
Best Regards,
Darek

Here is a link to conan recipe repository: https://github.com/DariuszOstolski/conan_glog
Tested on linux only, but should work on all platforms

Do you want to support Conan package? I can move your package to Bincrafters repo and Bincrafters team will support package for your library. Also you of course can do it manually :-)

I've just joined bincrafters team, so soon it should be published officially under bincrafters flag.

https://cpplang.now.sh/


Text in specialization tree talent box does not update when changed.

When updating the talent text in a specialization tree the text does not update, see image:

Steps to reproduce the behavior:

Open any specialization tree
Open any talent
change the text and save
See error

Expected behavior
This worked in the previous version of foundry and SW-FFG
Version information:

Foundry Version: v12.330
System Version: 1.902
Hosted on Forge: NO

Found two oddities that I think are related to this bug:

When modifying the talent, the attribute "clickfromparent" is lost, preventing the specialization to be updated when closing the talent. When closing the talent without modifying it, "clickfromparent" is present and the specialization is updated, but ...
the description update code references formData.data.description that is unassigned, so the description in the specialization is never updated.

You can test this by:

open a talent
set it as "Force Talent: Yes"
close talent
specialization tree doesn't change
check that talent is modified in talent compendium
open same talent
close same talent
talent now has visual mark of a Force Talent.


ROX-11202: Add links to split CVE types

Description
This PR displays links to the new sub-categories of CVEs, when the ROX_FRONTEND_VM_UDPATES feature flag is on, but keeps the existing links to generic CVE lists, when the flag is off.
This only affects the following links:

Header tiles on the VM Dashboard
Entities "shortcut" menu on the Workflow List layout
Entities "shortcut" menu on the Workflow Entity (single) layout

Note: the 2nd and 3rd menus look the same, but had to be changed in 2 different layout files because they need access to the feature flag.
I though about abstracting the repeated function to its own file, but soon realized I would need a curried function to pass the flag in, and since this check will only survive one release, I decided the repeated code was actually simpler to reason about.
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added

Testing Performed
Manual testing with the flag ON
Dashboard:

Inside page:

Manual testing with the flag OFF
Dashboard:

Inside page:

/retest-required

/retest


Cannot Reproduce Result of ZINC

Hi,
I trained some models on PCQM4M-LSC, ogbg-molhiv, and ZINC following the setting in the paper, and the results of PCQM4M-LSC and ogbg-molhiv are same as the paper. I also run experiment on ZINC several times, but the MAE is always more than 0.14 (with or without adding --intput_dropout_rate 0), which should be about 0.12 according to the paper. Here is my command:
python3 entry.py --dataset_name ZINC --hidden_dim 80 --ffn_dim 80 --num_heads 8 --tot_updates 400000 --batch_size 256 --warmup_updates 40000 --precision 16 --intput_dropout_rate 0 --gradient_clip_val 5 --num_workers 8  --gpus 1 --accelerator ddp  --max_epochs 10000

Would you please kindly provide the detailed python environment info to let us reproduce your problem?

Thanks for your reply, 0.14 MAE is on validation set. I'm using python 3.8.8, and below are the packages in my environment.
absl-py                 0.14.1
aiohttp                 3.7.4.post0
async-timeout           3.0.1
attrs                   21.2.0
cachetools              4.2.4
certifi                 2021.5.30
chardet                 4.0.0
charset-normalizer      2.0.6
Cython                  0.29.24
einops                  0.3.2
fsspec                  2021.10.0
future                  0.18.2
google-auth             1.35.0
google-auth-oauthlib    0.4.6
googledrivedownloader   0.4
grpcio                  1.41.0
idna                    3.2
isodate                 0.6.0
Jinja2                  3.0.2
joblib                  1.0.1
littleutils             0.2.2
Markdown                3.3.4
MarkupSafe              2.0.1
multidict               5.2.0
networkx                2.6.3
numpy                   1.21.2
oauthlib                3.1.1
ogb                     1.3.2
outdated                0.2.1
packaging               21.0
pandas                  1.3.3
Pillow                  8.3.2
pip                     21.0.1
protobuf                3.18.0
pyasn1                  0.4.8
pyasn1-modules          0.2.8
pyDeprecate             0.3.1
pyparsing               2.4.7
python-dateutil         2.8.2
pytorch-lightning       1.4.9
pytz                    2021.3
PyYAML                  5.4.1
rdflib                  6.0.1
rdkit-pypi              2<IP_ADDRESS>
requests                2.26.0
requests-oauthlib       1.3.0
rsa                     4.7.2
scikit-learn            1.0
scipy                   1.7.1
setuptools              58.0.4
six                     1.16.0
tensorboard             2.6.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.0
threadpoolctl           3.0.0
torch                   1.9.1+cu111
torch-cluster           1.5.9
torch-geometric         2.0.1
torch-scatter           2.0.8
torch-sparse            0.6.12
torch-spline-conv       1.2.1
torchaudio              0.9.1
torchmetrics            0.5.1
torchvision             0.10.1+cu111
tqdm                    4.62.3
typing-extensions       <IP_ADDRESS>
urllib3                 1.26.7
Werkzeug                2.0.1
wheel                   0.37.0
yacs                    0.1.8
yarl                    1.7.0

Would you like to test your trained model on ZINC test set (according to the best valid MAE)? The MAE of 0.122 reported in the paper is on test set, where the valid MAE of 0.14+ (we have not reported valid MAE in the paper).

It seems that there is not problem for reproducing, but I still want to know why there are some difference between the result of validation and test set.

Glad to see your reproduction. The gap is caused by the distribution of valid and test set of ZINC is not strictly same.

Close this issue due to inactive. Feel free to raise a new one or reopen this one for any further question.


Initial definition for stylus

References:

https://developer.mozilla.org/en-US/docs/Web/CSS/pseudo-elements
https://developer.mozilla.org/en-US/docs/Web/CSS/Pseudo-classes

@LeaVerou Fixed merge conflict

@LeaVerou Please review this too

Isn’t there anything of value this can borrow from the CSS definition? I feel that a lot of stuff is repeated, but it might just be my impression.

@LeaVerou That was what I tried at first. But, it didn't workout.

Ok, merged, though I have my doubts about the CSS thing.
Anyway, thanks for contributing!!

Thanks! I'm open to suggestions.


How to specify to run on my institute's HPC cluster?

Hi,
I downloaded your pipeline on my local institution cluster with:
git clone https://github.com/nf-core/proteomicslfq.git

and can successfully run the pipeline with the test example and it even works with nextflow tower after adding my nextflow access token in the nextflow.config file.
cd proteomicslfq
nextflow run main.nf -profile singularity,test -with-tower --outdir test

I realised all the jobs are running on the login cluster of my institution (CRG ANT-CLUSTER), while I'd like to run the pipeline the on the high performance computing cluster of my institution (CRG, Barcelona, Spain). To do this I would usually use qsub for example If I'd like to run STAR
qsub -q <QUEU_NAME> -V -cwd -pe smp <NUM_PROCESSES> \
         -l virtual_free=<NUM_RAM>G -l h_rt=<NUM_HOURS>:05:15 \
         -N STAR_TEST \
         -b y\
         STAR bla bla bla

In the past I achieve such a thing with nextflow simply by writing something like this in the nextflow.config file:
process {
      container = '/path/to/envs/or/container'
     	queue='long-sl7,short-sl7'
     	memory='12G'
     	cpus='4'
     	time='6h'
 }

Now, to my understanding in this pipeline nextflow.config file there's no such information cause the process requirments are specified in the conf/*.config files so that one has more flexibility on the processes demands, plus everything is nicely controlled by the check_max() function, which is fine for me.
I also read in the documentation that:

The pipeline also dynamically loads configurations from https://github.com/nf-core/configs when it runs, making multiple config profiles for various institutional clusters available at run time

however, I believe there's no config profile available for my institutional cluster, or I'm not doing it right (-profile crg failes for example).
So now what is the best practise to follow? Should I make my own my_institute.config to run on my institutional cluster or there's something that I'm doing wrong?
Thanks,
Nicco
> nextflow info

  Version: 20.07.1 build 5412
  Created: 24-07-2020 15:18 UTC (17:18 CEST)
  System: Linux 3.10.0-514.16.1.el7.x86_64
  Runtime: Groovy 2.5.11 on OpenJDK 64-Bit Server VM 1.8.0_131-b11
  Encoding: UTF-8 (UTF-8)

Dear @Ni-Ar,
all available configs can be found here.
If there's no matching config for the CRC, then I would like to suggest that you read Adding new configs and add one.
If you have questions feel free to ping me.
Cheers
Lukas

Hi @Zethson,
thanks for you reply. I'm afraid there's CRG config (even though it seems weird as nextflow was originally developed here in this institute). Anyway, is there no half-way solution? I mean I honestly believe I lack the competence and skill to make my institution's config "available at run-time to all nf-core pipelines". (Let's face it, I'm a n00b). So, I guess I could use -c for a local custom config I could write, no?

Yes, I was surprised too haha. I'll see whether I can dig something up from them.
But yes of course. You can as you mention just supply your own local config using -c.

Hi!
Of course, you can do that. It would even be the first step for making it public anyway ;)
Usually, I split my profiles in three parts:

One institutional profile which specifies, which cluster is used, which max capacities it has, and other specific information of the HPC environment. See the public institution profiles for an example.
One data size profile, which refines the default requested resources for the different process types we have, based on the size of your data. The default may not be enough for very large datasets. We intend to do some smart resource estimation in the next releases. See here for the defaults.
And optionally a dataset profile with the actual parameters of the pipeline specific to the used dataset (in case you want to reuse or share the analysis with the dataset).

The last two actually could/should be combined.

In the end I made a new config in the baseDir which is a sort of merged nextflow.config + conf/base.config and add:
process {
  queue='long-sl7,short-sl7'
  executor='sge'
  penv='smp'
  cpus = { check_max( 2 * task.attempt, 'cpus' ) }
  memory = { check_max( 8.GB * task.attempt, 'memory' ) }
  time = { check_max( 4.h * task.attempt, 'time' ) }

_omissis_

  withName:get_software_versions {
    cache = false
  }
}

And now it can run on the HPC.
Thanks a lot for the support!

@lucacozzuto

Eing? :)


RestClient::SSLCertificateNotVerified: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

from /usr/local/rvm/gems/ruby-2.3.1/gems/rest-client-2.0.0/lib/restclient/request.rb:809:in rescue in transmit'         from /usr/local/rvm/gems/ruby-2.3.1/gems/rest-client-2.0.0/lib/restclient/request.rb:698:in transmit'
from /usr/local/rvm/gems/ruby-2.3.1/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in execute'         from /usr/local/rvm/gems/ruby-2.3.1/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in execute'
from /mnt/campus/qlik_test/lib/qlik_api.rb:74:in test'         from (irb):1         from /usr/local/rvm/gems/ruby-2.3.1/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in start'
from /usr/local/rvm/gems/ruby-2.3.1/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in start'         from /usr/local/rvm/gems/ruby-2.3.1/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in console'
from /usr/local/rvm/gems/ruby-2.3.1/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in run_command!'         from /usr/local/rvm/gems/ruby-2.3.1/gems/railties-4.2.6/lib/rails/commands.rb:17:in <top (required)>'
from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in require'         from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in block in require'
from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in load_dependency'         from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in require'
from /mnt/campus/qlik_test/bin/rails:9:in <top (required)>'         from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in load'
from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in block in load'         from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in load_dependency'
from /usr/local/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in load'         from /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require'
from /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require'         from -e:1:in '
It appeared when i use the restclient to execute the request,like this:
qlik_response = RestClient::Request.execute(:method => :post,
:url => 'https://qv:4243/qps/ticket?Xrfkey=0123456789abcdef',
:payload => {'UserId' => 'user1', 'UserDirectory' => 'qv', 'Attributes' => []},
:headers => {'X-Qlik-Xrfkey' => '0123456789abcdef'},
:verify_ssl => true,
# :ssl_ca_path => '/etc/ssl/certs/',
:ssl_ca_file => 'pem/<IP_ADDRESS>/client.pem',
# :ssl_client_cert => OpenSSL::X509::Certificate.new(File.read('pem/<IP_ADDRESS>/client.pem')),
:ssl_client_key => OpenSSL::PKey::RSA.new(File.read('pem/<IP_ADDRESS>/client_key.pem')),
:content_type => :json)
but when i use the curl tool, it works well, like this:
curl -v -X POST "https://qv:4243/qps/ticket?Xrfkey=0123456789abcdef" --cert /mnt/campus/qlik_test/pem/<IP_ADDRESS>/client.pem --key /mnt/campus/qlik_test/pem/<IP_ADDRESS>/client_key.pem --header "x-qlik-xrfkey:0123456789abcdef" -d'{"UserDirectory":"qv","UserId":"user1"}' -H "Content-Type: application/json"
I tried this issue, but it doesn't work for me.
Any help is appreciated!

qlik_response = RestClient::Request.execute(:method => :post,
:url => 'https://qv:4243/qps/ticket?Xrfkey=0123456789abcdef',
:payload => {'UserId' => 'user1', 'UserDirectory' => 'qv', 'Attributes' => []}.to_json,
:headers => {'X-Qlik-Xrfkey' => '0123456789abcdef','Content-Type' => 'application/json'},
:verify_ssl => OpenSSL::SSL::VERIFY_NONE,
:ssl_ca_path => '/etc/ssl/certs/',
:ssl_ca_file => 'pem/<IP_ADDRESS>/client.pem',
:ssl_client_cert => OpenSSL::X509::Certificate.new(File.read('pem/<IP_ADDRESS>/client.pem')),
:ssl_client_key => OpenSSL::PKey::RSA.new(File.read('pem/<IP_ADDRESS>/client_key.pem'))
)
I use this parameters to request, and it work form me. But ignore the ssl verify is not a secure solution, is there any better suggestion?

I agree with @zhanjianmin, ignoring :verify_ssl => OpenSSL::SSL::VERIFY_NONE is not secure. Any other options?


TLSv1.3: Correct the Id for the TLS1.3 ciphersuite

Description of change

We have one TLS1.3 ciphersuite, but there is a typo in the id that should
be corrected.

Merged. Thanks.