Annotated StructuredDataset: add nested types support

Tracking issue
_https://github.com/flyteorg/flyte/issues/4241_

Why are the changes needed?
Currently StructuredDatasets only support flat schemas.
This PR aims to support nested types as form of dict/json, dataclass, named args/kwargs, or a mix from all of them.
What changes were proposed in this pull request?

flatten_dict() in base_task.py for nested types dict from kwtypes().

No data= in kwtypes(data=...) needed.
MUST use kwtypes() to pass types. Check comments.

After we get a list of SUPPORTED_TYPES, we select by series of key joined by ..

how to flat?   e.g.,

{
  "a": {
  "b": {
        "c": {
            "d": "vvv"
        }
      }
    },
    "e.f": "www",
}

to  {'a.b.c.d': 'vvv', 'e.f': 'www'}

One CANNOT pass only the sub keys. like "c.d".

How was this patch tested?
WIP
Setup process

Build a docker image with pandas then upload to locahost:30000 registry.
Run pyflyte run --image localhost:30000/basic:latest --remote ./dataclass.py struct_record_wf.
Result should filter out dataframe columns which those not set in kwtypes()

Screenshots
Check all the applicable boxes

[ ] I updated the documentation accordingly.
[ ] All new and existing tests passed.
[x] All commits are signed-off.

Related PRs

Docs link

from dataclasses import dataclass
from flytekit import task, workflow, kwtypes
from flytekit.types.structured import StructuredDataset
from typing_extensions import Annotated
from flytekit import ImageSpec, task

import pandas as pd

# custom_image = ImageSpec(
#     name="basic:latest",
#     registry="localhost:30000",
#     packages=["pandas", "numpy"],
#     apt_packages=['git'],
#     python_version=3.11,
# )

# if custom_image.is_container():
    # import pandas as pd

@dataclass
class DetailField():
    age: int
    sex: str

@dataclass
class RecordField():
    name: str
    detail: DetailField
    # city: str

Schema = Annotated[StructuredDataset, kwtypes({"Name": str}, {"Detail": {"Age": int, "Sex": str}})]
# Schema = Annotated[StructuredDataset, kwtypes(Name=str, Detail=DetailField )]
# Schema = Annotated[StructuredDataset, kwtypes({"Detail": {"Age": int, "Sex": str}})]
# Schema = Annotated[StructuredDataset, kwtypes(Detail=DetailField)]
# Schema = Annotated[StructuredDataset, kwtypes(Name=str)]#, however...
# Schema = Annotated[StructuredDataset, kwtypes({"Detail": DetailField}, Name=str)]
# Schema = Annotated[StructuredDataset, kwtypes(Name=str, Detail={"Age": int, "Sex": str})]

@task
def write_df() -> Schema:
    df = pd.DataFrame({"Name": ["Tom", "Joseph", "Alyssa"], "Detail": [{"Age":20, "Sex": "M"}, {"Age":22, "Sex": "M"}, {"Age":24, "Sex": "F"}], "City": ["Madrid","Paris","London"]})
    return StructuredDataset(dataframe=df)

@task
def read_df(structds: Schema):
    # print(structds.columns())
    df = structds.open(pd.DataFrame).all()
    print(df)

@workflow
def struct_record_wf():
    out = write_df()
    read_df(structds=out)

if __name__ == "__main__":
   struct_record_wf()

FROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y
RUN pip install -U git+https://github.com/austin362667/flytekit.git@ba969bdda617981d1b58ec6c550142a62d2257b1
RUN pip install -U pandas

testing: Schema = Annotated[StructuredDataset, kwtypes({"Name": str}, {"Detail": {"Age": int, "Sex": str}})]

testing: Schema = Annotated[StructuredDataset, kwtypes(Name=str, Detail=DetailField )]

testing: Schema = Annotated[StructuredDataset, kwtypes({"Detail": DetailField},{"City": str},{"City": str}, Name=str)]

@gitgraghu , @dylanwilder
Please help me check with any unexpected behavior. Thank you~
cc @pingsutw

@austin362667 it doesn't work for me. Try below example,
from typing import Annotated

import pandas as pd
from flytekit import StructuredDataset, kwtypes, task, workflow, ImageSpec

data = {
    'company': 'XYZ pvt ltd',
    'location': 'London',
    'info': {
        'president': 'Rakesh Kapoor',
        'contacts': {
            'email'<EMAIL_ADDRESS>            'tel': '9876543210'
        }
    }
}

# MyDataset = Annotated[StructuredDataset, kwtypes(company=str)]
MyDataset = Annotated[StructuredDataset, kwtypes(info={"president": str})]

@task
def create_bq_table() -> StructuredDataset:
    df = pd.json_normalize(data, max_level=0)
    print("dataframe: \n", df)
    return StructuredDataset(dataframe=df)
    # return StructuredDataset(
    #     dataframe=df, uri="bq://dogfood-gcp-dataplane:dataset.nested_type"
    # )

@task
def print_table(sd: MyDataset) -> pd.DataFrame:
    t = sd.open(pd.DataFrame).all()
    print(t)
    return t

@workflow
def wf():
    sd = create_bq_table()
    print_table(sd=sd)

@austin362667 it doesn't work for me. Try below example,

@pingsutw  Thank you for providing me useful use cases.
These examples can now work in remote mode.
And bq or arrow is still the tricky part.

Original DataFrame
  data = {
        'company': 'XYZ pvt ltd',
        'location': 'London',
        'info': {
            'president': 'Rakesh Kapoor',
            'contacts': {
                'email'<EMAIL_ADDRESS>                'tel': '9876543210'
            }
        }
    }

By Dict

By Dataclass

@gitgraghu  @dylanwilder
Please give it a try if it works in your GCP bq scenario. Thank you!
It pass in my own bucket and bigquery with my own GCP account, both in local and remote mode. However, I'm not 100% sure of other cases.

@dataclass
class Levels():
    # level1: str
    level2: str
Schema = Annotated[StructuredDataset, kwtypes(age=int, levels=Levels)]

@task(container_image=image)
def mytask_w() -> StructuredDataset:
    df = pd.DataFrame({
        "age": [1, 2],
        "levels": [
            {"level1": "1", "level2": "2"},
            {"level1": "2", "level2": "4"}
        ]
    })
    return StructuredDataset(dataframe=df)

@task(container_image=image)
def mytask_r(sd: Schema) -> pd.DataFrame:
    t = sd.open(pd.DataFrame).all()
    print("dataframe: \n", t)
    return t

@workflow
def mywf():
    sd = mytask_w()
    mytask_r(sd=sd)

Local

Remote

Add files field for ImageSpec() in order to copy gcp_keys.json
file into container.

@pingsutw  Should we leverage FlyteFile mentioned by @Future-Outlier so that avoiding modify ImageSpec() fields by adding files propriety.

Or separate ImageSepc() modification as another PR (merge it first due to dependency).


pull in master

This confused me at first, but eventually I figured out that you've made a pull request into your branch from master (which seems odd, but is in principle OK).


Fix: no need to do the qual-clause twice

fix #ISSUE_Number

Change logs
CBDB already support pushdown qual-clause. If current qual-clause have been process in access method, Then ExprState inside PlanState should reset to NULL, In this time, ExecScan should not exec it again.
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
Contributor's Checklist
Here are some reminders and checklists before/when submitting your pull request, please check them:

[ ] Make sure your Pull Request has a clear title and commit message. You can take git-commit template as a reference.
[ ] Sign the Contributor License Agreement as prompted for your first-time contribution(One-time setup).
[ ] Learn the coding contribution guide, including our code conventions, workflow and more.
[ ] List your communication in the GitHub Issues or Discussions (if has or needed).
[ ] Document changes.
[ ] Add tests for the change
[ ] Pass make installcheck
[ ] Pass make -C src/test installcheck-cbdb-parallel
[ ] Feel free to request cloudberrydb/dev team for review and approval when your PR is ready🥳

This is my plan for heap table:
postgres=# explain analyze verbose select * from int8_tbl where q1 in (select c1 from inner_text);
                                                                  QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice1; segments: 3)  (cost=0.00..1.05 rows=2 width=16) (actual time=2.100..2.103 rows=2 loops=1)
   Output: int8_tbl.q1, int8_tbl.q2
   ->  Seq Scan on public.int8_tbl  (cost=0.00..1.02 rows=1 width=16) (actual time=1.257..1.262 rows=2 loops=1)
         Output: int8_tbl.q1, int8_tbl.q2
         Filter: (hashed SubPlan 1)
         SubPlan 1
           ->  Broadcast Motion 3:3  (slice2; segments: 3)  (cost=0.00..150.00 rows=11600 width=32) (actual time=1.031..1.033 rows=2 loops=1)
                 Output: inner_text.c1
                 ->  Seq Scan on public.inner_text  (cost=0.00..150.00 rows=11600 width=32) (actual time=0.098..0.099 rows=1 loops=1)
                       Output: inner_text.c1
 Settings: optimizer = 'off'
 Planning Time: 0.471 ms
   (slice0)    Executor memory: 117K bytes.
   (slice1)    Executor memory: 316K bytes avg x 3x(0) workers, 414K bytes max (seg0).
   (slice2)    Executor memory: 111K bytes avg x 3x(0) workers, 111K bytes max (seg0).
 Memory used:  128000kB
 Optimizer: Postgres query optimizer
 Execution Time: 2.809 ms


Kan operations of fcom types, stage 1.

@jonsterling Lol. You have not seen nothing yet!

The examples showed that it works for the easiest cases. I think we need to improve the infrastructure a little bit before moving on (see the above discussion), and I should go back to the paper draft for the moment. @jonsterling I suppose I can merge this now? Feel free to take over pi1s1.prl if you have time.

sounds good, feel free to merge!

On Nov 1, 2017, at 11:48 AM, favonia (Sai-hong サイホン<EMAIL_ADDRESS>wrote:
The examples showed that it works for the easiest cases. I think we need to improve the infrastructure a little bit before moving on (see the above discussion), and I should go back to the paper draft for the moment. @jonsterling I suppose I can merge this now? Feel free to take over pi1s1.prl if you have time.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


feature request: @Version handling

It would be nice to have a way to handle columns that are designated to handle the version number for optimistic locking - marked with the @Version annotation. It would be great to see this feature integrated into the modeling and also in code generation and reverse engineering. Also shortcuts for adding the version flag to all tables/entites would be helpful for initialization.

Now I see that you have a feature to add generic annotations to any columns. That's good so far, but a tight integration would be better. Another feature idea: In the default setting all version related columns could be hidden because as a database designer i don't care about them normally. They should onyle be there for JPA. Maybe their presence can be visualized by an icon in the header bar of the table.

@Version support is already available in the jpa modeler and you don't need to use the generic annotation stuff for these, please check the following links :
http://jpamodeler.github.io/page.html?l=tutorial/AttributeMapping#VM
Thanks for the second suggestion but from my view, version column should not be hidden in the database modeler, may be developer want to change the column name.

Great, that's what i was looking for.
I noticed an issue with the diagram creation from reverse engineering: When the existing @Version properties have other annotation then the process doesn't recognize them properly.
Working:
@Version
private int version;

Not Working:
@Basic(optional = false)
@NotNull
@Column(nullable = false)
@Version
private int version;

@Version and @Basic could not be used for same property.
Correct me if I am wrong by proper details and jpa spec sections

You're probably right. It's up to you what to do only when creating the JPA DIagram from reverse engineering. Even if it's wrong it might stll be useful to apply the @Version annotation, even if the definition isn't clean.

Yes, we have already tried to captured and fix, most of the corrupted jpa entities class , relationship case scenario to diagram in reverse engineering action but there may be "N" case scenario.
For this case scenario, If Version will have priority then one day some will report that @Basic information is lost.


PendingMigrationError in test environment

My app has three migrations, with the following versions:
20170317113511
20170317115110
20170317123256
Calling FactoryGirl.create_list in one of the specs causes a PendingMigrationsError to be called for the specs it is related to (it is the only place it is called). But this does not happen if I just call FactoryGirl.create.
The migrations have definitely been run using rake neo4j:migrate RAILS_ENV=test. Immediately after this command completed, I ran rspec and everything behaved normally. However, further runs of rspec produce the following error:
Neo4j::PendingMigrationError:
       Migrations are pending:
<PHONE_NUMBER>3511
<PHONE_NUMBER>5110
<PHONE_NUMBER>3256
        To resolve this issue, run:

               bin/rails neo4j:migrate RAILS_ENV=test

Attempting to run bin/rails neo4j:migrate RAILS_ENV=test again produces the following error, which indicates that the migration has already taken effect:
Neo4j::MigrationError: Duplicate constraint for PlanSegment#uuid
And indeed calling rake neo4j:migrate:status RAILS_ENV=test yields the following information:
Status     Migration ID         Migration Name
--------------------------------------------------
down<PHONE_NUMBER>3511       PlanSegment
down<PHONE_NUMBER>5110       Event
down<PHONE_NUMBER>3256       RemovePositionFromPlanSegment

However, going into Rails console for the test environment and running the following command:
r = Neo4j::Migrations::Runner.new
r.instance_variable_get("@up_versions")

returns the following:
#<SortedSet: {"20170317113511", "20170317115110", "20170317123256"}>

Which as you can see indicates that all the "down" migrations are up.
I have tried stopping and starting the server, and deleting the test database and starting again. Not sure what else to try.
Runtime information:
Neo4j database version:
neo4j gem version: 8.0.10
neo4j-core gem version: 7.0.8

That's definitely strange.  I wonder if you have your tests set up to delete the database.  I ran into a problem a while ago where my tests would delete the whole database, but that also means that it was deleting the SchemaMigration nodes in the DB which are used to keep track of which migrations have been run.  For the moment I do the following:
Neo4j::ActiveBase.current_session.query('MATCH (n) WHERE NOT n:`Neo4j::Migrations::SchemaMigration` DETACH DELETE n')

I think the best solution, though, is to implement something like what Rails / ActiveRecord have in a schema.rb file where every time you run migrations, a "schema" file is created which stores the state of the database.  For Neo4j for tests if you are starting with no nodes this means you'd only need to create the indexes and constraints.  This also has the advantage that you don't need to keep running migrations to set up your test database, which only gets slower and slower as you get more migrations.
Having a schema configuration file for tests is probably my highest priority for the next thing to implement, though I've been quite busy lately.  I'm happy to talk about how that might work if you're interested

I am interested, but have only a cursory understanding of with how this side of Rails works - so you might spend a while explaining this to me!
Further investigation of the bug get curiouser and curiouser:
I am using the database_cleaner gem, which has neo4j support, and I replaced it with the following snippet from the docs:
  config.around do |example|
    Neo4j::Transaction.run do |tx|
      example.run
      tx.mark_failed
    end
  end

I also tried your code snippet above, in the following code:
  config.after(:each) do
    Neo4j::ActiveBase.current_session.query('MATCH (n) WHERE NOT n:`Neo4j::Migrations::SchemaMigration` DETACH DELETE n')
  end

and I still have the same problem.
Furthermore the issue appears to arise when get method is called in a request spec,  after factory_girl.create_list has run. If I go into byebug and call the FactoryGirl.create_list and get methods manually then the former runs and the latter does not - the code does not even reach the controller.
I stepped through the process as far as I could, and got to:
.rvm/gems/ruby-2.4.0/gems/actionpack-5.0.2/lib/action_dispatch/middleware/callbacks.rb:38
but I couldn't trace the problem any further.
So I then placed a breakpoint at Neo4j::Migrations::CheckPending#call and stepped as far as Migrations::Runner#initialize. The results at this point might be interesting. The @schema_migrations variable equals [] after it has been initialised. Which struck me as odd since there are clearly several migrations.
check_for_pending_migrations! fails at this point as expected, which is obviously where the exception is being thrown.

So, there are a couple of things to clarify:

Unfortunately, while Neo4j supports transactions and it will even let you nest transactions, but if you do, getting a failure of the inner transaction rolls back the entire outer transaction.  So transactions are not, unfortunately, a good way to roll back the state of the database after each test because if your test gets an exception (even an expected one that is expected by your test), the entire transaction for the test setup will fail.  That is why I instead use that DETACH DELETE command (incidentally, I prefer putting it in my before instead of my after because I don't like the idea of depending on other tests cleaning up)
The other problem that you may be having (which is not unique to Neo4j) is that if your test is running inside a transaction and you're doing a request spec, the browser won't be able to see the objects that were created by the tests (because they were inside a transaction are are only visible to the test itself until it's committed).  Changing out to use DETACH DELETE instead of transactions should help with that too.

Ah, yes! I forgot all about the second issue. Don't use non-api Rails as much as I used to. OK, so I've moved the DETACH command into the before block and removed DatabaseCleaner entirely. Then I ran rspec again and it still failed, complaining about attributes being defined in the model. However, this was fixed simply by migrating the test environment.
Thank you so much for your help. You really made an effort.
Do you think it would be worth adjusting the "pending migrations" error message such that if the program is running in the test environment that it mentions your point about request specs?

There are potentially two issues (I think).  First, people shouldn't be running their tests in transactions (transactions in the tests are fine, just not transactions around the tests).  I could clarify that in the docs.
But I think that the pending migrations is a separate issue.  I've actually introduced some changes in the branch for the upcoming 8.1.x series which makes migrations not checking inside of tests.  See this comment on another issue:
https://github.com/neo4jrb/neo4j/issues/1368#issuecomment-289940339
Of course your database still needs to be in a good state, but you'll generally get other errors (like that an index or a constraint isn't there) and thus you'll need to setup the db (again, from the comment that would be with the neo4j:schema:load rake task, but it could also be via migrations)


Adding Firefox use counter data to HTTPArchive

Hey folks,
In https://github.com/HTTPArchive/legacy.httparchive.org/issues/59 Chrome's use counter data was added to httparchive.
The relevant code in this repo seems to be https://github.com/HTTPArchive/data-pipeline/blob/41fe511951797d25cebc71097726c8b65497212b/modules/import_all.py#L146
I'd like to have Firefox use counter data also be available in httparchive. (Maybe more things also, but starting with use counters.) To that end, I've filed https://bugzilla.mozilla.org/show_bug.cgi?id=1813593 so that the data can be extracted locally.
Are there considerations we should know about for this to work?
cc @emilio @janodvarko

We only run tests in Chrome, so I don't think this would be feasible. @pmeenan WDYT?

Why not? We'd need to also run tests in desktop Firefox, in addition to current desktop Chrome and Android Chrome. I don't think it's necessarily useful to collect and store everything for Firefox that is currently stored for Chrome, that would increase storage with 50%. But only use counter data seems negligible for storage.

It’s not just storage. It’s also crawl capacity.

As it stands right now, it takes ~25,000 VM's the better part of a week to collect the data.  Technically it is pretty easy to support but financially it would increase the running costs by ~30% (assuming we'd only run one config).  I'm guessing some form of additional sponsorship would be needed to cover the costs.

OK, thanks. What would that amount to in USD?

30% of our current crawl expenses would come out to about $20k per month.

That is likely more than the value Mozilla would get from the data. 🙂
For web compat analysis, the sample_data URLs (10k pages) would still be better than nothing. Assuming a full run would be 12,500,000 URLs (httparchive.pages.2023_01_01_desktop has 12,647,566 rows), 10k pages would be 0.08% of the cost, which is .... $16.
Would it be feasible to start there?

Update: https://bugzilla.mozilla.org/show_bug.cgi?id=1813593 is now fixed (thanks @emilio!). It's possible to set these prefs to log use counter data to stderr:

dom.use_counters.dump.document
dom.use_counters.dump.worker
dom.use_counters.dump.page

For the purpose of this issue, using page and worker but not document makes most sense. (document is for each document, including e.g. SVGs; these accumulate into page which is per top-level page.) worker use counters don't accumulate into page so need to be included separately.
The logged output looks like this:
USE_COUNTER_PAGE: USE_COUNTER2_DOCUMENTOPEN_PAGE - http://software.hixie.ch/utilities/js/live-dom-viewer/
USE_COUNTER_PAGE: USE_COUNTER2_CSS_PROPERTY_Display_PAGE - http://software.hixie.ch/utilities/js/live-dom-viewer/
USE_COUNTER_PAGE: USE_COUNTER2_CSS_PROPERTY_FontStyle_PAGE - http://software.hixie.ch/utilities/js/live-dom-viewer/
USE_COUNTER_PAGE: USE_COUNTER2_CSS_PROPERTY_FontWeight_PAGE - http://software.hixie.ch/utilities/js/live-dom-viewer/

You need to close the page for some of the use counters to be added to the log.


instaloader

i want to get the data from insta by instaloader to a sqlite database but i could not.if you can tell me why i would appreciate
import instaloader
from datetime import datetime
from itertools import dropwhile, takewhile
import time
import csv
import urllib3, socket
import sqlite3
from sqlite3 import OperationalError, connect
l = []
db = sqlite3.connect("database1.db")
cur = db.cursor()
cur.execute(
"""CREATE TABLE IF NOT EXISTS hedie(
id INTEGER PRIMARY KEY,
captions TEXT)"""
)
loader = instaloader.Instaloader()
loader.login('hedrashidii', '09227365701Hedi@')
def get_post_info_csv(hashtag):
with open(hashtag+'.csv', 'r', newline='') as file:
writer = csv.writer(file)
posts = instaloader.Hashtag.from_name(loader.context, hashtag).get_posts()
for post in posts:
print("post caption: " + str(post.caption))
for i in posts:
l.append(i)
posturl = "https://www.instagram.com/&__d=dis" + post.shortcode
print("post url: " + posturl)
writer.writerow(
["post", post.caption])
hashtag = "به"
get_post_info_csv(hashtag)
cur.executemany("""INSERT INTO hedie(captions)VALUES(?)""", l)
db.commit()
db.close()

This is not worth posting as an issue. This is more of a stackoverflow question, but likely won't get a response on stackoverflow because you haven't described what issue you're facing, and how you're trying to do it.

thank you


Security-Gateway Action remove-permissions deletes valid SG rules

When run a policy to remove any SG rule that has a source IP of <IP_ADDRESS>/0 and is not http or https the filter appears to match okay, but the action removes ALL SG rules where the source is <IP_ADDRESS>/0, including any with HTTP or HTTPS. Use of ingress: matched does not appear to make any difference.
Policy:
policies:
  - name: sg-test
    resource: security-group
    filters:
       - and:
         - type: ingress
           OnlyPorts: [22, 80, 443]
#
         - type: ingress
           Cidr:
              value_type: cidr
              op: eq
              value: <IP_ADDRESS>/0
#
    actions:
      - type: remove-permissions
        ingress: matched

the removal is currently against the ANDED set of things that matched each ingress filter independently, the default block operator is 'AND'. The your intended use case is had by specifying the full condition for the rule on a single ingress filter, ie. with OnlyPorts and Cidr` on the same ingress filter.

So i think you are saying the filter should be as follows?:
         - type: ingress
           OnlyPorts: [22, 80, 443]
           Cidr:
              value_type: cidr
              op: eq
              value: <IP_ADDRESS>/0

I think that I'm experiencing a similar problem, except I think that the issue might be related to OnlyPorts not working properly. Here's an example of a filter (the formatting is a bit weird due to having been code-generated from another markup language):
filters:
  - Cidr:
      op: eq
      value: <IP_ADDRESS>/0
      value_type: cidr
    OnlyPorts:
    - 22
    type: ingress
- actions:
  - ingress: matched
    type: remove-permissions

My expectation is that this filter would not match a security group whose only rule is to open port 22 to the world. However, it is matching the following SG:

Given this, I'm suspecting that the behavior that @timbo256 reported is due to the OnlyPorts condition not taking effect.
This issue also looks like a potential duplicate of #2484.

From some further testing I would agree with @JoshRosen, that the issue seems to be with OnlyPorts as if do a more specific test using Ports (for the items I want to remove) it works as expected.

@JoshRosen thanks for the example that seems definitely out of alignment to the intent of how that should work.

A colleague, @marcelluseasley, and I have been looking into this and we've identified the bug. It's an unhandled edge case (and untested as far as I can tell) when OnlyPorts and Cidr are specified together.
It's a very subtle issue in c7n.resources.vpc.SGPermission.process_ports:
    def process_ports(self, perm):
        found = None
        if 'FromPort' in perm and 'ToPort' in perm:
            for port in self.ports:
                if port >= perm['FromPort'] and port <= perm['ToPort']:
                    found = True
                    break
                found = False
            only_found = False
            for port in self.only_ports:
                if port == perm['FromPort'] and port == perm['ToPort']:
                    only_found = True
            if self.only_ports and not only_found:
                found = found is None or found and True or False
        return found

In the above method, the return value (found) defaults to None. If only_ports is specified and ports is not, AND the permission we're checking matches a port in only_ports, (i.e. only_found becomes True)... the method ends up never setting found to False, it's left at the default value of None.
So given the above, when only_ports is set and ports is not set, process_ports() returns True for ports that aren't in only_ports but None for ports that are in only_ports.
In the call method of c7n.resources.vpc.SGPermission, we have the following block of code for evaluating each permission on the SG:
            perm_matches = {}
            for idx, f in enumerate(self.vfilters):
                perm_matches[idx] = bool(f(perm))
            perm_matches['ports'] = self.process_ports(perm)
            perm_matches['cidrs'] = self.process_cidrs(perm)
            perm_matches['self-refs'] = self.process_self_reference(perm, sg_id)
            perm_match_values = list(filter(
                lambda x: x is not None, perm_matches.values()))

            # account for one python behavior any([]) == False, all([]) == True
            if match_op == all and not perm_match_values:
                continue

            match = match_op(perm_match_values)
            if match:
                matched.append(perm)

This code checks for permission matches for ports, cidrs, and self-references, and then builds a list of all of those values that are not None.
The end result of this is that when a port matches one of the OnlyPorts values, process_ports() returns None, which gets ignored when building perm_match_values. In this case, the port check is effectively ignored and if all other checks (CIDR or self_reference) are True, the permission is considered a match.
The fix for this is adding a simple branch at the end of process_ports to return False if only_ports is present and only_found is True - i.e. stop ignoring the negated match for only_ports and return False.
We'll be trying to work up a PR for this that includes the relevant tests; it appears that OnlyPorts is not currently tested in combination with cidr or self-reference.


Linking on columns with different names

comparison_columns in settings assumed columns both have the same name in two datasets. Fine for deduping, but not necessarily the case when linking (e.g. l.birth_date = r.dob)

Yep.  This will be a bit annoying for users who will have to rename columns.
But I think the additional complexity of code we'd need in sparklink isn't worth it - e.g. we'd have to fundamentally change the settings object.

I assume this is still not possible with splink3? I have a Phone and CellPhone column in 1 dataset and I want to link either of those to a Phone column in a different dataset.


Fix: Define test function as early as possible

Closes #68

What I did
Attempt on fixing #68 by injecting the __test script as early as possible
Checklist for Contributors
Manual testing
This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!
Documentation

[ ] Add or update documentation reflecting your changes in this repository
[ ] Request documentation updates in the test-runner docs website

Checklist for Maintainers

[ ] Make sure this PR contains one of the labels below:

Available labels

skip-release: Skip any releases, e.g., documentation only changes, CI config etc.
patch: Upgrade patch version (e.g. 0.0.x)
minor: Upgrade patch version (e.g. 0.x.0)
major: Upgrade patch version (e.g. x.0.0)

📦 Published PR as canary version: 0.17.1--canary.445.4041ab4.0

:sparkles: Test out this PR locally via:
npm install<EMAIL_ADDRESS># or
yarn add<EMAIL_ADDRESS>

With this patch I was able to resolve "__test in not defined" errors.


Ibax

Describe the bug
it needs to be refreshed several times for the coin to appear, and it happened again and again when I logged in.. hopefully it can be resolved
Reproduction
Good
System Info
Good

Logs
Good

Validations

[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

Problems with Internet connection, please try again. thanks.


Logo Strategy

I feel it's been a while since we've talked about Shields logos (the handful of ones we maintain and provide directly from https://github.com/badges/shields/tree/master/logo) and that it'd be helpful if we can articulate our current posture and ensure our logo documentation accurately reflects that posture.
We've had various discussions and considerations over the last few years (e.g. #7684), including whether we should drop them outright. We did semi-recently decide to keep the ones that we already have, or at least we didn't decide to delete them, but I'm unclear whether we took a position on whether we could/would potentially ever add any more (and obviously opinions can and do change over time).
As such I'd like to use this to discuss if we collectively feel there are circumstances in which we'd accept a new logo (and if so capture that in our docs), or even if we'd like to drop the logos altogether.
Framing as options:

Drop the existing set of logos we maintain, and do not take on any new ones, only provide logos via SI and the custom logo param (as detailed below in https://github.com/badges/shields/issues/9476#issuecomment-1681256345)
Keep the existing set but refuse to take on any new ones under any circumstances
Keep the existing set and potentially take on new ones after determining/updating guidelines under which we would

I confess I'm ambivalent on the topic myself. I definitely think that SimpleIcons should remain the first port of call, as I'm cognizant of the drawbacks we've experienced in the maintenance of the current set. I feel like if we are to consider accepting new logos that we'd need a set of rather high thresholds a proposed logo would have to clear, and I think that we would also still need to have the discretion to choose on a case by case basis (i.e. our docs would enumerate the key requirements, but we'd still retain the right to reject a proposed logo based on other considerations regardless of whether it cleared all the documented requirements).
That being said, assuming we were to keep logos, I'd find a hypothetical scenario with the following characteristics to be rather compelling:

a vendor/brand owner (or someone operating explicitly on their behalf) came to us with a PR to incorporate their logo because the monochrome SI route wasn't viable because doing so would ruin/break the brand identity
vendor/brand representative explicitly agreed they'd follow up if/when they had a branding change that needed to be reflected in the logo (as opposed to giving us flak for the "shields" logo being outdated)
there was a sizeable portion of our user base that wanted to use the logo
the logo was too big to be used via the custom logo feature in the url (node.js' hard limit on the header size)

Admittedly a fairly tall order, so I wonder if a hypothetical scenario that only had some of those characteristics (and/or other ones) would be sufficiently compelling.
Curious your thoughts @chris48s @PyvesB

As a contributing user who got tripped up by this policy, I would only ask that maintainers make some kind of effort to follow their own documentation, whatever that may be, and that if there is not an intent to ask a contributor to close a PR, that language should not be used to that effect.
I've contributed to a number of similar projects -- for example the icon set for Buildkite -- and I don't know why this one is different. While the custom logo feature works, embedding an SVG can be quite large and is suboptimal for diffs. Obviously embedded SVGs do not update when brands update, which is an unfortunate side effect that renders one of Shields' coolest features inert.
The ship has sailed on the PR I tried to file, as it was shot down immediately (or, least I thought it was), and then a strange and meandering discussion ensued. A clear policy would help to avoid this situation. Thanks for your guys' hard work on this project and while I realize it is not your day jobs, many many open source projects rely on it, so I would hope you would be open to contributions and a fair and clear process given your important position in the community.
As a fan and potential contributor to Shields, thanks for filing @calebcartwright.

I've not had a chance to read over the whole sage of https://github.com/badges/shields/pull/9474, https://github.com/badges/shields/issues/9475, https://github.com/badges/shields/issues/9477, and https://github.com/badges/shields/issues/9478
but I've seen it is there and I'll try to plough through it.
Tbh, I try to stay out of logo discussions, so I don't think I've ever actually written this down on GitHub but..
I'd ideally really like to completely delete the remaining custom icons and say the available options are SimpleIcons or base64. It would solve a lot of problems IMO.
That wasn't the direction we went in after https://github.com/badges/shields/issues/7684 but it does leave us in this limbo state where we have this slightly odd collection of custom icons for historical reasons (all of them were added before simple-icons), and it is difficult to justify why those but not others (gotta have that Travis CI icon because.. everyone uses that these days).
Whatever we do, someone is going to be unhappy, but I feel like if we just deleted the handful of remaining custom icons, we deal with one backlash, rip that band aid off, and that buys us:

a more consistent and explainable logo policy
less suprising behaviour that is easier to document/explain (currently logo is a simple-icons slug ..except when it isn't)
an unambiguous solution to https://github.com/badges/shields/issues/4947
a category of issue/content we have definitively decided not to deal with

That may or may not be a helpful opinion.

I'd ideally really like to completely delete the remaining custom icons and say the available options are SimpleIcons or base64
That may or may not be a helpful opinion.

Helpful indeed Chris and I've updated the issue description to be more explicit about the options because dropping them altogether certainly is one.
As I commented back in #7684 I do think our logos look "better" (as vague/subjective that may be), and I can understand why someone would want our logo vs. the other. However, I'd realized I was actually mildly disappointed we didn't decide to get rid of them back then.
I know there's an audience that strongly prefers them, but I feel like it's a relatively small subset of our user base that actually uses them, and that the logos eat up an inordinate amount of maintainer effort/energy/bandwidth cost in comparison.
Probably best for now to punt discussion of specifics around how we'd potentially go about doing so, but something that could make it more palatable for me would be if we provide a list of the corresponding precanned ?logo=... params to make it easier for people to copy/paste (acknowledging the Travis logo would be the one exception)
I believe I've a soft preference for option 1, followed fairly closely by option 3. I'm not feeling too fond about option 2, as while I suppose it's better than the current nebulous state of affairs, it would probably perpetuate the most of the current problems we have compared to the other options.

I think this is a place where you guys just need to make a decision.
Delegating to SimpleIcons is reasonable. I don't think anyone would fault Shields for doing that.
I agree that consistency would be a better user experience than the marginal benefit of the built in logos. I don't think there should be "special" logos.
If there is this much ambiguity with a contribution policy, it isn't really a policy and will only serve to upset people who follow it in good faith.

If there is this much ambiguity with a contribution policy, it isn't really a policy and will only serve to upset people who follow it in good faith.

I understand the benefits of an explicit policy with fully deterministic criteria, and why someone might prefer such a policy. However, it's important to note that what we currently have is a set of guidelines, and not a rigid policy.
This is intentional; it is a feature, not a bug. There are scenarios in which it is not possible to enumerate a fully exhaustive list of considerations and/or is beneficial and desirable to retain the ability to apply discretion, to make a judgement call on a case by case basis. Our current posture on logos is such a scenario.
Furthermore, I'd also note that in my opinion this is already well reflected in the existing logo documentation text in https://github.com/badges/shields/blob/master/doc/logos.md#contributing-logos, with several examples including (emphasis mine)

Our preferred way to consume icons is via the SimpleIcons logo. As a first port of call, we encourage you to contribute logos to the SimpleIcons project.

In some cases we may also accept logo submissions directly. In general, we do this only when:

We may also approve logos for other tools widely used by developers.

There is substantial benefit in using a multi-colored icon over a monochrome icon.

I don't think anyone should read that and come away thinking that any proposed logo will absolutely, unequivocally be accepted. That being said, if we decide against option 1 then I wouldn't be opposed to adding something more explicit in these docs (e.g. something to the effect of "We do not guarantee we will accept a new logo, and we encourage potential logo contributors to contact us first to determine viability"), especially if this is likely to be a recurring source of confusion/misinterpretation.
However, I'd be vehemently opposed to attempting to establish a policy with a binding, exhaustive set of conditions from the current maintainer team discretion with a general set of guidelines/considerations. I do not believe it would be possible for us to establish a fully future-proof, exhaustive list under which we must accept a logo.

Personally I'd have to go with option 1 here. It completely removes the burden on maintainers and for most cases base64 SVGs will do the trick.

My opinion has broadly not changed since last year:

I'd personally be leaning towards simplification and removing all our custom icons. If some users felt strongly about one icon or another, they can always provide their own encoded one via the logo parameter.

Excellent, sounds like a decision made then. I'll open up one or more issues to cover next steps, with links back to this one and other relevant discussions.

@calebcartwright > I don't think anyone should read that and come away thinking that any proposed logo will absolutely, unequivocally be accepted
The point was that there was no better case than Bazel for meeting this criteria, not that I believed it would be unequivocally accepted (I did and do not). I'm happy to at least see an outcome which can be followed consistency by people who contribute in good faith.


[Enhancement]: Optional notification message close

Click on notification message will be optional. As a result, any a href tags used inside the notification message will be clickable.
Fixes #90.

Thanks, any chance you can squash this into a single commit?

sorry @poteto for the multiple file commits. I have a global git username (primary) which is not being overridden by local git username (looks like some prob), so am unable to use the terminal for git commits to my personal repo. I will try to figure out a way to sort that out and send PRs in future as single commit.

Hey @abhilashlr, if you add me as a collaborator to your repo I can squash it for you :+1:

sure I will @poteto

Hello @poteto, I have added you as collaborator to my repo.

What's the status of this issue?


fix(ingest): use logger.warning instead of logger.warn

The logger.warn method is deprecated in favor of warning.
Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[x] Links to related issues (if applicable)
[x] Tests for the changes have been added/updated (if applicable)
[x] Docs related to the changes have been added/updated (if applicable)

@hsheth2 : seems like a black check failed


chore(main): release pixel-motion 1.0.1

:robot: I have created a release beep boop
1.0.1 (2024-10-21)
Bug Fixes

deps: update dependency framer-motion to v11.11.2 (4466f3a)
deps: update dependency framer-motion to v11.11.4 (0dd94c6)
deps: update dependency framer-motion to v11.11.7 (bd35112)
deps: update dependency framer-motion to v11.11.8 (78e04ce)
deps: update dependency framer-motion to v11.11.9 (e0f1f1f)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

pixel-motion-v1.0.1
:sunflower:


Payflow Remove Urlencoding

Per Payflow Gateway documentation specifically indicates NVP's should not be urlencoded.
Do Not URL Encode Name-Value Parameter Data
Do not URL encode your NVP data because it can cause problems with authentication and reporting.

Closing in favor of #167 - excluded first commit, which is not quite relevant (makes ppref optional, but at the same time makes pnref required).

Which requires to generate migration and addressed in the PR #118.


Unable to add a new source

Describe the bug
As part of the Quickstart I was playing around with the UI and CLI and wanted to add a new ingestion (MongoDB to be specific). However, no matter what I tried I encountered a "Connection refused" error related to GMS.
To Reproduce
Steps to reproduce the behavior:

Follow the steps in the Quickstart and and observe that datahub docker check returns that there are no issues detected.
In the UI click on Ingestion.
Click Create new source and select Mongo.
Enter mongodb://localhost for the connect_uri (I have Mongo running locally on port 27017), and also the correct credentials for the username and password fields.
Leave the sink's server address as http://localhost:9002/api/gms
Skip setting up a schedule and finish the setup.
Click execute within the UI.

After a few seconds the job fails and from the logs I see the following.
"establish a new connection: [Errno 111] Connection refused')': /api/gms/config\n"
           '[2022-04-14 23:03:53,631] WARNING  {urllib3.connectionpool:810} - Retrying (Retry(total=1, connect=None, read=None, redirect=None, '
           "status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f153f0d64c0>: Failed to "
           "establish a new connection: [Errno 111] Connection refused')': /api/gms/config\n"
           '[2022-04-14 23:04:01,618] WARNING  {urllib3.connectionpool:810} - Retrying (Retry(total=0, connect=None, read=None, redirect=None, '
           "status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f153f0d6310>: Failed to "
           "establish a new connection: [Errno 111] Connection refused')': /api/gms/config\n"

If I change the server's sink address to http://localhost:8080 I get the same error but instead of /api/gms/config the logs contain /config.
Expected behavior
The ingestion of the data in my MongoDB would be successful and I'd see the data in the UI.
Desktop (please complete the following information):

OS: macOS Big Sur 11.6.2
Browser: Chrome Version 100.0.4896.88 (Official Build) (x86_64)

@putnam120 I think you are doing it wrong, instead of using "localhost:9002/api/gms" and "localhost:8080" you should use 'http://datahub-gms:8080'
Since you have not enabled Access token functionality you are unauthorized to make a request.
When you enable access token generation in your you can use "localhost:9002/api/gms" with token config.

You can refer to this link for more details : https://datahubproject.io/docs/ui-ingestion/

Yes - That is the correct solution.
Thanks @nj7 for replying, and thanks @putnam120 for raising the issue.
Closing this one out.
John


Content-type header missing when using the API Client without a proxy

When sending requests without a proxy, the content-type header seems to be missing.
Example spec: https://github.com/FotieMConstant/lanadoc/blob/main/public/resource/openapi-spec-sample.yaml
Screenshot: https://user-images.githubusercontent.com/42372656/277362781-ae63be4f-032f-43e7-8eba-b29549e0d507.png
Further context: https://github.com/scalar/scalar/discussions/351

This issue seems to have been fixed as of ^0.8.2 for some reason.

It’s fixed, and I’ve added a test and improved the sendRequest method in #384. :v:


I found  flink-cdc supported connectors using YAML definition,its very convenient,i love it.so my question is  when other pipeline is  supported?for example,i want sync data form oracle to pg or pg to mysql.Thanks!

Please don't check the box below

[X] I'm aware that bugs and new features should be reported on Apache Jira or Flink mailing list instead of here

Again, please don't check the box below

[X] I'm aware that new issues on GitHub will be ignored and automatically closed.

请不要勾选以下选项

[X] 我已知悉缺陷和新功能需要在 Apache Jira 或 Flink 邮件列表中反馈，而不是在这里创建新 issue。

也请不要勾选以下选项

[X] 我已知悉 GitHub 上的新 issue 会被忽略且自动关闭。

As required by Apache Flink, please report bugs or new features on Apache Jira under the project Flink using component tag Flink CDC.
You can also initiate discussions in Apache Flink's dev mailing list<EMAIL_ADDRESS>for technical issues.
If you have any questions about using Flink CDC, please use Flink user mailing list<EMAIL_ADDRESS>


Fix issue #2221 by adding a newline character to generated code

Add a newline prefix to the code generated for non-immutable harmony exports. This
resolves a case where export statements without a trailing semicolon would cause
webpack to generate a bundle with invalid syntax.

Thanks


Simplify authorization configuration code

Using updated APIs to simplify the configuration of authorization policies.

/cc @BillWagner - this should do the same thing as the current code, but less verbose.


refactor tox & circle configurations

it all looks good to me but maybe it would be a good idea if we just remove the commented out code

@mochic808 consider those "notes to self" :laughing:


Update kubelet reference

The reference for kubelet is manually maintained due to upstream code changes.
This PR updates the reference according to the help message from the 1.23 version of kubelet executable.

/approve
Thanks @tengqm

Preview: https://deploy-preview-31464--kubernetes-io-main-staging.netlify.app/docs/reference/command-line-tools-reference/kubelet/

Thanks @tengqm .
/lgtm


🚀 Enhancement: Show notification after adding (and verifying) authenticator

🔖 Enhancement description
After adding (and verifying) an authenticator, the console should show a notification saying:

Authenticator has been added

similar to how we show a notification after enabling/disabling MFA:
https://github.com/appwrite/console/blob/f664172c338f39242bf8b5f589eddb51d150efb7/src/routes/console/account/updateMfa.svelte#L33-L36
🎤 Pitch
This gives user feedback that something happened successfully.
👀 Have you spent some time to check if this issue has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Hi, I'd like to work on this! Just to clarify, the notification should be shown both after adding and after verifying? If this is the case should the messages then be "Authenticator has been added" (for addAuthenticator()) and "Authenticator has been verified" (for verifyAuthenticator())?


Prisma 5 drops undefined from Arrays when using Json fields with Postgres

Bug description
Since Prisma 5, undefined values in an array are dropped when writing them to a Json field using Postgres. In Prisma 4,
undefined values were turned into null, mirroring JSON.stringify behavior in JavaScript.
I expect undefined values not to get dropped from arrays and the encoding behavior to match JSON.stringify.
How to reproduce
Schema:
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model Todo {
  id    Int     @default(autoincrement()) @id
  content String
  checked  Boolean @default(false)
  test Json?
}

Create call:
await prisma.todo.create({
    data: {
      content,
      checked: false,
      test: ["here", undefined, "is", null, "a", 0, "value"],
    },
  });

Bug:
const todos = await prisma.todo.findMany();

console.log(todos[0].test) // ["here, "is", null, "a", 0, "value"]
// Expected: ["here, **null**, "is", null, "a", 0, "value"]

Repros:
Prisma 4: CodeSandbox
Same container but with Prisma 5: CodeSandbox
You can run node in those containers and execute this:
const PrismaClient = require('@prisma/client').PrismaClient;
const prisma = new PrismaClient();
const t = await prisma.todo.findMany();
console.log(t);

To inspect the data returned from Prisma.
Expected behavior
JavaScript turns JSON.stringify([undefined, null]) into '[null, null]'. I expect Prisma 5 to match this behavior.
Concretely, the above code should return ["here, null, "is", null, "a", 0, "value"] like in Prisma 4 instead of ["here, "is", null, "a", 0, "value"]. Notice that the second element (null) is missing in the Prisma 5 version.
Environment & setup
Latest of literally everything, macOS and Alpine Linux.
Prisma Version
5.0.0

Thanks, @cpojer
I could reproduce this difference in behavior that is triggered by using jsonProtocol (which became the default in Prisma 5)
From what I see the types do not allow this in Prisma 4 or 5 but without type-checking the runtime behavior differs like you found out -> undefined is removed from the array with jsonProtocol
Sample code
import { PrismaClient } from "@prisma/client";

async function main() {
  const prisma = new PrismaClient();

  console.debug(
    JSON.stringify(["here", undefined, "is", null, "a", 0, "value"])
  );
  // will output the following:
  // ["here",null,"is",null,"a",0,"value"]

  const data = await prisma.todo.create({
    data: {
      content: "hello world",
      checked: false,
      // @ts-ignore is required here as `undefined` is actually not allowed by the types
      // Type error because of `undefined`:
      //     Type '(string | number | null | undefined)[]' is not assignable to type 'NullableJsonNullValueInput | InputJsonValue | undefined'.
      // Type '(string | number | null | undefined)[]' is not assignable to type 'InputJsonObject'.
      //   Index signature for type 'string' is missing in type '(string | number | null | undefined)[]'.ts(2322)
      test: ["here", undefined, "is", null, "a", 0, "value"],
    },
  });
  console.debug(data);
  // 4.16.2 (and before)
  // {
  //   id: 3,
  //   content: 'hello world',
  //   checked: false,
  //   test: [ 'here', null, 'is', null, 'a', 0, 'value' ]
  // }
  // 4.16.2 jsonProtocol (and same in v5 as `jsonProtocol` became the default)
  // {
  //   id: 4,
  //   content: 'hello world',
  //   checked: false,
  //   test: [ 'here', 'is', null, 'a', 0, 'value' ]
  // }

  prisma.$disconnect();
}

main();

I'm using https://github.com/arthurfiorette/prisma-json-types-generator which likely masks this issue, and for about a year before that I had to cast things into Prisma Json types to make it not complain about various other issues with using Json types.
I would propose this path forward:

Restore the pre-jsonProtocol Prisma 4 behavior to align with JSON.stringify's behavior as that default likely matches the expectation most people have
Align the types with this behavior to allow undefined

Thanks for the suggestion and extra info, I'll check with @SevInf what he thinks about this when he's back on Monday!

Thank you for aligning the behavior!


When merging cards/adding new cards. Current Card Index is Reset to 0

I want to dynamically add cards. when merging like in the example:
this.setState({
          cards: this.state.cards.concat(cards2),
          outOfCards: true
})

if (nextProps.cards !== this.props.cards) {

      if (this.cardAnimation) {
        this.cardAnimation.stop();
        this.cardAnimation = null;
      }

      currentIndex[this.guid] = 0; //Reset to first?
      this.setState({
        cards: [].concat(nextProps.cards),
        card: nextProps.cards[0]
      });
    }

After concat I get shown the first card of cards1 array. How to preserve the current card and aggregate cards?

the lib concats the new cards itself.
currentIndex[this.guid] = 0; //Reset to first?       this.setState({         cards: nextProps.cards,         card: nextProps.cards[0]       });
should be working


Is it possible to get SSO to O365 when using Azure AD Domain Services?

We are using Azure AD DS to provide managed domain services to remote desktop servers deployed in Azure.  These RDS servers are running Office 365 apps.  Users can use same credentials to login to Office 365 apps but there is no single sign on.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8a91a6cd-5871-d2a0-4da9-19b611c353de
Version Independent ID: 0f38ffd2-d32c-77af-ca06-a32899ce9fbb
Content: Common deployment scenarios for Azure AD Domain Services
Content Source: articles/active-directory-domain-services/scenarios.md
Service: active-directory
Sub-service: domain-services
GitHub Login: @iainfoulds
Microsoft Alias: iainfou

@chrisjhahn Thank you for your query. We will check and update on the same.

@chrisjhahn, Users wont be able to experience the SSO in this case since Azure AD Domain Services like On-Prem AD, only uses the legacy auth protocols like Kerberos or NTLM, where as these Apps like O365 apps, uses web protocols for auth. These apps uses something called PRT (Primary Refresh token) to provide the SSO experience to the users.
Inorder to get a PRT issued to the device, the device can be Hybrid  Azure AD joined (since these machines are already joined to the managed domain in Azure AD Domain Services).
You can read more about Hybrid AAD join here: https://docs.microsoft.com/bs-cyrl-ba/azure/active-directory/devices/hybrid-azuread-join-managed-domains
Hope this helps.

I've seen that article, unfortunately it doesnt help as there is no Azure AD Connect. Both hybrid domain join and SSO are typically done with Azure AD connect.  In an Azure AD Domain Services domain the accounts are synced automatically so there is no Azure AD Connect.
How is hybrid domain join and or SSO accomplished when using Azure AD Domain services where there is no Azure AD Connect?

@chrisjhahn, I would like to take back my recommendation of using Hybrid AAD join here. Hybrid AAD join is not a good idea to be honest. But, in order to experience SSO, you can get you machines Azure AD registered. In simple terms, since in your current scenario, the RDS server is domain joined to Azure ADDS, so the user can use the Work or School account and get the machine Azure AD registered.
You can click on the connect button as you see in the screen shot and once prompted, use your work or school account to get the device Azure AD Registered.

If the RDS server would have been in the WORKGROUP state, then you could have also used Azure AD join method, but in your case, the suitable option is to go ahead and get the device Azure AD registered. After getting the device Azue AD registered, if you access any O365 Apps, you should experience SSO.

@chrisjhahn, I wanted to touch base with you to check if there are any more queries around this response. If not they we would like to close this thread out.
Do let us know your thoughts.

@chrisjhahn, We are closing this thread for now. Please feel free to reopen this thread in case further help is needed.


Occasional missing data even though heatmap cell indicates anomalies

Occasionally, a heatmap cell summary will indicate an anomaly present, but when clicked on, shows 0 available anomalies.
The anomaly summaries and the anomaly data are fetched in 2 different calls, so likely the issue has to do with the time bounds being different between the two queries. If an anomaly is on the edge, it may be getting included in the summary, but not included in the raw results, leading to the discrepancy.
Screenshot of the error:

Opensearch version <IP_ADDRESS>
I realized this odd behavior too, but this is not occasionally, in our case it does not seem to work at all.
The live dashboard shows live anomalies. These live anomalies are not present in the anomaly overview of  the heat map or vice versa. If you click on an anomaly entity (red square) in the heatmap it should show at least the anomaly occurence in the table, but shows 0 as described above, and I think it should also show anomaly occurences in the anomaly grade/confidence graph, but instead it displays Click on an anomaly entity to view data.
In  OpenSearch version <IP_ADDRESS>, the entity name appeared in the anomaly, confidence graph. and it showed the anomaly occurences as i can remember, but it never showed a confidence/anomaly graph. This is only the case if we explicitly do not use a Categorical field but instead a data filter for exaclty one entity. Then we can see a confidence graph and anomaly detection.

@prpaluch hello, thanks for bringing this issue up. Regarding the differences between <IP_ADDRESS> and <IP_ADDRESS>: there was some changes in <IP_ADDRESS> which reformatted the layout of the charts shown below the heatmap, but should not have affected their functionality regarding showing populated anomalies or not. Additionally, both versions show the same content (anomaly results table, anomaly results chart, and feature breakdown).
If you are seeing "Click on an anomaly entity to view data", then that means the chart isn't recognizing a heatmap cell has been selected. If it's selected, but there is no results, then you should see the error like in the screenshot above ("There are no anomalies currently."). If you are consistently seeing the former when trying to click on a heatmap cell, then that may be because of cluster load (taking a long time to update the charts), or some separate issue. If so, can you open a separate issue describing how to reproduce that bug? I would be happy to help assist and deep dive that problem, thanks.

Hi @ohltyler, is there an option to debug this behavior or maybe find out the cluster load, or that the cluster is overladed? The detector runs on several indices with a wildcard  * option. Each index could grow to 1-2 tb on data. The data for a specific index can delay 1-2 days, means data can be written to an index that is 2 days old.  Currently the whole configuration runs as a POC to see how anomaly detection performs with our data situation and runs on 3 nodes. with  a total of 264 Gb of Memory.  So currently nothing happens if we click on a heat map square except that all other cells are greyed out, the chart stays the same also after 1 Minute waiting.

One note to say, over GET _plugins/_anomaly_detection/stats and POST _plugins/_anomaly_detection/detectors/results/_search for each detector, data seems to be available.

@prpaluch thanks for those details. When a cell is selected, the plugin makes a call to fetch the individual anomaly results for the detector filtered on the cell's time range. My assumption is that request is timing out due to cluster load, or there is a bug in displaying the results.
To check for errors, can you try selecting a heatmap cell and waiting a few minutes, while monitoring the console output using Chrome's dev tools (or another browser's version of it)? That should give you more information on if the request is timing out, or any errors occurring.
You can check real-time cluster-level stats using the node stats API (GET /_nodes/stats) to see if there is high pressure on the nodes / if there is failures. It sounds like the anomaly detectors themselves are working, just an issue in displaying on the webpage.

Hi @ohltyler
thanks for the suggestions you made to solve the problem, really appreciate your help,
these errors could be found in the chrome dev. console during loading the detector and clicking on a heatmap cell.
I guess this could be the problem we experienced?
The first exception is thrown if i visit the side of  our configured detector:
anomalyDetectionDashboards.plugin.js:1 Uncaught (in promise) Configured indices are not found: [.opendistro-alerting-config]
And next if  I click on a cell, this exception is thrown in the console:
a_nomalyDetectionDashboards.chunk.2.js:1 Uncaught TypeError: entityListAsString.split is not a function
at convertToEntityL_ist (anomalyDetectionDashboards.chunk.2.js:1)
at EventEmitter.handleHeatmapClick (anomalyDetectionDashboards.chunk.2.js:1)
at EventEmitter.emit (anomalyDetectionDashboards.chunk.1.js:15)
at HTMLDivElement.plotObj.emit (anomalyDetectionDashboards.chunk.1.js:63)
at emitClick (anomalyDetectionDashboards.chunk.1.js:63)
at Object.click (anomalyDetectionDashboards.chunk.1.js:63)
at Object.clickFn (anomalyDetectionDashboards.chunk.1.js:63)
at HTMLDocument.onDone (anomalyDetectionDashboards.chunk.1.js:63)
Here the two screenshots of the console

@prpaluch great, this is really helpful. The first exception (missing alerting config) is okay and will not interrupt the plugin from working. The second exception, when clicking on a cell, seems to be where the issue is. Might be due to an NPE on entityListAsString, which is possibly behind the other errors regarding negative value widths. I'll investigate more on the root cause and see what I can find.

Looks to be an NPE happening on this line, which means selectedEntityString (declared on this line is returning null or undefined.
If the heatmap chart is populated with anomalies and if there are values on the y-axis, those should be persisted in the chart data, where points[0].y (the entity list) shouldn't be able to be empty. Can you confirm what the value on the y-axis is showing? Additionally, can you share what the values are when hovering your mouse over one of the cells you are trying to select?
Apologies for all of the requested info. I'm unable to reproduce this myself and am trying to figure out what data could lead to this error being thrown.
To help unblock you as well, you may look into upgrading to <IP_ADDRESS>. A lot of the logic around the heatmap cells has been refactored, and the function that is causing the NPE here has actually been removed. Regardless, this is helpful to try to fix this bug in order to provide a patch for users using <IP_ADDRESS> (and potentially <IP_ADDRESS>). Thanks!

hi sorry for the late response, yes the values are available on the y axis, the entity is shown that we selected for the category field.
if you hover with the mouse over a square it shows the entity name, time, anomaly grade and occurrence. But what we can see is, that the square lengths seemed to be not scaled in an appropriate manner, means the hight of the cells is somehow randomly choosen and not equal. I attach two screenshots, first is for 7 days time range, and the last is for 24 hours. The one for 7 days show entity 288. On the y axis you can see entities that are overlapped, maybe no anomalies detected, this looks not okay seems to be a display bug somehow, and the second picture shows last 24 hours. Here the entities are totally mixed up within one cell. Entity 288 is completely not available and it shows anomalies for entity 101 that it did not show for the last 7 days. Could it simply be that the x,y positions are not correct for a cell and therefore it cannot find the selectedEntityString because it cannot display it in proper way in terms of scalable cells.? Does this make sense? Why are the squares different in height, this does not look correct to me? I will try out version 1.2 if available.
Thanks four your help.
![hover](https://user-images.githubusercontent.com/94357865/145002581-3f13c7b7-44b3-49bd-97a7-8fb4c568b5f1.png

)

Checked version 1.2.0 and our problems are gone, it seems that the refactoring in version 1.2.0 helped a lot to make the heatmap work. Now it is possible to click on a square and you get the anomalies and also the confidence graph as it should.
One  problem that still exists in version 1.2.0 is the overlapping y axis and the heat cells that are not scaled correctly. But currently we are happy to view anomalies by clicking on the cells.

Thanks for the detailed information! I'm glad that 1.2 is working better for you, but sounds like there is still issues with the y-axis scaling. Your assumption about the overlapping y-axes causing an empty selectedEntityList in 1.1 are probably related - I'm assuming that the aggregated data within the cell is probably corrupt because of this, and leads to the errors (which isn't handled properly in 1.1).
If you don't mind, could you provide some more details on the index and detector configuration? I'd like to try to reproduce locally to root cause the bug. Some useful info would be:

the index fields and mappings
detector feature configuration
detector category field configuration

hi yes of course. We are analysing ha-proxy logs. Here is the index template for the log data. It contains the settings and mappings. You can set it up via dev tools.
POST /_template/haproxy_example
{
"order" : 0,
"index_patterns" : [
"haproxyus*"
],
"settings" : {
"index" : {
"number_of_shards" : "6",
"number_of_replicas" : "1"
}
},
"mappings" : {
"properties" : {
"server_name" : {
"type" : "keyword"
},
"srvconn" : {
"type" : "integer"
},
"actconn" : {
"type" : "integer"
},
"Ta" : {
"type" : "integer"
},
"Tc" : {
"type" : "integer"
},
"client_port" : {
"type" : "integer"
},
"popId" : {
"type" : "keyword"
},
"http_method" : {
"type" : "keyword"
},
"backend_name" : {
"type" : "keyword"
},
"beconn" : {
"type" : "integer"
},
"hostIpAddress" : {
"type" : "ip"
},
"client_ip" : {
"type" : "ip"
},
"Tr" : {
"type" : "integer"
},
"bytes_uploaded" : {
"type" : "long"
},
"frontend_ip" : {
"type" : "ip"
},
"Tw" : {
"type" : "integer"
},
"http_uri" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"http_status_code" : {
"type" : "integer"
},
"termination_state" : {
"type" : "text"
},
"feconn" : {
"type" : "integer"
},
"srv_queue" : {
"type" : "integer"
},
"http_version" : {
"type" : "keyword"
},
"bytes_read" : {
"type" : "long"
},
"data_field" : {
"type" : "keyword"
},
"frontend_port" : {
"type" : "integer"
},
"backend_queue" : {
"type" : "integer"
},
"retries" : {
"type" : "integer"
},
"frontend_name" : {
"type" : "keyword"
},
"TR" : {
"type" : "integer"
}
}
},
"aliases" : { }
}
And here a snippet how the data looks like, all xxxxxxx are string values
"hits" : [
{
"_index" : "haproxyus-logging-x",
"_type" : "_doc",
"_id" : "_lcMp30BxL-IX7Ry8qBn",
"_score" : 1.0,
"_source" : {
"Ta" :<PHONE_NUMBER>,
"frontend_name" : "xxxxxxxxxxx",
"Tt" :<PHONE_NUMBER>,
"popId" : "123",
"Tw" : 0,
"termination_state" : "----",
"@version" : "1",
"backend_name" : "xxxxxxxxxxx",
"server_name" : "xxxxxxxxx",
"client_port" : 65141,
"bytes_read" : 7411465,
"TR" : 0,
"hostIpAddress" : "xxxxxxxxxx",
"frontend_port" : "8080",
"backend_queue" : 0,
"Tc" : 0,
"feconn" : 123615,
"Th" : 0,
"Tr" : 3,
"Ti" : 7,
"http_version" : "HTTP/1.1",
"host" : "xxxxxxxxxxxxx",
"retries" : 0,
"srv_queue" : 0,
"srvconn" : 2452,
"actconn" : 146466,
"http_uri" : "xxxxxxxxxxx",
"@timestamp" : "2021-11-23T04:24:24.554Z",
"client_ip" : "xxx.xxx.xxx.xxx",
"beconn" : 140468,
"bytes_uploaded" : 4708527,
"Td" :<PHONE_NUMBER>,
"Tq" : 7,
"http_status_code" : 200,
"type" : "xxxxxxxxxxx",
"http_method" : "CONNECT",
"frontend_ip" : "xxx.xxx.xxx.xxx"
}
}
]
Here the detector configuation
GET _plugins/_anomaly_detection/detectors/g61xN30Bju0JQ98M1BaN
{
"_id" : "g61xN30Bju0JQ98M1BaN",
"_version" : 1,
"_primary_term" : 1,
"_seq_no" : 0,
"anomaly_detector" : {
"name" : "ha_proxy_rate_detector",
"description" : "check the index rate for pops",
"time_field" : "@timestamp",
"indices" : [
"haproxyus-logging-*"
],
"filter_query" : {
"match_all" : {
"boost" : 1.0
}
},
"detection_interval" : {
"period" : {
"interval" : 6,
"unit" : "Minutes"
}
},
"window_delay" : {
"period" : {
"interval" : 0,
"unit" : "Minutes"
}
},
"shingle_size" : 8,
"schema_version" : 0,
"feature_attributes" : [
{
"feature_id" : "gq1xN30Bju0JQ98MthYl",
"feature_name" : "count_documents",
"feature_enabled" : true,
"aggregation_query" : {
"count_documents" : {
"value_count" : {
"field" : "http_status_code"
}
}
}
}
],
"ui_metadata" : {
"features" : {
"count_documents" : {
"aggregationBy" : "value_count",
"aggregationOf" : "http_status_code",
"featureType" : "simple_aggs"
}
},
"filters" : [ ]
},
"last_update_time" :<PHONE_NUMBER>635,
"category_field" : [
"popId"
],
"user" : {
"name" : "admin",
"backend_roles" : [
"admin"
],
"roles" : [
"own_index",
"all_access"
],
"custom_attribute_names" : [ ],
"user_requested_tenant" : "user"
},
"detector_type" : "MULTI_ENTITY"
}
}
If you need more information, i am glad to help you.

@prpaluch I'm able to reproduce the issue on 1.2 after setting up a similar local environment (similar indices, detector config). Will respond back once I can dive deeper into the root cause.

Found out Plotly will automatically try to determine the data type based on the axis data given. In this case, it looks like it is labeling it as "linear" by default. I've tested a change of specifying the "type" as "category", and it now looks to show properly:

I'll work on a patch for this and will update this issue with the latest progress.

Heatmap chart axes fixed as part of #167. Will still leave this issue open for tracking the root cause of occasional empty/missing data in the cells.


move {{property.propertyName}} from Class.liquid to a separate file

I have already described the sentence here, but it seems I have not been heard
I propose to move the generation of the property from the  Class.liquid. This will allow you to customize the name of the generated properties by overriding the file without having to override the entire Class.liquid.

How would you do this snake case to camel case conversion in liquid only?

{% assign words = 'my_test_snake_case_string' | split : '_' %}
{% assign camelCase = words | first %}
{% for word in words offset : 1 %}
	{% assign wordCap = word | capitalize %}
  {% assign camelCase = camelCase | concat : wordCap  %}
{% endfor %}

{{ camelCase | join : '' }}

https://jumpseller.com/support/liquid-sandbox/

I think for this we need to answer this question:
https://gitter.im/dotliquid/dotliquid?at=602b9ec847585464db800da9

Hey everyone, I've a question regarding "MakeThreadSafe()": According to the docs "you can't use template owned variables anymore" - my question would be: Can i still use localVariables?

            return template.Render(new RenderParameters(CultureInfo.InvariantCulture)
            {
                LocalVariables = hash,
                Filters = new[] { typeof(LiquidFilters) },
                ErrorsOutputMode = ErrorsOutputMode.Rethrow
            }).Replace("\r", "").Trim();

in my unit tests i see that this does not work with MakeThreadSafe()
To me my usage looks completely stateless within the render method, i.e. all state is passed with RenderParameters, Render() can maintain a state per call (thread-safe) and then return the result.

This fix here could lead to a big performance hit if we load the template for every property name:
https://github.com/RicoSuter/NJsonSchema/commit/81d79c47265e45ca3b0138d142e81342b27872f3


Database configuration for running Spring.Data tests?

What type of database and configuration are expected when Spring.Data tests?

You can check how AppVeyor build sets the database, the scripts can be found here: https://github.com/spring-projects/spring-net/tree/master/build-support . The files with appveyor- prefix.
You need to create db with schema and users and then create alias that is recognized.
I think this is a bit too complicated as it tries to overcome problems with different SQL Server setups that nowadays could be circumvented by using a Dockerized version running on specific port. This setup could be shared via Docker Compose for easier setup.


Master flat creation is requesting master bias even when --ignore-bias is active

Master bias is not ignored when creating master flats.
The keyword --ignore-bias should allow the pipeline to run on files when no bias is available to construct a master bias therefore applying a bias correction, but that's not the case for the method that builds the master flats for Spectroscopy and for Imaging.

This will not be fixed by #291 because the function is being relocated by #287 and it will be fixed most likely inside #287


python310Packages.peaqevcore: 0.0.16 -> 0.0.19

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.peaqevcore is: "Library for interacting with Peaqev car charging"
meta.homepage for python310Packages.peaqevcore is: "https://github.com/elden1337/peaqev-core"
meta.changelog for python310Packages.peaqevcore is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
Warning: a test defined in passthru.tests did not pass
found 0.0.19 with grep in /nix/store/bbwwbqqnjx5nf4nqrcjjpgh9cbjvgalr-python3.10-peaqevcore-0.0.19
found 0.0.19 in filename of file in /nix/store/bbwwbqqnjx5nf4nqrcjjpgh9cbjvgalr-python3.10-peaqevcore-0.0.19

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

2 package rebuild(s)

2 x86_64-linux rebuild(s)
2 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
python310Packages.peaqevcore
python39Packages.peaqevcore

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/bbwwbqqnjx5nf4nqrcjjpgh9cbjvgalr-python3.10-peaqevcore-0.0.19 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.peaqevcore https://github.com/r-ryantm/nixpkgs/archive/3f907642422cb19c256287d2dbe43e9c2ba34763.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/bbwwbqqnjx5nf4nqrcjjpgh9cbjvgalr-python3.10-peaqevcore-0.0.19
ls -la /nix/store/bbwwbqqnjx5nf4nqrcjjpgh9cbjvgalr-python3.10-peaqevcore-0.0.19/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

python310Packages.peaqevcore
python39Packages.peaqevcore

Maintainer pings
cc @fabaff for testing.

Result of nixpkgs-review pr 170718 run on x86_64-linux 1

2 packages built:

python310Packages.peaqevcore
python39Packages.peaqevcore


Syntax error when running BUILD2.BAT

When I run the BUILD2.BAT file I get the error
C:\Users\James\Downloads\CEVidium-master\CEVidium-master\CEVENCODER>BUILD2.BAT -h
  File "C:\Users\James\Downloads\CEVidium-master\CEVidium-master\CEVENCODER\tools\toolkit2.py", line 1
    print "Loading libraries..."
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?

Hello,
Add me on Discord and I can help you from there.
jlaw#6710

I figured it out, thanks though!

@Linux-Error

I figured it out, thanks though!

should tell people what the problem was, and close the issue. (GitHub etiquette)
in this case it was you using python 3.x instead of python 2.7.x (but you already knew that)
@cheesemagic good to see you offering help to others after i helped you :)

@Linux-Error

I figured it out, thanks though!

should tell people what the problem was, and close the issue. (GitHub etiquette) in this case it was you using python 3.x instead of python 2.7.x (but you already knew that)
@cheesemagic good to see you offering help to others after i helped you :)

why are you using python 2.7 in the first place, pip doesn't even go back to python 2.7 so installing packages does not work, which means I can't install PIP for python2.7

Python 2.7 seems to be the only version that works for CEVidium. You can install packages still. I don't remember how because I did it last year, but I do know that it works.

@FloweyProgramer
Pip does work I believe, that's how I installed packages needed. this project is just old that's why python 2.7 is used and not python 3.x. no-idea behind the motivation beyond that.

ah yeah forgot about that, when I was last using it pip still worked (somewhere last year indeed). in that case you will need to install dependencies manually. don't know how to do that out of the top of my head but I'm sure the internet has solutions for that.

Because the python code was written for python 2, its not compatable with python 3. Use Python 2 to 3 converter online to convert the python code in the tools folder into python 3 code. :)


Overbroad Except Clauses

To start with this is my first time looking at the code, so take this as the first impression that it is. I found the except: clause a code smell. It reminds me of this anti-pattern Overbroad Except Clauses Can we be more specific on what we are catching? When we are using it to catch an assert, would an if be clearer?
As always just starting a discussion, and willing to help implement any fix.

Yes, this sounds like a good idea!
There are probably cases here and there where try/except is better, like if we're testing whether an Orca object can be generated without errors, but mostly we can get rid of it.
I may have had a similar thought at some point, because it looks like the injectable assertions near the end of orca_test.py mostly use if statements.
If you feel like implementing this, go right ahead!

I'll second that motion.  If you have the time to do this and issue a pull request, Jacob, run with it!

Do you have advice for the TODO's?

Whoops, i replied to that in the PR, which is now closed.
No, i'm not sure how we should handle the backtraces. One of the things i haven't gotten around to is adding Python 3 cross-compatibility, which might give us better options. (I'll open a new issue for that.)


Custom localization

https://github.com/HerbertHe/for-editor-herb/blob/9f6a6d8c28f2356fdccfa0a01fffe10c159c66b2/src/index.tsx#L212
If the language is not a string but an object with the translations inside it that will allow us to use other languages and not only only the 4 included.

I am preparing to do this


Enable bootstrapping statefulsets

What this PR does:
This PR enables bootstrapping StatefulSets. The controller:

Fetches the EtcdCluster resource.
Checks for an existing StatefulSet:

If missing and Spec.Size > 0, creates a StatefulSet with replicas = 0.

Sets ownership using SetControllerReference for proper garbage collection.

It follows the workflow mentioned here.

Thanks @abdurrehman107 for the PR. Based on our discussion in last WG meeting, @Gaurav Dasson is supposed to deliver a PR for this. I am not sure what's his progress so far. Let me ping him in slack.
Also I'd like to see a complete PR to transfer all the implementation of syncHandler (around 300 -400 lines of code).

@abdurrehman107 since there is no response from Gaurav Dasson, please feel free to continue to work on this. Could you deliver a complete PR to transfer all the implementation of syncHandler (around 300 -400 lines of code)?

The goal is to implement the initial Reconcile workflow based on syncHandler .  Please ensure that you verify it in a K8s cluster. thx

Could you deliver a complete PR to transfer all the implementation of syncHandler (around 300 -400 lines of code)?

Yes I can. I'll ask for help in the wg slack channel.

Please ensure that you verify it in a K8s cluster. thx

Aighty.

@abdurrehman107 , @ahrtr : Sorry, I had to be away from work as I had some viral symptoms. I'll start looking at this work from today onwards. Thanks.

I hope you're well @gdasson . I have made a lil progress. I dont mind getting on a call later today or tomorrow briefing you about it. Please lemme know over slack of when we can schedule a syncup.

Thanks both @gdasson and @abdurrehman107  for the feedback. We definitely need more contributors. But in the very first step, I'd like to we have a skeleton implementation for the Reconcile as soon as possible, so as to unblock & breakdown the following tasks.
@abdurrehman107 Since you have already started working on this for 1+ weeks, please continue to work on it. Are you able to deliver a PR per https://github.com/etcd-io/etcd-operator/pull/12#issuecomment-2553666892 before our next working group meeting (Jan. 7th)?

Yes we can have the sync handler function implementation by 7th Jan.

@abdurrehman107 @ahrtr : I worked on this over this weekend and I have now moved most of the logic from syncHandler under the Reconcile loop here. I have kept my code in a forked branch. Please review here:
https://github.com/etcd-io/etcd-operator/compare/main...gdasson:etcd-operator:main
If this looks fine for a starting point/skeleton, we can then start to refine it further. If you guys feel fine after the review, I can either add this in Abdur's PR or open a new one.

Please review here:
main...gdasson:etcd-operator:main

Thanks @gdasson. It is almost identical to the existing implementation, and it's exactly what I want. I just expect a base/foundation asap for anyone to continue to work on it.  So please raise a PR, thanks.
As a new team, we need to work together more to get used to each other and avoid overlapping tasks. I’m hopeful that our teamwork will be much smoother going forward.

Hey @gdasson, thank you for the work. Umm I was completely unaware that you were working on this. I've been working at defining the skeleton, and I was almost done. @ahrtr do I drop on it and work on this scaffold @gdasson has pushed?


Column-Pivoted QR Decomposition

Please:

[x] Check for duplicate requests.
[x] Describe your goal, and if possible provide a code snippet with a motivating example.

Hi, I received the following error when executing jax.scipy.linalg.qr(A, pivoting=True),

NotImplementedError: The pivoting=True case of qr is not implemented.

I'm curious if there is a plan to implement Column-Pivoted QR Decomposition in the near future? Thanks in advance for the help!

Hi - thanks for the question. Currently JAX implements QR decomposition via geqrf, which doesn't support pivoting. For that, we would need geqp3 (which scipy uses for pivoting=True)
Adding new lapack routines is somewhat involved, because JAX needs to be able to target appropriate implementations on all relevant backends (i.e. CPU, GPU, etc.), and it appears that cuBlas (where JAX gets its geqrf implementation for nvidia GPU) does not currently have any implementation of geqp3 (see https://docs.nvidia.com/cuda/cublas/index.html)
With that in mind, I think it's unlikely that pivoting=True will be supported in the near future, but I'll leave this issue open to track the request.

Understand; thanks for the explanation!

What hardware platform?
If you need it only on CPU it wouldn't be a huge lift to add it. Even on GPU it's possible (MAGMA has magma_?geqp3) we could support it. I'm not sure we'll have time to do it but maybe. And contributions are welcome.

Personally, it'd be great to have it on GPU (and CPU if that's easy, but GPU alone would be fine). Would this be easy?


LIP-4 72h Aragon votes

The proposal to increase Aragon voting time from the current 24h to 72h.

@vshvsh incorporated the feedback


Do not format with fewer fields

Format does not take effect if there are few fields or conditions
less case：

many case：

Probably not related to number of fields:
CREATE TEMP TABLE T (long_field_name_ABC INT, long_field_name_XYZ INT);

unchanged when formatted.
CREATE TEMP TABLE T (long_field_name_ABCD INT, long_field_name_XYZ INT);

becomes
CREATE TEMP TABLE T (
    long_field_name_ABCD INT,
    long_field_name_XYZ INT
);


Test: editor error placeholders

Refs: https://github.com/microsoft/vscode/issues/142875, https://github.com/microsoft/vscode/issues/130971

[ ] anyOS @TylerLeonhardt
[x] anyOS @chrmarti

Complexity: 2
Create Issue

The editor placeholder in case of error was updated and also shows in more cases now. The easiest is to create a HTML file with a link to a file that does not exist and click it to open as editor.

verify you still get asked to create the file from a modal dialog
if you don't create a file, the editor place holder properly conveys the error and offers the same action as the dialog
if you create the missing file in this flow, the editor place holder goes away

Alternatively you can:

open a file
close VSCode
delete the file
start VSCode
the editor restores but with the error placeholder since the file was deleted on disk

Works well!


DO NOT MERGE Connect automatically

So I added the autoconnect back, however:

In napalm_connection we will try to get napalm_options and pass them as optional_args (we can do the same for the rest.
Instead of having connections work with either a task or a host I call the connection task from the host. For that I needed to have access to the parent brigade object from the host so tasks that are configurable could read it from there. For instance: https://github.com/napalm-automation/brigade/blob/develop/brigade/plugins/tasks/connections/paramiko_connection.py#L18
I also added available_connections so the connections that are available for the tasks can be customized

Aside from my small comment above I think it looks good.
The available_connections will need some documentation. So perhaps a few lines about that in the class.

Yeah, there are a few things that need documentation here. I was mostly waiting for agreement before doing that :P

:) I heard that Documentation Driven Development is the new black.

I didn’t say it would be a pretty way to remove connections :) Just meant that it wouldn’t be hard.
My main point is that I think my scenario is more probable to happen than the one you describe.
It could be solved with a key in Config like external_connections_policy which takes “merge” or “replace”.
Don’t know if this just makes something simple overly complex.

On 15 Dec 2017, at 16:54, David Barroso<EMAIL_ADDRESS>wrote:
@dbarrosop commented on this pull request.
In brigade/core/init.py:

@@ -68,6 +70,10 @@ def init(self, inventory, dry_run,
level=logging.ERROR,
format=format,
)

   if available_connections is not None:

       self.available_connections = available_connections

   else:

       self.available_connections = connections.available_connections

And I agree this is going to be strange (specially disabling connections), if anything the second case and I think it's equally easy to accommodate with both yours and my solution while mine makes less assumptions of how things should work :P
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

It could be solved with a key in Config like external_connections_policy which takes “merge” or “replace”.

I'd rather not have keys to change the behavior of things if we can help it. As your proposal and mine result in almost the same convenience for the user I'd rather stick with something that is generic and does't make assumptions on how users are supposed to use things.
Also, each key we add increases the chances users might get something wrong :P

In any case, no matter what we decide we need the available_connections argument because of how the filter works so I'd suggest not getting stuck with this conversation as what you are suggesting can be an improvement later on.
@ktbyers do you have any opinion about this PR? if you are good with it let me know what you want me to add to netmiko (or push a commit yourself directly to this branch)

+1 works nicely (even if the default behaviour for available_connections is wrong :) )

Everything else looks good to me.

As people seems to agree overall I am going to merge. We can iterate if there is something to be changed.


Properly use DateTimeOffset and TimeSpan

Due to issue: https://github.com/Azure/autorest/issues/825 I had to generate from a local build where I hard coded the value of UseDateTimeOffset to True for these two clients. This update uses DateTimeOffset instead of DateTime and properly uses TimeSpan for timeSpan values.

@begoldsm How are you going to regenerate if there is a hotfix?

Hey @azuresdkci retest this please since it looks like it is stuck.


services/horizon: Add verify-range Batch job manual GitHub action

This PR creates a GitHub action which allows manually running a verify-range AWS Batch job
See https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/ for details on how this is supposed to work.
Addresses part of #2761

It seems we need to merge this into master to test it :(

What's the motivation for having this be a GitHub Action vs being a Jenkins pipeline?

What's the motivation for having this be a GitHub Action vs being a Jenkins pipeline?

Unfamiliarity with Jenkins plus easier access.
Why would we use Jenkins instead?

@Shaptic It seems we are switching to Jenkins, I will remove the github job definition and just leave the script.


default value with set_header

Hi,
I have a 3 column header-less CSV that looks like this:
# comment
a,b,c
c,a,b

1,2,3
red,orange,blue
tall,medium
apple,orange,tomatoes

I am attempting to read it with:
io::CSVReader<3, io::trim_chars<' '>, io::double_quote_escape<',', '\"'>, io::single_and_empty_line_comment<'#'>> csv_reader(filename);

csv_reader.set_header("Col A", "Col B", "Col C");

string col_a, col_b, col_c;

while (csv_reader.read_row(col_a, col_b, col_c)) {
  cout << col_a << " " << col_b << " " << col_c << "\n";
}

I get an exception thrown (io::error::too_few_columns) at the "tall,medium" line as it is missing a column.  Is there a a way to add a default value (such as a simple null or "") for any missing column like it is possible with read_header?  I saw this reported issue, but there wasn't really a solution other than adding a header in the file (I have no control on the file).
An example would be appreciated!

Hi,
there is no way to properly solve this other than fixing the broken
input file.
Consider for example you input file. Which column is missing in the
"tall,medium" line? Is it the first, second, or third? This problem is
further complicated when you consider automatic column reordering and
selecting as csv_reader.read_header provides.
Best Regards
Ben Strasser
On 25.07.2019 16:11, elimpnick wrote:

Hi,
I have a 3 column header-less CSV that looks like this:
comment
a,b,c
c,a,b
1,2,3
red,orange,blue
tall,medium
apple,orange,tomatoes
I am attempting to read it with:
io::CSVReader<3, io::trim_chars<' '>, io::double_quote_escape<',',
'"'>, io::single_and_empty_line_comment<'#'>> csv_reader(filename);
csv_reader.set_header("Col A", "Col B", "Col C");
string col_a, col_b, col_c;
while (csv_reader.read_row(col_a, col_b, col_c)) {
cout << col_a << " " << col_b << " " << col_c << "\n";
}
I get an exception thrown (io::error::too_few_columns) at the
"tall,medium" line as it is missing a column. Is there a a way to add
a default value for any missing column like it is possible with
read_header? I saw this reported issue [1], but there wasn't really a
solution other than adding a heading the file (I have no control on
the file).
An example would be appreciated!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [2], or mute the
thread [3].
Links:
[1] https://github.com/ben-strasser/fast-cpp-csv-parser/issues/18
[2]
https://github.com/ben-strasser/fast-cpp-csv-parser/issues/82?email_source=notifications&email_token=AC3IBD3YPMU2VOG2HCMYQLLQBGYC5A5CNFSM4IG3CR42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HBPXFQQ
[3]
https://github.com/notifications/unsubscribe-auth/AC3IBD6YIRBTZGGRZF4NM3TQBGYC5ANCNFSM4IG3CR4Q

Hi,
The way I see it, in the case of "tall,medium", the 3rd column is missing because the separator is there.  I have it like that in my head:
tall,,medium #missing second column
,tall,medium #missing first column
tall,medium #missing third column
Which would translate to:
tall,NULL,medium
NULL,tall, medium
tall,medium,NULL
Even if you add a header, it won't make the missing data suddenly appear.  To me, having the heading in the file or hardcoded in the source code seems the same.
Cheers!

An example of such as file is this wireshark OUI file, if you CTRL-F for "00:00:13", you will see an example of the first 2 columns being present and the 3rd missing.

Hi,
if you have separators, then you should not get the error that you are
seeing.
The following are clear:
tall,,medium
,tall,medium
tall,medium
The following, which is your original example, is not:
tall,medium
I do not understand your usecase.
Best Regards
Ben Strasser

If you look at the link above and you search for "00:00:13", you will see an example where there are 2 columns available, the 3rd is missing.  If you look in the entire file, there is always 2 columns, and about 95% the 3rd column has data.

I still do not understand your point.
"a,b," is correctly parsed. That is not an issue. "a,b" cannot be
correctly parsed as there is no way to tell which column is missing.

To the OP:
It sounds like you want to grep each line, and add a trailing comma to
lines that contain only 2 commas (or whatever your field separate is). As
Ben has noted, this is outside the purview of the csv parser, but should be
easily to accomplish with standard regex tools...
Best,
Christian
On Thu, Jul 25, 2019 at 12:03 PM elimpnick<EMAIL_ADDRESS>wrote:

If you look at the link above and you search for "00:00:13", you will see
an example where there are 2 columns available, the 3rd is missing. If you
look in the entire file, there is always 2 columns, and about 95% the 3rd
column has data.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ben-strasser/fast-cpp-csv-parser/issues/82?email_source=notifications&email_token=AAEXA5IJNM7TBGPKWZ5C4GLQBHFFNA5CNFSM4IG3CR42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2Z54NI#issuecomment-515104309,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEXA5J37JKPJPAM2VHWLM3QBHFFNANCNFSM4IG3CR4Q
.

--
A man, a plan, a cat, a ham, a yak, a yam, a hat, a canal – Panama!
http://www.x14n.org

Regex it is, thanks.