[SPARK-17346][SQL] Add Kafka source for Structured Streaming

What changes were proposed in this pull request?
This PR adds a new project  external/kafka-0-10-sql for Structured Streaming Kafka source.
It's based on the design doc: https://docs.google.com/document/d/19t2rWe51x7tq2e5AOfrsM9qb8_m7BRuv9fel9i0PqR8/edit?usp=sharing
@tdas did most of work and part of them was inspired by @koeninger's work.
How was this patch tested?
The new unit tests.

/cc @marmbrus

Test build #65401 has started for PR 15102 at commit c4db857.

Test build #65401 has finished for PR 15102 at commit c4db857.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65401/
Test FAILed.

Merged build finished. Test FAILed.

Just from a very brief look, this duplicates egregious amounts of code from the existing Kafka submodule and doesn't handle offset ordering correctly when topicpartitions change.
I'd -1 on this, wholesale, if I had the right to.

Test build #65404 has started for PR 15102 at commit 51d251f.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65404/
Test FAILed.

Test build #65405 has started for PR 15102 at commit d3c0754.

@koeninger Thanks a lot for looking at this PR.

The duplicated codes are mostly because we don't want to depend on Spark DStream in structured streaming source. Anyway, we can create a common project for these codes if that's really a big deal.
This PR doesn't support topicpartitions changes for now. I'm not sure how to implement he strict ordering for Kafka offsets when topicpartitions changes on the fly. The strict ordering is important to StreamingExecution otherwise it may skip some data. Do you have any thoughts? I looked your PR but seems you have not yet resolved this problem?

Test build #65405 has finished for PR 15102 at commit d3c0754.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65405/
Test PASSed.

This already does depend on most of the existing Kafka DStream implementation.  The fact that most of it was copied wholesale proves that.  If you're just saying that you don't want it to have a transitive dependency on the spark-streaming-X module, I can refactor that.  I would much rather take the time to do that work personally now, as opposed to the maintenance problems later.

Users are going to change topicpartitions whether you want them to or not, and this PR ostensibly supports SubscribePattern, so it must handle changing topicpartitions.  Kafka does not have a global order  (Kafka doesn't even have a per-partition order that's guaranteed contiguous, see SPARK-17147, but that's another can of worms).  Given an offset from topicpartition A and an offset from topicpartition B, it is impossible to determine ordering without some other information.  I brought some of these concerns up during the structured streaming design document, and it was handwaved as we'll just make structured offset an interface and try to figure it out later.  I believe my PR does the least bad thing possible given this interface, in that it does allow changing partitions, does not violate hashcode, and does have a stable order (if we really want to deal with hashcode collisions, we can sort and concat all topicpartitions or something).  If you see something else wrong there, by all means let me know.  Yes, if someone publishes data to a topicpartition right before deleting it, the stream may not consume that data, but... really, what did they expect to happen in that case?  We can talk about whether it might be possible to use time indexing in kafka as an alternative ordering, but what's there right now doesn't cut it.

My thoughts at this point:

It's clear from the jira this shouldn't get rushed into 2.0.1, let's do this as right as possible given the circumstances.
How can we collaborate on a shared branch?  You guys manually copying stuff from my fork doesn't make any sense.
@marmbrus Can you give some specific technical direction as to how users can communicate the type for key and value, without having to map over the stream as is done in this PR?

This already does depend on most of the existing Kafka DStream implementation....

I pushed for this code to be copied rather than refactored because I think this is the right direction long term.  While it is nice to minimize inter-project dependencies, that is not really the motivation.  While the code is very similar now, there a bunch of things I'd like to start changing:

I don't think that all the classes need to be type parameterized.  Our interface  SQL has its own type system, analyser, and interface to the type system of the JVM (encoders).  We should be using that.  Operators in SQL do not type parameterize in general.
To optimize performance, there are several tricks we might want to play eventually (maybe prefetching data during execution, etc).

These are just ideas, but given that DStreams and Structured Streaming have significantly different models and user interfaces, I don't think that we want to tie ourselves to the same internals.  If we identify utilities that are needed by both, then we should pull those out and share them.

Users are going to change topicpartitions whether you want them to or not...

I think we need to take a step back here and look at what is actually required by the Offset interface.  We don't need to handle the general problem of is kafka Offset A from Topic 1 before or after kafka Offset B from Topic 2.  We'll only ever look at full KafkaOffsets that represent slices in time across a couple of topicpartitions and were returned by the Source (possibly with other annotations the Source wants to add).  The only questions that the the StreamExecution will ask today are:

Does x: KafkaOffset == y: KafkaOffset (i.e. is there new data since the last time I checked)?
Give me the data between KafkaOffset x and KafkaOffset y for all included topicpartitions.

The final version of this Source should almost certainly support wildcards with topicpartitions that change on the fly.  Since it seems this is one of the harder problems to solve.  As a strawman, I'd propose that we only support static lists of topics in this PR and possibly even static partitions.  I want to get users to kick the tires on structured streaming in general and report whats missing so we can all prioritize our engineering effort.

It's clear from the jira this shouldn't get rushed into 2.0.1, let's do this as right as possible given the circumstances.

Agreed, if this doesn't make 2.0.1 thats fine with me.

How can we collaborate on a shared branch? You guys manually copying stuff from my fork doesn't make any sense.

I typically open PRs against the PR author's branch when I want to collaborate more directly.

Can you give some specific technical direction as to how users can communicate the type for key and value, without having to map over the stream as is done in this PR?

I'd like this to work the same as other calculations in SQL, using column expressions.
df.withColumn("value", $"value".cast("string"))

I'd also like to create expressions that work well with kafka specific deserializers, as well as integrations with our json parsing and maybe Avro too.  The nice thing about this path is it fits with other SQL APIs and should play nicely with Dataset Encoders, UDFs, our expression library, etc.
Does that seem reasonable?  Is this missing important cases?

I pushed for this code to be copied rather than refactored because I think this is the right direction long term. While it is nice to minimize inter-project dependencies, that is not really the motivation. While the code is very similar now, there a bunch of things I'd like to start changing:

A bigger concern for me at this point is that the code was copied, and then modified in ways that don't seem to have anything to do with the necessities of structured streaming (e.g. your "why is this a nested class" comment).
Options from my point of view, from best to worst

Refactor to a common submodule.  Based on how little I had to change to get the common functionality in my branch, I think this is going to initially leave most things untouched.  If things change in the future, they can be refactored / copied as necessary.  I think this minimizes the chance that someone fixes a bug in Dstream cached consumer, and forgets to fix in sql cached consumer, or vice versa.

Copy without changes, make only minimal changes necessary at first.  This is going to make what happened more obvious, and make it easier to maintain changes across both pieces of code

Copy and make unnecessary changes (what seems to have been done currently).  This seems like a maintenance nightmare for no gain.

I don't think that all the classes need to be type parameterized. Our interface SQL has its own type system, analyser, and interface to the type system of the JVM (encoders). We should be using that. Operators in SQL do not type parameterize in general.
To optimize performance, there are several tricks we might want to play eventually (maybe prefetching data during execution, etc).

Kafka consumers prefetch data already, that's the main reason the CachedKafkaConsumer exists.  My thought here is that there isn't much gain to be had with something more than a thin shim around a Kafka rdd, or at least not for a while.  Kafka's data model doesn't really allow for much in terms of pushdown optimizations (you basically get to query by offset, or maybe time).  About the only idea I've heard that might have promise was Reynold suggesting scheduling straight map jobs as long-running kafka consumers in a poll loop on the executors, to avoid batching latency.  But that seems to open a whole can of worms in terms of deterministic behavior, and is probably much further down the road.  If we get there, what's the harm in cutting shared dependencies at that point rather than now?

These are just ideas, but given that DStreams and Structured Streaming have significantly different models and user interfaces, I don't think that we want to tie ourselves to the same internals. If we identify utilities that are needed by both, then we should pull those out and share them.

At this point, the shared need is basically everything except KafkaUtils' static constructors, and the parts of the DirectStream related to the DStream interface.  You still need an rdd, a cache for consumers, offset ranges, a way to configure consumers, a way to configure locality, a consumer running on the driver to get latest offsets...

We don't need to handle the general problem of is kafka Offset A from Topic 1 before or after kafka Offset B from Topic 2.
Does x: KafkaOffset == y: KafkaOffset (i.e. is there new data since the last time I checked)?

We do need to handle it comparing completely different topicpartitions, because it's entirely possible to have a job with a single topicpartition A, which is deleted or unsubscribed, and then single topicpartition B is added, in the space of one batch.  I have talked to companies that are actually doing this kind of thing.  If all we need to do is be able to tell that one sql offset (that we already knew about) is different from another sql offset (that we just learned about), then I think it's pretty straightforward - your three cases are

error.  ahead in some common topicpartitions, and behind in others.
equal.  same kafka offsets for same topicpartitions
not equal.  different offsets for same topicpartitions, and/or different topicpartitions

That does imply that any ordering of sql Offsets is by when we learn about them in processing time, which sounds suspect, but...

The final version of this Source should almost certainly support wildcards with topicpartitions that change on the fly. Since it seems this is one of the harder problems to solve, as a strawman, I'd propose that we only support static lists of topics in this PR and possibly even static partitions. I want to get users to kick the tires on structured streaming in general and report whats missing so we can all prioritize our engineering effort.

The problem I see with that approach is that doing it right may require changing the structured streaming interface, which gets progressively harder to do the more users and more implementations there are relying on it.  I think it's generally best to learn about your items of highest risk first (even if you don't totally solve them in version 1.0).

I typically open PRs against the PR author's branch when I want to collaborate more directly.

Sorry, I meant this less in terms of github mechanics and more in terms of process - are you going to consider pull requests against this branch, what pieces make sense to split off, etc.  For a specific initial question, is your mind made up about the submodule thing, or if I do that refactor will you at least look at it?

df.withColumn("value", $"value".cast("string"))
I'd also like to create expressions that work well with kafka specific deserializers, as well as integrations with our json parsing and maybe Avro too. The nice thing about this path is it fits with other SQL APIs and should play nicely with Dataset Encoders, UDFs, our expression library, etc.
Does that seem reasonable? Is this missing important cases?

Yeah, that seems reasonable.  Yeah, the builtin types, json, and maybe avro are the ones to hit.  I'd be 100% on board if we had one working example of a specific deserializer, that didn't require access to private spark classes, so people could tell how to do it for whatever serialization scheme they are using.
My guess is someone's going to want an implicit api to let them just call .stringKeyValue or whatever on the dataframe because all they care about is those two columns, but that's really straightforward for people to do themselves.

We do need to handle it comparing completely different topicpartitions, because it's entirely possible to have a job with a single topicpartition A, which is deleted or unsubscribed, and then single topicpartition B is added, in the space of one batch.

Yeah, it's also possible that when a batch is running, a single topicpartition C is created then deleted. But the Kafka source doesn't even know C. The root issue is currently the approach is polling metadata, and any metadata changes between two pollings are missing.

I'm not concerned about people deleting partitions before messages have
been processed, because they can take care of that problem themselves, by
not deleting things until consuming has caught up. This is fairly intuitive
IMHO.
I am concerned about shipping an implementation that can't handle partition
changes at all, because that's a big step backwards.
On Sep 18, 2016 11:41 PM, "Shixiong Zhu"<EMAIL_ADDRESS>wrote:

We do need to handle it comparing completely different topicpartitions,
because it's entirely possible to have a job with a single topicpartition
A, which is deleted or unsubscribed, and then single topicpartition B is
added, in the space of one batch.
Yeah, it's also possible that when a batch is running, a single
topicpartition C is created then deleted. But the Kafka source doesn't even
know C. The root issue is currently the approach is polling metadata, and
any metadata changes between two pollings are missing.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/15102#issuecomment-247910429, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGAB4dbMY6oP4EYl062Dd3chUtOk520ks5qrhKQgaJpZM4J9QvR
.

Test build #65636 has started for PR 15102 at commit f5c57f5.

Test build #65636 has finished for PR 15102 at commit f5c57f5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65636/
Test FAILed.

Test build #65640 has started for PR 15102 at commit b64d104.

Test build #65640 has finished for PR 15102 at commit b64d104.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65640/
Test FAILed.

You should not be assuming 0 for a starting offset for partitions you've just learned about.  You should be asking the underlying driver consumer what its position is.  This is yet another argument in favor of keeping the implementations the same until they have to be different, because the existing implementation at least tries to do this correctly.
My bigger concern is that it looks like you guys are continuing to hack in a particular direction, without addressing my points or answering whether you're willing to let me help work on this.
Have you made up your mind?

My bigger concern is that it looks like you guys are continuing to hack in a particular direction, without addressing my points or answering whether you're willing to let me help work on this.
Have you made up your mind?

Cody, I think we have been addressing your points, though I know we are not done yet. It would be helpful if you could make specific comments on the code, preferably with pointers to what you think the correct implementation would look like.  Otherwise its hard to track which points you think have been resolved and which are still in question.
I appreciate that you are concerned that some of this code is duplicated, but I'm going to have to respectfully disagree on that point.  I think this is the right choice both for the stability of the DStream implementation and our ability to optimize the SQL implementation.

You should not be assuming 0 for a starting offset for partitions you've just learned about. You should be asking the underlying driver consumer what its position is.

I'll let Ryan comment further here, but I'm not sure if this is correct.  It sounds like if we rely on Kafka to manage the its position there will be cases where partial failure could result it data loss.  In general, I think we need to be careful about relying on Kafka internals when our end goal is to provide a much higher level abstraction.

Absent a direct answer, I'm going to read that as "Yes I've made up my mind, no I will not consider PRs to the contrary."
If you want pointers to what I think the correct implementation would look like, I'd say look at the existing DStream and my prototype structured streaming fork.  With the main difference that, as I said above, I don't think the current sql Offset interface makes sense if all you need is to measure equality | not equal | error.
My main concern is making sure end users can do their jobs using Kafka.  If your main concern is a higher level abstraction that may or may not actually work well with Kafka, I wish you luck in that endeavor, but I don't think it makes sense for me to spend any more time arguing with you about it.

You should not be assuming 0 for a starting offset for partitions you've just learned about. You should be asking the underlying driver consumer what its position is.

Yes, there are two approaches for getting new partitions' offsets:

From a user perspective, if I set “auto.offset.reset” to latest, I would like to process new data after the query starts. So we should start from the earliest offset for a new partition.
Follow auto.offset.reset as what you did in your fork.

However, option 2 makes the query result indeterminate when auto.offset.reset is latest, depends on how the query runs. E.g., if you add new partitions and push new data very quickly, the new data may be lost; if you add new partitions and push new data when the app is being recovered, the new data may be lost as well.

My fork is not following auto.offset.reset, it's following what the
(potentially user-provided) consumer does when it sees a new partition.
Maybe that's auto.offset.reset, maybe it's something else.
Either way, who are you to presume that a user doesn't know what she is
doing when she configured a consumer to start at a particular position for
an added partition?  You have no guarantee that offset 0L even exists at
that time.
On Tue, Sep 20, 2016 at 2:52 PM, Shixiong Zhu<EMAIL_ADDRESS>wrote:

You should not be assuming 0 for a starting offset for partitions you've
just learned about. You should be asking the underlying driver consumer
what its position is.
Yes, there are two approaches for getting new partitions' offsets:

From a user perspective, if I set “auto.offset.reset” to latest, I
would like to process new data after the query starts. So we should start
from the earliest offset for a new partition.
Follow auto.offset.reset as what you did in your fork.

However, option 2 makes the query result indeterminate when
auto.offset.reset is latest, depends on how the query runs. E.g., if you
add new partitions and push new data very quickly, the new data may be
lost; if you add new partitions and push new data when the app is being
recovered, the new data may be lost as well.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/15102#issuecomment-248414530, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGAB1U_gCyEfg0GHzuymzWp3wtZ4zNoks5qsDmYgaJpZM4J9QvR
.

Either way, who are you to presume that a user doesn't know what she is
doing when she configured a consumer to start at a particular position for
an added partition?

I feel like we are talking past each other.  We are not giving the developer the option to manually configure a consumer in this way for this PR precisely because I don't think we can while still maintaining the semantics that structured streaming has promised.  This is consciously a higher level API than is provided by DStreams.  Neither of these is right or wrong. I do want to be able to support as many real use cases as possible in the Structured Streaming API, but not at the expense of correctness for the queries that we support. If there are low level use cases that we can't support, which require users to use DStreams instead, that is okay, as it is okay that some users still program against RDDs.

Test build #65685 has started for PR 15102 at commit 6bc2994.

We are not giving the developer the option to manually configure a consumer in this way for this PR precisely because I don't think we can while still maintaining the semantics that structured streaming has promised.

You've got this backwards.  As soon as you give someone the ability to set auto.offset.reset to largest, it opens up the can of worms as to whether resets should happen at the beginning of a stream, during a stream, and/or when a partition is added.  Giving people the ability to configure a consumer doesn't cause that problem, it allows them to solve that problem until such time as the Kafka project has a unified way to solve it.  Similarly, as soon as you allow pattern subscriptions, it opens up the can of worms as to adding/removing topics and whether the sql Offset interface as is makes sense for Kafka.  Just saying you aren't going to handle deletions for right now doesn't solve that problem.
If you really don't want to consider changing the Offset interface, and want to tell people who need the details of Kafka in order to work done to use the DStream, then you should probably eliminate all configuration options except brokers, a list of topics, and maybe SSL.
I'll try one more time, and then I'm done:

Months ago you came up with an interface that realistically will only work with Kafka / Kinesis / lookalikes, yet had no implementation for any of those.
Actually attempting an implementation raised some notable differences between what the interface allowed for and what the implementation needed.
I offered some specific suggestions, including considering changes to the interface
I offered to help with implementation

Your response, from my point of view, has been

Decline to consider changes to the interface
Decline any assistance with actual implementation
Only (re)implement a subset of Kafka functionality that you can see is "safe", regardless of whether it's congruent with the way Kafka is already being used by users.

Under those circumstances, I'm happy to answer specific directed questions you may have, but I'm not interested in continuing to argue.  If you guys say you've got this and you're going to do it your way, then you've got it.
Let me know if you change your mind, I'll still be around.

Test build #65694 has started for PR 15102 at commit 881b206.

Test build #65685 has finished for PR 15102 at commit 6bc2994.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 sealed trait ConsumerStrategy
  case class SubscribeStrategy(topics: Seq[String], kafkaParams: ju.Map[String, Object])
  case class SubscribePatternStrategy(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65685/
Test PASSed.

Test build #65694 has finished for PR 15102 at commit 881b206.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65694/
Test PASSed.

Merged build finished. Test PASSed.

I asked @koeninger to clarify the specific suggestions he is referring to above, here's my response:

[Comments here and on JIRA relating to concerns with the Offset implementation]

I'd be happy to consider a PR (either against this branch now or against master if this patch is merged) to use a different offset for KafkaSource, or even change the offset API in all of structured streaming. A change that large however would need to be a different PR, and should not block this one unless there are correctness issues. I'd want to see some test cases though that show why the current implementation is wrong from an end-user perspective if it needs to block merging initial kafka support.

[Comments here and on JIRA about using String -> String for configuration]

I'd also be happy to consider PRs to improve the DataFrameReader/Writer interface.  That would also need to be in a separate PR.  It would need to work well across languages and be backwards compatible (i.e. what happens when you are using an older data source with the new configuration system?).
That said, we already have something that we've been using since Spark 1.3, even if its not perfect. It does work across languages and I don't see a problem with supporting comma separated strings long term for topic lists (kafka even does this today in its own configuration for things like bootstrap.servers.).  As a result, I do not think it is reasonable to suggest we block merging this patch on an overhaul of the DataSource API configuration system.
I think @koeninger made a good suggestion to block accepting certain kafka configurations.  In particular, auto.offset.reset does not make sense in a world where we are managing the offsets ourselves (kafka documentation suggests you set this to false if you maintain offsets externally).  Similarly, letting the user set the group id could cause data to get split with a different query and thus could affect correctness.  My only question here is if we should have a whitelist or a blacklist.  Scanning though the list of possible configuration options, I think it could go either way.  I tend to error on this side of more power and go with the blacklist.
I disagree that prefixing configuration that should be passed to kafka with kafka is a hack.  We do this in other places in the DataFrameReader/Writer API and have not had any complaints or confusion.  I could be convinced otherwise if someone can point out a case where this would be confusing or ambiguous to a user.

I'd want to see some test cases though that show why the current implementation is wrong from an end-user perspective if it needs to block merging initial kafka support.

PR with failing test indicating at least one reason why it's wrong from an end-user perspective:
https://github.com/zsxwing/spark/pull/4

I do not think it is reasonable to suggest we block merging this patch on an overhaul of the DataSource API configuration system.

Here's what I actually said:
'if you know your plan down the line is to use json for structured configuration, you should use it now, and provide more convenient ways to construct json later, not use "convenient" non-json hacks now.'
No hyperbole about blocking on a complete overhaul, nothing that isn't backwards compatible.  I'm just saying that, if the design document already recognizes that json is necessary to work around the string -> string interface... start using structured json strings now, and make it more convenient later.
Or do you actually think that stuff like
option("assign", "topicA:1:1,topicA:2:2,topicB:3:3")
makes it clear what the arguments are?

I think @koeninger made a good suggestion to block accepting certain kafka configurations.

In case it wasn't clear, I was not suggesting that preventing users from doing things they could otherwise do with Kafka is actually a good idea.  I think it's a bad idea, but if you're going to run with it, you might as well be consistent about it.

PR with failing test indicating at least one reason why it's wrong from an end-user perspective:

@koeninger Thanks for writing the test. Yes, we are aware of this issue. However, it's unlikely that we can support deleting topics using the current Source API. You can take a look at how StreamExecution checks the new data here: https://github.com/apache/spark/blob/976f3b1227c1a9e0b878e010531285fdba57b6a7/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L320
Using the hash code to compare offsets has a potential issue, it may make the latest offset be smaller than the old offset, then StreamExecution won't process the new data.
I think one possible solution is StreamExecution doesn't compare the offsets, instead, it just assumes getOffset will always return the latest offset, and it never rollback to an old offset. This needs more discussion anyway. Hence I suggest we don't block this PR for this. Deleting topics can be supported in a later PR when we make an agreement on how to resolve the issue.

This is pretty much the fundamental issue.  Kafka offsets alone aren't
capable of meeting the SQL Offset interface as defined.  I think that means
the Offset interface needs to be reconsidered or eliminated.  I think that
work needs to be done now, before even more work gets put into a
fundamentally unworkable interface.  I don't see what the rush is, because
this isn't going to make it into 2.0.1 anyway.
On Thu, Sep 22, 2016 at 12:29 PM, Shixiong Zhu<EMAIL_ADDRESS>wrote:

PR with failing test indicating at least one reason why it's wrong from an
end-user perspective:
@koeninger https://github.com/koeninger Thanks for writing the test.
Yes, we are aware of this issue. However, it's unlikely that we can support
deleting topics using the current Source API. You can take a look at how
StreamExecution checks the new data here: https://github.com/apache/
spark/blob/976f3b1227c1a9e0b878e010531285fdba57b6a7/sql/core/src/main/
scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L320
Using the hash code to compare offsets has a potential issue, it may make
the latest offset be smaller than the old offset, then StreamExecution
won't process the new data.
I think one possible solution is StreamExecution doesn't compare the
offsets, instead, it just assumes getOffset will always return the latest
offset, and it never rollback to an old offset. This needs more discussion
anyway. Hence I suggest we don't block this PR for this. Deleting topics
can be supported in a later PR when we make an agreement on how to resolve
the issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/15102#issuecomment-248971360, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGAB0a4SZx4Zjq4hMkI6tSLIDwJLW8Bks5qsrsUgaJpZM4J9QvR
.

Test build #65785 has started for PR 15102 at commit 786af2f.

Test build #65785 has finished for PR 15102 at commit 786af2f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65785/
Test FAILed.

@tdas moving this conversation back to the PR that's linked from the public jira

yeah, i am trying to figure out all the options and write up something to so that we are clear on the pros and cons of each approach. At a high level, they match the ones you suggested. Though I am trying to tease out what needs to be done if deletion is to supported, and what needs to be done deletion+recreation of same topic needs to be supported.

Also, at a high level I think that supporting deletion of topics does not require timestamps, its supporting deletion+recreation of the same topic would require more disambiguating information like timestamps. Here are a few questions,
Just to confirm, there are no unique guid kind of thing for topics in Kafka?
When a topic is deleted and recreated, what happens to the offsets? Does the recreated topic's offset start from 0? Or does it start from where the previous topic left off?

I don't think it's useful to focus too hard on deletion, it's a symptom not just a cause.  Subscription changes for other reasons would also expose the same issue.   I think the underlying issue is that the Offset interface is asking for a global monotonic order, which is hard in a distributed system.
To answer the questions, a topicpartition is a folder on disk (containing messages and offsets), and a node in ZK.  Both are deleted if the topicpartition is successfully deleted, so offsets start over.  ZK nodes have a cZxid, but I don't think it's a good idea to rely on Kafka having ZK internals, and I'm not sure what it buys you.  Say you do have zxid, and say you have two consumer states:
State A:  topic A, partition 0, offset 1, zxid 0x20
State B: topic B, partition 0, offset 1, zxid 0x21
At one point in time the consumer is in state A, and different point in time it is in state B.  Without more information, how can you tell if A < B or B < A ?

For streaming you already know what the global order is, because you know when you asked for A and B.  I agree that we should probably remove the comparable requirement from Offset in favor of just having equality.  At the time it was a useful safe guard, but its clearly causing more confusion than anything.
Assuming A was retrieved before B, then it seems like you emit a warning that data was possibly missed from A (since it was deleted before we could get it) and you start a new batch on topic B from offsets 0-1.  Right?

Or do you actually think that stuff like option("assign", "topicA:1:1,topicA:2:2,topicB:3:3") makes it clear what the arguments are?

We don't support assign.  When we do add that support, that is not super easy to follow, I don't feel strongly that its better or worse than JSON though if its unambiguous.
Are there arguments that we do support that you think are confusing?

For streaming you already know what the global order is, because you know when you asked for A and B. I agree that we should probably remove the comparable requirement from Offset in favor of just having equality.

Sounds good, as long as the execution portion of it isn't e.g. storing timestamps for A and running into issues on driver failover to a machine with clock drift.

Assuming A was retrieved before B, then it seems like you emit a warning that data was possibly missed from A (since it was deleted before we could get it) and you start a new batch on topic B from offsets 0-1. Right?

I think in the absence of prior information about the position in a topicpartition, you start a new batch on topic B starting from wherever the consumer's position was at the time it acquired the subscription, which might not be 0.  I.e. you call position() before seekToEnd().  This might mean you need to record 2 kafka offsets in an SQL Offset if it's the first time you've seen that topicpartition.

Are there arguments that we do support that you think are confusing?

I think the main thing that would be confusing is to specify topics in one way (custom-delimited string) for one configuration, and in another way (structured json) for another configuration.

@koeninger
I did some independent brainstorming with @zsxwing on topic deletion, and yeah I agree with you that attempting to account for deleted topics in the offset in the KafkaSourceOffset such that compareTo is satisfied is more complicated than just eliminating compareTo. That said, there are still a few corner case - of the same topic being deleted and recreated. I am not familiar with how often this can happen (let us know your thoughts). But the general idea we can implement that that we attach a unique id to the topic in the KafkaSourceOffset. Whenever the new topic is detected (while running or across query restarts), generate a unique id so that it is consider as a new topic. Here are the options
Option 1: When getOffset detects new topic, if the topic existed in previous offset, create new (topic, unique id)

Pro: Simple
Con: Cannot detect if topic gets deleted+recreated between triggers (possibly, across query restarts),

Option 2: Use RebalanceListener to know when topic has been deleted

Pro: Handles topic deletion+recreation between triggers while query is active
Con: Misses deletion+recreation during query restarts
Con: Listener called on different thread, so possible race conditions

Option 3: Use the creation time / cZxid of topic info stored in ZK to disambiguate

Pro: Zookeeper maintains uniques ness across any component restarts
Con: Requires depending on full Kafka + ZK,
Con: Requires knowing the exact ZK path where topics are saved, but this can be tested and made sure that it never fails when we upgrade Kafka

I feel that we should just keep it simple for now, and go for Option 1. What do you think?

Comparable requirement removed in #15207.

I think in the absence of prior information about the position in a topicpartition, you start a new batch on topic B starting from wherever the consumer's position was at the time it acquired the subscription, which might not be 0. I.e. you call position() before seekToEnd().

Why do you care when it acquired it?  If it appeared in-between the the last batch and now, don't you want to consume all of the available data from it?  Otherwise the answer is going to depend on the specifics on when you see the topic, which seems counter to the model of Structured Streaming.

I think the main thing that would be confusing is to specify topics in one way (custom-delimited string) for one configuration, and in another way (structured json) for another configuration.

Are you proposing users have to type "[\"topic1\", \"topic2\"] (or pull in a json library) instead of "topic1,topic2"?  Seems we could pretty seamlessly add support for JSON in the future, while still making the common case easy to type.

@tdas I think as long as marmbrus' PR to remove comparable from the interface works for sane variations of subscription changes it's the best way to go.  I'm honestly fine with someone getting what they deserve if they delete and recreate a topic in the space of a single batch or while a stream is down.
@marmbrus

Why do you care when it acquired it?

This isn't so much a temporal thing, as a let the consumer do its job thing.  This sort of configuration should ideally be handled by auto.offset.reset, and we shouldn't bake in too much second guessing about it.  There's plenty of use case for "I want to be able to add a topicpartition mid stream, but I don't want to start it from the beginning."

Are you proposing users have to type

I'm saying that you guys proposed json as a workaround for the string->string thing.  Given that, yeah, I think consistency in using json for any non-scalar values is worth 2 extra characters per topic and 4 at the ends.

"I want to be able to add a topicpartition mid stream, but I don't want to start it from the beginning."

I see, I was thinking only of new topics that appear that match your pattern.  I agree that if/when we add that ability to add existing partitions midstream we'd probably need to add two offsets in to the SQL offset for new partitions.

I think consistency in using json for any non-scalar values is worth 2 extra characters per topic and 4 at the ends.

I'd also support JSON here, but I would not mandate it (i.e. try json parsing and fall back to comma separation).  Its not ambiguous, supports consistent usage, and doesn't penalize the simple use cases.

I agree that if/when we add that ability to add existing partitions midstream we'd probably need to add two offsets in to the SQL offset for new partitions.

It's not just existing partitions.  If you have a low-value high-volume stream (which is the kind of situation where you'd want auto offset reset latest to begin with), you may not even want your first batch to have however many messages got in between creation and subscription rebalance.  I dunno, I just don't want to assume too much.

I'd also support JSON here, but I would not mandate it (i.e. try json parsing and fall back to comma separation). Its not ambiguous, supports consistent usage, and doesn't penalize the simple use cases.

Cool, seems reasonable.

Just curious looking at this, if you are comparing "sequential" offsets across partitions a rebalance would definitely affect this and, unless something has changed, it probably not a good idea to compare offsets from kafka across partitions. You could simply add an id/timestamp to the producer and send it with the message rather than using this methodology or if you must use offset query the broker for the full list and compare what you consumed to that list (small increase in latency btwn consumption and processing).  This is from the Kafka paper, which makes me question your scheme: "...Note that our message ids are increasing but not consecutive. To compute the id of the next message, we have to add the length of the current message to its id." This means simply comparing which offsets are larger will not necessarily yield you the most recent message across partitions and definitely won't hold in a rebalance during which time some broker logs will be on hold and not consumed. In my own implementation, the offsets are great for message guarantees (eg delivery/consumption checks), because the broker has a full ordered list, but not for cross partition ordering.

We aren't comparing ordering of offsets across partitions, and I don't
think that was ever in consideration.  At this point the most likely
candidate for the global ordering is implicit in the single driver thread
that is asking for getOffset
On Sep 23, 2016 12:41 AM, "Jay White Bear"<EMAIL_ADDRESS>wrote:

Just curious looking at this, if you are comparing "sequential" offsets
across partitions a rebalance would definitely affect this and, unless
something has changed, it probably not a good idea to compare offsets from
kafka across partitions. You could simply add an id/timestamp to the
producer and send it with the message rather than using this methodology or
if you must use offset query the broker for the full list and compare what
you consumed to that list (small increase in latency btwn consumption and
processing). This is from the Kafka paper, which makes me question your
scheme: "...Note that our message ids are increasing but not consecutive.
To compute the id of the next message, we have to add the length of the
current message to its id." This means simply comparing which offsets are
larger will not necessarily yield you the most recent message across
partitions and definitely won't hold in a rebalance during which time some
broker logs will be on hold and not consumed. In my own implementation, the
offsets are great for message guarantees (eg delivery/consumption checks),
because the broker has a full ordered list, but not for cross partition
ordering.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/15102#issuecomment-249107463, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGAB8MjSGueEHbEv3gXHkFiV5_CfsTqks5qs2Z2gaJpZM4J9QvR
.

Which offset does getOffset() return one from the partition or something you created? Because it looked like when you were hashing them, you were returning partition offsets.

Source.getOffset returns an sql Offset, not a kafka offset.  At this point
the current plan is to remove the ordering requirement for sql Offset,
which makes the whole discussion of comparing ordering for kafka offsets
irrelevant.  The order is the order in which the driver thread saw sql
Offsets as it called getOffset.
On Fri, Sep 23, 2016 at 1:24 PM, Jay White Bear<EMAIL_ADDRESS>wrote:

Which offset does getOffset() return one from the partition or something
you created? Because it looked like when you were hashing them, you were
returning partition offsets.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/15102#issuecomment-249266737, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGABzXhKpBRDeRAoSIroXJJA_HzEZRXks5qtBlxgaJpZM4J9QvR
.

Test build #65845 has started for PR 15102 at commit 5f33eb4.

Test build #65845 has finished for PR 15102 at commit 5f33eb4.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SparkContext(config: SparkConf) extends Logging
class ChiSqSelector @Since(\"2.1.0\") () extends Serializable
case class ShowColumnsCommand(tableName: TableIdentifier) extends RunnableCommand
abstract class CompactibleFileStreamLog[T: ClassTag](
class FileStreamSinkLog(
  case class FileEntry(path: String, timestamp: Timestamp, batchId: Long) extends Serializable
class FileStreamSourceLog(
trait Offset extends Serializable

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65845/
Test FAILed.

Merged build finished. Test FAILed.

Test build #65926 has started for PR 15102 at commit 755ceaa.

Test build #65926 has finished for PR 15102 at commit 755ceaa.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
        // ('scaled' = +Infinity). However in the case that this class also has
        // 0 probability, the class will not be selected ('scaled' is NaN).
  final val thresholds: DoubleArrayParam = new DoubleArrayParam(this, \"thresholds\", \"Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0 excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold\", (t: Array[Double]) => t.forall(_ >= 0) && t.count(_ == 0) <= 1)
    thresholds = Param(Params._dummy(), \"thresholds\", \"Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.\", typeConverter=TypeConverters.toListFloat)
case class SortOrder(child: Expression, direction: SortDirection, nullOrdering: NullOrdering)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65926/
Test PASSed.

Test build #65937 has started for PR 15102 at commit 852f607.

Test build #65937 has finished for PR 15102 at commit 852f607.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65937/
Test FAILed.

Ok, finished a line-by-line compare + comment.
The biggest thing I'm having trouble reconciling is the stated emphasis on limiting user options in order to give guarantees, yet throwing those guarantees away with only warn level logging (e.g. the handling of offset out of range exception).

Right now since there are some arguments about how to handle various failures, I'm listing what I found via stress test to discuss:
(1) Kafka APIs fail because we cannot connect to Kafka cluster. Solution: fail the query.
(2) getOffset fails temporarily because some of topics are deleted at the same time. consumer.position will throw NPE due to this race condition. Solution: retry (This is why withRetries is added)
consumer.poll(0)
val partitions = consumer.assignment()
consumer.position(p)

(3) In getBatch, some partitions are new because they are not in fromOffsets. Then we will call fetchNewPartitionEarliestOffsets to fetch these partitions. However, some of these new partitions may be deleted due to topic deletion, then they won't appear in consumer.assignment(). Solution: log a warning.
(4) In getBatch, some partitions are new because they are not in fromOffsets. Then we will call fetchNewPartitionEarliestOffsets to fetch these partitions. Similiar to (2), consumer.position may throw NPE due to this race condition. Solution : retry.
(5) Topics are deleted when a Spark job is runinng, which may cause OffsetOutOfRangeException. (I'm not sure if there are more types of exceptions, may need to investigate) Solution: log a warning.
(6) A topic is deleted then added. This may make untilOffset is less than fromOffset. Solution: log a warning.

We can add an option to allow the user failing the query instead of just logging the warning.

Ok, so this kind of thing is why I was concerned about the copy, paste, randomly change things approach to developing this module.

(5) Topics are deleted when a Spark job is runinng, which may cause OffsetOutOfRangeException. (I'm not sure if there are more types of exceptions, may need to investigate) Solution: log a warning. Note: if a Spark job fails, then the query will fail as well.

OffsetOutOfRangeException basically means you asked Kafka for an offset, and it wasn't there.  The most common reason this happens isn't because a topic got deleted, it's because messages expired out of retention before they got read.
Just logging at warning level and continuing in this situation is catastrophically, someone-loses-their-paying-job-not-their-spark-job, bad.
The existing kafka DStream integrations that have been around for 7 spark versions will just let that exception be thrown, resulting in errors / failed tasks, which make it pretty obvious that something is really wrong.
If you think that behavior is incorrect, let's figure out a unified behavior for how to deal with exceptional situations that break fundamental assumptions, and make it reallllly obvious to users how to get the behavior they need across both modules.  But having the structured stream behave in significantly different ways seems like a recipe for trouble.

Just pushed to verify if the workaround for https://issues.apache.org/jira/browse/KAFKA-1894 does work on Jenkins. Not ready for another round of review.

Test build #66005 has started for PR 15102 at commit 59a93a5.

Test build #66005 has finished for PR 15102 at commit 59a93a5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class KafkaSourceOffset(partitionToOffsets: Map[TopicPartition, Long]) extends Offset
abstract class StreamExecutionThread(name: String) extends UninterruptibleThread(name)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66005/
Test FAILed.

FYI: #15274 adds support for parsing JSON from the key/value into a Spark SQL StructType

Test build #66047 has started for PR 15102 at commit 9d95d52.

Test build #66047 has finished for PR 15102 at commit 9d95d52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66047/
Test PASSed.

I spent a while playing around with this today on a real cluster, and overall it is pretty cool!  I have a few suggestions we should implement in the long run, but these can probably be done in follow-up PRs

It would be nice to be able to do something other than earliest/latest.  In particular, I have some low volume streams where I want to go back a bit as I'm working interactively.  Going back all the way results in a really long job.  If there was a time index, I'd love to say -10 minutes but even just `-100 offsets would be great.
When specifying earliest, you end up with really big partitions.  As a result you get virtually no progress reporting until its done.  I'd consider having a max number of offsets in a task.  It should be easy to split them up in getBatch.  One question, is it a problem if two tasks are pulling from the same topic partition in parallel?  Does this break the assumptions of our caching?
Similarly, you get no results until the end.  We might also want to have a maxOffsetsInBatch similar to what we have in the FileStreamSource.

I can do a final pass over the code, but do we think we are getting close to something that we can merge and iterate on?

It would be nice to be able to do something other than earliest/latest.

That's what Assign and the starting offset arguments to the Subscribe strategies are for.  The implementation was already there.

When specifying earliest, you end up with really big partitions.

Again, spark.streaming.kafka.maxRatePerPartition and the associated implementation was already there.  If you don't want the coupling to time, it's pretty straightforward.  The bigger question is when / if / how you're going to do backpressure.

One question, is it a problem if two tasks are pulling from the same topic partition in parallel? Does this break the assumptions of our caching?

This breaks fundamental assumptions of Kafka (per-topicpartition ordering) and really shouldn't be done.

I can do a final pass over the code, but do we think we are getting close to something that we can merge and iterate on?

I think we're in much better shape than when we started, but I still honestly think this implementation made a bunch of user-visible behavioral and configuration changes from the DStream that really have nothing to do with the inherent differences between it and structured streaming.  This isn't just me whining about "you changed my code",  it really is going to make it harder to explain to people and harder to maintain.

Test build #66265 has started for PR 15102 at commit e883062.

Test build #66274 has started for PR 15102 at commit d154532.

Test build #66265 has finished for PR 15102 at commit e883062.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66265/
Test PASSed.

Test build #66274 has finished for PR 15102 at commit d154532.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66274/
Test PASSed.

Test build #66283 has started for PR 15102 at commit 77208d1.

Test build #66285 has started for PR 15102 at commit ccadd81.

Test build #66287 has started for PR 15102 at commit f78c990.

Test build #66287 has finished for PR 15102 at commit f78c990.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66287/
Test FAILed.

Test build #66288 has started for PR 15102 at commit 7ff1059.

Test build #66283 has finished for PR 15102 at commit 77208d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66283/
Test PASSed.

Test build #66289 has started for PR 15102 at commit a6c4970.

Test build #66285 has finished for PR 15102 at commit ccadd81.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66285/
Test PASSed.

Merged build finished. Test PASSed.

Test build #66288 has finished for PR 15102 at commit 7ff1059.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66288/
Test FAILed.

Test build #66289 has finished for PR 15102 at commit a6c4970.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66289/
Test FAILed.

Test build #3294 has started for PR 15102 at commit a6c4970.

Test build #3294 has finished for PR 15102 at commit a6c4970.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #66338 has started for PR 15102 at commit d50a05e.

Test build #66345 has started for PR 15102 at commit 4316906.

Test build #66338 has finished for PR 15102 at commit d50a05e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66338/
Test PASSed.

Test build #66345 has finished for PR 15102 at commit 4316906.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66345/
Test PASSed.

Test build #66396 has started for PR 15102 at commit d9d848c.

Test build #66398 has started for PR 15102 at commit 4754125.

Test build #66396 has finished for PR 15102 at commit d9d848c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66396/
Test PASSed.

Test build #66397 has finished for PR 15102 at commit 7d658f1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66397/
Test PASSed.

Test build #66398 has finished for PR 15102 at commit 4754125.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66398/
Test PASSed.

Merged this to master, @zsxwing do you have another PR for 2.0.

Merged this to master, @zsxwing do you have another PR for 2.0.

See #15367


Closes #55-Fix error prone error code from Codacy

[Current coverage][cc-pull] is 79.78% (diff: 100%)

Merging [#64][cc-pull] into [develop][cc-base-branch] will not change coverage

@@            develop        #64   diff @@
==========================================
  Files            16         16
  Lines           277        277
  Methods           0          0
  Messages          0          0
  Branches         23         23
==========================================
  Hits            221        221
  Misses           47         47
  Partials          9          9

Powered by Codecov. Last update [41708ee...77e8f63][cc-compare]
[cc-base-branch]: https://codecov.io/gh/modernmaster/katas/branch/develop?src=pr
[cc-compare]: https://codecov.io/gh/modernmaster/katas/compare/41708ee1daa272f63d54d140968631beee3c2ee7...77e8f63856b1eebb8e87b985c08040eff264dc50?src=pr
[cc-pull]: https://codecov.io/gh/modernmaster/katas/pull/64?src=pr


Add Transaction details

Add details to transaction pages which includes transaction size, script sig details and any data points we have. This should be in an "advanced" (show/hide) view, especially for scritpsig
Internal Use
Acceptance Criteria

Definition of Done

[ ]  Tested against acceptance criteria
[ ] Tested against the assumptions of the user story
[ ] The project builds without errors
[ ] Unit tests are written and passing
[ ] Tests on devices/browsers listed in the issue have passed
[ ] QA performed & issues resolved
[ ] Refactoring completed
[ ] Any configuration or build changes documented
[ ] Documentation updated
[ ] Peer Code Review performed

These details are all available in chainquery. Just need to added the UI for it.


Allow setting smaller radius in options

The options page currently allows various radii, but 1/2 mile (the default) is the smallest. Let's add a 1/4 mile option as well.

Done. 1/2 mile is still the default.


Installing mongoDB

While doing the project, I couldn't get passed the MongoDB installation part as I tried to install it.
I got this error after running "Sudo apt-get install MongoDB-org" command
"Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package MongoDB-org"
I found a link that seemed to work but when I tried it, I  had an issue with missing some dependencies
MongoDB-org-mongos : Depends: libssl1.1 (>= 1.1.1) but it is not installable
MongoDB-org-server : Depends: libssl1.1 (>= 1.1.1) but it is not installable
MongoDB-org-shell : Depends: libssl1.1 (>= 1.1.1) but it is not installable
I am using ubuntu 22.04 from AWS ec2 for this project. Here I found a solution that solved this problem for me
the steps from this link got it to work for me.
[(https://www.cloudbooklet.com/how-to-install-mongodb-on-ubuntu-22-04)/]
I've got MongoDB up and active now
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-06-08 05:21:30 UTC; 9h ago
Docs: https://docs.mongodb.org/manual
Main PID: 2730 (mongod)
Memory: 68.8M
CPU: 2min 15.243s
CGroup: /system.slice/mongod.service
└─2730 /usr/bin/mongod --config /etc/mongod.conf
Jun 08 05:21:30 ip-172-31-29-122 systemd[1]: Started MongoDB Database Server.


Option for detaching script from node (#6934).

Fixes #6934

@vnen Right! Sorry, that comment slipped. Fixed.


Display/Font issues on Ubuntu xfce4 desktop

VSCode Version: Code 1.18.1 (929bacba01ef658b873545e26034d1a8067445e9, 2017-11-16T18:34:22.110Z)
OS Version: Linux x64 4.10.0-40-generic
Extensions:

Extension
Author (truncated)
Version

code-gnu-global
aus
0.2.2

cpptools
ms-
0.14.3

Steps to Reproduce:

install editor
see weird dots between letters, and sometimes between lines. If you look at the attached screenshot you should be able to see what i'm talking about. I've tried about 10 different fonts in different sizes, each one has similar issues, although the problem spots tend to move around a bit with different fonts.

Reproduces without extensions: Yes/No

I fixed it. I had to chanage my Sub-pixel order from RGB to None.

@Gman47 great


Opens app select instead of going directly to default mail app on Android?

Using this simple convenience method:
Device.OpenUri(new Uri("mailto:contact@contact.com"));

The default email app is launched as expected.
Using the plugin:
var emailMessenger = CrossMessaging.Current.EmailMessenger;
if (emailMessenger.CanSendEmail)
{
<EMAIL_ADDRESS>"", "');
}

Seems to launch a more general text share intent (it's using the simple API there, but even when using the builder and filling all cc, bcc, etc fields, it still doesn't launch the default app).
Is there something I'm missing?

The plugin follows the guidance provided by Xamarin on how to send an email using an Intent.   The guidance states that by setting the Intent mime type using email.SetType ("message/rfc822") will cause applications capable of handling email to launch.  If you have multiple apps capable of doing this, you will get a list of apps to choose from.
From reading the guidance provided by Google they state that using the more specific ACTION_SENDTO intent and the mailto uri scheme will ensure that only email apps handle the action.  This seems to be the behaviour your are getting by using your convenience method.
Do you get a different list of apps when using the plugin vs your convenience method, or does the plugin show a list of apps to choose from, whilst your convenience method launches the email application directly?  If the plugin shows a list of applications capable of handling the intent, does this not include the default email application?

I'm not OP, but I can confirm this behavior. The default e-mail client DOES show up in the list, but so do other apps clearly not capable of sending e-mails.
Most prominently, these include: What's App, Microsoft OneNote, Microsoft OneDrive, Google Cloud
We actually got user feedback that this list is confusing. We use this plugin for a "Feedback" option and intend the user to write an e-mail to us, which pre-includes some technical data in the body. The users might actually expect that they can use What's App to send their feedback to us, while we do not even have a What's App account.
On top of that, on iOS and Windows the default e-mail app launches without this list. I would expect a similar behavior on Android.
I'd love for this plugin to filter only "true" e-mail apps or just show the default e-mail app. Maybe make this an optional parameter if you choose to do so.
Thank you.

I've added a new UseStrictMode to theEmailMessageBuilder that uses the Google recommended practice to limit the list to only email apps.  This works fine from my testing but unfortunately sending attachments using the Strict Mode doesn't seem to work from my testing.  Seems I'm not the only one running into these issues.
StrictMode will be available as part of the v4 release.


fix id_feature's number_buckets uage with fg，add more info for using

id_feature's number_buckets uage
add dingding group
remove gpu in cluster config
add more information

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

id_feature's number_buckets uage
add dingding group
remove gpu in cluster config
add more information


readme.md contains typo

The package is called sql-escape-string but the readme.md suggests require 'sql-string-escape, causing an error because the name is not correct.

String theory is over rated, it will never reach escape velocity or get a sequel
Please close


PIP-69 Add deps vuln scan

PIP-69 Use nvd-clojure to scan for vulnerable deps

What's the eviction policy on the NVD owasp cache? I didn't see anything explicit

Heh, a little weird. So you'll notice it is explicitly banned in the general deps cache (just above), then it gets its own cache of just the owasp path (containing the db)

The storage/eviction policy is a little more complicated. The github cache action only ever saves on a pass, so if there is any vuln it won't save any updates it found that go-round. Not ideal, but not horrible. Unfortunately you can not currently change the DB directory in the tools.cli version of nvd-clojure.

I pulled and ran nvd locally and obviously nothing SEVERE, but is there any actions behavior like a failure if there is?

nvd-clojure returns a nonzero exit status (by design) on any vuln, yeah

You can see it failing (doing its job) here: https://github.com/yetanalytics/xapipe/runs/4523404235?check_suite_focus=true

You can see it failing (doing its job) here: https://github.com/yetanalytics/xapipe/runs/4523404235?check_suite_focus=true

Cool, that seems great.


Improve CodeWriter newlines and indentation

This commit simplifies and improves how the CodeWriter handles the
automatic injection of newlines and indentation. Now, the only difference
between write() and writeInline() is that write() appends a newline
(there was previously a lot of special-cased code for writeInline).

writeInline() now properly handles indentation when necessary (e.g., if
a previous call injected a newline). This includes a few edge cases
like if the first call to the CodeWriter is to writeInline() and uses
indentation.
The character used for newlines is now tracked by the current state rather
than globally.
Trailing newlines are now always inserted to the given text if enabled
There was a bug previously where they were not appended if the last call
on the CodeWriter was a writeInline(). This could potentially affect
usage of CodeWriters that only wrote with inline calls.
The automatic appending of a newline by methods like openBlock(),
closeBlock(), and write() can now enabled or disabled using
enableNewlines() and disableNewlines().

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The automatic appending of a newline by methods like openBlock(), closeBlock(), and write() can now enabled or disabled using enableNewlines() and disableNewlines().

Would this allow additional helpers like openBlockInline to be defined, so I could have an inline block, but not add the trailing newline?

I'm not 100% sure what openInlineBlock would do, but it should be possible with this change. There's a test case that does something like that: https://github.com/awslabs/smithy/pull/529/files#diff-07be89dbdaaf4a400e8e226ec79bac02R606

This could potentially affect usage of CodeWriters that only wrote with inline calls.

Anything in particular we need to look out for w/ existing usage of CodeWriter?


Add calendar to homepage

Add a calendar like the one on http://wafflejs.com so that people can see the current month's speakers as well as last month's speakers to get an idea of what kind of talks are at the event. Link all past event titles to the youtube for the talk :)

Yay, we have a past speakers page now!


builder pattern for blob content streaming

A solution for the build pattern (issue #90)
It has some lifetime issues, please check them.

Lease_Id, Content_request_id
Clone for (blob) client
No settings for increment (some default value is used)

The lease and client request lifetime issue is solved by copying the data.

Awesome! thank you!


As podcaster I want to be able to deploy the CMS on a sub folder

https://github.com/podlove/radiator-cms/issues/114


about 'train_custom' coco  dataset

Hello,
The annotations file, person_keypoints_train_custom.json, is not mentioned on the readme page.
The file may be modified from person_keypoints_train2017.json
What changes have been made? Could you please upload the custom files?
Thanks for your time.

Thank you for your interests,
"train_custom" is for user's custom dataset. Its format would be same as original coco dataset.
You can change it to ''train2017" for original coco dataset.
Regards,
Jay.


Feature Request: Option To Automatically Update Major and Minor Version Tags

release-please-action's documentation describes how one can hand-roll an approach to automatically keeping vX and vX.Y (or simply X and X.Y in keeping with Commitizen's existing style) tags up to date with the latest corresponding patch version. GitHub Action's own official documentation recommends such a tagging procedure: "Using the specific major action version allows you to receive critical fixes and security patches while still maintaining compatibility. It also assures that your workflow should still work." It would be super convenient to offer an input that just does this automatically. Not sure if this issue would be better suited to the commitizen repo or here.

You would like this action to offer v1, v2, etc. Or for this action to create 2 tags? one with the x.y.z and another with just x? for the later I think we need 2 changes:
in commitizen
add to the cz version a flag to get the different chunks, like cz version --project --format 'v$MAJOR'
in commitizen-action
add a flag "major_tag" which would execute:
if [[ $INPUT_MAJOR_TAG == 'true' ]]; then
    git tag $(cz version --project --format 'v$MAJOR')
fi

I'm open to this feature, but I don't have the capacity to add the flag to commitizen, PRs are welcome

2 tags (or 3 if we also include minor version) would be in keeping with GitHub Actions' guidance. I think it's okay to leave this open in case someone is interested in working on it eventually.


Unhandled Promise rejection due to babel-polyfill

[x] I have searched for existing issues
[x] I have provided detailed instructions that can reproduce the issue (including code and data necessary)

Package versions:
{
    "react": "16.0.0-alpha.12",
    "react-native": "0.48.3",
    "react-native-pathjs-charts": "0.0.31",
    "react-native-svg": "5.4.1"
}

Traceback:
console.error: "Unhandled promise rejection", {}

console.error
    YellowBox.js:67:10
<unknown>
    es6.promise.js:97:18
module.exports
    _perform.js:3:26
<unknown>
    es6.promise.js:91:15
<unknown>
    JSTimers.js:295:17
_callTimer
    JSTimers.js:148:6
_callImmediatesPass
    JSTimers.js:196:6
Object.callImmediates
    JSTimers.js:464:11
MessageQueue.__callImmediates
    MessageQueue.js:282:13

Removing import 'babel-polyfill' solves the problem.
Issue #62 and related PR #64 that introduced babel-polyfill. Can we try to reproduce #62 on the latest RN release to make sure we still need the polyfill?

I'm seeing the same error on:
{
"react": "16.0.0-alpha.12",
"react-native": "0.47.1",
"react-native-pathjs-charts": "0.0.31",
"react-native-svg": "^6.0.0-rc1"
}
and it looks like removing import 'babel-polyfill' fixes the issue. I also think we should check if polyfill is still needed.

I have the same problem @elbuild ,had you sovle this problem?and  how?thans!

Closing now that #211 has been merged into master

With this fix, we're seeing the return of:
undefined is not a function (evaluating '_iteratortypeof Symbol === 'function' ? Symbol.iterator: '@@iterator'')
Specifically on android

Curious what imports you have in your app? When I removed babel-polyfill, I also updated the example app and replaced:
import 'babel-polyfill'

with
import 'core-js/es6/symbol'
import 'core-js/fn/symbol/iterator'


Basic regular expression doesn't work

This basic regular expression doesn't seem to be working: [a-zA-Z0-9_]*
var word = 'DiegoPardo12'
var reg = /[a-zA-Z0-9_]*/;
var regex = new Regex(reg);
if(regex.test(word))... is always false
Thanks!

Happened the same with me.
var word = 'ws://';
var reg = /(ws:\/\/)/;
var regex = new Regex(reg);
console.log(regex.test(word)); // always false :/

yikes

It doesn't work because I haven't implemented it. Sorry. I'm a little busy with college work nowadays. Hence, I might not be able to fix this issue anytime soon.


Only hash ./poetry.lock to compute the cache key for venvs

This avoids the post run step failing when a test creates unreadable
files or directories within the working directory.
Signed-off-by: Sean Quah<EMAIL_ADDRESS>

Thanks! Are you okay to cut a release and move the v1 tag forward?

Sure, I'll do that after I've done some testing.

Thanks! Are you okay to cut a release and move the v1 tag forward?

Done!


souporcell_pipeline.py: command not found

Hi again! Sorry if I cause a lot of errors, really appreciate all your help!

Caused by:
Process HADGE:run_multi:gene_demultiplexing:demultiplex_souporcell:souporcell (sampleID) terminated with an error exit status (127)
Command executed:
     mkdir souporcell_sampleID
     mkdir souporcell_sampleID/souporcell_out
     touch souporcell_sampleID/params.csv
     echo -e "Argument,Value

bamfile,sampleID__filtered_bam_file.bam
barcode,barcodes.tsv.gz
fasta,genome.fa
threads,5
clusters,2
ploidy,2
min_alt,10
min_ref,10
max_loci,2048
restarts,None
common_variant,No common variants are given.
known_genotype,/path/to/project/results/demultiplexing/split_wgs_vcf/project.sampleID.vcf
known_genotype_sample,No known sample names are given.
skip_remap,True
ignore,False " >> souporcell_sampleID/params.csv
     souporcell_pipeline.py --threads 32 -i sampleID__filtered_bam_file.bam -b barcodes.tsv.gz -f genome.fa -t 5 -k 2 --ploidy 2 --min_alt 10 --min_ref 10 --max_loci 2048               --known_genotypes unzipped.vcf  --skip_remap True  -o souporcell_sampleID/souporcell_out

Command exit status:
127
Command output:
(empty)
Command error:
.command.sh: line 26: souporcell_pipeline.py: command not found

Could this be due to the corrupted singularity image (https://github.com/theislab/hadge/issues/49)? I am cleaning my cache now and trying again

How did you call the pipeline? Use which profile then?

How did you call the pipeline? Use which profile then?

I use the flag -profile cluster in my nextflow run .. call
Should I also put singularity.enabled    = true into the cluster profile, then?

yes exactly, you can also do thay by using the existing profiles --profile cluster,conda_singularity. The second profile enables both conda and souporcell (check the latest doc, e.g. section "Quick start")

How did you call the pipeline? Use which profile then?

I use the flag -profile cluster in my nextflow run .. call
Should I also put singularity.enabled = true into the cluster profile, then?

yes exactly, you can also do thay by using the existing profiles --profile cluster,conda_singularity. The second profile enables both conda and souporcell (check the latest doc, e.g. section "Quick start")

Ah sorry, overlooked that. Will try! Thanks for the quick help, really appreciate it!

Worked, thanks


Drop shelf_router[_generator]

Moved to https://github.com/dart-lang/shelf/

Oh! Do you have the keys to transfer issues?

@kevmoo I don't think issues can be transfer outside the organization. We'll probably have to copy/pasta issues.


clarify naming scheme for getters / setters

We've got a number of getters and settters, and the names have become a little less consistent.
extract_i16_as_i32_m128i! should probably be "get" like the rest.
Also some of the "getters" don't get raw data they do a round, so that should be a "convert" maybe.
Naming is a mess.
--
Our naming rules are...

"cast" preserves the exact bit patterns involved.
... ?

Hopefully this will be fixed as part of https://github.com/Lokathor/safe_arch/pull/33

Closed by https://github.com/Lokathor/safe_arch/pull/33


fix example relation between files

Describe this PR

Changes

What issue does this fix?

Any other relevant information

Thanks @naor88 ! We'll get a tech review on this and get it merged; your contributions are appreciated!

So, the first change to remove .ts from the file name in the import is certainly correct. I'm trying to validate the others as well 👍

@TasinIshmam put on his detective hat and followed the types and the createUser signature from the article is:
export async function createUser(user: CreateUser, ctx: Context) {
  if (user.acceptTermsAndConditions) {
    return await ctx.prisma.user.create({
      data: user,
    })
  } else {
    return new Error('User must accept terms!')
  }
}

So it does expect only the context object and not an instance of PrismaClient, so all changes should be correct. Thanks so much for catching this @naor88 🙏

Thanks so much @naor88 !!

Happy to help! :)


Fix bundle relocation override

not sure if using the same branch again messes with the merge, but this should be much nicer. Tested and tested and tested again.

define+export is very nice, didn't know about that.


jetbrains: update

Used ./update.pl, no other changes.
Tested by building and running a Scala project in idea-community

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

CLion works fine :+1:

Goland flat out crashes with a SIGSEGV on my machine:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8252561692, pid=12900, tid=12901
#
# JRE version: OpenJDK Runtime Environment (11.0.7) (build 11.0.7+0-adhoc..source)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+0-adhoc..source, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x82a692]  InstanceKlass::initialize_impl(Thread*)+0x492
#
# Core dump will be written. Default location: Core dumps may be processed with "/nix/store/vac1gmzh1xmk3s7w9pbjvirxqsg1npn0-systemd-243.7/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/konrad/www/nixpkgs/core.12900)
#
# An error report file with more information is saved as:
# /home/konrad/java_error_in_GOLAND_12900.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
/nix/store/d2zaxh9z4v1bsv551dqs088dn1ckvb85-goland-2020.2.2/goland-2020.2.2/bin/goland.sh: line 229: 12900 Aborted                 (core dumped) "$JAVA_BIN" -classpath "$CLASSPATH" ${VM_OPTIONS} "-XX:ErrorFile=$HOME/java_error_in_GOLAND_%p.log" "-XX:HeapDumpPath=$HOME/java_error_in_GOLAND.hprof" -Didea.vendor.name=${PRODUCT_VENDOR} -Didea.paths.selector=${PATHS_SELECTOR} "-Djb.vmOptionsFile=$VM_OPTIONS_FILE" ${IDE_PROPERTIES_PROPERTY} -Didea.platform.prefix=GoLand com.intellij.idea.Main "$@"

I've been seeing this lately with the current version as well so it might not be related to the update itself but rather some issue in my setup.

On another note, there have also been updates to phpstorm, would you mind updating that as well in this PR or should i open another one?

there have also been updates to phpstorm

AFAICS the update script updates all components - I guess that phpstorm update wasn't available yet when I opened this PR 11 days ago?

would you mind updating that as well in this PR or should i open another one?

I think the easiest way forward is if you create a new PR (that would then supersede this one).

AFAICS the update script updates all components - I guess that phpstorm update wasn't available yet when I opened this PR 11 days ago?

I think so, yes.

I think the easiest way forward is if you create a new PR (that would then supersede this one).

Is up: #97359

closing this in favour of #97359 then


[BUG] MaterialUI Themed not apply for imports from @mui

Describe the bug
I'm using refine and I also have imports from @mui. But the theme I set from refine header (dark/light mode) is not apply to the components imported from @mui, only to the components build in refine.
Steps To Reproduce

Install a refine project
Import a component from @mui
Change the theme

Expected behavior
The theme is applied to components imported from @mui.
Screenshot
No response
Desktop
No response
Mobile
No response
Additional Context
No response

Hey @MetaCorp, can you elaborate a bit more and provide a repro or a guide to repro? 🤔 I've just created a Material UI + CRA app with create-refine-app and added a button from @mui/material.
<Button variant="contained">Contained Button</Button>

When I try to change the theme, button reflect the changes in the theme 🤖

Closing this for now. Do not hesitate to re-open when you can provide a repro 🙏


-m"Microsoft SQL Server Management Studio - Query".

On SSCM, try to add -m"Microsoft SQL Server Management Studio - Query", received "The parameter is invalid"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5a9b38b8-0f80-6a0f-9b65-a138ef4b22e6
Version Independent ID: 4165ecb2-9e9b-bf79-6a23-6eb10b898462
Content: Start SQL Server in Single-User Mode - SQL Server
Content Source: docs/database-engine/configure-windows/start-sql-server-in-single-user-mode.md
Product: sql
Technology: configuration
GitHub Login: @MikeRayMSFT
Microsoft Alias: mikeray

@wzcgh2019 -- thank you for your feedback.  Please provide more information & context about your general task.  Meanwhile, you may find answers here:

DBA Stack Exchange
Stack Overflow

I am following this document to start up sql server in single mode
https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/start-sql-server-in-single-user-mode?view=sql-server-ver15
You know by adding -m to the "Startup Parameters" of sql server configure manager and then restart the sql server instance is the way to make the instance to be single user mode.
According to the document, you can use -m"Microsoft SQL Server Management Studio - Query" as well. However, the problem is when you put that line into SSCM and click "add", it errors out with a pop window saying "The parameter is invalid".

@wzcgh2019 -- thank you for clarifying.
@MikeRayMSFT -- please look into this issue.

I am also not able to put the -m"Microsoft SQL Server Management Studio - Query" in the SQL Server Configuration startup parameters. I'm going to check a few things before I close the issue. Some configuration managers behave a little differently.
What you can do is start the command with net start "SQL Server (MSSQLSERVER)" -m"Microsoft SQL Server Management Studio - Query" to get the effect you want.
Update the string for a named instance, of course.


How to Open Smart Object;

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. Windows10, Windows7]
Photoshop Version: [e.g. Photoshop-2020]
Python Version: [e.g. python-3.6]

Additional context
Add any other context about the problem here.

How to open a smart object in my PSD？

    def openSmartObject(self):         a = ps.Application().stringIDToTypeID("placedLayerEditContents")         d = ps.ActionDescriptor()         ps.Application().executeAction(a, d)
hello i Solved this problem on my own

Thank you very much. I have succeeded in using your plan. Thank you.

    def openSmartObject(self):         a = ps.Application().stringIDToTypeID("placedLayerEditContents")         d = ps.ActionDescriptor()         ps.Application().executeAction(a, d)
hello i Solved this problem on my own

前辈你这个问题帮助了我很多，我还有一些问题想请教，我现在想在你的解决方法的基础上，替换新打开的智能对象中文字，我现在打开智能对象后就不知道该如何去操作了，情不能指教一下，非常感谢！
import photoshop.api as ps
App = ps.Application()
doc = App.load('*.psd')
a = App.stringIDToTypeID("placeLayerEditContents")
b = ps.ActionDscriptor()
App.executeAction(a,b)
now I open the smart object,  but i have no idea to change the text in the opened smart object, how to seleted the opened smart object layer,? And how to change the text in the opened smart object layer  ?

Mhh.. that is not working for me. Photoshop complains (translated):
The command "" is not available"
Any ideas?