Raw JSON Layer Design Proposal

Raw JSON Layer design
Why
Currently, on Tool Layer, for each plugin, we read data from API, extract and transform fields we need and store into predefined tables, this approach poses some limitations:

data lost: those fields that we didn't extract got throwed away.
data distortion: some fields got transformed to type or format that not exactly ideal.
development/test: in order to develop/test our collection logic, we have to manually run code against a real endpoint, and the responses from api endpoint varies from time to time.

which leads to following difficulties:

User: whenever a user set a wrong parameter for a collection task, he/she have to recollect data from api all over again, which is time consuming.
Test: no stable fake data for automated test, and QA verification. it's hard to setup and maintain a mock up deployment for each and every plugin.
Development: it's unnecessary to actually fetch data from api when we are just developing some simple transformation logic.

How
In order to overcome these drawbacks, most of us had reached some agreements:

A raw json layer must be added before transformation to preserve the full information from api responses.
This layer should be managed by framework, plugin can saves/loads their raw json by using functions or interfaces provided by framework, plugin should not access raw json tables directly.
Each tool layer entity has its own respected raw json table, the relationship between them is OneToOne.

What
I propose the following solution:

pseudocode for jira issue collector
func CollectJiraIssues(apiClient *JiraApiClient, source *JiraSource, boardId int) error {
    rawJiraIssueTable := RawDataTable(&RawJiraIssue{})
    rawJiraBoardIssueTable := RawDataTable(&RawJiraBoardIssue{})

    return apiClient.FetchPages(func(res *http.Response) error {
        if res.StatusCode != 200 {
            return fmt.Errorf("unexecpted status code")
        }
        jiraApiIssuesResponse := &JiraApiIssuesResponse{}
        err := core.UnmarshalResponse(res, jiraApiIssuesResponse)
        if err != nil {
            return err
        }
        // process issues
        for _, jiraApiIssue := range jiraApiIssuesResponse.Issues {
            err = rawJiraIssueTable.Store(&RawJiraIssue{
                SourceId: source.ID,
                IssueId: jiraApiIssue.Id,
                RawData: jiraApiIssue,
            })
            if err != nil {
                return err
            }
            rawJiraBoardIssueTable.Store(&RawJiraBoardIssue{
                SourceId: source.ID,
                BoardId: boardId,
                IssueId: jiraApiIssue.Id,
            })
            if err != nil {
                return err
            }
        }
    })
}

which store data into raw_jira_issues table like following:

source_id
issue_id
raw_json
created_at
updated_at

1
1067629368
{"id":<PHONE_NUMBER>, "self": "...", "fields": ...}
2022-01-26 09:52:17.697000000
2022-01-26 09:52:17.697000000

1
1009381716
{"id":<PHONE_NUMBER>, "self": "...", "fields": ...}
2022-01-26 09:52:17.697000000
2022-01-26 09:52:17.697000000

pseudocode for jira_issue_transformer
func TransformJiraIssue(source *JiraSource, boardId int) error {
    rawJiraIssueTable := RawDataTable(&RawJiraIssue{})
    return rawJiraIssueTable.Iterate(func(rawJiraIssue *RawJiraIssue) error {
        // extract information from rawJiraIssue and create/update JiraIssue accordingly
        return nil
    })
}

By this approach, we can solve mentioned difficulties:

Users no long have to re-collect data from api if they set a wrong customfield
We can collect and copy those raw json tables for testing and QA verification, whenever a new type of data source came in, we just collect and added it to our copying list, no need to manually fake any data.
Speed up our new fields collection and transformation logic debug.

I have some suggestions about abstracting the process from plugins. By defining  3 interfaces Paginator, RawJsonExtractor and Store, we could fouce on the implementation of these interfaces.
type Paginator interface {
	HasNext() bool
	GetNext() (*http.Response, error)
}

type RawJson struct {
	Id        string
	Blob      []byte
	CreatedAt time.Time
}
type RawJsonExtractor interface {
	Extract(resp *http.Response) ([]*RawJson, error)
}
type Store interface {
	Save(context.Context, []*RawJson) error
}

func Collect(ctx context.Context, paginator Paginator, extractor RawJsonExtractor, store Store) error{
	for paginator.HasNext() {
		select {
		case <-ctx.Done():
			return ctx.Err()
		default:
		}
		resp, err := paginator.GetNext()
		if err != nil {
			return err
		}
		objs, err := extractor.Extract(resp)
		if err != nil {
			return err
		}
		err = store.Save(ctx, objs)
		if err != nil {
			return err
		}
	}
	return nil
}

@mindlesscloud good idea, how to address board_issues relationship with this approach? this relationship can not be found anywhere inside the raw response, but merely from parameter that user passed in.

In my opinion, we can keep only sub task flow in framework, and put the interfaces(or say flow of specific sub task) of one sub task in plugins layer. We still have a lot of data sources to integrate, and in large probability, there are lots of differences in realizing a sub task among multiple datasources. @mindlesscloud 's proposal can restrain the code structure by both inside and external developers, so it will be easier to maintain by us in the future. And in framework, we can pay more attention on how to manage sub tasks.

BTW, I still think it may bring too many tables if we use one table to map a kind of entity of one datasource.  For current datasources, in tool layer, if we store raw json for notes, comments, events separately, it will bring about 40 more tables, almost one datasource need 20 tables in total. With 50 more datasources, we may need hundreds of tables. In general, I think we'd better keep raw_json layer as simple as possible, and do more work when store the data.

In my opinion, we can keep only sub task flow in framework, and put the interfaces(or say flow of specific sub task) of one sub task in plugins layer. We still have a lot of data sources to integrate, and in large probability, there are lots of differences in realizing a sub task among multiple datasources. @mindlesscloud 's proposal can restrain the code structure by both inside and external developers, so it will be easier to maintain by us in the future. And in framework, we can pay more attention on how to manage sub tasks.

This can be true if and only if the interface is complete, so it is stable and never need to be changed. which is quite like the concept of zero friction in physis, no so practical in reality. So, Whenever we change the interface a little bit, it breaks existing 3rd plugins.
On the other hand, the library approach has no such issue, because code got compiled into plugin and run on its own, plugin developer can decide when to upgrade their code, and users can still use existing outdated plugins as they see fit.
As always, I stand against using interface as a norm, check the git history of https://github.com/merico-dev/lake/blame/main/plugins/core/plugin.go this simple interface has been modifed couple of times. This would not be ok if we had some impactful 3rd plugins in market.

BTW, I still think it may bring too many tables if we use one table to map a kind of entity of one datasource. For current datasources, in tool layer, if we store raw json for notes, comments, events separately, it will bring about 40 more tables, almost one datasource need 20 tables in total. With 50 more datasources, we may need hundreds of tables. In general, I think we'd better keep raw_json layer as simple as possible, and do more work when store the data.

Can you post the detail of your solution to this matter? What do you suggest we do in order to keep it simple?

New Design  2022/2/25
Summary of opinions

Be more general
Single responsibility principle
Framework in control, declarative plugin task approach
Future oriented, consider tasks management, multi api version compatibility

func CollectApiIssues(
	apiClient *JiraApiClient,
	source *models.JiraSource,
	boardId uint64,
	since time.Time,
) error {
	// --------- skip this part begin --------
	// user didn't specify a time range to sync, try load from database
	if since.IsZero() {
		var latestUpdated models.JiraIssue
		err := lakeModels.Db.Where("source_id = ?", source.ID).Order("updated DESC").Limit(1).Find(&latestUpdated).Error
		if err != nil {
			logger.Error("jira collect issues:  get last sync time failed", err)
			return err
		}
		since = latestUpdated.Updated
	}
	// build jql
	jql := "ORDER BY updated ASC"
	if !since.IsZero() {
		// prepend a time range criteria if `since` was specified, either by user or from database
		jql = fmt.Sprintf("updated >= '%v' %v", since.Format("2006/01/02 15:04"), jql)
	}
	query := &url.Values{}
	query.Set("jql", jql)
	// --------- skip this part end --------

	const SIZE = 100

	collector, err := core.NewApiCollector(core.ApiCollectorArgs{
		ApiClient: apiClient,
		PageSize:  SIZE,
		/*
			url may use arbitrary variables from different source in any order, we need GoTemplate to allow more
			flexible for all kinds of possibility.
			Pager contains information for a particular page, calculated by ApiCollector, and will be passed into
			GoTemplate to generate a url for that page.
			We want to do page-fetching in ApiCollector, because the logic are highly similar, by doing so, we can
			avoid duplicate logic for every tasks, and when we have a better idea like improving performance, we can
			do it in one place
		*/
		UrlTemplate: "agile/1.0/board/{{ Params.BoardId }}/issue?startAt={{ Pager.StartAt }}&maxResults={{ Pager.Size }}",
		/*
			Some api might do pagination by http headers
		*/
		//Header: func(collector *core.ApiCollector) http.Header {
		//},
		/*
			Sometimes, we need to collect data based on previous collected data, like jira changelog, it requires
			issue_id as part of the url.
			We can mimic `stdin` design, to accept a `Input` function which produces a `Iterator`, collector
			should iterate all records, and do data-fetching for each on, either in parallel or sequential order
			UrlTemplate: "api/3/issue/{{ Input.ID }}/changelog"
		*/
		//Input: databaseIssuesIterator,
		/*
			This struct will be JSONEncoded and stored into database along with raw data itself, to identity minimal
			set of data to be process, for example, we process JiraIssues by Board
		*/
		Params: JiraApiParams{
			SourceId: source.ID,
			BoardId:  boardId,
		},
		/*
			Reduce records come out from api
		*/
		Query: query,
		/*
			ApiCollector will try to `UnmarshalResponse` into this variable, and handle failure
		*/
		ResponseData: &JiraApiIssuesResponse{},
		/*
			Accept response and Unmarshalled body, return raw data either a single object or list
		*/
		OnData: func(res *http.Response, body interface{}) (interface{}, error) {
			issuesBody := body.(*JiraApiIssuesResponse)
			return issuesBody.Issues, nil
		},
		/*
			For api endpoint that returns number of total pages, ApiCollector can collect pages in parallel with ease,
			or other techniques are required if this information was missing.
		*/
		GetTotalPages: func(res *http.Response, body interface{}) (int, error) {
			issuesBody := body.(*JiraApiIssuesResponse)
			return issuesBody.Total / SIZE, nil
		},
		/*
			Table store raw data
		*/
		Table: "jira_api_issues",
	})

	if err != nil {
		return err
	}

	return collector.Execute()
}

@hezyin

@klesh This is definitely going in the right direction. Let me quickly summarize my understanding and questions:
Understanding of current design

ApiClient used to be responsible for handling pagination and parallelizing requests. It seems in the new design both responsibilities are extracted out of ApiClient and assigned to ApiCollector.
How this works is that we tell ApiCollector what API to query, what parameters to use, how to handle pagination (if any) and ApiCollector will take care of the collection process and store API responses in a database.

Questions for current design

Does Query get appended to UrlTemplate? If so, why keep them separate?
Why do we unmarshal API response into a struct? Is this for data validation purposes? I'm thinking that ResponseData and OnData may belong somewhere else or at least we need some more explanation/justification.
For pagination, what happens if API doesn't return total pages, do we fall back to get next page strategy or something else?

@klesh This is definitely going in the right direction. Let me quickly summarize my understanding and questions:
Understanding of current design

ApiClient used to be responsible for handling pagination and parallelizing requests. It seems in the new design both responsibilities are extracted out of ApiClient and assigned to ApiCollector.
How this works is that we tell ApiCollector what API to query, what parameters to use, how to handle pagination (if any) and ApiCollector will take care of the collection process and store API responses in a database.

Questions for current design

Does Query get appended to UrlTemplate? If so, why keep them separate?
Why do we unmarshal API response into a struct? Is this for data validation purposes? I'm thinking that ResponseData and OnData may belong somewhere else or at least we need some more explanation/justification.
For pagination, what happens if API doesn't return total pages, do we fall back to get next page strategy or something else?

Yes, you got it right.
Answers to Questions:

Yes, it does. Query is separated because it is optional, it is easier for plugin task to pass down query string if any, we can then do staff like omitting the query key altogether if value was missing.
Validation and error handling, unmarshal logic is needed for pretty much every collection task, by doing so, we can do it in one place to avoid error-swallowing, and provide a unity error interace for users.
Yes, maybe page by page or step concurrency technique, haven't quite decided yet, but we can keep improve the strategy in ApiCollector without touching plugin tasks.

@klesh

Whether we separate query string with URL or not is a minor design choice so I don't have a strong preference. I'm leaning towards keeping them together since most web crawling frameworks I've used (e.g., scrapy in python) only ask for a single URL and it's the user's responsibility to build that URL.
If we agree on the following responsibility definition for ApiCollector, then I think we should delay the unmarshal logic to a later point, possibly when developers process the stored API response. ApiCollector is only concerned with where to send API requests (URL), how to handle pagination and parallelism, and checking that we're getting HTTP 200 for those API requests. ApiCollector should be oblivious to the internal data structure.

How this works is that we tell ApiCollector what API to query, what parameters to use, how to handle pagination (if any) and ApiCollector will take care of the collection process and store API responses in a database.

Ok, I concur.

Updated design:

Query function is optional, developer can plug them into UrlTemplate directly, or use this function when url is getting too long to maintain
ApiCollector is not doing unmarshal now

func CollectApiIssues(
	apiClient *JiraApiClient,
	source *models.JiraSource,
	boardId uint64,
	since time.Time,
) error {
	// --------- skip this part begin --------
	// user didn't specify a time range to sync, try load from database
	if since.IsZero() {
		var latestUpdated models.JiraIssue
		err := lakeModels.Db.Where("source_id = ?", source.ID).Order("updated DESC").Limit(1).Find(&latestUpdated).Error
		if err != nil {
			logger.Error("jira collect issues:  get last sync time failed", err)
			return err
		}
		since = latestUpdated.Updated
	}
	// build jql
	jql := "ORDER BY updated ASC"
	if !since.IsZero() {
		// prepend a time range criteria if `since` was specified, either by user or from database
		jql = fmt.Sprintf("updated >= '%v' %v", since.Format("2006/01/02 15:04"), jql)
	}
	// --------- skip this part end --------

	const SIZE = 100

	collector, err := core.NewApiCollector(core.ApiCollectorArgs{
		ApiClient: apiClient,
		PageSize:  SIZE,
		/*
			url may use arbitrary variables from different source in any order, we need GoTemplate to allow more
			flexible for all kinds of possibility.
			Pager contains information for a particular page, calculated by ApiCollector, and will be passed into
			GoTemplate to generate a url for that page.
			We want to do page-fetching in ApiCollector, because the logic are highly similar, by doing so, we can
			avoid duplicate logic for every tasks, and when we have a better idea like improving performance, we can
			do it in one place
		*/
		UrlTemplate: "agile/1.0/board/{{ Params.BoardId }}/issue",
		/*
			(Optional) Return query string for request, or you can plug them into UrlTemplate directly
		*/
		Query: func(pager *core.Pager) (*url.Values, error) {
			query := &url.Values{}
			query.Set("jql", jql)
			query.Set("startAt", fmt.Sprintf("%v", pager.Skip))
			query.Set("maxResults", fmt.Sprintf("%v", pager.Size))
			return query, nil
		},
		/*
			Some api might do pagination by http headers
		*/
		//Header: func(pager *core.Pager) http.Header {
		//},
		/*
			Sometimes, we need to collect data based on previous collected data, like jira changelog, it requires
			issue_id as part of the url.
			We can mimic `stdin` design, to accept a `Input` function which produces a `Iterator`, collector
			should iterate all records, and do data-fetching for each on, either in parallel or sequential order
			UrlTemplate: "api/3/issue/{{ Input.ID }}/changelog"
		*/
		//Input: databaseIssuesIterator,
		/*
			This struct will be JSONEncoded and stored into database along with raw data itself, to identity minimal
			set of data to be process, for example, we process JiraIssues by Board
		*/
		Params: JiraApiParams{
			SourceId: source.ID,
			BoardId:  boardId,
		},
		/*
			Accept response, return raw data either a single object or list
		*/
		OnData: func(res *http.Response) (interface{}, error) {
			issuesBody := &JiraApiIssuesResponse{}
			err := core.UnmarshalResponse(res, issuesBody)
			if err != nil {
				return nil, err
			}
			return issuesBody.Issues, nil
		},
		/*
			For api endpoint that returns number of total pages, ApiCollector can collect pages in parallel with ease,
			or other techniques are required if this information was missing.
		*/
		GetTotalPages: func(res *http.Response) (int, error) {
			issuesBody := &JiraApiIssuesResponse{}
			err := core.UnmarshalResponse(res, issuesBody)
			if err != nil {
				return 0, err
			}
			return issuesBody.Total / SIZE, nil
		},
		/*
			Table store raw data
		*/
		Table: "jira_api_issues",
	})

	if err != nil {
		return err
	}

	return collector.Execute()
}

Extractor Example
func ExtractApiIssues(
	source *models.JiraSource,
	boardId uint64,
	since time.Time,
) error {
	extractor, err := core.NewApiExtractor(core.ApiExtractorArgs{
		Table:       "jira_api_issues",
		Params: JiraApiParams{
			SourceId: source.ID,
			BoardId:  boardId,
		},
		RowData: &JiraApiIssue{},
		Extractors: []core.RawDataExtractor{
			func(row interface{}, params interface{}) (interface{}, error) {
				p := params.(*JiraApiParams)
				issue := row.(*JiraApiIssue)
				issueId, err := strconv.ParseUint(issue.Id, 10, 64)
				if err != nil {
					return nil, err
				}
				return &models.JiraIssue{
					SourceId: p.SourceId,
					IssueId:  issueId,
					// other fields
				}, nil
			},
			func(row interface{}, params interface{}) (interface{}, error) {
				p := params.(*JiraApiParams)
				issue := row.(*JiraApiIssue)
				issueId, err := strconv.ParseUint(issue.Id, 10, 64)
				if err != nil {
					return nil, err
				}
				return &models.JiraBoardIssue{
					SourceId: p.SourceId,
					BoardId:  p.BoardId,
					IssueId:  issueId,
				}, nil
			},
		},
	})

	if err != nil {
		return err
	}

	return extractor.Execute()
}

Hi, @hezyin , I ran into some difficulty when trying to implement fetch pages in parallel
Check the THIS WONT WORK part, res.Body is a stream which can only be read once.
func (collector *ApiCollector) fetchPagesAsync() error {
	// make sure table is created
	db := collector.args.Ctx.GetDb()
	err := db.Exec(GetRawTableCreationSql(collector.table)).Error
	if err != nil {
		return err
	}
	// flush old data
	err = db.Exec(GetRawTableDeletionSql(collector.table), collector.params).Error
	if err != nil {
		return err
	}

	if collector.args.GetTotalPages != nil {
		/* when total pages is available from api*/
		// fetch the very first page
		err := collector.fetchAsync(nil, func(res *http.Response) error {
			// gather total pages
			totalPages, err := collector.args.GetTotalPages(res)
			if err != nil {
				return err
			}
			// handle response body of first page
			err = collector.handleResponse(res) // THIS WONT WORK: because res.Body already got read in GetTotalPages
			if err != nil {
				return err
			}
			// fetch other pages in parallel
			for page := 2; page <= totalPages; page++ {
				err = collector.fetchAsync(&Pager{
					Page: page,
					Size: collector.args.PageSize,
					Skip: collector.args.PageSize * (page - 1),
				}, collector.handleResponse)
				if err != nil {
					return err
				}
			}
			return nil
		})
		if err != nil {
			return err
		}

	} else {
		/* when total pages is available from api*/
		// fetch page by page in sequential?
		// use step currency technique? fetch like 10 pages at once, if all went well, fetch next 10 pages?
		panic("not implmented")
	}
	return nil
}

@klesh

In order to read twice from resp.body, we can construct a new io.ReadCloser after reading once and hand that to the next reader, see the following example from here:

bodyBytes, _ := ioutil.ReadAll(req.Body)
req.Body.Close()  //  must close
req.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))

resp.body needs to be closed after reading to avoid any resource leakage. Please check that we're properly closing them now.
It seems we only removed OnResponse from the ApiCollector interface but OnData is still there. Shall we also move OnData to somewhere else?
Now that we have made some progress with these examples, I suggest that we spend some time writing the documentation for the raw data layer, clearly explaining the interface and responsibilities of ApiCollector and ApiExtractor, and including an example usage. Note that these don't include implementation description, they're external-facing docs for developers wishing to implement their own ApiCollector and ApiExtractor.

@hezyin

I see, we can make do with that. But, still, it will get unmarshal twice, do we really need to do it this way?
core.UnmarshalResponse closes it properly
I dont understand
Will do

closed by #1287


Remove EI analytics docs

Purpose
Fixes https://github.com/wso2/docs-mi/issues/357

Let's reword

https://mi.docs.wso2.com/en/latest/mi-analytics/mi-elk-installation-guide/

Let's reword

https://mi.docs.wso2.com/en/latest/mi-analytics/mi-elk-installation-guide/

Done with 345f4b5


[SDP-913] stellar-multitenant: add-tenants cmd

What
This PR adds a new CLI command in the stellar-multitenant package. This CLI is responsible for registering new tenants on the main database schema. The tenant name is unique.
mtn add-tenants myorg

Why
We should be able to register new tenants.
Known limitations
This process only registers the new tenant in the main database schema but we are missing creating the database schema and configuring the basics for it.
Checklist
PR Structure

[x] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR does not mix refactoring changes with feature changes (split into two PRs otherwise).
[x] This PR's title starts with the name of the package, area, or subject affected by the change.

Thoroughness

[x] This PR adds tests for the new functionality or fixes.
[x] This PR contains the link to the Jira ticket it addresses.

Configs and Secrets

[x] No new CONFIG variables are required -OR- the new required ones were added to the helmchart's values.yaml file.
[x] No new CONFIG variables are required -OR- the new required ones were added to the deployments (pr-preview, dev, demo, prd).
[x] No new SECRETS variables are required -OR- the new required ones were mentioned in the helmchart's values.yaml file.
[x] No new SECRETS variables are required -OR- the new required ones were added to the deployments (pr-preview secrets, dev secrets, demo secrets, prd secrets).

Release

[ ] This is not a breaking change.
[ ] This is ready for production.. If your PR is not ready for production, please consider opening additional complementary PRs using this one as the base. Only merge this into develop or main after it's ready for production!

Deployment

[ ] Does the deployment work after merging?

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr76.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr76.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr76.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr76.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr76.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr76.previews.kube001.services.stellar-ops.com


Prelude module

Would be nice to export main stuff, like http://hackage.haskell.org/package/diagrams-lib-<IP_ADDRESS>/docs/Diagrams-Prelude.html

Deferring for now


MIMIR-780 save data from statbank saved tables to repo

Add fetch data functions for statbank saved
Change datasource input to support url or id
Add request for html table
Save html and json format of result
MIMIR-780

Thanks for feedback!


enh(tables): improve support for comma-separated lists

Description
This change to the tables query dust app (https://dust.tt/w/78bda07b39/a/b4f205e453) updates the prompt to help the model better handle comma-separated lists.
Risk
very limited blast radius
Deploy Plan
simple front deploy

@spolu PMRW


[SPARK-22600][SQL] Fix 64kb limit for deeply nested expressions under wholestage codegen

What changes were proposed in this pull request?
SPARK-22543 fixes the 64kb compile error for deeply nested expression for non-wholestage codegen. This PR extends it to support wholestage codegen.
This patch brings some util methods in to extract necessary parameters for an expression if it is split to a function.
The util methods are put in object ExpressionCodegen under codegen. The main entry is getExpressionInputParams which returns all necessary parameters to evaluate the given expression in a split function.
This util methods can be used to split expressions too. This is a TODO item later.
How was this patch tested?
Added test.

This patch needs #19800 to fix another issue in codegen.

Test build #84158 has started for PR 19813 at commit 1cf6a48.

Test build #84158 has finished for PR 19813 at commit 1cf6a48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84158/
Test FAILed.

Merged build finished. Test FAILed.

We need to clearly define what is the current input according to the codegen context. For normal code path, it's always ctx.INPUT_ROW, which means when we split codes to methods, we just need to pass InternalRow ctx.INPUT_ROW to those methods.
However for whole stage codegen path, it's way more complex:

some of ctx.currentVars are just variables, their codes have already been generated before. But some are not. For those whose codes are not generated, they are not valid inputs.
ctx.currentVars is not null but has null slots, and ctx.INPUT_ROW is not null. Then both ctx.currentVars and ctx.INPUT_ROW are valid inputs.

Test build #84182 has started for PR 19813 at commit 65d07d5.

However for whole stage codegen path, it's way more complex:

some of ctx.currentVars are just variables, their codes have already been generated before. But some are not. For those whose codes are not generated, they are not valid inputs.
ctx.currentVars is not null but has null slots, and ctx.INPUT_ROW is not null. Then both ctx.currentVars and ctx.INPUT_ROW are valid inputs.

Yes, this is correct.
So, for 1, only the variables not evaluate yet, we don't include them as parameters.
For 2, null slots in ctx.currentVars won't be included as parameters too. ctx.INPUT_ROW will be included only if it is not null.

Test build #84182 has finished for PR 19813 at commit 65d07d5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84182/
Test FAILed.

If we have a clear rule, I think it makes more sense to do this in CodegenContext, i.e. having a def splitExpressions(expressions: Seq[String]): String, which automatically extract the current inputs and put them into the parameter list.

Test build #84207 has started for PR 19813 at commit 9f848be.

Test build #84207 has finished for PR 19813 at commit 9f848be.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExprCode(
case class ExprInputVar(val expr: Expression, val exprCode: ExprCode)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84207/
Test FAILed.

Merged build finished. Test FAILed.

retest this please.

Test build #84208 has started for PR 19813 at commit 9f848be.

@cloud-fan @kiszk ctx.currentVars and ctx.INPUT_ROW are not the only sources for expression evaluation under wholestage codegen. There are also eliminated subexpressions and the input rows and variables referred by deferred expressions.
In an API like def splitExpressions(expressions: Seq[String]): String, seems to me those sources are not easily to access.

Test build #84211 has started for PR 19813 at commit 57b1add.

Test build #84208 has finished for PR 19813 at commit 9f848be.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExprCode(
case class ExprInputVar(val expr: Expression, val exprCode: ExprCode)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84208/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84211/
Test FAILed.

splitExpressions is the most common way we use in the codegen framework to deal with large code. If we can't make it work with whole stage codegen, we are not making many values.

A more advanced version of splitExpressions may work. We can provide necessary function parameters to it.

BTW splitExpressions doesn't work with subexpressions since the beinning, it's another topic to integrate them.

Test build #84239 has started for PR 19813 at commit d051f9e.

Test build #84241 has started for PR 19813 at commit 6368702.

Test build #84239 has finished for PR 19813 at commit d051f9e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84239/
Test FAILed.

Test build #84241 has finished for PR 19813 at commit 6368702.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84241/
Test FAILed.

Merged build finished. Test FAILed.

Test build #84260 has started for PR 19813 at commit 8c7f749.

Test build #84260 has finished for PR 19813 at commit 8c7f749.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84260/
Test FAILed.

retest this please.

Test build #84275 has started for PR 19813 at commit 8c7f749.

Test build #84275 has finished for PR 19813 at commit 8c7f749.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84275/
Test FAILed.

Test build #84303 has started for PR 19813 at commit 7f00515.

Test build #84303 has finished for PR 19813 at commit 7f00515.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84303/
Test PASSed.

Merged build finished. Test PASSed.

Test build #84323 has started for PR 19813 at commit 777eb7a.

Test build #84327 has started for PR 19813 at commit 7230997.

Now the way to extract necessary parameters is moved out from Expression. We can use it to prepare parameters for an API such as splitExpressions.

Test build #84323 has finished for PR 19813 at commit 777eb7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84323/
Test PASSed.

Test build #84327 has finished for PR 19813 at commit 7230997.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84327/
Test PASSed.

Test build #84330 has started for PR 19813 at commit fd87e9b.

cc @cloud-fan @kiszk @maropu @mgaido91 for review. Thanks.

Test build #84331 has started for PR 19813 at commit 57a9fb7.

Test build #84330 has finished for PR 19813 at commit fd87e9b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ExpressionCodegenSuite extends SparkFunSuite

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84330/
Test FAILed.

Test build #84331 has finished for PR 19813 at commit 57a9fb7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84331/
Test PASSed.

Merged build finished. Test PASSed.

@mgaido91 Thanks for the review.

Test build #84367 has started for PR 19813 at commit 0d358d6.

Test build #84370 has started for PR 19813 at commit 98850a6.

Test build #84371 has started for PR 19813 at commit aa3db2e.

Test build #84367 has finished for PR 19813 at commit 0d358d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84367/
Test PASSed.

Merged build finished. Test PASSed.

Test build #84370 has finished for PR 19813 at commit 98850a6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84370/
Test FAILed.

Test build #84371 has finished for PR 19813 at commit aa3db2e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84371/
Test PASSed.

thanks @viirya , LGTM

ping @cloud-fan @kiszk Do you have more review on this? Thanks.

Test build #84422 has started for PR 19813 at commit 429afba.

Test build #84422 has finished for PR 19813 at commit 429afba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84422/
Test PASSed.

Test build #84486 has finished for PR 19813 at commit 48add65.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExprInputVar(expr: Expression, exprCode: ExprCode)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84486/
Test PASSed.

ping @kiszk @cloud-fan any more thoughts or review for this?

LGTM, good job!

Test build #84641 has started for PR 19813 at commit 7fc3e04.

Test build #84642 has started for PR 19813 at commit 9443011.

Test build #84642 has finished for PR 19813 at commit 9443011.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExprInputVar(exprCode: ExprCode, dataType: DataType, nullable: Boolean)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84642/
Test PASSed.

Test build #84641 has finished for PR 19813 at commit 7fc3e04.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExprInputVar(exprCode: ExprCode, dataType: DataType, nullable: Boolean)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84641/
Test PASSed.

Merged build finished. Test PASSed.

ping @cloud-fan Previous comments are all addressed. Please review this again. Thanks.

Test build #84710 has started for PR 19813 at commit 2f4014f.

Test build #84710 has finished for PR 19813 at commit 2f4014f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84710/
Test FAILed.

Merged build finished. Test FAILed.

Test build #84741 has started for PR 19813 at commit 655917c.

Test build #84742 has started for PR 19813 at commit c083a79.

Test build #84741 has finished for PR 19813 at commit 655917c.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84741/
Test FAILed.

Test build #84742 has finished for PR 19813 at commit c083a79.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84742/
Test FAILed.

Test build #84756 has finished for PR 19813 at commit 1251dfa.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84756/
Test FAILed.

Test build #84760 has started for PR 19813 at commit c4f15f7.

Test build #84760 has finished for PR 19813 at commit c4f15f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84760/
Test PASSed.

Merged build finished. Test PASSed.

Test build #84769 has started for PR 19813 at commit f35974e.

Test build #84769 has finished for PR 19813 at commit f35974e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84769/
Test PASSed.

thanks, merging to master!

This needs a design doc.

After some more thoughts, this PR makes a new contract that Spark doesn't promise before: Expression.genCode must output something that can be used as parameter name or literal.
I do remember in some places we just output statement like a + 1 for codegen, but I could be wrong. At least we need to check all the places and document this new contract before merging this PR.
Another solution is to not make this contract. By a quick look this seems hard to do, because at the time of doing this, the code(method body) is already generated and we don't know how to replace statement like a + 1 with the generated parameter name, inside the method body. We may need to do this fix earlier in the codegen procedure.
I'm going to revert it, let's have a proper design doc and resubmit this. Sorry for the inconvenient!

@cloud-fan Ok. Thanks.

Also needs to revert #19969 which is based on this.

@cloud-fan Do you think we should still allow something like a + 1 as the output of codegen?

I'm not sure, it depends on how many places we already doing it and what's the drawback if we forbid it. Let's have a design doc and gather more feedbacks.
Thanks for your understanding! both PRs are reverted.

I wrote a design doc at https://paper.dropbox.com/doc/Split-deeply-nested-expressions-under-wholestage-codegen-WXkQ9iIlN3zkGdn8MHgiW
The design is based on what did in this PR.
Please give me feedbacks if you have time to go through it. Thank you.
cc @cloud-fan @kiszk @mgaido91 @maropu @gatorsmile

can we have a google doc so that we can leave comments inline? thanks!

Probably, we'd better to move this discussion to the jira?

can we have a google doc so that we can leave comments inline? thanks!

Sure. The google doc is at https://docs.google.com/document/d/1By_V-A2sxCWbP7dZ5EzHIuMSe8K0fQL9lqovGWXnsfs/edit?usp=sharing

Probably, we'd better to move this discussion to the jira?

I summary it and also post the design doc link to the jira.

I'm wondering if an output statement like a + 1 is a problem.
If we only include codegen output with valid java variable names, an output like a + 1 won't be included as parameter. We only need to make sure the creation of ExprCode in genCode has correctly linked with its input row and variables (this is done in this PR). So an output like a + 1 depends on other input a. This a will be included as parameter if necessary.
cc @cloud-fan @kiszk @mgaido91 WDYT? Or do I miss something?

Note: One possible problem is in a + 1, a is not an input variable from an operator.

I am sorry, but I cannot give a precise answer. I can't think of any case where this can happen. There are many places where we assume that what we get is a literal  or a variable name. But there may be some cases I am not aware of where this is not true. I think that making the proposed contract is the easiest thing to do. Otherwise we might have to revisit the whole code generation method.

@mgaido91 Thanks for the comment. I agreed that to make the contract is the easiest way. If we don't make this contract, seems to me a significant change is needed.

Another solution is to not make this contract. By a quick look this seems hard to do, because at the time of doing this, the code(method body) is already generated and we don't know how to replace statement like a + 1 with the generated parameter name, inside the method body. We may need to do this fix earlier in the codegen procedure.

@cloud-fan Can you say more about this limitation? Can't we just replace all occurrences of a + 1 with a parameter name like varA in method body if we are going to use it as parameter name in the split method?
E.g., from a method body like:
void splitMethod1(int a + 1, int b, boolean c) {
 ...
 int d = a + 1 + b;
 if (a + 1 > 10 && c) {
    ...
 }

to
void splitMethod1(int varA, int b, boolean c) {
 ...
 int d = varA + b;
 if (varA > 10 && c) {
    ...
 }

replacing them via string is too dangerous, logically we wanna replace some nodes in AST, which needs an AST based codegen framework, or we need to refactor the current framework a little bit to do it safely.

AST based codegen framework sounds a too far step from current point. I think we either follow the new contract or refactor the current framework a little bit.

I agree that string replacement is too dangerous (e.g. a + 1 = a + 10 with a + 1).
How about a contract with adding assertions?

I found a few problems that this PR can ideally solve. If this is not available soon, I will use workaround in upcoming PRs.

@kiszk Yea, I have now checked through the codegen. I didn't find the places that can cause that issue (a + 1 as the codegen output value) yet. I may submit another PR to let us easily identify such codegen output so we can easily do adding assertions for it.
As we don't use such statement as codegen output, I think the easiest approach is adding assertions. @cloud-fan WDYT?

I think that this PR is necessary to fix SPARK-22868 and SPARK-22869

To me whole stage codegen compilation fix is less important as we can fallback to non whole stage codegen, so we don't need to rush.

As we don't use such statement as codegen output, I think the easiest approach is adding assertions

What about future? Will we need to output statement for some reason? like reducing the usage of local variables?

What about future? Will we need to output statement for some reason? like reducing the usage of local variables?

I think that we won't have strong motivation to use output statement. The reason is, although it helps to reduce the usage of local variables (is it any beneficial like reducing global variables?), it also means the chained evaluation of expressions needs to be run at every occurrence.

it also means the chained evaluation of expressions needs to be run at every occurrence.

We can introduce some mechanism to save statement to local variables if it's going to be re-computed. A possible benefit is reducing code size. Anyway I think this is a valid possibility to improve the codegen framework and we should not totally give it up.

Not all of expressions can use statement as output. Seems to me there are only very few scenarios (e.g., a chain of expressions that all can use statement output by coincidence) we can really save local variables. If we also consider possible optimization of JIT, the benefit might be only marginal. That is why I am not sure if it is necessary to consider statement output generally.

IMHO, in general, the output ev.value would be declared as local variable by parent as
s"""${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};

Such as cases cannot have an expression in ev.value.
As @viirya pointed out, I imagine there are a few scenarios. Would it be possible to show an example and place in source code where an expression is used as output in order to correctly understand the issue?

I did a search but can't find one in the current codebase, but I do think this is a valid idea, e.g. a simple example would be a + b + .... z, if expressions can output statement, then we just generate code like
int result = a + b ... + z;
boolean isNull = false;

instead of
int result 1 = a + b;
boolean isNull1 = false;
int result2 = result1 + c;
boolean isNull2 = false;
...

This can apply to both whole stage codegen and normal codegen, and reduce the code size dramatically, and make whole stage codegen less likely to hit 64kb compile error.
Another thing I'm working on is: do not create global variables if ctx.spiltExpression doesn't spit. This optimization should be much more useful if combined with this optimization.

This is only valid when by coincidence the all expressions involved can use statement as output. As I looked at the codebase, I think only few expressions can output statement. This may not apply generally to reduce code size.

I think all arithmetic, predicate and bitwise expressions can benefit from it, and they are very common expressions in SQL. More importantly, allowing expressions to output statement may have other benefits that we haven't discovered yet, I don't think we should sacrifice it just for supporting splitting code in whole stage codegen, which is only for performance not stability.
For now I think we can fix the 64kb compile error caused by the whole stage codegen framework not expressions. I remember @maropu has a PR to fix that and I prefer to take priority to review that PR.

Which one is more common? A chain of arithmetic expressions? Or a deeply nested expression? I don't see strong evidence that supports statement output from the discussion. The only one possibility for now is to reducing code size. This is also for performance, not stability. On the contrary, isn't using local variable more stable? Don't forget we need to introduce other mechanism to fix the problem of statement output like re-evaluation I pointed out above.
I'm not saying it is not good to support statement output. But for now, the reason to support it is very vague.

Do we have to sacrifice one of them? If we do then I agree deeply nested expression is more common than a long chain of arithmetic expressions and we should get this patch. I think we should explore more about how to split methods in whole stage codegen before making this decision, at least now I'm not convinced that we have to forbid expressions to output statement.

I agree that the best is we can have both of them.
I have a proposal to replace statement output in split methods. Maybe you can check if it sounds good.
By #20043, we have a StatementValue wrapping statement output. Instead of immediately embedding the statement in codes, we use a special replacement like %STATEMENT_1% for it. Normally we replace this with actual statement. If we need split methods, we replace this with a generated variable name. As it is special replacement, I think it should be safer.
This is the idea to more safely replace statement with generate variable name under the string based framework.

This is a pretty cool idea that can work with the current string based codegen framework, LGTM!

Cool. LGTM, too.

LGTM, great work!

@cloud-fan Since #20043 was merged now, I will go to polish this and implement the above idea. Will submit a PR for this when it is ready.


Documentation insufficient

The Link mentioned on this page is dead:
https://github.com/joeyates/imap-backup/blob/main/docs/configuration.md
Unfortunately it is unclear how to config a server for usage of  STARTTLS on Port 143. Default seems to be SSL/TLS on port 993.
Thanks for the support and for this project
Birk

I've corrected the link, thanks.
The page https://github.com/joeyates/imap-backup/blob/main/docs/configuration.md now has a link to the Ruby Standard Library documentation for the configuration that can be set.


Question about DVD dumping

I know that, if the disc is CSS encrypted, there will be a text file with the css key for the VOB files.
But the backup wouldn't be decrypted, right?
I tested dumping a few DVDs here and there. On some I had it that there were "unrecoverable errors" during dumping. But when e.g. ripping all video files with MakeMKV there was not a single error. Could it be some kind of protection? I cannot tell yet, if it's the same thing like in https://github.com/saramibreak/DiscImageCreator/issues/34
Using /c on a DVD makes the dumping incredibly slow. What's so different? the DiscInfo log file becomes huuuuuge.

But the backup wouldn't be decrypted, right?

You can dump the encrypted iso file. Because it is illegal to create a decrypted file.

Could it be some kind of protection?

I'm not sure. Please use A-Rayscanner or ProtectionID to detect protection.

I scanned the disc with various tools:
ProtectionID Protection Report (txt format):
17:09:56 | DISC D:\ [THE_PINK_PANTHER] | Detected CSS / CPMM Protection! (0x00000001) | Region Lock Detected -> RegionBitMask: 00000002
[.] Region(s) allowed : 2
A-Ray Scanner:
[17:10:59] DvdCrypt CSS/CPPM detected -> D:\THE_PINK_PANTHER  [F40CEA60]
DVDFab Passkey:
Invalid PGCs, CELLs, VOBUs protection is removed.
Invalid PTTs, PGCIs, TMAPs protection not found.
8 fake vts (8, 10, 13, 17, 18, 19, 20, 21) protections are removed.
Potential bad sector (vts:1 title begin:0 end:178) is removed.
Potential bad sector (vts:2 menu begin:0 end:357) is removed.
Potential bad sector (vts:2 title begin:195 end:194653) is removed.
Potential bad sector (vts:2 title begin:1863115 end:2281496) is removed.
Potential bad sector (vts:3 menu begin:0 end:178) is removed.
Potential bad sector (vts:11 title begin:0 end:178) is removed.
Potential bad sector (vts:14 title begin:82769 end:84200) is removed.
Potential bad sector (vts:15 title begin:0 end:178) is removed.
8 potential bad sector protections are removed.
Structure protection (ARccOS, RipGuard, etc.) is removed.
So it seems like there are bad sectors as a kind of protection.

Is there a visible ring on your disc?

No, see for yourself:
https://imgur.com/a/W63QvlO
One image from the label side, two of the data side.

Some Sony pictures DVDs have ARccOS https://en.wikipedia.org/wiki/ARccOS_protection
I can't support it yet because I don't have its protected disc.

One can't tell DIC to just continue even after bad sectors, can we? After an unrecoverable error, the dumping quits.

Hmm strangely, DIC skips bad sectors on another disc (though this is a new drive of the same type I had before, the old one stopped reading anything):
NumberOfProgramChain: 31212
Detected irregular title number
Detected protection [MVSNRGFP]. LBA 0 to 0
Detected Anchor Volume Descriptor Pointer: LBA 3284767
Outputted to _xxxKey.txt
Error sectors range: LBA 385480 to 385511 = 32 -> Filled with 0x00
Error sectors range: LBA 470112 to 470142 = 31 -> Filled with 0x00
Error sectors range: LBA 471504 to 471534 = 31 -> Filled with 0x00
Disc dumping is still running currently. What is this mentioned about the LBA 0?

Detected protection [MVSNRGFP].

It's RipGuard.

LBA 0 to 0

This is useless for this protection. I'll update so as not to show it.

As a little side info, I found a typo. In the *_disc files for DVDs, there I've seen "Skiped" more often. I assume it should be "Skipped"?

I assume it should be "Skipped"?

Yes, I'll fix it.

DiscImageCreator_test.zip

added: detecting ARccOS

I tested Stranger than Fiction (0 43396 15407 0) and Pursuit of Happyness, the (0 43396 15085 0)

How long did the dumping need for these? I tried dumping the Pink Panther DVD again.
StartTime: 2021-08-02T17:49:33+0200
that's now nearly an hour ago, and it's only at LBA 649492, because it finds errors and errors and errors, then reads back some sectors, then finds other errors.  The disc has 4169920 LBAs

I am not sure, that this is right.... for one, the process took now 3 hours and 10mins.
Then the _disc file does NOT list all erroneous sectors, but the _mainerror does. I attached all but the image files.
THE_PINK_PANTHER.zip

the process took now 3 hours and 10mins.

Yes, it took the same time for my disc.

Then the _disc file does NOT list all erroneous sectors, but the _mainerror does.

2589129 is sometimes readable and at other times not.
DiscImageCreator_test.zip

added: /nss option in dvd

If you use "/nss 3", reading error sector is reread at 3 times.

If checking the ISO file afterwards with Isobuster and let it check the readability of files, there's a whole bunch of files with the errors, though they're all at the "end" of the file list:
https://i.imgur.com/sZsWF5n.png
VTS_02 still contains over 4GB alone, where the main movie is. so those other VTS files are bogus files I assume, only there to make it difficult to rip (which btw. is possible without any problems, because the main movie is in files which are all ok)

Will close this from my side, as it is more useful to have the disc already decrypted on the disk done with another tool. Dumping Video DVDs isn't the purpose of DIC I would think.
I hope the addition for the ARccOS was still useful for others.


Create cvc5-cloud

Adds submission info for cvc5-cloud

I see a "merge submissions" commit - are those going to be updated after the deadline today?

Yes.


Card, manage Tap on Action area 1

@mccart77 @B3nz011 d in card title first, do we consider the header a part of the area 1 ?
In small card, the area is all the card ?
Need illustration in DSM.

@mccart77 @B3nz011 d in card title first, do we consider the header a part of the area 1 ? In small card, the area is all the card ? Need illustration in DSM.

Yes, i think this is a point that needs to be changed, because in Android the header in part of the primary action area, so it should be the same for iOS. A visual has been added to the DSM specification to describe this.


Make sure value passed in obeys the useUTC flag

Fixed the UTC unit tests, added more

This is to fix the issue reported here: https://github.com/edgycircle/ember-pikaday/issues/69

Thank you so much for the great contributions @DanLatimer! I just released v2.0.0-beta.2, please give it a spin.

thanks @stravid! Will do!


No tests found after installing buefy

Version
3.0.0-rc.3
Reproduction link
https://github.com/tony19/vue-cli-broken-tests-demo.git
Steps to reproduce
Run the following commands:
vue create vue-test2
# and enable unit tests at prompt
cd vue-test2
npm i -S buefy
npm run test:unit

What is expected?
The unit tests run without error
What is actually happening?
An error:
➜  vue-test2 git:(master) ✗ npm run test:unit

><EMAIL_ADDRESS>test:unit /Users/tony/src/tmp/vue-test2
> vue-cli-service test:unit

No tests found
In /Users/tony/src/tmp/vue-test2
  11 files checked.
  testMatch: /Users/tony/src/tmp/vue-test2/(tests/unit/**/*.spec.(js|jsx|ts|tsx)|**/__tests__/*.(js|jsx|ts|tsx)) - 0 matches
  testPathIgnorePatterns: /node_modules/ - 11 matches
Pattern:  - 0 matches
 ERROR  jest exited with code 1.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>test:unit: `vue-cli-service test:unit`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>test:unit script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      /Users/tony/.npm/_logs/2018-06-27T20_17_10_928Z-debug.log

The contents of /Users/tony/.npm/_logs/2018-06-27T20_17_10_928Z-debug.log:
0 info it worked if it ends with ok
1 verbose cli [ '/Users/tony/n/bin/node',
1 verbose cli   '/Users/tony/n/bin/npm',
1 verbose cli   'run',
1 verbose cli   'test:unit' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'pretest:unit', 'test:unit', 'posttest:unit' ]
5 info lifecycle vue-test2@0.1.0~pretest:unit<EMAIL_ADDRESS>6 info lifecycle vue-test2@0.1.0~test:unit<EMAIL_ADDRESS>7 verbose lifecycle vue-test2@0.1.0~test:unit: unsafe-perm in lifecycle true
8 verbose lifecycle vue-test2@0.1.0~test:unit: PATH: /Users/tony/n/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/Users/tony/src/tmp/vue-test2/node_modules/.bin:/Users/tony/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/tony/n/bin
9 verbose lifecycle vue-test2@0.1.0~test:unit: CWD: /Users/tony/src/tmp/vue-test2
10 silly lifecycle vue-test2@0.1.0~test:unit: Args: [ '-c', 'vue-cli-service test:unit' ]
11 silly lifecycle vue-test2@0.1.0~test:unit: Returned: code: 1  signal: null
12 info lifecycle vue-test2@0.1.0~test:unit: Failed to exec test:unit script
13 verbose stack Error<EMAIL_ADDRESS>test:unit: `vue-cli-service test:unit`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/Users/tony/n/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:304:16)
13 verbose stack     at EventEmitter.emit (events.js:182:13)
13 verbose stack     at ChildProcess.<anonymous> (/Users/tony/n/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:182:13)
13 verbose stack     at maybeClose (internal/child_process.js:961:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:5)
14 verbose pkgid<EMAIL_ADDRESS>15 verbose cwd /Users/tony/src/tmp/vue-test2
16 verbose Darwin 17.6.0
17 verbose argv "/Users/tony/n/bin/node" "/Users/tony/n/bin/npm" "run" "test:unit"
18 verbose node v10.5.0
19 verbose npm  v6.1.0
20 error code ELIFECYCLE
21 error errno 1
22 error<EMAIL_ADDRESS>test:unit: `vue-cli-service test:unit`
22 error Exit status 1
23 error Failed at the<EMAIL_ADDRESS>test:unit script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

A workaround is to delete and reinstall node_modules. Note that installing the project from scratch after a git clone does not reproduce the problem. The issue happens specifically after installing buefy after creating a Vue project from the cli.

Environment

node 10.5.0
npm 6.1.0
macOS High Sierra 10.13.5
MacBook Pro

Full command output

➜  tmp vue create vue-test2
Vue CLI v3.0.0-rc.3
? Please pick a preset: Manually select features
? Check the features needed for your project: Babel, Linter, Unit
? Pick a linter / formatter config: Basic
? Pick additional lint features: Lint on save
? Pick a unit testing solution: Jest
? Where do you prefer placing config for Babel, PostCSS, ESLint, etc.? In dedica
ted config files
? Save this as a preset for future projects? No
Vue CLI v3.0.0-rc.3
✨  Creating project in /Users/tony/src/tmp/vue-test2.
🗃  Initializing git repository...
⚙  Installing CLI plugins. This might take a while...
yarn install v1.7.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
success Saved lockfile.
✨  Done in 33.19s.
🚀  Invoking generators...
📦  Installing additional dependencies...
yarn install v1.7.0
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 4.32s.
⚓  Running completion hooks...
🎉  Successfully created project vue-test2.
👉  Get started with the following commands:
$ cd vue-test2
$ yarn serve
➜  tmp cd vue-test2
➜  vue-test2 git:(master) npm i -S buefy
npm WARN deprecated<EMAIL_ADDRESS>1.2.0 should have been a major version bump

<EMAIL_ADDRESS>uninstall /Users/tony/src/tmp/vue-test2/node_modules/yorkie
node bin/uninstall.js

husky
uninstalling Git hooks
done
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/jest as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/jest
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/uglifyjs as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/uglify-js
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/regjsparser as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/regjsparser
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/nopt as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/nopt
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/jsesc as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/jsesc
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/esparse as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/esprima
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/esvalidate as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/esprima
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/browserslist as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/browserslist
npm WARN rm not removing /Users/tony/src/tmp/vue-test2/node_modules/.bin/babylon as it wasn't installed by /Users/tony/src/tmp/vue-test2/node_modules/babylon

<EMAIL_ADDRESS>install /Users/tony/src/tmp/vue-test2/node_modules/fsevents
node install

[fsevents] Success: "/Users/tony/src/tmp/vue-test2/node_modules/fsevents/lib/binding/Release/node-v64-darwin-x64/fse.node" already installed
Pass --update-binary to reinstall or --build-from-source to recompile

<EMAIL_ADDRESS>install /Users/tony/src/tmp/vue-test2/node_modules/yorkie
node bin/install.js

setting up Git hooks
done
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.

<EMAIL_ADDRESS>added 234 packages from 56 contributors, removed 89 packages, updated 1245 packages and audited 72619 packages in 47.473s
found 0 vulnerabilities

➜  vue-test2 git:(master) ✗ npm run test:unit

<EMAIL_ADDRESS>test:unit /Users/tony/src/tmp/vue-test2
vue-cli-service test:unit

No tests found
In /Users/tony/src/tmp/vue-test2
11 files checked.
testMatch: /Users/tony/src/tmp/vue-test2/(tests/unit//*.spec.(js|jsx|ts|tsx)|/tests/*.(js|jsx|ts|tsx)) - 0 matches
testPathIgnorePatterns: /node_modules/ - 11 matches
Pattern:  - 0 matches
ERROR  jest exited with code 1.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>test:unit: vue-cli-service test:unit
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>test:unit script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      /Users/tony/.npm/_logs/2018-06-27T20_17_10_928Z-debug.log

I've encountered something similar, I believe this may be a Jest 23.2.0 bug. See https://github.com/facebook/jest/issues/6546

@sickp Thanks, I'll watch that issue.

FYI, I verified a workaround: remove the testMatch property from jest.config.js per https://github.com/facebook/jest/issues/6546#issuecomment-400588471.

duplicate of #1686


chore(ci): Limit git clone depth

This may speed up cloning on CI services by reducing the clone depth.
Change-Type: patch

Any reason why not doing depth 1?

Yeah, from https://docs.travis-ci.com/user/customizing-the-build/#Git-Clone-Depth:

Please note that if you use a depth of 1 and have a queue of jobs, Travis CI won’t build commits that are in the queue when you push a new commit.

And I figured we probably want to avoid that.

I wonder if that means that (effectively) there can only be as many jobs in the Travis build queue as the selected clone depth?
Given that Travis CI is often quite slow, and we often have many jobs queued up, I guess time will tell if a depth of 3 is 'enough' ;-)

I wonder if that means that (effectively) there can only be as many jobs in the Travis build queue as the selected clone depth?

Oh, is that indeed the case @jhermsmeier? As far as I can see, Travis CI clones the main branch of the repo (which after this change means the last 3 commits), but then calls git fetch on the branch like this:
git fetch origin +refs/pull/1272/merge:

Which means that the branch will always be fetched from the internet, and therefore the depth here doesn't apply.
Am I missing something?

Honestly, I don't really know, but from what I understood depth: 1 is a special case, so build behavior shouldn't change.

@lurch was right. This is causing failures when the Travis CI backlog is greater than 3. See:

https://travis-ci.org/resin-io/etcher/jobs/220811455
https://travis-ci.org/resin-io/etcher/jobs/220811463
https://travis-ci.org/resin-io/etcher/jobs/220802663

I reverted this commit for now.

@lurch was right.

Hah, lucky guess ;-)

Oh boy, sorry about that :/ Lesson learned.


201 created as a succesfull post request response code

To be able to work with in harmony with django rest framework's CreateModelMixin 201 Created added as a successfull post request response, according to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

@rnicholus Can you halp with this pull request please ?

Is this for a simple upload request, or for one of the requests sent by Fine Uploader S3/Azure (signature and success)?

Based on the issue you files, #1354, I'm guessing this is related to the uploadSuccess POST sent by Fine Uploader S3/Azure.
Your pull request is against master, which only contains released code.  Regardless, we'll be sure to make this adjustment in #1314 as part of 5.2.0.

@rnicholus Thank you so much, it was uploadSuccess POST request. It seems like this is going to be adjusted in 5.2.0 . Thanks!


Test: terminal drag and drop between windows

Refs: https://github.com/microsoft/vscode/issues/126387

[x] macOS @donjayamanne
[ ] linux @connor4312
[ ] windows @claudiaregio

Complexity: 3
Authors: @Tyriar, @meganrogge
Create Issue

Try dragging and dropping between windows:

terminals in the editor area to the tabs list
tabs to the editor area
tabs to the tabs list
terminals in the editor area to the editor area

Check that all of those still work in a single window.

Works beautifully

Closing as tested and all worked for me :)


[Q] Linux Build: How to disable Dektec and others?

After catching up to the latest version, excluding DekTec doesn't work anymore.
This is about static building on Linux.
I used to have  STATIC=1 NOPCSC=1 NOCURL=1 NODTAPI=1 NOTEST=1 in the make command, but this does not longer work to exclude DekTec.
How can I exclude DekTec?
Also, there are two plugin folders now:

src/libtsduck/plugin
src/tsplugins

I know I can exclude plugins from the latter by adding something like
    export TSPLUGINS := $(filter-out tsplugin_aes, $(TSPLUGINS))

to Makefile.tsduck
How can I exclude plugins from the first folder?

After catching up to the latest version, excluding DekTec doesn't work anymore.

What do you mean by that? NODTAPI means do not use the DTAPI and do not depend on it. But the tools and plugins are still built. Using them reports an error:
$ tsdektec
tsdektec: This version of TSDuck was compiled without Dektec support
$

Also, there are two plugin folders now:

src/libtsduck/plugin
src/tsplugins

The plugins folder is still src/tsplugins. Everything that is under src/libtsduck is the TSDuck library. A long time ago, it was a flat directory which ended up with more that 1000 files.  So, it was reorganised as a tree of files. The subdirectory plugin contains classes pertaining to the plugin management. Please read the libtsduck/README.txt file.
If you worked on a very old base line, the code may have substantially changed since then.

If you worked on a very old base line, the code may have substantially changed since then.

I have started "from scratch" with the latest tsDuck code and I have manually re-applied all the changes that I had made over time.

What do you mean by that? NODTAPI means do not use the DTAPI and do not depend on it.

I build with NODTAPI, but then I run tsp -lall, I get this:
List of tsp input plugins:

  dektec ... Receive packets from a Dektec DVB-ASI or demodulator device
  dvb ...... DVB receiver device input
  file ..... Read packets from one or more files
  fork ..... Fork a process and receive TS packets from its standard output
  hls ...... Receive HTTP Live Streaming (HLS) media
  ip ....... Receive TS packets from UDP/IP, multicast or unicast
  null ..... Generate null packets
  srt ...... Receive TS packets from Secure Reliable Transport (SRT)

List of tsp output plugins:

  dektec ... Send packets to a Dektec DVB-ASI or modulator device
  drop ..... Drop output packets
  file ..... Write packets to a file
  fork ..... Fork a process and send TS packets to its standard input
  hls ...... Generate HTTP Live Streaming (HLS) media
  ip ....... Send TS packets using UDP/IP, multicast or unicast
  srt ...... Send TS packets using Secure Reliable Transport (SRT)

List of tsp packet processor plugins:

  analyze .. Analyze the structure of a transport stream
  file ..... Write packets to a file and pass them to next plugin
  filter ... Filter TS packets according to various conditions
  fork ..... Fork a process and send TS packets to its standard input
  pes ...... Analyze PES packets
  stats .... Report various statistics on PID's and labels
  timeshift  Delay transmission by a fixed amount of packets
  until .... Copy packets until one of the specified conditions is met

Just in case you missed it: I'm doing a STATIC build.

Roughly, nothing changed. Compiling with NODTAPI, NOCURL, etc. never removed the wrapper C++ classes on top of these external libraries and never removed the corresponding plugins. The idea of the NOxxx options is to remove dependencies on external libraries if you do not want to install them or if you have a problem to get or compile them on such rare flavour of Linux (it happens). But there is also a requirement that applications using the TSDuck library (tsduck tools, plugins or third-party applications) continue to compile normally. For instance, an application using the class ts::DektecDeviceInfo shall continue to compile. Otherwise, that would create a mess. But, if the TSDuck library was compiled with NODTAPI, when you run the application, you get the error message I mentioned in my previous post.
Now, one little thing has changed a couple of versions ago about the way plugins are registered. From the beginning, for purely technical link-edit reasons, the code of some plugins such as the dektec or srt plugins was already inside the TSDuck library. The shared library tsplugin_dektec.cpp or tsplugin_srt.cpp contained nothing but the macro for the registration of the plugin. This was a bit ridiculous to build and install a shared library just to register code which was otherwise already present inside tsduck.dll or libtsduck.so. So, now, the macro for the registration of those plugins are inside the code of the plugins, in the TSDuck library. The corresponding tsplugin_xxx have been removed.
So, in practice, we now have "built-in self registered plugins" which are always there and traditional plugins from shared libraries which are registered only when the corresponding shared library is loaded (or object file is linked in the case of a static build). Note that the effect of tsp -l is to scan and load all tsplugin_xxx to list them.
What does this change for you? I understand that you explicitly remove the tsplugin_dektec shared library (or static object file in your case). So, previously, you removed the registration of the dektec plugin. But the code was there anyway and continues to be there now, except that the registration is now performed internally in the TSDuck library and you can see it with tsp -l. Anyway, since you compiled with NODTAPI, using this plugin will result in the error message. And static vs. dynamic build does not change anything regarding the plugin registration.

Hi Thierry, thank you very much for the detailed explanation.
It also explains why there are those two "plugin dirs" now. I had just thought looked at this in a very simple way: Previously, Dektec was not listed as an available plugin and now it is listed => something must be wrong.
But when you say that it's just listed now, even though it's not included, then it's just a bit weird, but I think I can live with it ;-)
Thanks again!


Fix: when activity is not present on the task.

When the activity is not present as a param in the task list, the
activity fails – it should instead succeed.
@rantonmattei

Coverage remained the same at 85.79% when pulling 57cea06dff5082f8a6e71137118dd577099ee271 on mo-15-march-remove-activity into b213769cd1420c87cafd5b946b704431c76cc06e on master.


Many warnings with perfomance testing. See solution below.

bash benchark_all.sh
/home/inetstar/Kamenev/programming/ZenDnn/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/inetstar/Kamenev/programming/ZenDnn/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
alexnet
/home/inetstar/Kamenev/programming/ZenDnn/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/inetstar/Kamenev/programming/ZenDnn/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)

Solution:
diff --git a/tools/validate_network.py b/tools/validate_network.py
index 8825eed..ef8e1ff 100644
--- a/tools/validate_network.py
+++ b/tools/validate_network.py
@@ -206,7 +206,7 @@ def get_config():

 def main(args):
-    m = getattr(torchvision.models,args.model)(pretrained = True)
+    m = getattr(torchvision.models,args.model)(weights="IMAGENET1K_V1")
     #print("Mean",m.bn1.running_mean.tolist()[:4])
     #print("Var",m.bn1.running_var.tolist()[:4])
     #print("W",m.bn1.weight.tolist()[:4])

Fixed some time ago.


Feat: Implement login and signup backend core logic using DB

Fixes #961 Adding core logic of login and signup using database.
Require model in Signup.js

Checking existing user in DB if it is not found then call to createUser function:

Create User password not stored directly because it is not secure to store same password i.e. I used bcrypt hashing for storing the password in the hashed form:

Login function logic bcrypt.compare() it check the current password and hashed form of password:

Database Schema for userModel:)

Pull Request Checklist

[x] I have synced the latest fork with my local repository and resolved any conflicts.
[x] I have mentioned the issue number which I created before making this PR .(format to mention issue number is : fixes #(issue number) )
[x] I understand that if any the above conditions are not met , my PR will not be MERGED .

@ANSHIKA-26 Please assign the level-2,3..it require alot of time.

@hiteashgupta1  @ANSHIKA-26   Please merge it:)


endpoint: Implement new Invalid endpoint state

See commit msgs.
Endpoint (EP) creation can fail for many reasons due to invalid data, such as IP conflict or if it's created with reserved labels. When EP creation failed, EPs can be stuck in the "waiting-for-identity" state, and left to be garbage collected. However, even though the object would be garbage collected, the metric representing which state the EP is in, is never decrementing, thus a leak.
This PR fixes this by introducing a new EP state representing an "invalid" EP. When EP creations fails, the EP will transition from its current state to "invalid", thus decrementing the current state metric.
All commits are reviewable, except for the ones marked as a revert or "dropme". They will be removed for the final change. They are kept around for debugging / saving previous revisions of this PR.

@joestringer @jrajahalme This PR fixes an issue that was reported. I'd like your thoughts on this approach. This issue occurs because we don't delete the endpoint when its creation fails due to invalid data. I'm curious to hear if there was a decision in the past to not delete endpoints. Also, it'd be good to know if there are any additional side-effects of this as well.

test-me-please

Coverage increased (+0.05%) to 36.985% when pulling fd86c7371b3994e78b5b75b3b01e7181a399a2bb on christarazi:pr/christarazi/fix-missed-decrement-metric-for-endpoints into 8ba1f84d2ffe5d59b1ca0a94c8ab804a9d43feac on cilium:master.

test-me-please

retest-net-next

CI went all green :tada:. Removing reverted commits and "dropme" commits.

test-me-please


Sequential training and predicting (LSTM)

I have a conceptual question about LSTM and sequences.  Consider we have only one input sequence (a sequence of numbers with some patterns (not random) ) and we want to predict the future according to past. The first architecture that comes into my mind is as follows: (borrowed from: http://colah.github.io/posts/2015-08-Understanding-LSTMs/)

For training we can have two approaches:
•	First approach:
n = length of the input sequence
X_train = input sequence (length = n1)
Y_train = shifted input sequence (only one) (length = n1)
So, for each value in X_train we have one corresponding value in Y_train (next value in the sequence). Then for training I fedd values in the X_train one by one (batch_size = 1) and update weights by using an optimization method. Here I assume the whole input as a continuous sequence. And the memory of LSTM should be able to extract long and short time dependencies in input sequence. After training the network we have a RNN that can predict one step ahead for each input. For predicting multiple steps ahead (for example 3 steps ahead) we should do as follows: feed x1 to have y1, then feed y1 to have y2, then feed y2 to have y3.
•	Second approach:
The proposed approach in:
https://github.com/fchollet/keras/blob/master/examples/stateful_lstm.py
Here the approach is making some subsequences. Here we want to predict the future of the sequence based on the history. I considered the length of the history length_history and we want to predict future for the length length_future. I transformed the data to following format:
X_train is  n x length_history  numpy array and y_train is n x length_future numpy array (for details check the above link) . Then we can train model for predicting multiple steps ahead in future.
I implemented both of these approaches in Keras (LSTM with stateful mode)
My questions:

Which approach is correct? I myself think that the first approach is better and more logical. The reason is that the whole input is a sequence and making subsequences might be wrong!
Is the second approach sequence to sequence training?
How should I manage model internal states. I mean in prediction step when I should reset the state for both approaches?
I fed my input to a NN without any hidden layer (just a dense layer) and for both approaches this neural network (without any hidden layer) worked better than RNN with LSTM hidden layers! How is it possible?

Here,

So, for each value in X_train we have one corresponding value in Y_train

X_train[0] or any X_train for that matter  has 1 value or a time series data itself?
Similarly what about Y_train[0] ?

@suraj-jayakumar : Indeed Y_train is shifted (shifted by one) vector of X_train. for example  X_train[2] = y_train[1]

I have also tried your first approach and right now working on your second approach.
Following the reference for your second approach, where they set lahead=1, I am in fact not sure whether the performance will be affected if we change lahead = 1 to lahead = a value > 1.
How was your accuracy, mininaNik?

@yangfantrinity : I cannot remember the accuracy and I have not access to that!!

Thanks, mininaNik. Understand you cannot remember the exact accuracy. But how about the comparison? which one is better, method one or two?


Publish the k-vscode extension on Open VSX

This is the open version of the Microsoft Extension Marketplace
https://open-vsx.org/
By default, Arch packages code, an open-source alternative to VSCode, that doesn't have access to MEM for possible legal reasons. The alternative is above.
Thank you @geo2a for the suggestion.

Here's the issue that provides context on why this is the case: https://github.com/microsoft/vscode/issues/31168

Done:https://open-vsx.org/extension/RuntimeVerification/k-vscode
and automated by: https://github.com/runtimeverification/k-editor-support/pull/44


[Improvement](RoomHeaderInfo Slot) Update userStatus with user typing info

Is your feature request related to a problem? Please describe
On mobile device, the user can't know if the member is typing or not (because the sidebar is not displayed)
Describe the solution you'd like
Update the userStatus when the user is typing to show the info on

Can't you use typingUsers data from room-header-info slot?

@antoine92190
Ok my bad, I missed this data slot


[Feature Request] Support of Linux Kernel 6.1 (LTS) in Mariner

acknowledged.  Thank you for your suggestion.

Hi @jslobodzian, can you please re-open this has yet to be done ?

What is the status update about this work?

Meanwhile this ticket was closed improperly by 4th March, the first 6.1 tag https://github.com/microsoft/CBL-Mariner-Linux-Kernel/releases/tag/rolling-lts%2Fmariner-3%2F<IP_ADDRESS> only appears by 5th, May.


Fix Typos in What Are DWNs Page

Fix Typos in What Are DWNs Page
🚀 Goal
We want to ensure our website's documentation don't have any mistakes.
🤔 Background
This issue focuses specifically on fixing typos across our docs. Your contributions here make a difference!
🤝 How You Can Contribute
We've broken it down into specific typos for you to address. Every contribution counts!
🔑 Tasks and Acceptance Criteria
❗You must only assign yourself one task at a time to give everyone a chance to participate.❗
You may assign yourself your next task after your current task is reviewed & accepted.

Fix Typos in What are DWNs page

[ ] Add space between "Identifer(DID)" to be "Identifier (DID)"

[ ] Change "up the application layer" to be "up to the application layer" ("to" for clarity)

Make sure all tests are passing.

🌟 Resources

What Are DWNs page
Docs, Developer Website

Getting Started

Comment ".take" on this issue to get assigned
Fork the repository and create a new branch for this task
Follow the tasks outlined above
Submit a pull request with your changes
Respond to any feedback during the review process

Questions?
If you have any questions or need clarification, please comment on this issue or join our Discord community.
Happy coding! 🎉

.take

Thanks! @Lymah123


autoscaling.keda.sh/paused-replicas: "0" not working

Report
The annotation autoscaling.keda.sh/paused-replicas: "0" is not behaving as expected. Even after applying this annotation to the KEDA ScaledObject, deployments are continuing to get scaled in.
Expected Behavior
The autoscaling.keda.sh/paused-replicas annotation with a value of "0" should prevent any scaling actions (scaling in) from occurring. This should scale down the deployment to zero or 1.
Actual Behavior
Deployments are still being scaled in even after applying the annotation.
Steps to Reproduce the Problem

Create or use an existing KEDA ScaledObject.
Add the annotation autoscaling.keda.sh/paused-replicas: "0" to the ScaledObject definition, intending to prevent scaling down.
Observe that deployments continue to get scaled in, despite the annotation indicating a value of "0."

Logs from KEDA operator
1.6952876629502664e+09	DEBUG	controller.scaledobject	ScaledObject is defined correctly and is ready for scaling	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "" ........

KEDA Version
2.6.1
Kubernetes Version
1.24
Platform
Google Cloud
Scaler Details
No response
Anything else?
No response

@ianebot is correct, closing as this is not a bug.


[BUG] - there is not way to change border bg custom Input

NextUI Version
2.1.17
Describe the bug
there is not way to change the border color once you select the input it stays as gray doesnt matter the classnames props added
Your Example Website or App
No response
Steps to Reproduce the Bug or Issue
trying to set custom styles
classNames={{
label: "text-black/50 dark:text-white/90",
input: [
"bg-transparent",
"text-black/90 dark:text-white/90",
"placeholder:text-default-700/50 dark:placeholder:text-white/60",
],
innerWrapper: "bg-transparent",
inputWrapper: [
"shadow-xl",
"bg-default-200/50",
"dark:bg-default/60",
"backdrop-blur-xl",
"backdrop-saturate-200",
"hover:bg-default-200/70",
"dark:hover:bg-default/70",
"group-data-[focused=true]:bg-default-200/50",
"dark:group-data-[focused=true]:bg-default/60",
"!cursor-text",
],
}}
Expected behavior
I would like to know the option to set that border color
Screenshots or Videos
No response
Operating System Version
macOS
Browser
Chrome

@Andrestheran I don't see you've set the border in inputWrapper. Below is an example by adding border-2 border-red-300 to inputWrapper.

@wingkwong I paste the example thta we could found on the documentation hover when you click on the field there is a focus gray color and there is not any way to put any other color

@Andrestheran Can you share the code of your screenshot?

sure, thank you so much for helping me out!
<Input                   id="firstName"                   label="First Name"                   value={form.values.firstName}                   onChange={form.handleChange}                   classNames={{                     label: ["border-none", "bg-white"],                     input: ["border-none", "bg-white"],                     inputWrapper: ["border-none", "bg-white"],                     innerWrapper: ["bordernone", "bg-white"],                   }}                   disabled={!isEditMode}                 />

I tried all the options and colors but trying and trying and always getting that gray shadow

@Andrestheran Try the following to see if this is what you want.
inputWrapper: ["border-none", "bg-white", "data-[hover=true]:bg-white", "group-data-[focus=true]:bg-white"],

it works!!1 thk u so much!

@Andrestheran Thanks for the confirmation. I'm going to close this issue then.


release: 0.12.2

Good news / bad news / good news scenario.
Good news
Travis CI did actually build the previous master branch with the release at exactly 11:31
https://travis-ci.com/lightelligence-io/styles/builds/135006244

So we have our package released
Bad news
We lost the previous gh-pages branch with all released docs
Good news
I found a fresh copy of the gh-pages ( https://github.com/SmolinPavel/styles/tree/gh-pages ) from a fellow contributor, which has all the docs till alpha-3 :
	0.12.1-alpha.0	Deploy lightelligence-io/styles to github.com/lightelligence-io/style…	29 days ago
	1.0.0-alpha.1	Deploy lightelligence-io/styles to github.com/lightelligence-io/style…	19 days ago
	1.0.0-alpha.2	Deploy lightelligence-io/styles to github.com/lightelligence-io/style…	5 days ago
	1.0.0-alpha.3

I guess we can recover the docs from there once we release the new alpha branch. In this case I think we should at least rename this PR before we merge it, so we don't "deploy" twice.


update August 2020 (version 1.49) and July 2020 (version 1.48) in Notebook UX not working

Issue Type: Feature Request
the Notebook UX updates not working despite of i update to least version or not
VS Code version: Code 1.49.0 (e790b931385d72cf5669fcefc51cdf65990efa5d, 2020-09-10T13:20:50.359Z)
OS version: Linux x64 5.4.0-47-generic

Currently it's only supported in Insiders, https://code.visualstudio.com/insiders/

Issue also exist vscode insider


chore: add multiple identifiers

Review Checklist

Summary*

[ ] fix: Bug fix for Add the items tackled here as checklists
[x] feat: Completed task
[ ] chore: Incomplete task

[ ] test: Sub-task 1

Structure*

[ ] The Pull Request has a proper title that conforms to our MR title standards
[ ] The Pull Request has one commit, and if there are more than one, they should be squashed
[ ] The commit should have a proper title and a short description
[ ] The commit must be signed off
[ ] Unused imports are not present
[ ] Dead code is not present
[ ] Ensure dry running library to confirm changes

Tests

[ ] Proper and high quality unit, integration and acceptance(if applicable) tests have been written
[ ] The coverage threshold should not be lowered

Sign off*

[ ] All comments have been resolved by the reviewers
[ ] Approved by Czar {replace_with_czar_name}
[ ] Signed off by second reviewer {replace_with_name}
[ ] Ensure all checks are done before merging :warning:
[ ] All PRs needs to be signed before merging :warning:

N/B:

Add a checklist if more than one item is done.
Add screenshots and/or images where necessary
Indicate any breakages caused in the UI :exclamation:
Where necessary, indicate which issue the Pull Request solves (Closes #)
Any new files are updated in the folder structure in the README

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

16 of 16 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 83.106%

Totals

Change from base Build<PHONE_NUMBER>0:
0.3%

Covered Lines:
487

Relevant Lines:
586

💛  - Coveralls


System stabilization: Put all deployment-related files under version control

Place crontasks and configuration files under version control, which improves the ease of updating, troubleshooting, and keeping an eye on deployments.

This is set-up! It certainly streamlines the Metro deployment process.


Encapsulation in a class TimeLapseCreator

Encapsulation in a class TimeLapseCreator. This will allow different instances to be created if someone wants to follow web cams in different parts of the world, where several hours of difference play a big role in terms of light changes.

Encapsulated in a class.


Round BPM to nearest half-beat

For most modern songs, it is highly unlikely, that they have been recorded without metronome or drum computer. It is not very likely that musicians or producers would deliverately set the BPM to something with decimals.
My suggestion would therefore be, when you create a new song to round the BPM to the nearest half-integer before doubling/tripling/quadrupling the BPM.

I don't think that tripling was ever a thing, it is rather (potentially repeated) doubling.

I've seen it a couple of times. e. g. just stumbled across USDB ID 6308 where the BPM is 324. The actual BPM is 108, so I suppose, somebody tripled it. But doesn't matter, rounding it will do.

duplicate: #29


vscode use subrocess,check_output 'utf-8' codec can't decode byte 0xd3 in position 79: invalid continuation byte error

          This looks like an issue with your code. If you think it's an issue with vscode's python support, please file an issue on the python extension with more detail

Originally posted by @roblourens in https://github.com/microsoft/vscode/issues/210957#issuecomment-2123767845
The issue with vscode. The issue has not been resolved. Should not be closed
The code file is new. no any string
The issue with vscode. The issue has not been resolved. Should not be closed

Please give enough detail for us to understand and reproduce the issue

Thanks for creating this issue! We figured it's missing some basic information or in some other way doesn't follow our issue reporting guidelines. Please take the time to review these and update the issue.
Happy Coding!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


How to save the entire network including the projection?

Hi,
is is possible so save the entire network including the projection layers?
In your exmaple you just show how to save the improved representation network.
# save your improved network
torch.save(resnet.state_dict(), './improved-net.pt')

Sorry if that's a dumb question, but I only work with keras and never worked with pytorch before.

@albert-92 ohh, that's easy, just replace resnet with learner