Introducing a CF domain variable

Introducing a CF domain variable
Moderator
@dblodgett-usgs
Moderator Status Review [last updated: 2020-09-23]
The proposal has been submitted and preliminarily reviewed by the moderator. Attention should be called to the potential for this proposal to subtly but fundamentally alter how CF-NetCDF data fields and domains are treated. Review from authors of CF-NetCDF client software is necessary here.
Requirement Summary
The concept of a domain that describes data locations and cell properties is not currently mentioned in the CF conventions, because it does not correspond to any single entity in the netCDF file. Instead, the domain is stored implicitly in a number of other variables and attributes that are linked to the data variable in various ways defined by the conventions.
The domain is, however, well defined in the CF data model as an abstract concept (as opposed to a data model construct) that provides the linkage between the field construct and the metadata constructs that describe the relevant data locations and cell properties. There is currently no "domain construct" in the data model, since there is no corresponding CF-netCDF entity.
There is a need to be able to describe a domain independently of any data variables, which is currently not possible. Use cases include:

Curated data streaming services for which it is impractical to send very large domain descriptions with every file.

Storing time-dependent coordinates from remote sensing applications.

Storing geometries without any timeseries data.

For such use cases, it is not satisfactory to try to locate an appropriate multidimensional data variable that describes the required domain, nor to create a dummy data variable for this purpose, which has no physical meaning.
Therefore, the inclusion of CF-netCDF domain variables that can encode a domain independently of any data, and a corresponding data model domain construct, will enhance CF by meeting these use cases.
Technical Proposal Summary
NetCDF encoding
A new "domain variable" will be introduced that is of arbitrary type since it contains no data. This variable will act as a container to bind together other variables that collectively define a domain, in a similar manner to how a data variable performs the same task.
It will support the same CF attributes as are allowed on the data variable for describing a domain, with exactly the same meanings and syntaxes: cell_measures, coordinates, geometry, and grid_mapping. These will be indicated as domain variable attributes by the additional "Do" indicator (short for Domain) in the "Use" column of Appendix A: Attributes.
Any future CF attributes that a data variable may use to describe its domain will be similarly transferred to the domain variable, meaning that keeping the domain variable up to date with other enhancements will be a well defined and easy task.
There is no mechanism for referencing a domain variable from a data variable, i.e. a data variable must still encode its domain in the current, implicit manner. This is to preserve backwards compatibility with all existing software libraries that understand the current structure of a data variable; and to reduce redundancy or incompatibility issues that may arise if a data variable encodes its own domain and references a domain variable.
A domain variable may exist in a file with or without other data variables.
Data model
The domain in the data model will be transformed from an abstract concept into a "top-level" construct, i.e. one that can exist in the absence of any other constructs. Currently, the field construct (corresponding to a CF-netCDF data variable) is the only top-level construct.
The new domain construct will replace the current domain concept, replicating it every in every way apart from that it will be related to the field construct via an aggregation relationship, rather than by the current composition relationship of the abstract domain concept. This makes it clear that the domain construct can exist independently from the field construct.
It is of no consequence to the data model that a CF-netCDF data variable will not be able to explicitly reference a CF-netCDF domain variable. That is an encoding choice that does not affect the logical structure.
Location in the conventions document

The domain variable will be described in a new section: 5.8 Domain Variables

The following appendices will updated:

Appendix A: Attributes

Appendix I: The CF data model

CF Conformance Requirements and Recommendations

Benefits
All those who meet the use cases described in the Requirements summary will benefit from the new domain variable.
Status Quo
At present, a domain can only be encoded implicitly via a data variable, leading to ambiguities when retrieving a domain from a dataset.
Associated pull request
#302
Detailed Proposal
Conventions text has been proposed in chapter 5, appendices A and I, and the conformance document in pull request #302

@davidhassell -- I'm in support of this in concept and would be willing to moderate the discussion. I will review the PR in detail soon.
Others, please review. Comments on detailed aspects of the PR can be in line, but please put all general discussion here.

@dblodgett-usgs Thanks for moderating

@davidhassell -- I left a couple comments on your PR #302 to seed some further discussion.
See https://github.com/cf-convention/cf-conventions/pull/302#discussion_r493579512
We need to be very aware that this change will loosen / modify the field-variable-centric nature of CF. I've always seen this as a central tenant of CF that was both annoying and super useful. This addition will make a lot of confusing things possible. Not really an argument for or against, but something to keep in mind.

The other thing is that I'm a little confused about how a domain variable works when it doesn't reference any dimensions. @davidhassell points to scalar coordinate variables as a case where this is valid. This does make sense but calls out the need for inclusion of examples showing how this would work.
A coordinates attribute would be required in the case that any coordinate variables that make up the domain are scalar coordinates.

Those are two general comments for people to consider.
I do want to pose a potential change for consideration.
Considering that this domain concept is new, we have an opportunity to require some things. I think it would be worth requiring all coordinate variables be declared in coordinates whether they use the dimension name or not. If there are other metadata that would otherwise by inferred, I think they should be required in this new domain variable construct. I'm thinking about this as a developer who wants a well-described domain declaration that doesn't require any special knowledge or inference to fully construct the spatio-temporal domain.
Regards, Dave

Hi Dave,
Thanks for the comments. I very much appreciate your having taken the time to look over it.

This addition will make a lot of confusing things possible.

This is a good point, and we should be sure that the motives for introducing it are valid. It would be good to hear from some of the people whose use cases I ever so briefly mentionedabove, to provide a better picture of why this domain variable will be worth while. My personal interest is just in helping CF along - so I'm not really qualified to speak on their behalf. For visibility, I'll flag  @ajelenak @AndersMS @erget @oceandatalab @dblodgett-usgs who may be able to help with this (thanks!).

Considering that this domain concept is new, we have an opportunity to require some things. I think it would be worth requiring all coordinate variables be declared in coordinates whether they use the dimension name or not. If there are other metadata that would otherwise by inferred, I think they should be required in this new domain variable construct. I'm thinking about this as a developer who wants a well-described domain declaration that doesn't require any special knowledge or inference to fully construct the spatio-temporal domain.

I wholly appreciate this stance, and indeed originally had that in mind. However, I came round to think that, as far as possible, the mechanics of the new domain variable should be identical to the equivalent mechanics of a data variable, e.g. that coordinate variables may be omitted form, or included in, the coordinates attribute.
This

makes it easier to describe in the conventions and
ensures the maximum consistency between the two ways of describing a domain (implicitly on a data variable and explicitly on a domain variable)

These points remove the need for duplicating parts of the conventions with partial modifications and reduce the possibility of misunderstandings between two almost identical, but not quite the same, encodings.
The second point also makes it much easier for developers who already deal with data variables for extracting domains. This is because they already have the machinery for decoding the domain from a data variable. I can say from experience that this is the case, having just today implemented the reading of the proposed domain variable in a branch of the cfdm library. This only needed ~30 lines of new code to modify the existing read-a-data-variable function to be able to read data variables or domain variables. This worked so easily because the attributes are are parsed and processed identically in both cases.
Remember that compression (DSG raggered arrays, gathering) also has to be considered. By stating that "things are the same as for a data variable" we get compression "for free", in terms of documentation, on the domain variable.

This does make sense but calls out the need for inclusion of examples showing how this would work.

Agreed. There is already an example showing a scalar coordinate variable, but not yet one with out any named dimensions. The more examples the better, I think.

Dear David
Thanks for this proposal, which I support in its current form, with some minor points:

Maybe Section 5 should be renamed "Coordinate systems and domain" to recognise this new construct.
I think the presence of a dimensions attribute can be taken as defining the variable as a domain variable rather than a data variable - is that right? If so, and if it's not stated, I think it should be. Maybe also it should not be allowed to have a variable to have both dimensions and a dimensions attribute, to avoid confusion.
A scalar data variable has only one data value. A domain variable also has one data value (you can't have a netCDF variable with no data values). Is there really a need to allow domain variables for scalar domains, with the possibly surprising empty dimensions attribute?
You say, "It is of arbitrary type since it contains no data." I think it would be clearer to say e.g., "The variable should be a scalar (i.e. it has no dimensions) of arbitrary type, and the value of its single element is immaterial."
The conformance document would be more future-proof if you didn't explicitly list the attributes which aren't recommended, and refer instead to Appendix A.
I find the sentence describing this attributes as rather hard to understand. It says

It is recommended that a domain variable does not have any other attributes that are also used to directly describe data values, defined in [attribute-appendix] as those attributes that are used for non-coordinate data which also do not have domain variable nor global use.

for which would suggest

It is recommended that a domain variable does not have any of the attributes marked in Appendix A as applicable to data variables except those which are also marked as applicable to domain variables.

Typo in "blank separated list of the dimensions names".

Jonathan

Dear @davidhassell , I also support this proposal. This will be of benefit to remote sensing users and would ensure that we can implement the ongoing work discussed in https://github.com/cf-convention/discuss/issues/37 in a way that is compatible with existing processing systems.

@JonathanGregory  Thanks for your thoughtful comments. Responses inline ...

Maybe Section 5 should be renamed "Coordinate systems and domain" to recognise this new construct.

Good idea

I think the presence of a dimensions attribute can be taken as defining the variable as a domain variable rather than a data variable - is that right?

That's right.

If so, and if it's not stated, I think it should be. Maybe also it should not be allowed to have a variable to have both dimensions and a dimensions attribute, to avoid confusion.

It is already stated that "The presence of a dimensions attribute will identify the variable as a domain variable" (https://github.com/cf-convention/cf-conventions/pull/302/files#diff-0eab4e85fe4c323f70ce4bce0229dbe6R782-R783). It is, however, quite far down that paragraph. It may be better to promote the statement to the first sentence and strengthen it a bit, i.e.
The dimensions of the domain must be stored with the **`dimensions`**
attribute, and the presence of a **`dimensions`** attribute on  a scalar variable will identify the
variable as a domain variable.

I like the idea of being clearer that if  scalar variable has the dimensions attribute then it has to be  a domain variable. This is slightly different to disallowing the attribute on non-scalar variables.

You say, "It is of arbitrary type since it contains no data." I think it would be clearer to say e.g., "The variable should be a scalar (i.e. it has no dimensions) of arbitrary type, and the value of its single element is immaterial."

That's better. (Aside: We should also update the text for other similar containers. I cut-and-paste my text from grid mappings.)

The conformance document would be more future-proof if you didn't explicitly list the attributes which aren't recommended, and refer instead to Appendix A.

OK

I find the sentence describing this attributes as rather hard to understand. It says

I see what you mean. I like your text better. We should also change Appendix A from saying "D for variables containing non-coordinate data" to "D for data variables", then.

A scalar data variable has only one data value. A domain variable also has one data value (you can't have a netCDF variable with no data values). Is there really a need to allow domain variables for scalar domains, with the possibly surprising empty dimensions attribute?

I can only argue by counter example, here. Consider the domain of:
dimensions:
variables:
    double x ;
        x:standard_name = "global_average_sea_level_change" ;
        x:coordinates = "time" ;
        x:units = "rod" ;
    double time ;
        time:units = "days since 2020-09-24" ;

It would be:
dimensions:
variables:
    char domain ;
        x:dimensions = "" ;
        x:coordinates = "time" ;
    double time ;
        time:units = "days since 2020-09-24" ;

... the presence of a dimensions attribute on  a scalar variable will identify the
variable as a domain variable.

There is some confusion here ... Are we saying that a domain variable must be a scalar? We don't insist on that for grid mapping and geometry variables (although it is recommended for grid mapping variables). If we say that for a domain variable, we should say the same for grid mapping and geometry variables - which I think would be a defect change (on the grounds that these variables were never intended to contain meaningful data arrays).
Either way, this brings me back to @JonathanGregory's suggestion of disallowing dimensions as a data variable attribute, which I now think is OK (for CF versions starting at the one in which the domain variable goes in). So that first sentence could become
The dimensions of the domain must be stored with the **`dimensions`**
attribute, and the presence of a **`dimensions`** attribute identifies the
variable as a domain variable. Therefore the *`dimensions`** attribute must
not be present on any variables that are to be interpreted as data variables.

The phrase variables that are to be interpreted as data variables means that variables that are referenced by data variables (such as auxiliary coordinate variables) may indeed have a dimensions attribute, but in that case such a variable cannot also be used as independent data variable (see, for example, the section "Interpreting CF-netCDF files" in https://doi.org/10.5194/gmd-10-4619-2017).

Dear @davidhassell ,
I also support this proposal. This will address some of the uses cases discussed in cf-convention/discuss#37, Standard way to define subsampled coordinates,, including the need for tools to be able to pre-process coordinates in a meaningful manner, without needing to access the data variables.

@davidhassell
A couple of comments on the proposed text:
Possibly it could be made clearer in the text that multiple domain variables may exist in a file. The text uses the plural form in a few places, like in the heading 5.8 Domain Variables, but mostly uses the singular form. The current CF convention document uses plural in most places when describing  variables, attributes and dimensions.
Also, does any particular restrictions apply when having multiple domain variables? I would assume that for a particular domain, only one domain variable is permitted?

Dear @davidhassell
You're right that we don't require container variables to be scalar, and I suppose we don't need to, but I assume that they would normally be scalar, since any data they contain is a waste of space. So "the presence of a dimensions attribute on a variable" indicates it's a domain variable. Thanks. My suggestion for disallowing domain variables for scalar domains is that the main purpose of the domain variable is to declare the domain without having to include any data. Since a scalar variable contains no more data anyway than a domain variable, that argument doesn't apply.
I can't see a need for more than one domain variable to describe any domain, but it seems harmless. A domain is defined by the things which are attached to a domain variable. Hence two domain variables describing the same domain would by definition have equal contents. That is redundant but doesn't cause a problem, just as one might have a copy of a data variable.
Cheers
Jonathan

Dear @AndersMS
Thanks (and to @erget) for the reference to the subsampled coordinates issue.

Possibly it could be made clearer in the text that multiple domain variables may exist in a file. The text uses the plural form in a few places, like in the heading 5.8 Domain Variables, but mostly uses the singular form. The current CF convention document uses plural in most places when describing variables, attributes and dimensions.

That would be fine. The last sentence in the new text could changed to read:
"Multiple domain variables may exist in a file, with or without other data
variables. Note that the data variable attributes describing its
domain can not be replaced by a reference to a domain variable."

Also, does any particular restrictions apply when having multiple domain variables? I would assume that for a particular domain, only one domain variable is permitted?

As @JonathanGregory says, this is no problem. I don't think that this needs special mention, as this is has always been true of all types of variables.
@JonathanGregory and all - I'll update the PR next week for the various new bits of text.

Hi @davidhassell and @JonathanGregory,

As @JonathanGregory says, this is no problem. I don't think that this needs special mention, as this is has always been true of all types of variables.

From a user point of view and for the ease of discovering the domains, it would appear attractive if:

there is only one domain variable per domain
if domain variables are used in the file, all domains must have a domain variable

It would also support better the use case stated by @oceandatalab for accessing coordinate variables without accessing data variables under Standard way to define subsampled coordinates #37.
For other data variables, we do need to permit multiple instances for things to work, and cannot excluded that some of these are copies of each other.
If there is no similar need for permitting copies of domain variables, I guess it would be cleaner and more beneficial not to permit copies.

Hi @AndersMS,
I don't see the use-case for restricting a dataset to have at most one domain variable, as datasets already can contain multiple implicit domains defined by data variables, so I think it makes sense to mirror that situation.
I open to saying that a file must contains either data variables or domain variables, but never both. What do others think?

Hi @davidhassell,
It was not my intention to propose only one domain variable per file, but to suggest one domain variable per domain (so no copies) :-)
So the if

there is only one domain variable per domain

and

if domain variables are used in the file, all domains must have a domain variable

then a user can easily search a file for all domain variables and the resulting list of domains will be complete and without copies. It would just appear convenient for discoverability.

Hi @AndersMS,
OK, I see. However, I don't see how we can enforce that any two domain variables in a dataset refer to different domains. It may be desired and of note to have multiple domains, some of which happen to be equal. We would also have to define "equal", which is another problem ...
I think that this is comes down to a user community choice - for example, a project could insist that, for its outputs, a dataset containing domains  must contain only one. This would similar to, say, the CMIP project which favours only one data variable per dataset - a local restriction that goes beyond CF but is useful to its users.

Should we instead identify a domain variable by having a cf_role attribute, with value "domain"? This would remove any ambiguity about what, or isn't, a domain variable.

It seems to me that the presence of a dimensions attribute is a sufficient and clear indication that it's a domain variable. Using cf_role as well would introduce redundancy and hence probable inconsistency.

Thanks, @JonathanGregory.
OK - that's fine by me. It's a good point about redundancy.

Hi @davidhassell

I don't see how we can enforce that any two domain variables in a dataset refer to different domains. It may be desired and of note to have multiple domains, some of which happen to be equal.

Thank you for the reply, I agree that it is better to keep that flexibility and withdraw my proposal regarding a single domain variable per domain.

Hello, I wrote before:

I open to saying that a file must contains either data variables or domain variables, but never both. What do others think?

I have since learned that there is neither a current desire nor use-case for restricting a mixture of domain and data variables, so I withdraw the suggestion.

We are also in support of @davidhassell 's proposal.
Several field variables may share the same domain (output parameters computed on the same grid for numerical model simulations, measurements and derived geophysical data acquired by an instrument in remote-sensing, etc...) but the current conventions define the domain only as an abstract concept which is implemented with attributes on the field variables: in order to identify the domains available in a file (one of our use cases), you have to analyze all the field variables available in the file, parse their attributes to extract domain-related information and then compare the extracted domains to remove duplicates. So listing domains is possible today but it sure is more involved than it should.
Materializing the domain variables as proposed here would make this process a lot easier and probably result in a clearer description of the data.
In the changes proposed in https://github.com/cf-convention/cf-conventions/pull/302, it is stated that:

The constructs contained by the field and domain constructs cannot exist independently, with the exception of the domain construct itself that may be part of a field construct or exist on its own

and

In CF-netCDF, domain information is stored either implicitly via data variable attributes (such as coordinates), or explicitly in a domain variable. In the latter case, the domain exists without reference to a data array.

Does it mean that when a domain construct is part of a field construct it has to be stored exclusively via attributes (as it is done with the current conventions) or is it possible to also have a reference to a domain variable?
Something like the following pseudo-CDL:
float some_field1(time, track, scan);
  : coordinates = "time lat lon";
  : grid_mapping = "crs";
  : domain = "some_domain";

float some_field2(time, track, scan);
  : coordinates = "time lat lon";
  : grid_mapping = "crs";
  : domain = "some_domain";

int some_domain;
  : dimensions = "time track scan";
  : coordinates = "time lat lon";
  : grid_mapping = "crs";

// Attributes for these variables have been omitted for conciseness
float lon(track, scan);
float lat(track, scan);
double time(time);
int crs;

It would still be compatible with existing software because domain information remains available as attributes on the field variables, but it would also clarify the relation between a domain variable and the field variables that use this domain.

Hi @oceandatalab,
Thank you describing your use case that would be benefited by a domain variable.

Does it mean that when a domain construct is part of a field construct it has to be stored exclusively via attributes (as it is done with the current conventions) or is it possible to also have a reference to a domain variable?

I am proposing that domain variable references should not be allowed from a data variable. This is to preserve backwards compatibility and to avoid redundancy (in the senses of design principles 10 and 6).
We must be careful not to confuse CF data model constructs with netCDF variables - the data model has been designed to be independent of the netCDF encoding. In the modified data mode proposed here, a field construct may contain a domain construct, but that in no way forces the netCDF representation of the field construct to contain an explicit reference to a domain variable.
Thanks,
David

Sorry for the confusion between field/data and construct/variable.
My question was about having a reference to the domain variable in addition to the attributes that already describe the domain (implicitly) on the data variable so I am not sure how it would break backwards compatibility.
I agree that it introduces some redundancy, but I would argue that is already the case when several data variables share the same domain and each of these data variables defines this very same domain implicitly with their attributes. Allowing a reference to the domain variable on data variables would add  a way to check that the attributes on these data variables (that are meant to describe the same domain) are consistent with each other.

Hi @oceandatalab  OK - we're in a slightly grey area here! This is where the design principles  can really help.
Principle 6 says
"To avoid potential inconsistency within the metadata, the conventions should minimise redundancy."
and principle 10 says
"... there is a strong preference against introducing any new capability to the conventions when there is already some method that can adequately serve the same purpose (even if a different method would arguably be better than the existing one)."
So to minimise redundancy, we should not allow both a domain variable reference and the other data variable attributes to exist at the same time; and we shouldn't allow a domain variable variable reference anyway because we already have adequate (even if improvable) means of conveying the same information.
My original comments about backwards compatibility weren't strictly right, I realise. Allowing a domain variable reference instead of the usual data variable attributes would not be a CF backward compatibility issue (though it would be a little tough on software writers), but it would fall foul of principle 10.
Thanks,
David

Point of order, I updated the moderator comments in the description above.

Thanks for the summary, Dave.
@oceandatalab - are you OK with not allowing a domain variable reference from a data variable?
There hasn't been any comment on the changes to the text of the data model. It would be great if someone could review the suggested changes to appendix I in PR #302.
All the best,
David

If I'm reading the right thing, the text of Appendix I contains the statement "It is not a construct of the data model, but is an abstract concept that is useful for understanding it." That should be deleted now (since that's the whole point :smile: )

I'm not sure what's going on here, but were you reading the rich diff? That seems to be having intermittent difficulties in showing the modified image caption (where that text was deleted from),  and also isn't showing the modified image). The side-by-side diff is OK, though, I think.

Yes, I was reading the rich diff. That must be it, then. Jonathan

@davidhassell Sorry for the delay I just came back from vacation.

My original comments about backwards compatibility weren't strictly right, I realise. Allowing a domain variable reference instead of the usual data variable attributes would not be a CF backward compatibility issue (though it would be a little tough on software writers), but it would fall foul of principle 10.

I think you meant that allowing a domain variable reference in addition to the usual data variable attributes would not be a CF backward compatibility issue, whereas replacing the usual attributes by a reference to a domain variable would break backward compatibility.
Allowing a domain variable reference from a data variable is not strictly necessary for our use case, so I do not consider this to be a blocking point. However, I still think it should be discussed because I am not sure rule 10 applies here:

there is a strong preference against introducing any new capability to the conventions when there is already some method that can adequately serve the same purpose

For me the reference to the domain variable does not serve the same purpose as the usual data variable attributes because this reference is meant to identify the domain uniquely, and this information is not provided by the usual attributes, so I would consider the reference as additional information, not a replacement/competitor.
Being able to clearly identify the domain of a data variable, and therefore the data variables that share a domain, is definitely an operation that could be made simpler and this goal could be achieved very easily by a domain variable reference. If the reference is an issue due to its nature, then one could simply replace it by a unique identifier string, but if domain variables are available then it would be a shame not to use them for that purpose too.

Dear @oceandatalab
I think that allowing data variables to refer to the domain with a single reference instead of providing the domain information by various references on the data variable would be a drastic change to the convention. Although not backwards incompatible in the sense that it wouldn't invalidate existing conventions or data, it would require all software to be rewritten to support this different method. I think that would be a bad decision. Alllowing a domain reference in addition to the other means of describing the domain by the data variable would be redundant, and therefore potentially inconsistent, which also doesn't sound good to me.
I understand your argument that you want to use the domain reference as a way to identify the domain uniquely, but I would argue that you can't really depend on that method. It will only work within a single file (within which one can depend on variable names as references) and netCDF datasets aren't necessarily contained in single files. Hence you still need to be able to decide whether domains are equal by inspecting the metadata and coordinates. You would have to be able to do that also if assembling a dataset from various sources.
Best wishes
Jonathan

Hi @JonathanGregory

I think that allowing data variables to refer to the domain with a single reference instead of providing the domain information by various references on the data variable would be a drastic change to the convention. Although not backwards incompatible in the sense that it wouldn't invalidate existing conventions or data, it would require all software to be rewritten to support this different method. I think that would be a bad decision.

I never suggested to use a single reference instead of  the usual data variable attributes. This is something that is only mentioned in https://github.com/cf-convention/cf-conventions/issues/301#issuecomment-709406316 and I think it was just due to a misunderstanding or a typo. So we agree that breaking backward compatibility would be a bad idea.

Alllowing a domain reference in addition to the other means of describing the domain by the data variable would be redundant, and therefore potentially inconsistent, which also doesn't sound good to me.

Here we disagree:

without this proposal, if there is a single domain shared by several data variables and each data variables describes this domain with the usual attributes, you already have redundancies as the single domain is described on each data variable, and there is no way to check that the domain description is consistent among these data variables so the inconsistency risk is quite high.

with this proposal but without the domain variable reference that I mentioned, we gain the ability to access domain information directly (which is a very good thing) but we create an additional description of the domain, which could conflict with the description provided on data variables (that may already conflict with each other as seen in 1.). So the risk of inconsistency is slightly higher than in 1.

with this proposal and with the domain variable reference, you still have all the redundant descriptions that were already there in 1. and 2. but now you have a tool that allows you to automatically detect consistency issues that arise from the pre-existing redundancy problem. So from my point of view you not only get a clearer description of the data but also a way to validate domain information across redundant definitions (which could be implemented or not in a software for automatic validation).

I understand your argument that you want to use the domain reference as a way to identify the domain uniquely, but I would argue that you can't really depend on that method. It will only work within a single file (within which one can depend on variable names as references) and netCDF datasets aren't necessarily contained in single files. Hence you still need to be able to decide whether domains are equal by inspecting the metadata and coordinates. You would have to be able to do that also if assembling a dataset from various sources.

I admit I have no experience with multi-file netCDF datasets so I may not fully grasp all the implications that adding the domain variable reference would have on this data structure. I quickly browsed the NcML documentation and it seems to allow the creation and modification of attributes on the variables of the multi-file dataset, so someone who wants to aggregate files from several sources could write a NcML file that correctly defines the domains and their references in the view offered by the multi-file dataset. But again, I have never worked with this kind of datasets so I may be completely wrong.
Cheers,
Sylvain

Hi Sylvain,

My original comments about backwards compatibility weren't strictly right, I realise. Allowing a domain variable reference instead of the usual data variable attributes would not be a CF backward compatibility issue (though it would be a little tough on software writers), but it would fall foul of principle 10.

I did indeed mean "instead of" rather than "in addition to". Allowing a domain variable reference instead of the usual attributes would neither disallow the usual attributes, nor change their meaning, so no backwards incompatibility. This is similar to the grid_mapping extension that was introduced at CF-1.7. In this case the old single grid mapping case was still supported in the new version, but a new syntax was created for multiple grid mappings. This new syntax is not understood by software built on CF-1.6.

Being able to clearly identify the domain of a data variable, and therefore the data variables that share a domain, is definitely an operation that could be made simpler and this goal could be achieved very easily by a domain variable reference. If the reference is an issue due to its nature, then one could simply replace it by a unique identifier string, but if domain variables are available then it would be a shame not to use them for that purpose too.

We shouldn't allow a domain variable instead of the usual domain definition because a) there was no use case for it and b) because it would require all software to be rewritten to support this different method. Even though allowing this would make it easier, in limited circumstances, to see "by eye" if two data variables shared a domain, I don't think that is a use case on its own. These limited circumstances only arise when informally comparing multiple data variables with domain references within the same file (as opposed to the same dataset). Library software would not generally benefit from this as it has to store the constituent parts of the domain (cell measure, grid mappings, coordinates, etc) regardless of how it was encoded. If a stronger use case were to present itself in the future I would welcome this being reviewed, but suggest that for now we do not allow this.
With regards the pre-existing redundancy issue, data variables are essentially independent entities. Therefore there is no redundancy if, say, two data variables have the same coordinates attribute value. We have to trust dataset providers to produce the datasets that they intend, and that is made easier by not allowing the same information to be encoded twice for each data variable. If this were allowed, and the two methods were inconsistent, we have no way of knowing which is correct.
Anyway, I think (if I've read everything correctly) we are in agreement that a domain variable variable reference should not be used in addition to nor instead of the usual domain definition (data variable coordinates attribute, etc). Which is good for the progress of this issue.
Thanks,
David

Hi David,

I did indeed mean "instead of" rather than "in addition to". Allowing a domain variable reference instead of the usual attributes would neither disallow the usual attributes, nor change their meaning, so no backwards incompatibility. This is similar to the grid_mapping extension that was introduced at CF-1.7. In this case the old single grid mapping case was still supported in the new version, but a new syntax was created for multiple grid mappings. This new syntax is not understood by software built on CF-1.6.

Ok, it was confusing because no one talked about using references to domain variables instead of the usual attributes before, so I thought you were replying to my in addition question.

We shouldn't allow a domain variable instead of the usual domain definition because a) there was no use case for it and b) because it would require all software to be rewritten to support this different method.

Agreed.

Even though allowing this would make it easier, in limited circumstances, to see "by eye" if two data variables shared a domain, I don't think that is a use case on its own. These limited circumstances only arise when informally comparing multiple data variables with domain references within the same file (as opposed to the same dataset). Library software would not generally benefit from this as it has to store the constituent parts of the domain (cell measure, grid mappings, coordinates, etc) regardless of how it was encoded. If a stronger use case were to present itself in the future I would welcome this being reviewed, but suggest that for now we do not allow this.

With regards the pre-existing redundancy issue, data variables are essentially independent entities. Therefore there is no redundancy if, say, two data variables have the same coordinates attribute value.

I get what you mean, but independence achieved by denormalization introduces redundancy as soon as two entities have some elements in common, and therefore makes the data prone to inconsistency issues. Even if each data variable has its own domain instance (i.e. its own set of coordinates, grid_mapping, etc... attributes) , if two or more data variables share the same domain (multiple parameters measured by the same instrument for example) then these instances of the domain are redundant, I don't see how it could be otherwise.

We have to trust dataset providers to produce the datasets that they intend, and that is made easier by not allowing the same information to be encoded twice for each data variable. If this were allowed, and the two methods were inconsistent, we have no way of knowing which is correct.

The idea is not to identify which definition is correct but to detect when two definitions of the same domain are incompatible or not as complete as they could. The goal is to offer a way for data producers to detect errors (multiple definitions of a single domain that are not compatible with each other) and consistency issues (when two variables share the same domain but one of them only provides a minimal definition while the other has a detailed description), therefore the means to improve the overall quality of the files they generate before these files are distributed to end users.
But again, it was just a suggestion for a small improvement of the proposal, it is absolutely not a blocking point for us.
Cheers,
Sylvain

Thanks for all of the discussion. I understand (from these comments and off-line conversations) that there are no objections to the pull request as it stands. @dblodgett-usgs would you agree?
It would be still be good to get some comment here on the data model changes.
Many thanks,
David

I agree @davidhassell and I don't think the subsequent conversation warrants any further summary above. Thanks for the god conversation all.

I've had a look at the latest draft and still support this proposal. The changes are mostly straightforward as they enshrine as a construct what was until now a concept that has served the community well. Thank you @davidhassell for the painstaking work here, I believe this will be a benefit to the community.


Too many open files even with large ulimits

Describe the bug
We are trying to deploy Mage in production using an ECS cluster setup as recommended for AWS (using Terraform).
But after a few minutes of uptime the Dashboard becomes inaccessible. Initially the thoughts were some networking/connectivity issue, but after a look at the Cloudwatch logs we have isolated it to a ulimit issue.
OSError: [Errno 24] Too many open files and zmq.error.ZMQError: Too many open files
We have increased the size of the task and the ulimits but still no luck. Our current settings are 512 CPU and 4GB Memory with the nofile ulimits defined as below in the Task Definition:
 "ulimits": [
        {
          "name": "nofile",
          "softLimit": 32768,
          "hardLimit": 65536
        }
      ]

There are no pipelines created, this is a fresh copy of Mage, with just the example_pipeline in it. Mage version is 0.9.8
Screenshots

Thanks for reporting it. We'll look into it.

Thanks @wangxiaoyou1993 we are currently blocked in production due to this. More contextual logs:

Thanks @wangxiaoyou1993  we are currently blocked in production due to this. More contextual logs:

Thanks @wangxiaoyou1993 we are currently blocked in production due to this. More contextual logs:

More information: The memory utilisation of the task (2GB set in the case below) steadily rises (no pipelines except the example Titanic one), and then starts failing. The drops in memory is when the new task was initialised to replace the one that was erroring out

Does setting a higher ulimit help?

Hi @wangxiaoyou1993 as mentioned in the bug report, the ulimit is set to a hard limit of 65536, which is very large. According to the mage docs: you can set the environment variable ULIMIT_NO_FILE to a larger number (e.g. 8192 or 16384) and our current value is x4 larger

I see. @dumim do you use the terraform scripts provided by Mage?

@wangxiaoyou1993 yes we do https://github.com/mage-ai/mage-ai-terraform-templates

@wangxiaoyou1993 yes we do https://github.com/mage-ai/mage-ai-terraform-templates

Ok, I'm looking into this issue.

Hey @dumim , can you run ulimit -n in your Mage terminal to check the limit?

And you can further increase the limit to 131072.

You can also try upgrading to latest version. The demo.mage.ai  runs with latest version and limit 32768 without problems

Thanks @wangxiaoyou1993 . The terminal cannot be interacted with for some reason... we have DISABLE_NOTEBOOK_EDIT_ACCESS set to 0. Am I doing this wrong, or should I raise a bug request for this?
Thanks we have now deployed v0.9.16 to production. Will report back after we observe the memory for at least 24 hours.

@wangxiaoyou1993 from an initial look it seems like the memory is rising again, with no usage from our end

CPU is stable FYI

Thanks for the updates! We're looking into the memory increase issue.

@wangxiaoyou1993
Can confirm that the memory kept rising by itself, until it errored out and the target group restarted it

CPU is .5 vCPU and Memory is set to 2GB at the moment FYI

@wangxiaoyou1993 Hi, just checking in on this. Aware that this is a complex issue. We are currently from being production-ready due to this. Attached is confirmation that the memory keeps rising until the health checks fail and the target group starts a new task

Hi @dumim , it's a bit hard to investigate this issue but we're prioritizing it this week. We'll definitely get it resolved this week. Sorry about the delay.

@wangxiaoyou1993 no worries, thanks for the update

@wangxiaoyou1993 checking in to see if there is any update on this? We are blocked from going live in production

@wangxiaoyou1993 checking in to see if there is any update on this? We are blocked from going live in production

Hi @dumim , we suspect the memory leak is from some of the dependency packages. So we upgraded the dependencies and currently doing some testing. Hope to get some results tomorrow.

Thanks for the update @wangxiaoyou1993

Thanks for the update @wangxiaoyou1993

Just to give an update here. We're releasing a fix for this issue tomorrow.

Thanks @wangxiaoyou1993 we are looking forward to testing it

@dumim The fix is included in mageai/mageai:latest. You can test it.

Thanks @wangxiaoyou1993 we will test this and report back

@wangxiaoyou1993 can confirm that after a day of letting it run, it is working properly. Thanks so much for the effort on your part to getting this tricky bug fixed!


Kudos for KVO/KVC with StatusBar and LongPress

Just a big thank you :+1:
Your implementation of handling the visibility of the StatusBar when the NavigationBar is hidden by gestures is far superior to the usually seen approach which suggests putting a UIView below the StatusBar. It fixes a visual glitch seen with this common approach, too: the separator of the NavigationBar was moving over the background view. To be honest, it took me a moment to fully understand what you were doing but keeping track of previous superView, adjusting visibility, y Position and alpha of SubViews is very elegant.
The same thank you for the longPress handling which adds the gesture to the BarButtonItem's View.
Both solutions are very elegant and made me fully understand KVO/KVC and the power it provides. I am using this in Swift in my own Code, not only related to an implementation of a Browser.
If I'd a vote, this should become part of UIKit, especially the StatusBar handling is something I always missed with .hidesBarsOnSwipe.
Cheers,
Frederik

You're welcome! 😃
Of course, it's a little bit risky depending on what you are trying to achieve, absolutely. But with nil checks and try/catch blocks safe enough for me to sleep good at night. I'll comment the code with 'Here be dragons' 😂
Cheers,
Frederik


PredMemOpt Extract at Store site instead of load site

This is the last missing piece that will allow me to land my patch updating pred mem opt for ownership.
rdar://31521023

@swift-ci test

@swift-ci test source compatibility


feature request: implement physical_path() for VirtualDevice

libevdev has libevdev_uinput_get_devnode for this.
It would be nice to have this feature.

Thanks for #72, could we have a release containing that?

@jian-lin evdev 0.11.6 is now available on crates.io.


jetbrains: 2024.2.4 -> 2024.2.5

Description of changes
Things done
Ran the update script.

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

hello, intellij idea 2024.2.3 and rustrover 2024.2.2 released since you made your pr, would it be possible to add them or should a separate pr be made?

@uku3lig thanks for the heads-up, re-updated.


Change page title for organizations that have requested or received sponsorship

For an organization that has requested or received sponsorship, we should make the HTML title of the page match the heading shown on the page itself. So, in both of those cases, it should be "Zulip Cloud billing for org-name", not "Request sponsorship...".

@amanagr FYI

I have gone through the issue and have a proper understanding of the issue. Can you please assign me this issue so I can start working on it? @alya

@alya can I work on this ?

Hey @alya @amanagr **I have changed the title of sponsorship page ** to display the title in the format mentioned above.

Can I raise a pr for it ?. **Thanks ** in anticipation

@timabbott thanks for the feedback sir. Will abide by it moving forward.


Add support for reading/writing objects with ReadOnlyMemory<byte> properties

I'm new to using EfficientDynamoDb, so apologies if I have this wrong, but I've been through the documentation and code and don't think its currently possible to read/write objects with binary properties of type ReadOnlyMemory<byte>. I need to write buffers to DynamoDB that have been produced by compression, and the data is written into a buffer that is larger than the final compressed data (which is typical for writing compressed data).
Currently, it seems I need to copy this data into a new byte array so I can set a byte[] property on the object I want to write to DynamoDB.
I propose modifying the BinaryAttributeValue struct to hold a ReadOnlyMemory<byte> field instead of a byte[] field and then add a new constructor accepting a ReadOnlyMemory<byte> parameter. The existing constructor would of course be retained (byte[] has an implicit cast to ReadOnlyMemory<byte>).
The BinaryAttributeValue.Write method would then be updated to:
public void Write(Utf8JsonWriter writer)
{
  writer.WriteStartObject();
  writer.WriteBase64String("B", _value.Span);
  writer.WriteEndObject();
}

And the BinaryAttributeValue.ToString method would be updated to:
public override string ToString()
{
  return Convert.ToBase64String(_value.Span);
}

This would allow me to write my own custom converter for a ReadOnlyMemory<byte> property. Although, it would be even better if the library shipped one out of the box.

Hi @bill-poole, thanks for raising this issue. You're right, with the current API you have to copy data into a new byte array to be able to serialize it.
Your proposal looks interesting and should generally work, but I have one concern about it. There is a public property byte[] Value in BinaryAttributeValue so changing _value field type to ReadOnlyMemory<byte> requires data copy to implement this property:
public byte[] Value => _value.ToArray();

I would like to avoid doing changes like this because it can introduce a silent performance degradation for other users. Also, we can't simply delete this property or change its type to ReadOnlyMemory<byte> because it will be a breaking change. In general, I'm not against introducing a breaking change but we need to think it through to make sure the change covers as many use cases as possible. I'd like to take some time on the weekends to think through this approach and check if there are other ways of implementing this one. If you have any ideas for this issue, feel free to share them as well.
Also one question about your use case that could be outside of the scope of the library but still. If your buffer is larger than the final compressed data, how would you know what part of the ReadOnlyMemory<byte> should be serialized? Do you plan to write a slice of it to the entity property or something else? Is there anything we can do on the library level to simplify the use case?

Thanks for considering this feature request @firenero. The compressor writes data to a byte array (byte[], which I rent from ArrayPool<byte>.Shared) and returns the number of bytes written. I can then get a ReadOnlyMemory<byte> from the byte array and slice it using the size returned from the compressor to create a new ReadOnlyMemory<byte> that is the subset of the larger byte array containing the compressed data. It's that slice that I then need to write to DynamoDB.
Alternatively, the compressor can write to an ArrayPoolBufferWriter<byte> (from the high performance community toolkit) instance, which has a WrittenMemory property, which gives me a ReadOnlyMemory<byte> over the compressed data.
One idea I had for how this could be approached:

create a new/separate dedicated attribute (e.g., BinaryAttribute<T> where there is no generic constraint on T, but where in the BinaryAttribute<T> constructor, it checks that T is either ReadOnlyMemory<byte> or byte[] (and possibly in the future, ArraySegment<byte> or any other type that can hold binary data);
add an AttributeValue.AsBinaryAttribute<T> method to convert to a BinaryAttribute<T> instance; and
add an implicit cast from BinaryAttribute<T> to AttributeValue, which checks the type of T (knowing it must be one of a specific set of supported types) and handles the cast differently for each type?

That would then allow support for more types in the future so this problem doesn't come up again. Alternatively, you could create a specific ReadOnlyMemoryBinaryAttribute specifically for ReadOnlyMemory<byte> properties.

Yes, your converter should work, good idea! I think including this converter by default in the library is the easiest way to support the ReadOnlyMemory<byte> properties without introducing any compatibility issues. I'll think about it over a few days before adding it, though. Meanwhile, I suggest using this converter so you're not left waiting for a new release of the library.

Okay great!
Given that the Read(in AttributeValue) and Write(ref ReadOnlyMemory<byte>) method implementations are not used when the Read(ref DdbReader) and Write(in DdbWriter, ref ReadOnlyMemory<byte>) methods are overridden, shouldn't the CustomIntConverter example in the documentation throw a NotImplementedException in those methods (like in the proposed ReadOnlyMemoryBinaryDdbConverter), rather than converting to/from NumberAttributeValue?
I was a bit confused by that example in the documentation having effectively two conversion implementations - one low-level conversion directly to/from JSON, and one higher-level conversion to/from NumberAttributeValue.
Furthermore, it seems the internal converters in the EfficientDynamoDb source code also do the same (e.g., BoolDdbConverter and GuidDdbConverter). Would it not be best to remove the unused higher-level conversion logic in those converters and just throw NotImplementedException?

Good point. I've realized that "basic" Read/Write method implementations are used in Document extensions ToObject<T>()/ToDocument<T>() calls so library converters have to provide them. Your converter will work fine as long as you don't convert objects to documents and vice versa so feel free to use it as a workaround for now.
Unfortunately, we can't ship this converter as it is in the library because it can lead to hidden errors in code. In theory, we can ship it with Read(in AttributeValue) and Write(ref ReadOnlyMemory<byte>) implementations that allocate but I'm looking for a better solution atm.

Okay thank you! I really appreciate it.

I found a way to write a converter that supports Document extensions ToObject<T>()/ToDocument<T>() without allocating in Write(ref T value) method. It's in PR #242 and going to be included in the next release.

Thanks @firenero. I took a look at the source code and I think there might be a problem with the code below from BinaryToMemoryDdbConverter.cs. The code uses MemoryMarshal.TryGetArray to attempt to get the array backing the ReadOnlyMemory<byte>, which returns an ArraySegment (if the ReadOnlyMemory<byte> is indeed backed by an array), but then disregards the ArraySegment's Offset and Count properties. The ReadOnlyMemory<byte> could be any arbitrary slice of an array (not to mention something else like unmanaged memory).
If the ReadOnlyMemory<byte> is not backed by an array, then the implementation allocates, which is not ideal. And if it is backed by an array, and is a slice of that array, then the whole array rather than the slice is written to the database.
public override AttributeValue Write(ref Memory<byte> value)
{
    var array = MemoryMarshal.TryGetArray((ReadOnlyMemory<byte>)value, out var segment)
        ? segment.Array
        : value.ToArray();
    Debug.Assert(array != null);

    return new AttributeValue(new BinaryAttributeValue(array));
}

then disregards the ArraySegment's Offset and Count properties.
if it is backed by an array, and is a slice of that array, then the whole array rather than the slice is written to the database.

You're right about this, I need to narrow down the happy path only to cases when Offset == 0 and Count == value.Length.

the implementation allocates, which is not ideal

Agree, it's not ideal but on practice the path with allocation will be very rare. In most cases, Memory<byte> is backed by an array and there is only one case (or two, but they are similar) when the allocation can possibly happen. The conditions are:

Entity is created manually in code (not read from the DDB)
The memory field is either not backed by an array OR it's a slice of an array. Note that if the memory field is backed by a not-sliced array, this condition is not true.
context.ToDocument(item) is used to convert the entity to document.

In all other cases, it won't allocate because either this converter method is not called or the happy path without allocation is possible.
Example of the code that will allocate:
public class TestObj
{
    [DynamoDbProperty("mem")]
    public ReadOnlyMemory<byte> Memory { get; set; }
}

var arr = new byte[] { 0x69, 0xa7, 0x5d, 0xff};
var item = new TestObj
{
    Memory = arr.AsMemory(1, 2) // Take only 0xa7 and 0x5d bytes
};

// This conversion allocates new array for 0xa7 and 0x5d bytes
var doc = context.ToDocument(item);

Similarly, it will allocate if your memory represents an unmanaged data but that's also happens quite rarely. In all other cases it will NOT allocate so I think it's a decent tradeoff overall. Let me know what you think about it.

I suspect that if someone is using Memory<byte>/ReadOnlyMemory<byte> instead of byte[], they are likely relying on the slicing ability provided by Memory<byte>/ReadOnlyMemory<byte>, meaning that the converter will allocate, otherwise I assume they could just use byte[].
That is actually my scenario. i.e., I'm using ReadOnlyMemory<byte> instead of byte[] because I need to pre-allocate an array to write compressed data into, and I don't know the size of the compressed data until the compression completes. At that time, I can either copy the compressed data into a newly allocated exact-sized byte[], or I can use a slice of the ReadOnlyMemory<byte>, but then the converter you propose would copy that to a new exact-sized byte[] anyway, albeit only when invoking ToDocument.
I assume the concern/issue with updating BinaryAttributeValue to use a ReadOnlyMemory<T> value instead of a byte[] value is maintaining backwards compatibility of the public API? If so, have you considered updating BinaryAttributeValue as follows:
[StructLayout(LayoutKind.Explicit)]
public readonly struct BinaryAttributeValue
{
    [FieldOffset(0)]
    private readonly ReadOnlyMemory<T> _value;

    public byte[] Value => MemoryMarshal.TryGetArray(_value, out var segment)
        && segment.Count == segment.Array.Length
        ? segment.Array
        : _value.ToArray();

    public Memory<byte> AsMemory()
        => Value;

    public ReadOnlyMemory<byte> AsReadOnlyMemory()
        => _value;

    public BinaryAttributeValue(byte[] value)
    {
        _value = value;
    }

    public BinaryAttributeValue(Memory<byte> value)
    {
        _value = value;
    }

    public BinaryAttributeValue(ReadOnlyMemory<byte> value)
    {
        _value = value;
    }

    public void Write(Utf8JsonWriter writer)
    {
        writer.WriteStartObject();
        writer.WriteBase64String(DdbTypeNames.Binary, _value.Span);
        writer.WriteEndObject();
    }

    public override string ToString() => Convert.ToBase64String(_value.Span);
}

As far as I can tell, this seems to allow BinaryAttributeValue to hold/use ReadOnlyMemory<byte> values while maintaining backwards compatibility.
Furthermore, it would only allocate if someone creates a BinaryAttributeValue using a ReadOnlyMemory<byte> slice, but then reads the value using the Value property or AsMemory() method, which would probably only happen if someone converts an item to a document (using ToDocument) and then back to an object (using ToObject). But I'd imagine that would be very rare.
You would then have a MemoryBinaryDdbConverter as follows:
public class MemoryBinaryDdbConverter : DdbConverter<Memory<byte>>
{
    public override Memory<byte> Read(ref DdbReader reader)
        => reader.JsonReader.GetBytesFromBase64();

    public override Memory<byte> Read(in AttributeValue attributeValue)
        => attributeValue.AsBinaryAttribute().AsMemory();

    public override void Write(in DdbWriter writer, ref Memory<byte> value)
    {
        writer.JsonWriter.WriteStartObject();
        writer.JsonWriter.WriteBase64String(DdbTypeNames.Binary, value.Span);
        writer.JsonWriter.WriteEndObject();
    }

    public override AttributeValue Write(ref Memory<byte> value)
        => new AttributeValue(new BinaryAttributeValue(value));
}

And you would have a ReadOnlyMemoryBinaryDdbConverter as follows:
public class ReadOnlyMemoryBinaryDdbConverter : DdbConverter<ReadOnlyMemory<byte>>
{
    public override ReadOnlyMemory<byte> Read(ref DdbReader reader)
        => reader.JsonReader.GetBytesFromBase64();

    public override ReadOnlyMemory<byte> Read(in AttributeValue attributeValue)
        => attributeValue.AsBinaryAttribute().AsReadOnlyMemory();

    public override void Write(in DdbWriter writer, ref ReadOnlyMemory<byte> value)
    {
        writer.JsonWriter.WriteStartObject();
        writer.JsonWriter.WriteBase64String(DdbTypeNames.Binary, value.Span);
        writer.JsonWriter.WriteEndObject();
    }

    public override AttributeValue Write(ref ReadOnlyMemory<byte> value)
        => new AttributeValue(new BinaryAttributeValue(value));
}

they are likely relying on the slicing ability provided by Memory/ReadOnlyMemory, meaning that the converter will allocate
or I can use a slice of the ReadOnlyMemory, but then the converter you propose would copy that to a new exact-sized byte[] anyway, albeit only when invoking ToDocument.

I think there is a misunderstanding. My version of converter will only allocate if you create an entity in code and then convert it to Document. This example from my previous comment will allocate:
public class TestObj
{
    [DynamoDbProperty("mem")]
    public ReadOnlyMemory<byte> Memory { get; set; }
}

var arr = new byte[] { 0x69, 0xa7, 0x5d, 0xff};
var item = new TestObj
{
    Memory = arr.AsMemory(1, 2) // Take only 0xa7 and 0x5d bytes
};

// This conversion allocates new array for 0xa7 and 0x5d bytes
var doc = context.ToDocument(item);

I can't imagine a scenario when someone need to create an entity then convert it to Document. I believe in most cases someone would want to convert an entity to Document after reading from the DB, that will NOT allocate:
var item = await context.GetItemAsync("pk", "sk");
var doc = context.ToDocument(item); // this won't allocate

Correct me if I'm wrong, but your use case is:

Create an entity with ReadOnlyMemory<byte> property that contains a slice of some byte[] array.
Write this entity to the DB.
At some point read it back from DB.

There is no allocation in the case of writing to DB because the optimized version Write(in DdbWriter writer, ref ReadOnlyMemory<byte> value) is called. The same applies to reading from the DB.

Your idea with using MemoryMarshal.TryGetArray in BinaryAttributeValue looks really interesting - I haven't thought about it before. I will test it in detail later this week.

Furthermore, it would only allocate if someone creates a BinaryAttributeValue using a ReadOnlyMemory slice, but then reads the value using the Value property, which probably shouldn't ever happen?

I have a feeling that it's a similar assumption to my previous one "No one creates an entity and converts it to a document right away" but I could be mistaken. No one is supposed to do it this way but we never know all the people use cases.
Anyways, I will check again your proposed API. Meanwhile, please let me know if we're on the same page about allocations in my current converter approach and if I understand your use case correctly.

I can't imagine a scenario when someone need to create an entity then convert it to Document.

Yes I agree; I couldn't think of a scenario either. However, the DynamoDbContext class has a ToDocument method, which seems to exists for that purpose, so I assumed that there was a use case I was missing. If the scenario were never to arise, then the converter could just throw a NotImplementedException in the Write(ref T) method.

please let me know if we're on the same page about allocations in my current converter approach and if I understand your use case correctly.

Yes I think we are on the same page and I think you understand my use case correctly. My scenario currently doesn't invoke the AttributeValue Write(ref ReadOnlyMemory<byte> value) of the ReadOnlyMemory<byte> converter, which is why I was able to get away with just throwing NotImplementedException. So the converters you proposed wouldn't allocate in my specific scenario.
However, tomorrow's scenarios are hard to predict, so I was trying to come up with a solution that would always allow allocations to be avoided under all circumstances.

@bill-poole I've checked the BinaryAttributeValue implementation you suggested in https://github.com/AllocZero/EfficientDynamoDb/issues/228#issuecomment-2106649684. I don't think we can make this constructor (at least publicly available):
public BinaryAttributeValue(ReadOnlyMemory<byte> value)
{
    _value = value;
}

The data itself is converted to mutable Memory so the read-only contract can be broken. I think for now I'll go with the fixed converters implementation #244 and evaluate doing the proper breaking change to BinaryAttributeValue in the major release. It works well for the majority of use cases and it has no downsides to current workflows since the Memory/ReadOnlyMemory were not supported at all.

This is now available in v0.9.16. Feel free to ree-open this ticket or open a new one if you need anything.

Okay, thanks @firenero.


Menu: Custom names for enchantments.

Describe the solution you'd like
This is one of the functions that I have been asking for since the plugin went on sale, it is to be able to place a different name of the enchantments in the menu so that it is not the same one that appears in the pickaxe.
Example:
Menu: &8> &e&lEFFICIENCY <
Pickaxe: &a¦ &eEfficiency

and support hexcode names y menus.
Example:
Menu: &8> &#d7f171&lEFFICIENCY <
Pickaxe: &a¦ &#d7f171&lEfficiency

@Drawethree Close was solved.
@marlon233-omc if you need more help open another ticket or use this one that reopens if you write.


fix: Fixing OpenAIPrompt class to be able to work with older models like text-davinci-003

Issue
While working with OpenAIPrompt class, I discovered that this class does not work with older models. Upon investigation, I found that for models specified in the list legacyModels, OpenAIPrompt class suppose to use OpenAICompletion instance, for all other model deployment names, OpenAIChatCompletion class is to be used. When setting parameter for OpenAICompletion, it tries to set parameters that are not available in OpenAICompletion, which causes instance creation to fail. There are no current unit tests or integration tests that verified this. In this PR, I am fixing this problem and adding unit test to validate this. I am also cleaning unit test and OpenAIPrompt class.
What changes are proposed in this pull request?
In this PR I am:

Ensuring that new prompt instances are created for each test. This will eliminate chance of one test interfering with another test.
Creating test to validate that for older model deployments, OpenAiCompletion instance is created in OpenAIPrompt class.
Fixing the bug that caused creation of OpenAICompletion object to fail.
Cleaning some code in OpenAIPrompt class.

How is this patch tested?

[X] I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change any dependencies?

[X] No. You can skip this section.
[ ] Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

[X] No. You can skip this section.
[ ] Yes. Make sure you have added samples following below steps.

/azp run

/azp run

/azp run

/azp run

/azp run


Possible side effects for complex queries

Perhaps it's worth expanding the documentation to explain possible side-effects when using paging cookies with complex queries when there is no uniqueness on the primary attribute in the resultset. See https://truenorthit.co.uk/2014/07/19/dynamics-crm-paging-cookies-some-gotchas/ for the detailed walkthrough and explanation.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0e94be8f-e9b5-5557-c450-5c4b851c9791
Version Independent ID: 73e131ef-eabd-16ab-20ed-f1f3168a3411
Content: Page large result sets with FetchXML (Developer Guide for Dynamics 365 Customer Engagement)
Content Source: ce/developer/org-service/page-large-result-sets-with-fetchxml.md
Service: crm-online
GitHub Login: @KumarVivek
Microsoft Alias: kvivek

@mayadumesh  I think that ideally paging cookies should simply work to improve performance in an application as they are intended to and not interfere with results if used incorrectly.
If we have a set of scenarios where paging cookies should not be used, or where they can't be expected to be returned, we ought to document them.
Can you help fill in the blanks?

@mayadumesh @georged
Please see When not to use paging cookies section added to this topic


Relaxing cryptography dependency

The current restriction is no higher than version 2, although for more than two and a half years the third version.
I am currently using 38.0.4 and everything works without any problems.

Hi!
Please, any chance this could get merged soon? @maintainers
Thank you!

Hi! Please, any chance this could get merged soon? @maintainers Thank you!

It is merged, thanks for the reminder!
I'll try to publish a new pypi version soon.

HI @ppawlak
Do you think this new version could be released in https://pypi.org/project/appstoreconnect/ ?
Thank you!

I've just published it on PyPI: https://pypi.org/project/appstoreconnect/0.10.1/
Sorry for the (very) long wait.


Cannot Add a child that doesn't have a YogaNode to a parent with out a measure function

I've been trying to get React Native up and running for a project I'm working on. It works beautifully for IOS, however on Android every time I try to use a Text component I keep getting the error
Cannot add a child that doesn't have a
YogaNode to a parent without a measure function!
Trying to add a ReactVirtualTextShadowNode to a NativeViewWrapper.

my entire code is
import React, { PureComponent } from 'react';
import { Text } from 'react-native';

class App extends PureComponent  {

render() {
    return (
      <Text>Hello World</Text>
    );
  }
}

export default App;

I'm using react native 0.4.3
Any ideas what could be causing this and how I could fix it?

Does it work if you wrap the Text inside a view? It's possible it doesn't work because you app is only a Text node and might require at least one View, at least on Android.

I just tried that and I still got the same error unfortunately

I'm not able to reproduce this on Snack, using the exact same code posted here: https://snack.expo.io/HJw8alx6e
Any chance you can provide a repro?

its a private repo, so unfortunately not, I can provide my package.json though to show all the dependencies and versions i'm using.

  "name": "app",
  "version": "0.1.0",
  "private": true,
  "devDependencies": {
    "eslint": "^3.18.0",
    "eslint-config-airbnb": "^14.1.0",
    "jest-expo": "^0.3.0",
    "react-test-renderer": "~15.4.1",
    "reactotron-apisauce": "^1.9.1"
  },
  "scripts": {
    "start": "react-native start",
    "android": "react-native run-android",
    "ios": "react-native run-ios",
    "test": "node node_modules/jest/bin/jest.js --watch"
  },
  "dependencies": {
    "apisauce": "^0.11.0",
    "babel-preset-react-native-stage-0": "^1.0.1",
    "immutable": "^3.8.1",
    "react": "16.0.0-alpha.3",
    "react-native": "0.43.0-rc4",
    "react-native-svg": "^5.1.6",
    "react-native-vector-icons": "^4.0.0",
    "react-redux": "^5.0.3",
    "reactotron-react-native": "^1.9.1",
    "reactotron-redux": "^1.9.1",
    "reactotron-redux-saga": "^1.9.1",
    "redux-form": "^6.6.0",
    "redux-persist": "^4.5.0",
    "redux-saga": "^0.14.3",
    "reselect": "^3.0.0"
  }
}

also this is the MainApplication.java
package com.readerzone;

import android.app.Application;

import com.facebook.react.ReactApplication;
import com.horcrux.svg.SvgPackage;
import com.oblador.vectoricons.VectorIconsPackage;
import com.facebook.react.ReactNativeHost;
import com.facebook.react.ReactPackage;
import com.facebook.react.shell.MainReactPackage;
import com.facebook.soloader.SoLoader;

import java.util.Arrays;
import java.util.List;

public class MainApplication extends Application implements ReactApplication {

  private final ReactNativeHost mReactNativeHost = new ReactNativeHost(this) {
    @Override
    public boolean getUseDeveloperSupport() {
      return BuildConfig.DEBUG;
    }

    @Override
    protected List<ReactPackage> getPackages() {
      return Arrays.<ReactPackage>asList(
          new MainReactPackage(),
            new SvgPackage(),
            new VectorIconsPackage()
      );
    }
  };

  @Override
  public ReactNativeHost getReactNativeHost() {
    return mReactNativeHost;
  }

  @Override
  public void onCreate() {
    super.onCreate();
    SoLoader.init(this, /* native exopackage */ false);
  }
}

Let me know if you need anything else

I just updated to the newest release of react native and it seems to have resolved this issue;

This issue is still prevailing on Android
React Native version: 0.49.1

I got this error in RN 0.49.1 in some conditional JSX that relied on a value's being truthy to decide whether a View shows up or not. I fixed the error using the classic double exclamation !!, i.e. "{!!value" instead of "{value}."

This is the stack trace i am getting
Cannot add a child that doesn't have a YogaNode to a parent without a measure function! (Trying to add a 'ReactVirtualTextShadowNode' to a 'LayoutShadowNode')
addChildAt
    ReactShadowNode.java:181
setChildren
    UIImplementation.java:398
setChildren
    UIManagerModule.java:310
invoke
    Method.java
invoke
    JavaMethodWrapper.java:363
invoke
    JavaModuleWrapper.java:162
run
    NativeRunnable.java
handleCallback
    Handler.java:739
dispatchMessage
    Handler.java:95
dispatchMessage
    MessageQueueThreadHandler.java:31
loop
    Looper.java:148
run
    MessageQueueThreadImpl.java:194
run
    Thread.java:818

I am using
react-native-cli: 2.0.1
react-native: 0.49.3

Aah silly error but still the stack trace is too confusing:
I had my button tag like:
  <Button
     transparent
     style={styles.nextContainer}
     onPress={props.onPress}
   >
     NEXT
   </Button>

which was causing the error . This used to work in old react-native. But now, no string values should be without the  tag.
So i resolved it by replacing the above code with
    <Button
      transparent
      style={styles.nextContainer}
      onPress={props.onPress}
    >
      <Text>NEXT</Text>
    </Button>

Same issue here on a simple Input Component.
react-native: 0.49.3
Cannot add a child that doesn't have a YogaNode to a parent without a measure function! (Trying to add a 'ReactVirtualTextShadowNode' to a 'LayoutShadowNode')
import React from 'react';
import { TextInput, View, Text } from 'react-native';

const Input = ({ label, value, onChangeText, placeholder, secureTextEntry }) => {
  const { inputStyle, labelStyle, containerStyle } = styles;

  return (
    <View style={containerStyle}>
      <Text>

      <Text style={labelStyle}>{label}</Text>
      <TextInput
        secureTextEntry={secureTextEntry}
        placeholder={placeholder}
        autoCorrect={false}
        style={inputStyle}
        value={value}
        onChangeText={onChangeText}
      />
      </Text>
    </View>
  );
};

const styles = {
  inputStyle: {
    color: '#000',
    paddingRight: 5,
    paddingLeft: 5,
    fontSize: 18,
    lineHeight: 23,
    flex: 2
  },
  labelStyle: {
    fontSize: 18,
    paddingLeft: 20,
    flex: 1
  },
  containerStyle: {
    height: 40,
    flex: 1,
    flexDirection: 'row',
    alignItems: 'center'
  }
};

export { Input };

Thank you @parthgandhi7

The approach presented by @parthgandhi7 works for me, too. I think the error message should be vastly improved, though.

I'm getting the same error on RN 0.50.3. Running on Android emulator API 23.

Writing each tags on a separate new lines has resolved the same problem for me.

Thank you @parthgandhi7 !!!

Thank you @parthgandhi7

Got the same error when trying to render the following JSX:
<Card title="All done!">   <Text style={{ marginBottom: 10 }}>There's no more content here!</Text>   <Button backgroundColor="#03A9F4" title="Get more!" />> </Card>
If you look very closely you'll see an extra '>' following the Button declaration. That's all I needed to remove for it to successfully render.

I had the same error with a semi column ";" wrongly placed inside the JSX on line 4.
So this error looks like generic for malformed JSX.

  case 'sun':
    return (
      <View style={styles.iconContainer}>
          <MyIcon name={''} size={18} color={colors.yellow} />;
      </View>
    );

this is still happening on 0.50.4

Some times you need to remove the extra spaces between the angle brackets to get rid of it.

@gopaldevra Thank you very much!!!

Removing comments inside Component solved it for me.
I wasted more than a day on this and found no useful solution anywhere. I got this error because I had put a comment inside a component. On removing that comment the app did run for me.

Also had this in something slightly different, to do with using && logic for component mounting.
I had something like
render {
  <View>
    { val === 'a' && renderA() }
    { val === 'b' && renderB() }
    { val === 'c' && renderC() }
  </View>
}

And fixed it by moving those out into it's own function
renderAlphabet (
  if (val === 'a') {
    return renderA();
  } else if (val === 'b') {
    return renderB();
  } else if (val === 'c') {
    return renderC();
  }

  return null;
)

render {
  <View>
    { this.renderAlphabet() }
  </View>
}

I had this when i add a space before comment in JSX
<View> {/*Reviews*/}   <-- error
</View>

and works fine after removing the space or putting the comment at the second line
<View>{/*Reviews*/} <-- works fine
</View>

<View>{/*Reviews*/} <-- works fine
</View>

There needs to be better feedback for debugging these issues

I was having problems at this time:
using ScrollView not set inside View
<ScrollView>  //error
  <View></View>
  <View></View>
  ...
</ScrollView>

<ScrollView>  //works well
  <View>
    <View></View>
    <View></View>
      ...
  </View>
</ScrollView>

In my case the issue was that I had an unnecessary character inside JSX:
render() {     return(         <View>             <Text>Lalala</Text>;         </View>     ) }
Which is the semicolon at the end of Text line. Removing it fixed the error.

This is always caused by a JSX syntax error, you can forget to close an element or have invalid characters in your JSX - there are endless possibilities. The error messaging for this error needs to improve and point to source of the issue. Right now, it doesn't tell you anything

i found this link . can anyone  help me  explaining the below code snippet from the following link .
https://www.javatips.net/api/ComicBook-master/node_modules/react-native/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactShadowNode.java
    // If a CSS node has measure defined, the layout algorithm will not visit its children. Even     // more, it asserts that you don't add children to nodes with measure functions.     if (mYogaNode != null && !mYogaNode.isMeasureDefined()) {       YogaNode childYogaNode = child.mYogaNode;       if (childYogaNode == null) {         throw new RuntimeException(           "Cannot add a child that doesn't have a YogaNode to a parent without a measure " +             "function! (Trying to add a '" + child.getClass().getSimpleName() + "' to a '" +             getClass().getSimpleName() + "')");       }       mYogaNode.addChildAt(childYogaNode, i);     }     markUpdated();
i am guessing that i have missed on to CSS in my native project.
i will look into that .
correct me if my guess is wrong .
hopefully, the above link could clear some doubts.

In my case, actionText was '' changing it to ' ', resolved the issue.

Inside the view you only have to write a text , i've seen many of you using ";" inside the view and that's wrong and causes this problem .

@Karim-Bensalah i tripled checked my entire directory.
here is the link to code.
Please check if you find anything help me correct it. I may have missed on some thing really small.
Thanks in Advance.

FWIW, I encountered this same (fairly useless) error stack with the following code:
const renderedItem = someOtherRenderFunc(); // My app has a lot of component nesting and reusability, making this a bit open-ended in what it might return.  Assumed to be JSX
return (
  <View style={someStyle}>
    {renderedItem}
  </View>
);

And was fixed by just replacing {renderedItem} with {renderedItem || null}.  By the look of it, something falsey was coming through here (above comments indicate it may be the empty string, though if memory serves I have had RN yell for passing undefined in JSX outside of a Text node, which would be more concerning.  Of course strings should be in Text, but undefined?) and the whole thing just crashed with the unhelpful stack trace.
In total agreement here that the trace could stand to be more helpful.  Will see if I can quickly reproduce this on Snack in a bit.

https://github.com/facebook/react-native/issues/13243#issuecomment-335488473 <-- js is weird
programmer: "does x exist?"
javascript: "???"
programmer: "...does x not... not exist?"
javascript: "yeah it exists!"

I got the same error. For me it was a null instead of undefined.
this.props.price && <MyText text={'$' + this.props.price} />
"price" comes from the API and in some cases it was "null" so  '$' + null produces this Error.
Changed to:
this.props.price !== undefined && this.props.price !== null &&  <MyText text={'$' + this.props.price.toString()} />

For me I was using immutablejs with forEach()
this.props.starred.keySeq().forEach(id=>{   return null })
Changing that to the ussual .map() fixed the issue
this.props.starred.keySeq().map(id=>{   return null })
I agree the error is very ambiguous  spent an hour on this and just used trial and error

This was horrible. I used <React.Fragment /> & it was giving me error & the version of React is 16.3.0-alpha.1.
Replaced it with <View /> & it works.
It was working for IOS & only Android had issue.
Example -
Not Working Solution

// About.js
const About = () => <React.Fragment><Text>About</Text></React.Fragment>;

// Home.js
import About from './About'

const Home = () => <About />;

Working Solution

// About.js
const About = () => <View><Text>About</Text></View>;

I put out all comments out of the render function.

To summarize, this issue is about what RN perceives to be malformatted JSX (usually occurring on Android render) and this thread has done a good job documenting the many forms this malformatted JSX can come in. Still doesn't solve the need for developers to comb through their code line by line looking for a random semicolons or spaces. If there is a way to have the stack trace be more specific about the specific offending character or error that would probably save devs hours of sad debugging

This seriously needs to be addressed, as the current behaviour is a total mess. Whatever IDE one is using and the stacktrace are completely useless in this situation.
It feels like i'm proofreading an essay written in notepad or nano...

crashlytics just reported the following crash
Cannot add a child that doesn't have a YogaNode to a parent without a measure function! (Trying to add a 'RCTRawText [text: />]' to a 'RCTView')
Thanks to the comments here I started looking in our code base for /> and after few minutes found the problem
<View>
<MyCustomButton
/>
/>
</View>

Notice the second /> It's sitting there, not crashing on iOS, but causing the issue on Android.

Knowing all the potential errors is helpful when the error has just appeared after changing some code but... how to figure it out when the error appears on a device after having developed a full app on another device/simulator? The error gives absolutely no feedback on which line or file the issue occurs.  The stacktrace seems to be limited to java files that are not part of my project. Any ideas on how to debug this?

I made this comment in #18880 as well but posting it here in case it can help someone not waste hours:
Here is another fun one that happens only in android (even in RN 0.55.3):
{this.props.loadingURL&&          <View style={{flexDirection:‘row’,alignItems:‘center’,justifyContent:‘center’,marginTop:40,marginBottom:10,}}>            <Text style={{marginRight:10,fontSize:20}}>{this.props.loadingURL}</Text>            <ActivityIndicator size=“large” color=“black” />          </View>        }
This blows up Android with the error reported in #18880  if loadingURL is an empty string.  It works fine if it's a non-empty string or a boolean or if I change it to this (basically convert it to a boolean):
{!_.isEmpty(this.props.loadingURL)&&
another 2 hours wasted!  👎

I also encountered the same wonderful problem, rn0.49.5 my code is as follows:
Return ()
And in FocusInstagram.js
<Text style={{
                Position: "absolute",
                Top: formatPx(115),
                Left: formatPx(158),
                fontSize: formatPx(36),
                Width: formatPx(303),
                Color: "#4a96e2",
            }}
                  numberOfLines={1}
            >{"# " + this.props.title}
I solved this problem by changing the value of the passed property: 'title'->'title_name'
Return ()
And in FocusInstagram.js
 <Text style={{
                Position: "absolute",
                Top: formatPx(115),
                Left: formatPx(158),
                fontSize: formatPx(36),
                Width: formatPx(303),
                Color: "#4a96e2",
            }}
                  numberOfLines={1}
            >{"# " + this.props.title_name}
However, so far, I still can not understand the real reason for this error.

@Joan Boixadós
make sure to provide View tag to wrap all stateless components in entire
project.
it solved my issue.
On Tue, 24 Apr 2018, 17:50 Joan Boixadós<EMAIL_ADDRESS>wrote:

I went through all my project files and have no:

{ results_in_false && whatever } structure that could lead to empty
strings
changed all title props to title_name
Formatted all the code with prettier so no syntax errors
Made sure to wrap every single string with

but the issue still persists, any other ideas? 😦
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/facebook/react-native/issues/13243#issuecomment-383909453,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APsQuIlha1RC4Zya9JwUDiFwxzjQ8VeOks5trxiUgaJpZM4MwLon
.

Thanks @Manish3323 , unfortunately I do not have stateless components :/

I found a "}" that was somewhere random in my JSX but had no corresponding opening "{". It was not marked as syntax error. But after removing it the problem was solved.

render with one line works.
two lines break
RN 0.49
VSCODE

My fix.

Cannot add a child that doesn't have a YogaNode to a parent without a measure function! (Trying to add a 'RCTRawText [text: return return]' to a 'RCTView') addChildAt
  import React , {Component} from 'react';
  import {Text ,View ,StyleSheet, TextInput,Button, AppRegistry} from 'react-native';

  export default class App extends Component {

              state={
              places:[],
              placename:""
              }

          placeholderhandler = val =>{
             this.setState({
             placename:val
        });
              };

              placesubmithandler =()=>{

        if(this.state.placename.trim() === ""){
        return

          }
     this.setState(prevState =>{
      return{
      places :prevState.places.concat(prevState.placename)
      };
      });
     };

                    render()  {

                        const placeoutput =  this.state.places.map(( place  ,i) =>(
                            <Text key ={i}>{place}</Text>
                        ));
                        return (
                          <View style={ styles.viewstyle} >

                          <View style={styles.inputcontent}>
                              <TextInput  style={styles.textinputstyle} placeholder="enter text"/>
                              <Button title="add" style={styles.buttonstyle} onPress={this.placesubmithandler} />
                         </View>
                      return
                      return
                         <View>{placeoutput}</View>
                         </View>
                      )
                    }
              }

                    const styles = StyleSheet.create({
                    viewstyle: {

                    padding:16,
                    backgroundColor:"#fff",
                    alignItems:"center",
                    justifyContent:"flex-start"
                    }
                    ,
                    inputcontent:{
                    width:"100%",
                   flexDirection:"row",
                   justifyContent:"space-between"
                    },
                    textinputstyle:{
                    width:"70%"
                    }
                    ,
                    buttonstyle:{
                    width:"30%"
                    }

                    }

                    );

                  // skip these lines if using Create React Native App
                  AppRegistry.registerComponent(
                    'AwesomeProject',
                    () => App);

help me please

@iivaschenko
I am developing with RN for over a year, and this is the first time I encountered this.
The amount of dev hours wasted over this is probably huge!

In my case I could solve it replacing an empty string for an empty component:
used:
const Media = ({ media }) => (media && <Thumbnail source={{ uri: media.src }} /></Left>) || <Text />;
instead of:
const Media = ({ media }) => (media && <Thumbnail source={{ uri: media.src }} /></Left>) || '';

Had the same problem... My part of code that caused this error was something like this in render
render(){
  return({ this.state.loading === false ?(
    <View>
      <Text>Loaded</Text>
    </View>
  ) : ""   })
}

It only crashed on android, ios worked fine. What i had to do was to change "" to null

this is a really weird case. I wasted two hours to find the problem, and then I found that @alainib got a similar solution: remove an space between tow blocks:
Before solution:
<Icon name="open" /> <Text>Open details</Text>
After solution
<Icon name="open" /><Text>Open details</Text>
The problem just go away. Only happend in Android.

Hi all! Sorry for the long wait but I think the fix for this just landed.
We were able to repro and fix this issue with this sample code:
function PlaygroundContent(props: {}) {
  return (
    <View>
      <Text>
        <View style={{width: 10, height: 10}}>
          <Image source={ANY_IMAGE_SOURCE} />
        </View>
      </Text>
    </View>
  );
}


Multipart support

How can I post a multipart form upload into an RPC that frodo is hosting?  Do you have any advice for me?  Or do I have to host a standard webserver to handle file uploads?

Sorry for the bananas late response to this. I'm sure you've moved on from this issue long ago. On the off chance some poor soul stumbles upon this later, here's my input on this. I punted on supporting multipart uploads. I did, at one point, start to implement it, but I absolutely hated the implementation and scrapped it and never came back to it.
The way I usually handle this case is to not have the service accept the file directly, at all. Instead, I implement this flow:

Client requests a service endpoint that responds with a pre-signed S3 upload URL (or your object store of choice) and some id unique to this upload.
Client uploads the file directly to S3 using that URL. The S3 location is usually in some temp bucket.
Client makes another request to the service with the unique upload id to the operation that actually needs the file. Once the work is done, we can either delete the file or move it to a bucket where it will live on.

Obviously that is very dependent on how/where you deploy, but I generally prefer this over having the services accept the file for these reasons:

It frees up a bunch of resources on your service to do what the service is really meant for. I've done it the other way and when dealing with large uploads or a lot of uploads, too many compute resources get spent moving bytes from one computer to another. This way, your service gets to stick to just the business logic work it was built for.
If you have a bunch of slow uploads, that can really mess with auto-scaling or health checks that are based on concurrent requests. Uploading directly to S3 means that those really long-lived connections are handled outside of your service infrastructure.
If I'm dealing with files, it's likely ending up in my object store anyway, so it cuts out the middleman.
Since you're putting it on your object store, you can use bucket policies to automatically purge files when their done or when they're abandoned w/o having been processed.

On the flip side, it does mean that something that was once 1 big request is now 3 smaller requests. It's a trade-off that I've typically been happy to make because the request overhead is usually overshadowed by the upload time anyway.

I ended up just base64 encoding a 30mb file using javascript and sending it up the rpcs as a string.  Worked fine.


fix: add missing spec input to action.yml

Add missing spec input to action.yml to prevent warnings in console log as on screenshot below:

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Perfect, thank you @DlgSHi


Starting day week fix

There was a bug in my startingDayWeek implementation: if the option was set to 1, events for the last day of the week were filtered out. If set to 2, the last two days, etc.

Dear twin,
The europian issue of the iso-week-standard is:
In the headers you see:
mon/tue/wed/thu/fri/sat/sun
in stead of the amirican standard:
sun/mon/tue/wed/thu/fri/sat
If I want to use your control, this issue must be solved.
Btw: I also need to nationalize the names of the the months/days
Kind regards:
<EMAIL_ADDRESS>

@Ak2Ps I have fixed the startingDayMonth issue in 0.1.6.
Feel free to have a test.
Now I only expose the datefilter configuration so that you can change the date format of the header (formatDayHeader) and the day (formatDay). Does that meet your internationalisation need?

Thank you for responding so quickly. I used your calendar as an example for making my own europian one. Other aspects of this calendar are so company-specific that I cannot publish this one.
Nevertheless, thanks very much and success.
<EMAIL_ADDRESS>