Rebass vs just using Styled System?

As of 3.0.0 (now that rebass is stripped right back), it seems to basically be just a thin layer over styled system that needs extending out by the end user.
So with all the extending required, why would someone use Rebass, instead of just using styled system? What is the rationale/intentions/benefits?
And where does https://github.com/jxnblk/superbox fit in with everything, as all the Rebass primitives are extensions of <Box /> anyway?

In fact, Box@rebass could be replace by superbox 🤔

@kristojorg I was actually surprised that Rebass didn't offer inputs as one of its default components.  Although it shouldn't be too hard to do these on your own, I would imagine the reason they're not included in Rebass was a deliberate decision.  Just curious,  why is that @jxnblk?

@cereallarceny once I understood the intended DX it made a lot of sense:
<Box
    as="input"
    m={3}
    value={value}
/>

Something like that. The base box is really just a set of functions applied to a default tag. You can change the tag as you wish. Rebass doesn't provide default input styling, but that didn't matter much for me as I would have changed it anyway.

@cereallarceny from my experience, input element styles tend to vary quite a bit from application to application and involve a lot of pseudoclasses, which makes it more difficult to create a one-size-fits-all component. I also think that most developers know enough CSS to be able to style an input for their specific application, and input styles are unlikely to cause any CSS side-effects outside of the component

Totally fair - I agree with that 100%.  Just simply found it interesting... I'm so used to seeing UI frameworks with inputs!  However, I also find that the vast majority of CSS overrides I write for any UI framework, are "undoing" CSS for inputs.  A bit ironic.
Would that be worth mentioning as a deliberate decision in the documentation?


No description field on the volume edit page

Steps to reproduce:

Create a volume, then edit it
On the edit page, check on the page

Result:
There is no description field on the edit page

Validated


Option to consider user model field for emails

Added a new option to consider user model field called "receive_emails" before actually sending an email. This could be used when the user wants to opt out of email notification messages to their respective emails

Looks good to me, minor note - maybe you could use more sensible variable name like "subscribed" rather than "actually_send" as it doesn't explain much really, but if you think it is ok then i'm fine.

You are right, subscribed sounds better, I'll change that
On Fri, 22 Apr 2016, 17:05 Elnur<EMAIL_ADDRESS>wrote:

Looks good to me, minor note - maybe you could use more sensible variable
name like "subscribed" rather than "actually_send" as it doesn't explain
much really, but if you think it is ok then i'm fine.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/p2pcourier/django-messages/pull/3#issuecomment-213493001


Shift extension ownership to (GitHub-)organization

Some time ago the popular Brackets Beautify extension was moved to the organization with the same name so we could maintain it together.
However, none of us now seems to be able to upload new version to the Brackets Extension Registry.
Is it possible to change the permission of that extension so that all members of the organization can upload new versions?
I don't want to upload it as separate extension, because that would require manual update work for each user.

@Hirse Brackets Registry currently only allows a single github id as owner. Can you setup a shared github id and use that account for all registry updates?

@redmunds Thanks for the advice.
I have set up a new account (brackets-beautify-updater). How can I transfer the ownership of the extension to it?

@Hirse Go to the Brackets Registry web page here: https://brackets-registry.aboutweb.com/  In the upper-right corner, click "Sign in via Github" link and sign in to current owner (Hirse). All extensions owned by logged in account will have a "Change Ownership" link.

@redmunds Thanks. It is not currently owned by me though.
@drewhamlett Could you try this, please?

@hirse Done!

@drewhamlett Sorry, I was not precise. We can't use the organization, so I made an updater account (whose password I would share with you), which should be owner of the extension in the registry.
@redmunds I guess, now we might have a problem as the current owner is our organization account, which we cannot use to login. Can you (or another admin) change to ownership to brackets-beautify-updater?

I don't have any authority on that system. If @drewhamlett can't make the change, and you can't change ownership of the your fork, then maybe submit a new extension from the updater account?

But having a new extension would have the serious downside of not providing the update mechanism, so most user would not receive the new version.
Is it possible to get the old one removed and upload a new one (from the updater account) with the same id so we have updates?


HDFS-16374 Fix error log format in AbfsListStatusRemoteIterator

JIRA:HDFS-16374
Fix error log format in AbfsListStatusRemoteIterator

not hdfs project issue,closing...

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 53s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

-1 :x:
mvninstall
34m 30s
/branch-mvninstall-root.txt
root in trunk failed.

+1 :green_heart:
compile
0m 37s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 31s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 24s

trunk passed

+1 :green_heart:
mvnsite
0m 37s

trunk passed

+1 :green_heart:
javadoc
0m 30s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m  1s

trunk passed

+1 :green_heart:
shadedclient
23m 24s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 30s

the patch passed

+1 :green_heart:
compile
0m 31s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 31s

the patch passed

+1 :green_heart:
compile
0m 27s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 27s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 16s

the patch passed

+1 :green_heart:
mvnsite
0m 29s

the patch passed

+1 :green_heart:
javadoc
0m 22s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 20s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m  3s

the patch passed

+1 :green_heart:
shadedclient
22m 48s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 51s

hadoop-azure in the patch passed.

+1 :green_heart:
asflicense
0m 30s

The patch does not generate ASF License warnings.

92m 46s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3767/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3767

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux f468f6ecd4b4 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / b0b050006668f39a7fadaa74eecb112d101657fd

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3767/1/testReport/

Max. process+thread count
524 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3767/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.


Remove unused FailAction parameter from KeptnEvaluation

Description
.spec.FailAction parameter of KeptnEvaluation is not used anywhere in the code and has no effect on the functionality. We should consider removing it.
https://github.com/keptn/lifecycle-toolkit/blob/main/lifecycle-operator/apis/lifecycle/v1beta1/keptnevaluation_types.go#L62

@odubajDT, I would like to do it

@odubajDT, I would like to do it

Hi @prakrit55 as I can see you have an opened PR and an issue already assigned. Please try to finish it first. Afterwards I will gladly assign this issue to you if it's still free!


Update docs for postgresql installation (#7268)

Pull request opened by the merge tool on behalf of #7268

Merged into branches iso7 in 6c336a716d7473864db6207de661a514d068bde6


Request: Some way to initialise a dynamically-sized state variable in TFLite

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): All
TensorFlow installed from (source or binary): Binary
TensorFlow version (or github SHA if from source): 2.9.1

Provide the text output from tflite_convert
tensorflow.python.ops.op_selector.UnliftableError: Unable to lift tensor <tf.Tensor 'zeros:0' shape=(None, None) dtype=float32> because it depends transitively on placeholder <tf.Operation 'arr' type=Placeholder> via at least one path, e.g.: zeros (Fill) <- Shape (Shape) <- arr (Placeholder)

Standalone code to reproduce the issue
I want to define a stateful graph where the shape of the state variable is defined when model is LOADED, not when it's saved.
As a simple example - lets say I just want to compute a temporal difference - ie. a graph that returns the difference between the input in two consecutive calls. The following should pass:
func = load_tflite_model_func(tflite_model_file_path)
runtime_shape = 60, 80
rng = np.random.RandomState(1234)
ims = [rng.randn(*runtime_shape).astype(np.float32) for _ in range(3)]
assert np.allclose(func(ims[0]), ims[0])
assert np.allclose(func(ims[1]), ims[1]-ims[0])
assert np.allclose(func(ims[2]), ims[2]-ims[1])

However as far as I know there is not way I can save a model, or define some function load_tflite_model_func for loading it, that would make this work for any runtime_shape, because variables in the graph can only be saved with a pre-determined size.
A full stand-alone notebook demonstrating the issue is here:
https://colab.research.google.com/drive/19CwkF1MSGlfMXKxlrndCNqTv7V4fx55Q
Request is for TFLite to be able to initialise state-variables, where the shape of the initialised variable can depend on the input-shape
See also https://stackoverflow.com/questions/75052366/how-do-i-to-save-a-stateful-tflite-model-where-shape-of-state-depends-on-input

@petered
Given Colab is not accessible. Could you please provide complete code or Colab gist to replicate the issue reported here?
Thank you!

My mistake - I shared the Colab Notebook

Great - thank you for following up.  I'm still unsure if there is a way to do this in Tensorflow Lite

If there is, it would benefit from better documentation
If there is not, I think it would be a very useful feature to have.

Hi @petered !
We can resize the tensor input by below command which will resize the image inputs automatically according to originals lite input shape.
interpreter.resize_tensor_input(0, [30, 40 ])
Attached resolved gist in 2.9, 2.10 and 2.11 for reference.
We can do the same using model binding feature from Android studio.  Do you suggest any PR in this section.
Thank you!

Thank you for your response @mohantym.
I looked at the modified notebook you provided, but I believe it does not resolve the issue.
The reason it works in your notebook is because you just changed the test so runtime shape to be identical to the compile-time shape.  The desired behaviour is that the function is able to run for any compile-time shape.  I've modified your notebook to use a different runtime-shape, and you can see it now fails.
To clarify... I am looking (in this example) to load a tflite model that can can implement a simple temporal difference, regardless of the shape of the input:
runtime_shape = 60, 80
func = load_tflite_model_func(tflite_model_file_path, input_shape=runtime_shape)
rng = np.random.RandomState(1234)
ims = [rng.randn(*runtime_shape).astype(np.float32) for _ in range(3)]
assert np.allclose(func(ims[0]), ims[0])
assert np.allclose(func(ims[1]), ims[1]-ims[0])
assert np.allclose(func(ims[2]), ims[2]-ims[1])

If I were to implement it in pure python, the solution would be
class DeltaT:
  def __init__(self):
    self._last = 0
  def __call__(self, x):
    delta = x-self._last
    self._last = x
    return delta

    ...
    func = DeltaT()

Is there a way to do this with Tensorflow Lite?

@petered !
Thanks for the update.
Auto-casting to the original shape in model would be a nice feature to have which are normally doing that through model binding from android studio and  metadata writer (by inputting metadata like input shape).
@sachinprasadhs !
Could you look this feature request.
Thank you!

To clarify - Auto-casting to the original shape in model is not what I'm asking for.  I'm asking for the ability to either initialise a variable in the model according to the input shape - or failing that, resize the saved variable according to the input shape, at time of loading the model.  See the example code above.
I just found out about model-binding in Android studio - from my understanding it just generates wrapper classes for the model.  I'm not sure how this would help - unless you're thinking of modifying the model to return the values of state variables and store them in a Java object instead - feeding them in to each function call - is that what you mean?

So is there really no way to change the shape of variables in a tflite Interpreter?

Ok well I found a solution that isn't ideal but does the job: Make the variable the largest size it could conceivably be at runtime, and just take and assign a slice of it.  Here is a modified notebook that does that.
A downside of this approach is that you end up with very large tflite-files (in my case, 24MB of zeros).
This issue remains - TFLite should provide some way to assign values to variables and reshape the downstream tensors, for example:
interpreter.resize_tensor_input(0, input_shape)

index_of_last_input_storage_var: int = 5  # e.g. look this up by name from interpreter.get_tensor_details()
interpreter.assign_variable(index_of_last_input_storage_var, np.zeros(input_shape, dtype=float32)
# ... the above method does not exist but should

interpreter.allocate_tensors()  # Allocate shapes downstream of both input and variables.

Thanks for sharing your solution, this would help others who will have the same issue.
Could you please close the issue if you don't have any further question. Thanks

Hi @petered,
Because the design philosophy of Tensorflow is to have immutable tensors, implementing this for TFLite which will be consistent with its design philosophy of being lightweight and high performance will be quite a heavy load. As such we do not plan to implement this feature in the near future and will close this issue for now. Please utilize the workaround for now.


Non uniform (still fixed) time grid for direct method (and direct shooting ?)

Allow to pass explicit time grid to solve().
Note: handling a non uniform grid is also a prerequisite of step control mechanisms...

I guess we would pass the time vector to solve() ?

@PierreMartinon question related to this discussion with @rand-asswad : is it possible to pass a (possibly non-uniform) grid to solve? Should not be a strong change but does not seem available form the API.

I can have a quick look at this, it should be rather simple since the time grid / step is not used at many places in the code.
I guess we can go for an optional argument time_grid=nothing for solve() ?

@ocots ~me too can write n'importe quoi then strike through it~ 🤭

~Of course it is possible.~

@PierreMartinon would be nice to interact with @rand-asswad who might have a nice use case that indeed requires a higher order scheme

I can have a quick look at this, it should be rather simple since the time grid / step is not used at many places in the code.
I guess we can go for an optional argument time_grid for solve(), with default value nothing for a uniform grid, or the exlicit vector of time steps. Probably add some internal checks for strictly increasing values, and consistency with initial and final times. I would say that providing an explicit time grid would override grid_size if also passed, maybe with a warning (especially if given size does not match the given vector)

This one is the non-uniform grid, not the high order scheme. Are we sleepwriting @jbcaillau ? :D

Done

Done
Regarding the arguments for the grid, we currently have

grid_size (integer) for uniform grid
time_grid (vector like) for explici grid
@ocots mentioned we could also have a function such as (t0,tf) -> grid

I think the best way is to have a single argument time_disc or something, whose type will be a union of the different possibilities

@PierreMartinon just posted in another issue (can't find it now 😅) that

I prefer two named rags (more readable / user friendly)
we ought to check what the current standard in Julia is for such things

Well, the two things are related and of interest for @rand-asswad example

This one is the non-uniform grid, not the high order scheme. Are we sleepwriting @jbcaillau ? :D


Investigate how we want to refactor guards like we did in productGuards [after 488]

Offshoot of #272
After we  refactor the routing parameters #488
Investigate how we want to refactor guards like we did in productGuards, and create ticktets for a team to go do it.

Kris says: The ProductGuard doesn't exist anymore, and I think we are OK with other guards for 1.0 (Refactor of CmsPageGuard should be considered only after 1.0 - it's a big thing). So I believe this ticket can be closed.


[SPARK-30519][CORE] Use spark.executorEnv.HADOOP_USER_NAME to set executor's hadoop user

What changes were proposed in this pull request?

Let executor can specify hadoop user by using spark.executorEnv.HADOOP_USER_NAME
Why are the changes needed?

We can use HADOOP_USER_NAME to specify hadoop user while submitting a job on driver, but invalid to set executor's hadoop user even by using spark.executorEnv.HADOOP_USER_NAME.
Does this PR introduce any user-facing change?

No
How was this patch tested?

This is a behavior change, and I don't think we should introduce this change by the approach proposed this way.
If both SPARK_USER and HADOOP_USER_NAME are specified in the env, we should prioritize SPARK_USER over HADOOP_USER_NAME. Also, I'm hesitate whether we should support HADOOP_USER_NAME at all.

This is a behavior change, and I don't think we should introduce this change by the approach proposed this way.
If both SPARK_USER and HADOOP_USER_NAME are specified in the env, we should prioritize SPARK_USER over HADOOP_USER_NAME. Also, I'm hesitate whether we should support HADOOP_USER_NAME at all.

But SPARK_USER is used for scheduler such as Mesos, Kubernetes and HADOOP_USER_NAME is used for Hadoop cluster such HDFS. At Spark on Kubernetes, SPARK_USER is not over HADOOP_USER_NAME


Support pypy3.2

I think it is currently supported except for module-level unicode docstrings:
https://bitbucket.org/pypy/pypy/issue/1952/pypy3-240-incorrect-tokenization-of#comment-None

This could be done by monkeypatching the tokenize module to support unicode literals -- however I don't think this is worth it.  It seems pypy hasn't really put much support into pypy3 after the initial release and in all likelihood they'll rewrite a newer iteration of python3 next time around

pyp3.5+ is fully supported, pypy3.2 is ~mostly supported unless you have u-prefixed docstrings


Update VERSION to master for Nexus images

This will make the Nexus image tag be master for the nightly builds. Needed for https://github.com/edgexfoundry/developer-scripts/pull/181.
Signed-off-by: Lisa Rashidi-Ranjbar<EMAIL_ADDRESS>

recheck

recheck

Codecov Report

Merging #2153 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2153   +/-   ##
=======================================
  Coverage   40.32%   40.32%
=======================================
  Files         178      178
  Lines       12270    12270
=======================================
  Hits         4948     4948
  Misses       7053     7053
  Partials      269      269

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5444d7...3ca188d. Read the comment docs.


Create Prometheus Alerting Rules per Backup Schedule

Summary
As K8up user
I want to have Prometheus alerting rules
So that I get notified if backups are failing
Context
Currently K8up only provides Prometheus metrics but no alerting rules. We have a set of default rules in the Commodore component: https://github.com/projectsyn/component-backup-k8up/blob/master/class/defaults.yml#L99
These rules are too generic and might lead to many false positives or might miss failed backups altogether.
Since K8up has all the necessary context to know what could be a good threshold for alerting, it could also generate said alerting rules.
Out of Scope

Providing a Prometheus stack

Further links

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrule

Acceptance criteria

When the feature is enabled, a PrometheusRule with appropriate rules and thresholds should be created for a Schedule

Implementation Ideas

Add a field to the schedule object which enables the creation of PrometheusRule objects per schedule.

Personal opinion incoming:
Not sure if it's something that k8up should handle. Alerting rules are something very use case specific. The use case you have may not apply for all users. The rules themselves have been out of scope, we only provided a set of default rules, that are now integrated in the component.
The component handles the alerting rules already so maybe invert the requirement. The component could be enhance in such a way that you can define schedules, it will create them and the correct rule for the schedule.

The idea behind this is that the current default rules are more or less unusable due to too many false positives and as you mentioned different use cases (e.g. hourly vs. daily schedules). And since the operator knows about the schedules it would have the necessary information to create useful rules.
I understand that it might be a stretch to make it the operator's problem but then again I think it would be a very common use case since a lot of users want alerts for failing/skipped backups.
Implementing it in the component would indeed be an option as well. This would then only work for schedules created via the component or it's helper Jsonnet lib and not for others. Also it might be quite challenging to implement such a logic in Jsonnet.

I'd also suggest to stop calling them defaults but rather examples. As they only work for schedules that are once a day as you've found out.

We also have to consider that if we add such rules to the schedule API, we'll add a hard dependency on Prometheus Operator and become subject to their API changes.
What if we instead of directly adding Prometheus rule spec to the schedule spec, add a reference to a ConfigMap that contains YAML document for each object to be generated. Similar to Espejo, K8up would then search and replace special variables with properties from the schedule spec? The difference is that such a ConfigMap gives the flexibility you seek, but K8up itself remains unopinionated about which objects?

How about monitoring mixin?
a unified approach that fits any prometheus configuration.
see node-mixin as example
https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin


Request to play music results in errors

Found speakers, got info on 'm, set default music service (Apple) but can't seem to play any music.
Get the following:
~~~~es.music_service | Unhandled SOAP Fault. Code: SOAP-ENV:Server. Detail: <Element 'detail' at 0x43de36c0>. String: There was an error processing your request
Traceback (most recent call last):
  File "/home/pi/mycroft-core/.venv/lib/python3.7/site-packages/soco/music_services/music_service.py", line 189, in call
    result_elt = message.call()
  File "/home/pi/mycroft-core/.venv/lib/python3.7/site-packages/soco/soap.py", line 324, in call
    raise SoapFault(faultcode, faultstring, faultdetail)
soco.soap.SoapFault: SOAP-ENV:Server: There was an error processing your request
 21:01:35.980 | ERROR    |   667 | SonosController | ('There was an error processing your request', 'SOAP-ENV:Server')

Any ideas?

Hey @MaartenW
I should not have merged the Apple Music music service. It is not working I would like yet.
For Apple Music support with Sonos, I would recommend to check this skill until I merge the full working code


The QA blob container doesn't  show as broken after renaming/deleting the original item

Storage Explorer Version: 1.8.1_20190523.2
Platform/OS: Linux Ubuntu/macOS High Sierra/Windows 10
Architecture: ia32/x64
Regression From: 1.8.1 release
Steps to reproduce:

Expand one normal account -> 'Blob Containers' node.
Right click one blob container then select 'Add to Quick Access'.
Try to rename or delete the original blob container.
Check whether the Quick Access blob container shows broken or not.

Expect Experience:
The Quick Access blob container shows as broken.
Actual Experience:
The Quick Access blob container doesn't show as broken.
More info:
This issue doesn't reproduce for file shares/queues/tables.

Hey @v-xuanzh , I think I might have fixed this on master. Can you verify? Thanks.

Hi @MRayermannMSFT
We verified this issue on the build master/20190624.4. It still reproduces.

Hey @v-xianya, I'm going to go ahead and close this. The way we are loading QA items now, we don't make a call to Azure. Instead we make the node from the information we have stored. It isn't till the user tries to do something with the container that we can find out it doesn't exist. So at that point should we break? Eh. Probably. But I don't think it's worth it w/o customer feedback that they'd like. So closing for now. Thanks for reporting though. 😄

Hi @MRayermannMSFT , Thanks for your sharing.
For this issue, We noticed it doesn't reproduce for QA file shares/queues/tables😲. Does the way loading QA file shares/queues/tables is different from QA blob containers？Could your help to take a look? Thank a lot！

@v-xianya Yes. It is different.


[WIP] Transcoding to UTF-32

Fixes #76
This should add transcoding to UTF-32 from UTF-8 and UTF-16 by the time it is done.

@lemire For now, I have only implemented for westmere and I did not test it yet so it might now be worth reviewing.

Thanks!

@lemire I have set up the testing, and right now I am stuck on pure ASCII conversion from UTF-8 to UTF-32. You can run convert_valid_utf8_to_utf32_tests to see by yourself. It seems that the store_ascii_as_utf32 in simd.h for haswell (and probably westmere) are wrong. I was not able to correctly use the instructions in that function. Could you look into it when you have time?

@NicolasJiaxin On my todo.

I was able to come up with something that seems to work, but there is probably a better to do it than what I did for ASCII transcoding in haswell.

I was able to come up with something that seems to work, but there is probably a better to do it than what I did for ASCII transcoding in haswell.

I'll have a look this afternoon. I was actually work on other PRs in this project. :-)

In my most recent commit, I added a line in the CMake file in src to fix an error regarding the modifications I have done in westmere/simd.h. Let me know if this is appropriate.

In my most recent commit, I added a line in the CMake file in src to fix an error regarding the modifications I have done in westmere/simd.h.

There are still some problems in the singleheader build as seen below. The line I added fixed when building normally, but the error is still there in the singleheader build.

I will get to it!

I am looking at the single-header issue. Just starting. It is probably a small thing.

We probably want to hook this into our benchmarks. It is simpler than it looks but it might require some additional effort and could be done as a separate PR, possibly... though it would be very nice to check the performance prior to merging.

We probably want to hook this into our benchmarks. It is simpler than it looks but it might require some additional effort and could be done as a separate PR, possibly... though it would be very nice to check the performance prior to merging.

Ok, I will try to do that!

@lemire I did not see prior to starting that there were also a little-endian and big-endian variations for UTF-32. The format I assumed unknowingly is UTF32-LE. Is that ok?

I have added a benchmark for valid UTF-8 to UTF-32 transcoding only for our implementation for now... The results I got for a 10KB random UTF-8 input are
>>> ./build/benchmarks/benchmark --random-utf8 10240 -P convert_valid_utf8_to_utf32
We define the number of bytes to be the number of *input* bytes.
We define a 'char' to be a code point (between 1 and 4 bytes).
===========================
Using ICU version 67.1
testcases: 1
input detected as UTF8
current system detected as haswell
===========================
convert_valid_utf8_to_utf32+fallback, input size: 10244, iterations: 2000
  12.425 ins/byte,   12.425 cycle/byte,    1.196 GB/s (2.7 %), 0.008786 b.misses/byte, 0.000000 c.mis/byte,    3.710 GHz,    4.005 ins/cycle
  30.886 ins/char,   30.886 cycle/char,    0.481 Gc/s (2.7 %), 0.021839 b.misses/char, 0.000000 c.mis/char,     2.49 byte/char
convert_valid_utf8_to_utf32+haswell, input size: 10244, iterations: 2000
   5.727 ins/byte,    5.727 cycle/byte,    1.761 GB/s (1.5 %), 0.001074 b.misses/byte, 0.000000 c.mis/byte,    3.725 GHz,    2.707 ins/cycle
  14.237 ins/char,   14.237 cycle/char,    0.708 Gc/s (1.5 %), 0.002669 b.misses/char, 0.000000 c.mis/char,     2.49 byte/char
convert_valid_utf8_to_utf32+westmere, input size: 10244, iterations: 2000
   6.094 ins/byte,    6.094 cycle/byte,    1.774 GB/s (1.6 %), 0.000683 b.misses/byte, 0.000000 c.mis/byte,    3.723 GHz,    2.905 ins/cycle
  15.149 ins/char,   15.149 cycle/char,    0.714 Gc/s (1.6 %), 0.001699 b.misses/char, 0.000000 c.mis/char,     2.49 byte/char

@NicolasJiaxin

I did not see prior to starting that there were also a little-endian and big-endian variations for UTF-32. The format I assumed unknowingly is UTF32-LE. Is that ok?

Correct. We have the same issue with UTF-16 where we only support UTF-16LE. Adding support for UTF-16BE is not very difficult (just flip the bytes). I will add it to our todo.

See https://github.com/simdutf/simdutf/issues/118

@NicolasJiaxin On your machine, how does convert_valid_utf8_to_utf32+haswell compare with convert_valid_utf8_to_utf16+haswell ? It should be 'similar'...
Note that you can test with actual files... Grab this:
https://github.com/lemire/unicode_lipsum
Then just pass a file name (-F myfile.utf8.txt).

@NicolasJiaxin Try the following to fix your amalgamation issue...

I applied the patch and I think it helped, but I still have the issue with avx2 intrinsics in the singleheader version, and removing the flag will also cause an error when building normally.

I will have a look again. :-)

I have added UTF-8 to UTF-32 transcoding with validation as well as the tests (convert_utf8_to_utf32_tests). I have also added a function utf32_length_from_utf8, analogous to utf16_length_from_utf8. It is exactly the same as count_code_points for utf8. I copied the code from that function rather than making a function call so that utf32_length_from_utf8 does not depend on that function although I don't think it could become a problem anyways.

@NicolasJiaxin
The following patch should allow you to build a proper westmere kernel:
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index c40b245..4322f3f 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -1,4 +1,3 @@
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
 add_library(simdutf-include-source INTERFACE)
 target_include_directories(simdutf-include-source INTERFACE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>)
 add_library(simdutf-source INTERFACE)
diff --git a/src/simdutf/westmere/simd.h b/src/simdutf/westmere/simd.h
index 37cf3cd..5e88935 100644
--- a/src/simdutf/westmere/simd.h
+++ b/src/simdutf/westmere/simd.h
@@ -23,8 +23,10 @@ namespace simd {
       _mm_storeu_si128(reinterpret_cast<__m128i *>(p+8), _mm_cvtepu8_epi16(_mm_srli_si128(*this,8)));
     }
     simdutf_really_inline void store_ascii_as_utf32(char32_t * p) const {
-      _mm256_storeu_si256(reinterpret_cast<__m256i *>(p), _mm256_cvtepu8_epi32(*this));
-      _mm256_storeu_si256(reinterpret_cast<__m256i *>(p+8), _mm256_cvtepu8_epi32(_mm_srli_si128(*this,8)));
+      _mm_storeu_si128(reinterpret_cast<__m128i *>(p), _mm_cvtepu8_epi32(*this));
+      _mm_storeu_si128(reinterpret_cast<__m128i *>(p+4), _mm_cvtepu8_epi32(_mm_srli_si128(*this,4)));
+      _mm_storeu_si128(reinterpret_cast<__m128i *>(p+8), _mm_cvtepu8_epi32(_mm_srli_si128(*this,8)));
+      _mm_storeu_si128(reinterpret_cast<__m128i *>(p+12), _mm_cvtepu8_epi32(_mm_srli_si128(*this,12)));
     }
     // Bit operations
     simdutf_really_inline Child operator|(const Child other) const { return _mm_or_si128(*this, other); }
diff --git a/src/westmere/sse_convert_utf8_to_utf32.cpp b/src/westmere/sse_convert_utf8_to_utf32.cpp
index 3b1878c..836dc2d 100644
--- a/src/westmere/sse_convert_utf8_to_utf32.cpp
+++ b/src/westmere/sse_convert_utf8_to_utf32.cpp
@@ -23,9 +23,11 @@ size_t convert_masked_utf8_to_utf32(const char *input,
       utf8_end_of_code_point_mask & 0xfff;
   if(((utf8_end_of_code_point_mask & 0xffff) == 0xffff)) {
     // We process the data in chunks of 16 bytes.
-    _mm256_storeu_si256(reinterpret_cast<__m256i *>(utf32_output), _mm256_cvtepu8_epi32(in));
-    _mm256_storeu_si256(reinterpret_cast<__m256i *>(utf32_output+8), _mm256_cvtepu8_epi32(_mm_srli_si128(in,8)));
-    utf32_output += 16; // We wrote 16 16-bit characters.
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output), _mm_cvtepu8_epi32(in));
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output+4), _mm_cvtepu8_epi32(_mm_srli_si128(in,4)));
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output+8), _mm_cvtepu8_epi32(_mm_srli_si128(in,8)));
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output+12), _mm_cvtepu8_epi32(_mm_srli_si128(in,12)));
+    utf32_output += 16; // We wrote 16 32-bit characters.
     return 16; // We consumed 16 bytes.
   }
   if(((utf8_end_of_code_point_mask & 0xffff) == 0xaaaa)) {
@@ -36,8 +38,9 @@ size_t convert_masked_utf8_to_utf32(const char *input,
     const __m128i ascii = _mm_and_si128(perm, _mm_set1_epi16(0x7f));
     const __m128i highbyte = _mm_and_si128(perm, _mm_set1_epi16(0x1f00));
     const __m128i composed = _mm_or_si128(ascii, _mm_srli_epi16(highbyte, 2));
-    _mm256_storeu_si256((__m256i *)utf32_output, _mm256_cvtepu16_epi32(composed));
-    utf32_output += 8; // We wrote 16 bytes, 8 code points.
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output), _mm_cvtepu8_epi32(composed));
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output+4), _mm_cvtepu8_epi32(_mm_srli_si128(composed,4)));
+    utf32_output += 8; // We wrote 32 bytes, 8 code points.
     return 16;
   }
   if(input_utf8_end_of_code_point_mask == 0x924) {
@@ -77,7 +80,8 @@ size_t convert_masked_utf8_to_utf32(const char *input,
     const __m128i ascii = _mm_and_si128(perm, _mm_set1_epi16(0x7f));
     const __m128i highbyte = _mm_and_si128(perm, _mm_set1_epi16(0x1f00));
     const __m128i composed = _mm_or_si128(ascii, _mm_srli_epi16(highbyte, 2));
-    _mm256_storeu_si256((__m256i *)utf32_output, _mm256_cvtepu16_epi32(composed));
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output), _mm_cvtepu8_epi32(composed));
+    _mm_storeu_si128(reinterpret_cast<__m128i *>(utf32_output+4), _mm_cvtepu8_epi32(_mm_srli_si128(composed,4)));
     utf32_output += 6; // We wrote 12 bytes, 6 code points.
   } else if (idx < 145) {
     // FOUR (4) input code-words

@NicolasJiaxin
The following patch should allow you to build a proper westmere kernel:

Yes, thanks!! It seems to fix the issue with westmere and the singleheader build. However, westmere no longer passes the test I implemented convert_valid_utf8_to_utf32 and convert_utf8_to_utf32. I can look into it.

Ah. Well, I just said compile not run. :-)

Ah. Well, I just said compile not run. :-)

Cannot deny that hahahaha. I think it should be fixed now. It was simply to use _mm_cvtepu16_epi32 instead of _mm_cvtepu8_epi32 since composed should be UTF-16 words and _mm_srli_si128(composed,8) instead of _mm_srli_si128(composed,4).

@lemire I have added all competitors that offer UTF-8 to UTF-32 transcoding to the benchmark, except CppCon2018. I am getting a similar error than the one we had with westmere build. I also did not include ICU in the benchmark although it is technically possible to transcode from UTF-8 to UTF-32 by using two conversions (UTF-8->UTF-16->UTF32), but I can if you think that is better. Here is a patch of my attempt to include CppCon2018:
diff --git a/benchmarks/src/benchmark.cpp b/benchmarks/src/benchmark.cpp
index c2b1e33..474a8f5 100644
--- a/benchmarks/src/benchmark.cpp
+++ b/benchmarks/src/benchmark.cpp
@@ -154,6 +154,11 @@ Benchmark::Benchmark(std::vector<input::Testcase>&& testcases)
         known_procedures.insert(name);
         expected_input_encoding.insert(std::make_pair(name,std::set<simdutf::encoding_type>({simdutf::encoding_type::UTF8})));
     }
+    {
+        std::string name = "convert_utf8_to_utf32+cppcon2018";
+        known_procedures.insert(name);
+        expected_input_encoding.insert(std::make_pair(name,std::set<simdutf::encoding_type>({simdutf::encoding_type::UTF8})));
+    }
 #endif
     {
         std::string name = "convert_utf8_to_utf16+hoehrmann";
@@ -231,6 +236,8 @@ void Benchmark::run(const std::string& procedure_name, size_t iterations) {
         // this is a special case
         if(name == "convert_utf8_to_utf16") {
           run_convert_utf8_to_utf16_cppcon2018(iterations);
+        } else if(name == "convert_utf8_to_utf32") {
+          run_convert_utf8_to_utf16_cppcon2018(iterations);
         } else {
           std::cerr << "unrecognized:" << procedure_name << "\n";
         }
@@ -757,6 +764,28 @@ void Benchmark::run_convert_utf8_to_utf16_cppcon2018(size_t iterations) {
     size_t char_count = active_implementation->count_utf8(reinterpret_cast<const char*>(data), size);
     print_summary(result, size, char_count);
 }
+/**
+ * Bob Steagall, CppCon2018
+ * https://github.com/BobSteagall/CppCon2018/
+ *
+ * Fast Conversion From UTF-8 with C++, DFAs, and SSE Intrinsics
+ * https://www.youtube.com/watch?v=5FQ87-Ecb-A
+ */
+void Benchmark::run_convert_utf8_to_utf32_cppcon2018(size_t iterations) {
+    using char8_t   = unsigned char;
+    const char8_t*  data = reinterpret_cast<const char8_t*>(input_data.data());
+    const size_t size = input_data.size();
+    std::unique_ptr<char32_t[]> output_buffer{new char32_t[size]};
+    volatile size_t sink{0};
+    auto proc = [data, size, &output_buffer, &sink]() {
+      sink = uu::UtfUtils::SseConvert(data, data + size, output_buffer.get());
+    };
+    count_events(proc, iterations); // warming up!
+    const auto result = count_events(proc, iterations);
+    if((sink == 0) && (size != 0) && (iterations > 0)) { std::cerr << "The output is zero which might indicate a misconfiguration.\n"; }
+    size_t char_count = active_implementation->count_utf8(reinterpret_cast<const char*>(data), size);
+    print_summary(result, size, char_count);
+}
 /**
  * Cameron, Robert D, A case study in SIMD text processing with parallel bit streams: UTF-8 to UTF-16 transcoding,
  * Proceedings of the 13th ACM SIGPLAN Symposium on Principles and practice of parallel programming, 91--98.
diff --git a/benchmarks/src/benchmark.h b/benchmarks/src/benchmark.h
index e22402c..3595535 100644
--- a/benchmarks/src/benchmark.h
+++ b/benchmarks/src/benchmark.h
@@ -92,6 +92,7 @@ namespace simdutf::benchmarks {
         * https://www.youtube.com/watch?v=5FQ87-Ecb-A
         */
         void run_convert_utf8_to_utf16_cppcon2018(size_t iterations);
+        void run_convert_utf8_to_utf32_cppcon2018(size_t iterations);
         /**
          * benchmarks/competition/u8u16 contains an open source version of u8u16, referenced in
          * Cameron, Robert D, A case study in SIMD text processing with parallel bit streams: UTF-8 to UTF-16 transcoding,

Also, do you want me to make a table comparing all the competitors on unicode_lipsum data?

I also did not include ICU in the benchmark although it is technically possible to transcode from UTF-8 to UTF-32 by using two conversions (UTF-8->UTF-16->UTF32), but I can if you think that is better.

It does not sound very enticing, now does it? It feels like it would not be reasonable as a competitor.

Also, do you want me to make a table comparing all the competitors on unicode_lipsum data?

Though it is fun, it is maybe too much work. I recommend picking a few... like latin/english, arabic, japanese...

@NicolasJiaxin
Here:
+        } else if(name == "convert_utf8_to_utf32") {
+          run_convert_utf8_to_utf16_cppcon2018(iterations);

You probably want to write run_convert_utf8_to_utf32_cppcon2018.
Second... try the following...
--- a/benchmarks/competition/CppCon2018/utf_utils.h
+++ b/benchmarks/competition/CppCon2018/utf_utils.h
@@ -368,7 +368,7 @@ UtfUtils::FastConvert(char8_t const* pSrc, char8_t const* pSrcEnd, char32_t* pDs
 ///     indicate an error was encountered.
 //--------------------------------------------------------------------------------------------------
 //
-KEWB_FORCE_INLINE ptrdiff_t
+ptrdiff_t
 UtfUtils::SseConvert(char8_t const* pSrc, char8_t const* pSrcEnd, char32_t* pDst) noexcept
 {

The problem is that CppCon2018 wants to forcefully inline the conversion function but we do not want that.
:-)

I wrote a script so it was not too bad. Here are the results for all implementations in our benchmark for transcoding various UTF-8 texts to UTF-32 (5000 iterations, data is from from unicode-lipsum).

CppCon2018 (GB/s)
fallback (GB/s)
haswell (GB/s)
hoehrmann (GB/s)
llvm (GB/s)
utf8lut (GB/s)
westmere (GB/s)
fallback(da) (GB/s)
haswell(da) (GB/s)
westmere(da) (GB/s)

Arabic-Lipsum.utf8.txt
0.672
0.573
2.255
0.376
0.266
1.770
1.810
0.503
2.130
1.697

Chinese-Lipsum.utf8.txt
0.913
0.686
2.773
0.458
0.572
2.303
2.270
0.589
2.599
2.104

Emoji-Lipsum.utf8.txt
0.982
0.738
2.257
0.461
0.590
0.681
1.581
0.627
2.145
1.493

Hebrew-Lipsum.utf8.txt
0.677
0.584
2.227
0.370
0.261
1.861
1.774
0.512
2.110
1.670

Hindi-Lipsum.utf8.txt
0.719
0.558
1.617
0.373
0.397
2.335
1.489
0.492
1.554
1.412

Japanese-Lipsum.utf8.txt
0.875
0.663
2.668
0.439
0.530
2.275
2.228
0.571
2.489
2.067

Korean-Lipsum.utf8.txt
0.840
0.698
1.907
0.380
0.324
2.035
1.747
0.597
1.821
1.646

Latin-Lipsum.utf8.txt
4.599
3.087
9.330
0.461
0.246
1.258
8.166
1.786
7.928
6.380

Russian-Lipsum.utf8.txt
0.593
0.445
2.224
0.330
0.265
1.827
1.816
0.402
2.117
1.705

da stands for dynamic allocation in the last three columns.
Sample command:  ./build/benchmarks/benchmark -F FILE_PATH -P convert_utf8_to_utf32 -I 5000

Second... try the following...
@lemire  Thanks, it worked!

I like these numbers. And wait till we throw AVX-512 at it!!!

@lemire How can I run the tests on an ARM machine?

@NicolasJiaxin Check your email.
:-)

It may be beneficial to sync with the main project.

These are the results for UTF-16 to UTF-32 conversions of a few implementations. Again, benchmarking using unicode-lipsum.

fallback
haswell
llvm
westmere
fallback(da)
haswell(da)
westmere(da)

Arabic-Lipsum.utf16.txt
2.938
17.887
1.475
13.752
2.251
10.756
7.969

Chinese-Lipsum.utf16.txt
2.936
19.535
1.474
14.032
2.242
11.915
7.857

Emoji-Lipsum.utf16.txt
2.601
0.641
2.456
2.499
2.167
0.625
2.191

Hebrew-Lipsum.utf16.txt
2.939
17.668
1.475
13.680
2.249
10.591
7.876

Hindi-Lipsum.utf16.txt
2.936
19.492
1.475
13.940
2.247
11.151
7.807

Japanese-Lipsum.utf16.txt
2.937
20.151
1.474
14.223
2.241
11.953
7.815

Korean-Lipsum.utf16.txt
2.936
17.806
1.474
13.593
2.247
10.614
7.687

Latin-Lipsum.utf16.txt
2.940
14.996
1.476
12.776
2.247
9.525
7.470

Russian-Lipsum.utf16.txt
2.941
15.955
1.475
13.151
2.249
10.112
7.651

For some reasons, haswell performs horribly on Emoji-Lipsum.utf16.txt (consists of surrogate pairs UTF-16).

For some reason, haswell performs horribly on Emoji-Lipsum.utf16.txt (consists only of surrogate pairs UTF-16, and is the only file in the list that contains such characters).

That's intriguing. Let us investigate.

@lemire I don't really understand where the decrease is coming from. The code for that case is basically the same as westmere, except that we are working with 256-bit vectors instead of 128. However, I have ran the test for UTF-16 to UTF-8 with the same file  (Emoji-Lipsum.utf16.txt) and haswell performs similarly to westmere and the fallback.

Yes. I will investigate. I was investigating something else earlier today. I will soon move to this issue you raise. Thanks.

@NicolasJiaxin  Would you update the README file with the new functionality?

This PR looks good to me. Let me investigate the performance issue.

The UTF-8 to UTF-32 over emoji looks good... (I am aware that it is not the issue you reported.)
$ ./build/benchmarks/benchmark -F unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt -P convert_utf8_to_utf32
We define the number of bytes to be the number of *input* bytes.
We define a 'char' to be a code point (between 1 and 4 bytes).
===========================
Using ICU version 66.1
testcases: 1
input detected as UTF8
current system detected as haswell
===========================
convert_utf8_to_utf32+cppcon2018, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt
  17.004 ins/byte,   17.004 cycle/byte,    0.776 GB/s (0.2 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.695 GHz,    3.572 ins/cycle
  68.012 ins/char,   68.012 cycle/char,    0.194 Gc/s (0.2 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf8_to_utf32+fallback, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt
  15.004 ins/byte,   15.004 cycle/byte,    0.867 GB/s (0.1 %), 0.000107 b.misses/byte, 0.000000 c.mis/byte,    3.695 GHz,    3.520 ins/cycle
  60.014 ins/char,   60.014 cycle/char,    0.217 Gc/s (0.1 %), 0.000427 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf8_to_utf32+haswell, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt
   4.975 ins/byte,    4.975 cycle/byte,    2.322 GB/s (0.6 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.699 GHz,    3.123 ins/cycle
  19.899 ins/char,   19.899 cycle/char,    0.580 Gc/s (0.6 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf8_to_utf32+hoehrmann, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt
  16.754 ins/byte,   16.754 cycle/byte,    0.461 GB/s (0.1 %), 0.000137 b.misses/byte, 0.000015 c.mis/byte,    3.694 GHz,    2.091 ins/cycle
  67.013 ins/char,   67.013 cycle/char,    0.115 Gc/s (0.1 %), 0.000549 b.misses/char, 0.000061 c.mis/char,     4.00 byte/char
convert_utf8_to_utf32+llvm, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt
  22.505 ins/byte,   22.505 cycle/byte,    0.557 GB/s (2.8 %), 0.000107 b.misses/byte, 0.000000 c.mis/byte,    3.694 GHz,    3.392 ins/cycle
  90.017 ins/char,   90.017 cycle/char,    0.139 Gc/s (2.8 %), 0.000427 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf8_to_utf32+utf8lut, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt
  13.952 ins/byte,   13.952 cycle/byte,    0.679 GB/s (0.4 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.693 GHz,    2.564 ins/cycle
  55.806 ins/char,   55.806 cycle/char,    0.170 Gc/s (0.4 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf8_to_utf32+westmere, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf8.txt
   7.174 ins/byte,    7.174 cycle/byte,    1.725 GB/s (8.3 %), 0.000107 b.misses/byte, 0.001053 c.mis/byte,    3.697 GHz,    3.347 ins/cycle
  28.695 ins/char,   28.695 cycle/char,    0.431 Gc/s (8.3 %), 0.000427 b.misses/char, 0.004211 c.mis/char,     4.00 byte/char

@NicolasJiaxin What do you think of this patch?
index 43caf8b..51eb976 100644
--- a/src/haswell/avx2_convert_utf16_to_utf32.cpp
+++ b/src/haswell/avx2_convert_utf16_to_utf32.cpp
@@ -56,9 +56,8 @@ std::pair<const char16_t*, char32_t*> avx2_convert_utf16_to_utf32(const char16_t
   const char16_t* end = buf + len;
   const __m256i v_f800 = _mm256_set1_epi16((int16_t)0xf800);
   const __m256i v_d800 = _mm256_set1_epi16((int16_t)0xd800);
-  const size_t safety_margin = 11; // to avoid overruns, see issue https://github.com/simdutf/simdutf/issues/92

-  while (buf + 16 + safety_margin <= end) {
+  while (buf + 16 <= end) {
     __m256i in = _mm256_loadu_si256((__m256i*)buf);

     // 1. Check if there are any surrogate word in the input chunk.

I am not reliably reproducing your performance issue... It is true that we are "relatively slow" on the Emoji-Lipsum dataset, but that's a trade-off... we simply do not consider that it is a likely scenario and we are happy to do away with high performance in that case. (Hopefully, with AVX-512, we will turn this around.)
Could you re-run your benchmark to see what you get?
$ ./build/benchmarks/benchmark -F unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt -P convert_utf16_to_utf32  -I 20000
We define the number of bytes to be the number of *input* bytes.
We define a 'char' to be a code point (between 1 and 4 bytes).
===========================
Using ICU version 66.1
testcases: 1
input detected as UTF16 little-endian
current system detected as haswell
===========================
convert_utf16_to_utf32+fallback, input size: 65542, iterations: 20000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.504 ins/byte,    6.504 cycle/byte,    0.568 GB/s (0.1 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.693 GHz,    1.000 ins/cycle
  26.013 ins/char,   26.013 cycle/char,    0.142 Gc/s (0.1 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+haswell, input size: 65542, iterations: 20000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.160 ins/byte,    7.160 cycle/byte,    0.635 GB/s (0.1 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.694 GHz,    1.231 ins/cycle
  28.640 ins/char,   28.640 cycle/char,    0.159 Gc/s (0.1 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+llvm, input size: 65542, iterations: 20000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   5.254 ins/byte,    5.254 cycle/byte,    2.105 GB/s (0.1 %), 0.000092 b.misses/byte, 0.000000 c.mis/byte,    3.699 GHz,    2.990 ins/cycle
  21.016 ins/char,   21.016 cycle/char,    0.526 Gc/s (0.1 %), 0.000366 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+westmere, input size: 65542, iterations: 20000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.191 ins/byte,    7.191 cycle/byte,    0.635 GB/s (0.1 %), 0.000046 b.misses/byte, 0.000000 c.mis/byte,    3.693 GHz,    1.236 ins/cycle
  28.765 ins/char,   28.765 cycle/char,    0.159 Gc/s (0.1 %), 0.000183 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char

I think I am also having trouble reliably reproducing the issue, but the last two times I have build and run the benchmark I got something like this...
$ ./build/benchmarks/benchmark -F ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt -P convert_utf16_to_utf32  -I 20000
We define the number of bytes to be the number of *input* bytes.
We define a 'char' to be a code point (between 1 and 4 bytes).
===========================
Using ICU version 67.1
testcases: 1
input detected as UTF16 little-endian
current system detected as haswell
===========================
convert_utf16_to_utf32+fallback, input size: 65542, iterations: 20000, dataset: ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.504 ins/byte,    6.504 cycle/byte,    0.590 GB/s (0.1 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.694 GHz,    1.040 ins/cycle
  26.014 ins/char,   26.014 cycle/char,    0.148 Gc/s (0.1 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+haswell, input size: 65542, iterations: 20000, dataset: ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.598 ins/byte,    6.598 cycle/byte,    2.364 GB/s (270.4 %), 0.000153 b.misses/byte, 0.002014 c.mis/byte,    3.705 GHz,    4.209 ins/cycle
  26.391 ins/char,   26.391 cycle/char,    0.591 Gc/s (270.4 %), 0.000610 b.misses/char, 0.008056 c.mis/char,     4.00 byte/char
WARNING: Measurements are noisy, try increasing iteration count (-I).
convert_utf16_to_utf32+llvm, input size: 65542, iterations: 20000, dataset: ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   5.254 ins/byte,    5.254 cycle/byte,    2.456 GB/s (0.1 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.700 GHz,    3.488 ins/cycle
  21.016 ins/char,   21.016 cycle/char,    0.614 Gc/s (0.1 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+westmere, input size: 65542, iterations: 20000, dataset: ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.629 ins/byte,    6.629 cycle/byte,    2.499 GB/s (0.2 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.699 GHz,    4.478 ins/cycle
  26.515 ins/char,   26.515 cycle/char,    0.625 Gc/s (0.2 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+fallback, input size: 65542, iterations: 20000, dataset: ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.760 ins/byte,    7.760 cycle/byte,    0.556 GB/s (0.1 %), 0.000122 b.misses/byte, 0.000000 c.mis/byte,    3.693 GHz,    1.169 ins/cycle
  31.038 ins/char,   31.038 cycle/char,    0.139 Gc/s (0.1 %), 0.000488 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+haswell, input size: 65542, iterations: 20000, dataset: ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.917 ins/byte,    6.917 cycle/byte,    2.013 GB/s (223.4 %), 0.000168 b.misses/byte, 0.001190 c.mis/byte,    3.696 GHz,    3.767 ins/cycle
  27.666 ins/char,   27.666 cycle/char,    0.503 Gc/s (223.4 %), 0.000671 b.misses/char, 0.004760 c.mis/char,     4.00 byte/char
WARNING: Measurements are noisy, try increasing iteration count (-I).
convert_utf16_to_utf32_with_dynamic_allocation+westmere, input size: 65542, iterations: 20000, dataset: ../unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.323 ins/byte,    7.323 cycle/byte,    2.200 GB/s (0.3 %), 0.000137 b.misses/byte, 0.000000 c.mis/byte,    3.697 GHz,    4.358 ins/cycle
  29.290 ins/char,   29.290 cycle/char,    0.550 Gc/s (0.3 %), 0.000549 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char

Now, it is fallback that is behind and westmere and haswell are fine.

@NicolasJiaxin Right. So when you look at the numbers, you see that the number of instructions is pretty much the same. What varies widely is the number of instructions retired per cycle. I will try it on an AMD processor to see if we can reproduce the issue.
I am pretty sure it has nothing to do with your code, per se.

@NicolasJiaxin Please consider the patch I proposed above.

Yes, I forgot to say. These numbers are with the patch applied.

On an AMD Rome processor, I do not encounter an issue... so I think we are good. You should have access to the AMD box.
$ ./build/benchmarks/benchmark -F unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt -P convert_utf16_to_utf32
We define the number of bytes to be the number of *input* bytes.
We define a 'char' to be a code point (between 1 and 4 bytes).
===========================
Using ICU version 67.1
testcases: 1
input detected as UTF16 little-endian
current system detected as haswell
===========================
convert_utf16_to_utf32+fallback, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.503 ins/byte,    6.503 cycle/byte,    2.580 GB/s (1.2 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.402 GHz,    4.932 ins/cycle
  26.013 ins/char,   26.013 cycle/char,    0.645 Gc/s (1.2 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+haswell, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.598 ins/byte,    6.598 cycle/byte,    2.463 GB/s (2.6 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.402 GHz,    4.776 ins/cycle
  26.390 ins/char,   26.390 cycle/char,    0.616 Gc/s (2.6 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+llvm, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   5.254 ins/byte,    5.254 cycle/byte,    3.386 GB/s (0.3 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.404 GHz,    5.226 ins/cycle
  21.015 ins/char,   21.015 cycle/char,    0.847 Gc/s (0.3 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+westmere, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.629 ins/byte,    6.629 cycle/byte,    2.459 GB/s (0.7 %), 0.000092 b.misses/byte, 0.000000 c.mis/byte,    3.401 GHz,    4.793 ins/cycle
  26.515 ins/char,   26.515 cycle/char,    0.615 Gc/s (0.7 %), 0.000366 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+fallback, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.759 ins/byte,    7.759 cycle/byte,    2.216 GB/s (0.5 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.400 GHz,    5.057 ins/cycle
  31.037 ins/char,   31.037 cycle/char,    0.554 Gc/s (0.5 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+haswell, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.916 ins/byte,    6.916 cycle/byte,    2.338 GB/s (2.4 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.401 GHz,    4.755 ins/cycle
  27.665 ins/char,   27.665 cycle/char,    0.585 Gc/s (2.4 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+westmere, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.323 ins/byte,    7.323 cycle/byte,    2.205 GB/s (0.8 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.400 GHz,    4.749 ins/cycle
  29.289 ins/char,   29.289 cycle/char,    0.551 Gc/s (0.8 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
dlemire@programming_station-for-dlemire:~/CVS/github/simdutf$ ./build/benchmarks/benchmark -F unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt -P convert_utf16_to_utf32
We define the number of bytes to be the number of *input* bytes.
We define a 'char' to be a code point (between 1 and 4 bytes).
===========================
Using ICU version 67.1
testcases: 1
input detected as UTF16 little-endian
current system detected as haswell
===========================
convert_utf16_to_utf32+fallback, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.503 ins/byte,    6.503 cycle/byte,    2.580 GB/s (2.3 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.401 GHz,    4.933 ins/cycle
  26.013 ins/char,   26.013 cycle/char,    0.645 Gc/s (2.3 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+haswell, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.598 ins/byte,    6.598 cycle/byte,    2.463 GB/s (2.6 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.402 GHz,    4.777 ins/cycle
  26.390 ins/char,   26.390 cycle/char,    0.616 Gc/s (2.6 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+llvm, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   5.254 ins/byte,    5.254 cycle/byte,    3.386 GB/s (0.3 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.404 GHz,    5.226 ins/cycle
  21.015 ins/char,   21.015 cycle/char,    0.847 Gc/s (0.3 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32+westmere, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.629 ins/byte,    6.629 cycle/byte,    2.461 GB/s (0.7 %), 0.000061 b.misses/byte, 0.000000 c.mis/byte,    3.401 GHz,    4.797 ins/cycle
  26.515 ins/char,   26.515 cycle/char,    0.615 Gc/s (0.7 %), 0.000244 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+fallback, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.759 ins/byte,    7.759 cycle/byte,    2.218 GB/s (0.5 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.400 GHz,    5.061 ins/cycle
  31.037 ins/char,   31.037 cycle/char,    0.554 Gc/s (0.5 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+haswell, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   6.916 ins/byte,    6.916 cycle/byte,    2.336 GB/s (2.4 %), 0.000076 b.misses/byte, 0.000000 c.mis/byte,    3.431 GHz,    4.710 ins/cycle
  27.665 ins/char,   27.665 cycle/char,    0.584 Gc/s (2.4 %), 0.000305 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char
convert_utf16_to_utf32_with_dynamic_allocation+westmere, input size: 65542, iterations: 2000, dataset: unicode_lipsum/lipsum/Emoji-Lipsum.utf16.txt
   7.323 ins/byte,    7.323 cycle/byte,    2.203 GB/s (1.1 %), 0.000092 b.misses/byte, 0.000000 c.mis/byte,    3.400 GHz,    4.745 ins/cycle
  29.289 ins/char,   29.289 cycle/char,    0.551 Gc/s (1.1 %), 0.000366 b.misses/char, 0.000000 c.mis/char,     4.00 byte/char

This PR looks good and we will merge it.

Sync with our main branch to catch this commit: https://github.com/simdutf/simdutf/commit/24dc56e4c09f607e606d0f69108c61e873acb718

@lemire like this?

like this?

I think not. I think you allocate char32_t and then you pass char, so you have to multiply your allocated region by four (since sizeof(char32_t)/sizeof(char) is 4).
I could be wrong... but I think that what we have looks ugly. It is not consistent.

So, like this?

This is now fine, I expect!

I will handle the merge.

It looks like it does not know in which order to merge both files. I can look into this @lemire

It is fine. I think I will get through it.
You should hear from me later today.


Fix ParticleFilter::updateWeights std_landmark[] argument description

A discussion with a fellow student lead us to believe that the description for the std_landmark[] argument for ParticleFilter::updateWeights() in particle_filter.h is incorrect.
Following the logic from lines 43 and 115 of main.cpp
double sigma_landmark [2] = {0.3, 0.3}; // Landmark measurement uncertainty [x [m], y [m]]

and
pf.updateWeights(sensor_range, sigma_landmark, noisy_observations, map);

respectively.
This suggests that the std_landmark argument reflects the measurement uncertainties of x and y, not range and bearing.
I hope I have correctly interpreted the ambiguity and that you think this correction is worth merging.

I've since pushed my solution to the project not realising it would impact this PR. I will create a new PR with only the relevant commit.

Yes, looks like the description of std_landmark in particle_filter.h is wrong.


T024: Pandas/jinja import error

On code cell 12 _translate_rescale_substitution_matrix().style.format("{:.2f}") the notebook fails withImportError: Pandas requires version '3.1.2' or newer of 'jinja2'. See here.

This issue might have resolved itself. See here


Avatar sizes increase to 48px globally.

Changes the avatar sizes found everywhere in the application to 48px. Since this is the size used by most other comparable messaging applications, 48px seems like a good option to go for (other reasons explained in the commit/s).
In the first commit .message-brief is set to padding: 0 16px 16px 73px. While this 73 might seem like an arbitrary number, it's confusing to me and I don't know why this is the case, but that is the only one which makes the messages perfectly aligned.

(The build failing doesn't have anything to do with the changes made in this code, it seems like. My other PRs also have the exact same error happening.)
Screenshots:

(The build failing doesn't have anything to do with the changes made in this code, it seems like. My other PRs also have the exact same error happening.)

(Yup -- see also #3526 .)

In the first commit .message-brief is set to padding: 0 16px 16px 73px. While this 73 might seem like an arbitrary number, it's confusing to me and I don't know why this is the case, but that is the only one which makes the messages perfectly aligned.

Huh, puzzling. That shouldn't be.
And in fact, when I run this, it looks slightly misaligned to me. Here's a screenshot, super zoomed in (a feature of the Android emulator -- Ctrl+Z, or the "zoom" icon in the emulator toolbar):

The PM conversations screen looks great, though!

Oh wow, here's a clue about the alignment issue, though -- in an emulated Nexus 5, the avatar winds up being some funny number of pixels wide! Here's the .avatar element:

Not 48px, but rather 38.828px.
(The actual avatar, the img, then gets scaled to that width and the corresponding height.)
I'm not sure what's controlling that... but that will be something for you to debug! :wink:
One related odd thing I notice: we set min-width. If that's doing anything at all, either before or after this change, then it's probably at least a clue as to what's going on here.

In the second commit:
@@ -42,7 +42,7 @@ class UserAvatarWithPresence extends PureComponent<Props> {
   static defaultProps = {
     avatarUrl: '',
     email: '',
-    size: 32,
+    size: 48,

Why change the default?
Are there other call sites this affects? If so, we'll want to look and make sure the effect is something we want.

One related odd thing I notice: we set min-width. If that's doing anything at all, either before or after this change, then it's probably at least a clue as to what's going on here.

 I don't know why, but setting min-width: 48 sets the width properly. I suspect it has something to do with the width of the phone (and hence the viewport), because on my device the width of the avatar before I changed min-width was ~ 40.821px. Something to do with used value in css, I think.
Checking width on MDN -

The min-width and max-width properties override width.

Which doesn't appear to be true, because if you remove the width property, the avatars become huge -- I believe the uploaded picture's width, and consequently the height (which is set to 48px!).  In any case, setting width to a value should set the width of the element in question as that value, without consideration! It's rather strange; I'm not able to figure out how the CSS works here.

Why change the default?
Are there other call sites this affects?

Shouldn't have changed the default. It does affect multiple sites, but at all call sites, for whatever reason, we override the default (to the same value as the default 😆). I've removed the change in the default size.

Thanks @ishammahajan for the PR and these updates! Looks good -- just merged.


Update jaspr in packages

I mostly care about jaspr_test because one of my packages depends on it. Every project in this repo should be using latest jaspr, but I don't want to deal with that so I just updated all the projects in packages since those are the most important.

Please make sure you run melos bootstrap to link packages together.

I ran it, but I'm not sure it actually did anything useful

Nevermind I see the path links now

Thx


chore(main): release 0.26.2

:robot: I have created a release beep boop
0.26.2 (2023-05-09)
Bug Fixes

Dialog: Increase the z-index of the dialog container (591bddb)
Layout: Sidebar background continues on scroll (591bddb)
Members: Do not display edit buttons if user has not the permission (591bddb)
Workspace: Hide countries property if empty (591bddb)
Workspaces: Various fixes based on MJL feedback (591bddb)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/BLSQ/openhexa-frontend/releases/tag/0.26.2 :sunflower: