[Question] Possible to (properly) implement simultaneous (custom) Kana-input layout?

Dear Google mozc team,
I've been using Google IME and mozc for some time and would first like to express my appreciation for releasing them to the world.
Apart from the Roma-ji and Kana layouts, there are other (sadly not very well-known) layouts out there. One of such layouts is 新下駄配列, which is one of the simultaneous Kana-input layouts.
https://wpedia.goo.ne.jp/wiki/新下駄配列 (Japanese)
The idea is that one can type different Hiragana, when 2 keys are hit (almost) simultaneously. For example in 新下駄配列,

'k' => 'い'
'd' => 'か'
'dk' => 'れ' (when typed simultaneously, otherwise "かい")
'kd' => 'れ' (when typed simultaneously, otherwise "いか")

I've forked and modified the mozc source code and implemented the PoC version of this functionality: https://github.com/kirameister/mozc
Here's how it looks like when in practice :

https://twitter.com/kirameister_ja/status/1412885637183721475
https://twitter.com/kirameister_ja/status/1412886174398623748

I'm wondering someone in Google mozc team would be able to implement this functionality. I consider my implementation to be nothing more than PoC, and would greatly appreciate it if you could implement it properly.
Of course I'd be more than happy to provide any information required for the implementation.
Best regards,
Akira K.

Hi Akira,
Thank you for your warm words. I'm glad that Mozc is used for your input method.
To understand the logic of 新下駄配列, please let me ask some questions.
Here are preconditions of the questions.

[K↓:10] means 'K' key was pressed after 10 millisecond from the first event.
100 milliseconds is the time threshold.

How to determine whether the two keys are pressed simultaneously or not.
Would you tell me if the following cases are determined as simultaneous input or not, or either is fine.

[D↓:0][K↓:10][D↑:20][K↑:25]
[D↓:0][K↓:10][K↑:20][D↑:25]
[D↓:0][K↓:200][D↑:210][K↑:220] (over the threshold)
[D↓:0][D↑:10][K↓:20][K↑:30] (within the threshold, but not overlapped)

If this is a simultaneous input, Which is "DKD" treated as "れか" or "かれ"?

Time threshold

Is 100 milliseconds the default number?
How often is this changed from the default? (= Is it acceptable not to be customizable?)

Implementation
Honestly speaking, we would not officially support additional input styles. On the other hand, it'd be nice if it is possible to customize input styles by customizing romaji composing rules or by applying patches to the code in a straightforward way.
Thank you,

Hello Hiroyuki-san,
I am super excited to have your reply!
How to determine whether the two keys are pressed simultaneously or not.
As far as I understood the mozc code, there's no key-code event sent to the mozc_server when the key is released (i.e., the client-server communication is done only when a key is pressed). Please correct me if that's not the case.
Having said that, following is the expected output at least from my implementation:

[D↓:0][K↓:10][D↑:20][K↑:25]

れ

[D↓:0][K↓:10][K↑:20][D↑:25]

れ (but key-event sequence is exactly the same as 1.. Perhaps that's a typo?)

[D↓:0][K↓:200][D↑:210][K↑:220] (over the threshold)

かい

[D↓:0][D↑:10][K↓:20][K↑:30] (within the threshold, but not overlapped)

れ (what counts is the 3rd event [K↓:20], which is below 100 threshold.

If this is a simultaneous input, Which is "DKD" treated as "れか" or "かれ"?

I understood this as something like [D↓:0][D↑:10][K↓:20][K↑:30][D↓:40][D↑:50]. If so, the output would be "れか" (and "か" would be stored in pending_ in Entry::Entry instance).

There are existing solutions for such simultaneous kana input. But (as far as I'm aware) none of them is implemented within the IME (therefore the timing when the key is released may need to be considered). In our context, we have the luxury to "update" what has already been rendered ("pending" vs. "result").
Time threshold

Is 100 milliseconds the default number?

In my implementation, I've created another slot in the protocol where this time threshold is stored. The number 100 just came from my intuition, nothing more.

How often is this changed from the default? (= Is it acceptable not to be customizable?)

I'm afraid it should be kept up to each end-user. Some may be fine with 50, others may need 200 (although I personally feel anything above 500 would do more harm than good, even for those who just started simultaneous kana input).
I was rather thinking of making the time-threshold for each simultaneous combination customizable (by adding anther column in the custom romaji-composition table -- more onto this point under Implementation below).

Implementation
I think I understand your position. On the other hand, I'm wondering following implementation-strategy would be feasible (in a sense if it could be accepted as "straightforward" patch):

Add one column in custom romaji-composition table (let's call it as prev_pending_limit), which is expected to be empty, or contain 0 or more integer value (if not qualified, the value will be discarded).
table.cc must be updated so that those values from the new (4th) column would be taken (so internally, Entry class would have another internal value to store).
As long as the prev_pending_limit of the entry is empty (or 0), or the elapsed time (from previous key-stroke) is equal or less than the value of prev_pending_limit, process goes as usual.
However, when the prev_pending_limit exists in entry and the elapsed time is greater than the value of prev_pending_limit, what was in the pending_ would be appended to conversion_ and then pending_ will be cleared. After that, process goes as before.

The advantages of above approach are:

The amount of changes in the internal data structure is kept minimal (e.g., protocol doesn't need to be changed).
We can let the end-user decide how much time (s)he would like to set as simultaneous threshold (for each combination!).

With that, the end user would need to update the custom romaji-table as following:

input
output
pending
prev-pending-limit

k

か

d

い

かd
れ

100

いk
れ

100

I hope this makes sense. From the outset, the content of the table may look a bit odd, but the existing romaji-table could be kept as is (and nothing would be different for those romaji-input users).
If any of my text above is unclear, or if you have anything you'd like to clarify, please let me know. Above implementation proposal is actually different from how I implemented it in my forked version. But if you'd think above is acceptable change (to the google/mozc repo), I'd be more than happy to do the implementation myself (although you'd most likely be faster and I'm most certain that your code would be cleaner than mine).
Best regards,
Akira K.

Thank you for the detailed descriptions. They are very helpful.
How to determine whether the two keys are pressed simultaneously or not.

As far as I understood the mozc code, there's no key-code event sent to the mozc_server

Correctly speaking, mozc.commands.KeyEvent.ModifierKey has KEY_DOWN and KEY_UP.
https://github.com/google/mozc/blob/master/src/protocol/commands.proto#L223
That's true that you don't need care about release events though.

[D↓:0][K↓:10][D↑:20][K↑:25]
[D↓:0][K↓:10][K↑:20][D↑:25]
れ (but key-event sequence is exactly the same as 1.. Perhaps that's a typo?)

3rd and 4th events are swapped. I think the result is れ anyway.

[D↓:0][D↑:10][K↓:20][K↑:30] (within the threshold, but not overlapped)
れ (what counts is the 3rd event [K↓:20], which is below 100 threshold.

If this is a simultaneous input, Which is "DKD" treated as "れか" or "かれ"?
the output would be "れか" (and "か" would be stored in pending_ in Entry::Entry instance).

It's good to know. So the missing part is basically time threshold.
Time threshold

I was rather thinking of making the time-threshold for each simultaneous combination customizable (by adding anther column in the custom romaji-composition table -- more onto this point under Implementation below).

How much do you customize the time for a single simultaneous input? If it is possible would you share your configurations for the reference?
Implementation
Romaji-composition table already has 4th column, named as 'attributes'.
https://github.com/google/mozc/blob/master/src/composer/table.cc#L423
You can probably use this 4th column for prev-pending-limit.
By the way, there is another way to make simultaneous input, if it is ok to use a single time threshold for all combinations.
Implementation with STOP_KEY_TOGGLING.
You might want to use the STOP_KEY_TOGGLING command.
https://github.com/google/mozc/blob/master/src/protocol/commands.proto#L452
This command was introduced for the toggle input on mobile devices.

If you tap "あ” twice within 1 second, "ああ" becomes "い".
If you tap "あ" twice slowly longer than 1 second, "ああ" remains "ああ".

To achieve the above behavior, mozc_client for mobile sends a STOP_KEY_TOGGLING command to mozc_server after 1 second if no event happens.
The composing rule can be a reference.
https://github.com/google/mozc/blob/master/src/data/preedit/toggle_flick-hiragana_intuitive.tsv
Strings inside {} are invisible. {!} indicates the STOP_KEY_TOGGLING command. The rules for you would be something like as follows. (I haven't confirmed them though).

input
output
pending
attributes

k

{?}か
NewChunk

{?}かd
れ

{?}か{!}
か

Thanks,

Hello Hiroyuki-san,
Thank you very much for your reply!
Again, I'll try to put my thoughts one by one..
How to determine whether the two keys are pressed simultaneously or not.

Correctly speaking, mozc.commands.KeyEvent.ModifierKey has KEY_DOWN and KEY_UP.
https://github.com/google/mozc/blob/master/src/protocol/commands.proto#L223

Ah, you're right! Apologies for missing those lines!

[D↓:0][K↓:10][D↑:20][K↑:25]
[D↓:0][K↓:10][K↑:20][D↑:25]
れ (but key-event sequence is exactly the same as 1.. Perhaps that's a typo?)

3rd and 4th events are swapped. I think the result is れ anyway.

Ah, again apologies for having missed that.. And yes, the expected output is "れ", which is determined on the 2nd event already.
Time threshold

How much do you customize the time for a single simultaneous input? If it is possible would you share your configurations for the reference?

Please find the following TSV file. Please notice that those lines with 4 columns (3 tab chars); those are the combinations where the top row of the keyboard (usually consisting numbers) is involved.
https://github.com/kirameister/mozc/blob/simul_kana_minimal_change_poc/src/data/preedit/shingeta.tsv
Please note that the value 200 is rather artificial; it may be that 150 may be enough. From my own experience (when I implemented simultaneous kana input in the past, before getting my hands onto the IME/mozc), I find it difficult to simultaneously type those keys, when one of them was on the top row, and thought it'd be nice to be able to customize the simultaneous threshold for each combination. However, it is nothing but "nice to have" feature and may not even be needed by most of the (potential) simultaneous kana-input users (at least I'm not aware of any existing solution, such as DvorakJ and やまぶきR where such a fine-grained configuration is possible). Personally, I'd be fine with not having this level of customizability, but being able to set a single threshold.
Implementation

You can probably use this 4th column for prev-pending-limit.

Yes, I thought of that when I was implementing my first PoC, and realized that it's been used with set of values (and that adding a new set of values may have unintended side-effect).
...However, now that I think of it again, one may actually update the existing set of values with much higher number, such that anything below those numbers would not have any impact on existing functionalities (i.e. updated internal representation would look something like the following):
enum TableAttribute {
  NO_TABLE_ATTRIBUTE = 1024,
  NEW_CHUNK = 2048,
  NO_TRANSLITERATION = 4096,
  DIRECT_INPUT = 8192,
  END_CHUNK = 16384,
};

Would you think it would work? One could set some sort of sanity-check (so that any value (even when it's numerical) below 1024 will be discarded.
https://github.com/google/mozc/blob/master/src/gui/config_dialog/roman_table_editor.cc#L182
Implementation with STOP_KEY_TOGGLING
Thank you very much for your pointer! I never thought of using the mobile-device functionality, but this may be more straightforward solution.
I've tried what you suggested by updating the custom romaji-table (in original mozc package, not my modified version) with {?}か{!} [tab] か and か{!} [tab] か on the 3rd row. Unfortunately neither worked (it always ended up in "/kd/" => "れ" no matter how long I waited before typing 'd').
Which implementation-approach would be more acceptable?
The big question for me is which implementation-approach would be more likely to be accepted into the google/mozc codebase. I've actually started implementing my suggested approach, and my very personal preference is to continue my path with extra column in roman_table_editor.cc (as I understand the STOP_KEY_TOGGLING approach involves the client side, which will involve multiple platforms and I would not be able to test on every platform).
However, if you think STOP_KEY_TOGGLING has higher chances to be accepted into google/mozc, I'd be more than happy to change my course and start looking into the relevant code.
cf. https://github.com/kirameister/mozc/tree/simul_kana_minimal_change_poc/src
Best regards,
Akira K.

(My reply would be next weekend or later. Sorry for your inconvenience).

Understood && Thank you very much for leaving this note!
In the meanwhile, I'll try to do my homework by implementing my suggested version of implementation-approach, which I hope to be able to present before this weekend.

Hello Hiroyuki-san ,
I've done yet another PoC implementation based on the approach that I proposed above (which is to add 4th column in the custom roman table, and use it for storing prev-pending limit by using the TableAttribute). You could see the list of the changes compared against the latest version of master branch of google/mozc:
https://github.com/google/mozc/compare/master...kirameister:simul_kana_minimal_change_poc?expand=1
And following is the STDOUT/STDERR output I observe by running the unit-test (command just copied from the instruction:
$ cd ~/src/mozc/src
$ bazel test base:util_test --config oss_linux -c dbg
bazel test base:util_test --config oss_linux -c dbg
INFO: Build option --compilation_mode has changed, discarding analysis cache.
INFO: Analyzed target //base:util_test (7 packages loaded, 846 targets configured).
INFO: Found 1 test target...
Target //base:util_test up-to-date:
  bazel-bin/base/util_test
INFO: Elapsed time: 50.267s, Critical Path: 5.30s
INFO: 310 processes: 126 internal, 184 linux-sandbox.
INFO: Build completed successfully, 310 total actions
//base:util_test                                                         PASSED in 0.0s
INFO: Build completed successfully, 310 total actions

From the outset, things seem to be working as expected.
For now (until you'd become available this weekend or later), I'll keep using it and see if there's any bug / unexpected behavior I missed.
As for the other approach (to use STOP_KEY_TOGGLING), I'll try to see how this could be utilized, but cannot promise how much progress I'd be able to make before the weekend, as I'm not even sure which part of the source-code files are responsible..
Best regards,
Akira K.

Hi again, sorry for my late response.
My future replies would be also on weekends only. Thank you for your understanding.
Time threshold
Thank you for the reference settings and background. I understood the simulation more. As the initial version, I'd like to minimize the modifications and the complexity. So I'd like to start from the version with a single threshold.
(Implementation)
This is not applicable, because I'd like to use a single threshold. It's just FYI.
In this case, I'd prefer adding key=value attribute and modifying the parser of the 4 column, rather than changing the enum value.
NewChunk TimeThreshold=100 NoTransliteration

Implementation with STOP_KEY_TOGGLING

I've tried what you suggested by updating the custom romaji-table (in original mozc package, not my modified version) with {?}か{!} [tab] か and か{!} [tab] か on the 3rd row. Unfortunately neither worked (it always ended up in "/kd/" => "れ" no matter how long I waited before typing 'd').

This requires to modify the client code to send the STOP_KEY_TOGGLING command.
Which implementation-approach would be more acceptable?
I'm still considering the best way to provide better customization (including simple code modification) minimizing the code complexity and the maintenance costs.
My current ideas are:

Use a single time threshold.
Base on the mechanism of STOP_KEY_TOGGLING.
Add a new field of timestamp to CharChunk and KeyEvent.

(If I was able to start from the scratch, the ideas would be different from the above.)
I will try to add the timestamp fields to CharChunk and KeyEvent.
Unittests
base:util_test is a test for the base/ directory.
In this case, please check cc_test_mozc rules in composer/BUILD.bazel and composer/internal/BUILD.bazel.
For example:
bazel test composer:composer_test composer/internal:char_chunk_test --config oss_linux

If I overlooked some of your questions/comments, please feel free to let me know.
Thank you,

Hello,
I'm completely fine with only-on-weekend communication, as long as move towards the implementation. Thank you for taking your time on this issue!
I understand that approach with STOP_KEY_TOGGLING will be more realistic approach (i.e., less maintenance costs). While I understand, I still have few questions:

Does "single time threshold" mean a fixed time threshold? Given that code snippet you wrote above, that seems like the case. If that's indeed the case, I wonder if I could ask to set the fixed time threshold to 50 (milliseconds) -- after playing (typing) around with the other implementation, I felt 100/200 milliseconds were too relaxed (i.e., leading to more typos) and 50 would be more realistic value.
Who will be implementing this? I read that you intend to implement this feature with STOP_KEY_TOGGLING approach. If that's the case, I am absolutely grateful! Of course, I'd be willing to contribute to the implementation, but I still feel lost how exactly the implementation would be done with STOP_KEY_TOGGLING (so it'd most likely take considerable amount of time for me to get it done).

Best regards,
Akira K.
PS: Thank you for pointing out about the scope of unit-test. There seems to be just a lot for me to learn :-)

Hello,

Does "single time threshold" mean a fixed time threshold? Given that code snippet you wrote above, that seems like the case. If that's indeed the case, I wonder if I could ask to set the fixed time threshold to 50 (milliseconds) -- after playing (typing) around with the other implementation, I felt 100/200 milliseconds were too relaxed (i.e., leading to more typos) and 50 would be more realistic value.

The default time threshold is probably 1000 msec. This is the current default value of STOP_KEY_TOGGLING used for the mobile input.

I think the patch to change it 100 msec should be straightforward.
It is still possible to customize the default in future, but please let me keep the current value while updating the code.

Who will be implementing this? I read that you intend to implement this feature with STOP_KEY_TOGGLING approach. If that's the case, I am absolutely grateful! Of course, I'd be willing to contribute to the implementation, but I still feel lost how exactly the implementation would be done with STOP_KEY_TOGGLING (so it'd most likely take considerable amount of time for me to get it done).

I or some of the Mozc team will implement it. I truly appreciate your PoC and kind offer for the implementation, however due to the internal policy, the files and directories we may accept pull requests are limited.
We start from refactoring of the composer/ directory. That refactoring changes perhaps break your patches first, but I believe it eventually improves code complexity and maintainability.
Thank you,

Hello,

It is still possible to customize the default in future, but please let me keep the current value while updating the code.

Understood. I'm fairly certain the configurability of the threshold will be demanded once people start using it, but I'm completely fine with having the simultaneous input functionality at first, and worry about the time threshold configuration later.

I or some of the Mozc team will implement it.

Thank you so much!

due to the internal policy, the files and directories we may accept pull requests are limited.

I was/am aware of this policy. However, I felt somewhat obliged to provide with the code (just to show the code, letting alone the pull request), as I was the one who brought up this question (which one may consider more like a feature-implementation request) in the first place.
本当にありがとうございます。
Best regards,
Akira K.

The commit 07c694e is a code refactoring. No behavior changes are expected yet.
The next commit (maybe next week) will add actual changes.

due to the internal policy, the files and directories we may accept pull requests are limited.
I was/am aware of this policy. However, I felt somewhat obliged to provide with the code (just to show the code, letting alone the pull request), as I was the one who brought up this question (which one may consider more like a feature-implementation request) in the first place.

Indeed, it made me think of a good way to support the request.

本当にありがとうございます。

My pleasure :)
Thanks.

Thank you so very much! I'm already excited to see your next commit :-)
I wonder if this feature would be implemented (reflected) on the Google IME (Google 日本語入力) as well. If so, I assume the potential impact to the society as a whole would be non-trivial.
Best regards,
Akira K.

Hi, the commit 296620c enables the simultaneous input with some configurations.
I hope new tests in composer_test.cc are helpful for you to see the necessary configurations.
https://github.com/google/mozc/commit/296620c04addd2fded7e1196f1051e867318a678#diff-2929ed83234fe579e57c1bf6c7808894abd80d1b8c36dbcd854f413b68f508d7
The configurations consist of two things:

Set 50 (ms) to Config.composing_timeout_threshold_msec
Create a composing table with "{!}", which represents timeout.

We do not have a plan to make a GUI for those configurations at this moment.

I wonder if this feature would be implemented (reflected) on the Google IME (Google 日本語入力) as well. If so, I assume the potential impact to the society as a whole would be non-trivial.

We cannot promise any things about Google Japanese Input here, although most code is shared each other.
Thank you for understanding.
If you have questions or problems for the configurations, please let me know.
Thank you :-)

Dear Hiroyuki-san,
Thank you very much for your implementation! Indeed, I still have few questions to ask..

Looking at the following lines, it seems that both timeout_threshold_msec and composing_timeout_threshold_msec are set to 0 by default (in the non-testing source code). I wonder if that was intentional.

https://github.com/google/mozc/commit/296620c04addd2fded7e1196f1051e867318a678#diff-06ca74cb8922d274a2f1c4cd892a3488688462ce31cad30fa8c760046667a196R284
https://github.com/google/mozc/commit/296620c04addd2fded7e1196f1051e867318a678#diff-c91e2c88434e76c14c3bdb5fc142dcd5730068ef3b61618d2eca2b7f61f0e0b6R322

I've locally modified above values to 50 and played around a bit (after importing the TSV file, of which the part of content is shown below). It seems, however, that "strange" chars are inserted before the 1st char to appear in the preedit (pre-composition) (as shown in the attached screeshot).

q		ー
ー{!}	ー
w		に
に{!}	に
e		は
は{!}	は
...
i		こ
こ{!}	こ
d		か
か{!}	か
...
かk	れ
いd	れ

I wonder if there's something I've missed (wrong insertion of STOP_KEY_TOGGLING, for example..).
Best regards,
Akira K.

Hello Akira-san,

Looking at the following lines, it seems that both timeout_threshold_msec and composing_timeout_threshold_msec are set to 0 by default (in the non-testing source code). I wonder if that was intentional.

Yes, this is intentional. If this value is 0, any behavior is not changed from the previous version. So we decided to keep it zero.

I see "strange" chars inserted before the 1st char

Ah, that's true. Please add the following line like toggle_flick-hiragana_intuitive.tsv

key
value
pending
attributes

{!}

NoTransliteration

I perhaps update the code not to require the above line later.

Dear Hiroyuki-san,
Thank you very much as always for your reply!

Looking at the following lines, it seems that both timeout_threshold_msec and composing_timeout_threshold_msec are set to 0 by default (in the non-testing source code). I wonder if that was intentional.

Yes, this is intentional. If this value is 0, any behavior is not changed from the previous version.

May I ask if there's any way to modify the [composing_]timeout_threshold_msec value(s) without modifying the source code (for example, by manually updating a file under $HOME/.mozc/)? I'm asking this because..

not being able to configure this value (apart from source-code update) would (most probably) introduce an extra hurdle for anyone (including myself, actually) to use this functionality.
it would be a bit tricky to file a bug report -- when you get a bug report, you'd probably expect it to be filed with a version that was officially released, instead of a version of which the source code was locally modified; but this may not be such a big issue, as the number of codes to be updated should be 2 (with current status of codebase).

Ah, that's true. Please add the following line like toggle_flick-hiragana_intuitive.tsv

I presume you meant to append such a line in the TSV file (of which I showed the snippet in previous comment) and import it within "Romaji table". However, I didn't observe any change in behavior after doing that. Looking at the roman_table_editor.cc and generic_table_editor.cc, it seems to me that the attributes (or 4th) column is not read at all. I do see the lines of code to read the attribute(s) from the stream in src/composer/table.cc (within Table::LoadFromStream(std::istream *is)). Did you mean to add the line with NoTransliteration within the code, such that it'd be included as part of custom_roman_table?

We cannot promise any things about Google Japanese Input here, although most code is shared each other.
Thank you for understanding.

Understood. I'll keep my fingers crossed that this functionality will be part of the Google Japanese Input in the future :-)
Best regards,
Akira K.

Dear Hiroyuki-san,
Just FYI, I've added a few lines to src/composer/table.cc (towards the bottom of Table::LoadFromStream(std::istream *is)) to artificially insert additional table entry with NoTransliteration and the strange behavior I described above seems to have disappeared.
Best regards,
Akira K.

Hi Akira-san,

May I ask if there's any way to modify the [composing_]timeout_threshold_msec value(s) without modifying the source code (for example, by manually updating a file under $HOME/.mozc/)?

$HOME/.mozc/config1.db contains both Romaji table and composing_timeout_threshold_msec. The file format is binary protocol buffer. config/config_hander.cc is the code to access config1.db.
(Note, the location may be $HOME/.config/mozc/config1.db).
There are some ways to modify config1.db.
Modify the GUI tool for config1.db.
As you previously modified it, you can add the composing_time_threshold_msec field to the GUI tool, and make the romaji table editor to append the special line of NoTransliteration on saving the data. The generated config1.db should be able to be usable on other environements.
Modify the config1.db file directly.
config1.db file is a binary protocol buffer. You can convert it to text protocol buffer, which is humanreadable format.
After you modify the text protocol buffer, you can convert it to config1.db in binary protocol buffer.
To convert text and binary protocol buffer formats each other, you can write a script in C++, Python and other languges.

https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.text_format
https://googleapis.dev/python/protobuf/latest/google/protobuf/text_format.html

In C++, please check PrintToString and ParseFromString functions.
In either way, you do not need to modify Mozc's main binaries. Once you create config1.db, you do not need even the modified GUI or scripts.
Please free feel to ask questions if you encouter additional questions.
Thanks,

Dear Hiroyuki-san,
Thank you very much for navigating me through the process to modify config1.db. I went with the path to update the file directly and it's working on my end!
Just for the record, I didn't actually have to call specific functions per se. Rather, it was a matter of decoding and encoding the binary file. FWIW following is what I've done after installing protoc:
cd $HOME/src/mozc
cat $HOME/.mozc/config1.db | protoc --proto_path=src/protocol --decode="mozc.config.Config" src/protocol/config.proto > temp_config.txt
UPDATE temp_config.txt such that "{!}\t\t\tNoTransliteration\n" is appended at "custom_roman_table" and add a new key-value pair "composing_timeout_threshold_msec: 50"
cat temp_config.txt | protoc --proto_path='src/protocol' --encode='mozc.config.Config' 'src/protocol/config.proto' > $HOME/.mozc/config1.db

I actually have 1 (most likely last) question:
From the outset, it seems that config1db binary file is platform-independent. In other words, could the modified config1.db still be used in macOS or Windows environment, even when the file was modified in Linux environment? It seems this is the case from the outset, but I wonder if you could confirm that.
Thank you very much for taking care of this request and guiding me through this journey. I hope/wish this functionality would be made available at Google IME (and I'd be checking the version number of my local (development-version of) Google IME, if the build value would be equal to or higher than 4450.
Best regards,
Akira K.

Actually, the overview page of the Protocol Buffers clearly states that it is platform neutral (which, I think, is equivalent to platform-independent).
https://developers.google.com/protocol-buffers/docs/overview
So all my questions have been answered. Thank you very much for your time and effort for interacting with me and implementing this functionality! May mozc live long and prosper!
Best regards,
Akira K.

I'm glad to hear that you succeeded the configuration.
I will close this issue after:

Update the document about config1.db
Consider how to handle {!} \t \t  NoTransliteration.

It was a nice time to discuss this feature with you.
Thanks,

Dear Hiroyuki-san,
My apologies for the delayed reply. Yes, I have no objection in closing this ticket. I'm very much thankful for those two points (to be done before closing the tickets) as well. Thank you again very much!
Best regards,
Akira K.

Hi Akira san,
It's no problem. I still action items to close this issue.

0a1d96f402f72e312d30d276924db1848de6f312 is the update of the document about config1.db.
Handling {!} \t \t NoTransliteration has not been done yet.

I recently added two changes related with this topic.

https://github.com/google/mozc/commit/446c5ba9ad3e9200e71b2cbf03cabc5c5801dd01 : The GUI remains invisible configurations (i.e. composing_timeout_threshold_msec)
https://github.com/google/mozc/commit/fe61216b768752bfe69b8b6a86d150303d273079 : The composer handles special characters like {!} without the fallback entries like {!} \t \t NoTransliteration.

FYI, if you want to make three key combinations keeping intermediate two key combinations, using special keys might be helpful (although I haven't tested it).

key
value
pending

a

あ

あb

あ{c}

あ{c}c
あい

あ{c}{!}
あ

Hello Hiroyuki-san,
Thank you very much for your follow-up!
I have few questions..

Combining those 2 bullet-point items, do I understand it correctly that the only missing piece in the puzzle (for realizing the simultaneous layout) is the GUI component to specify composing_timeout_threshold_msec?
How likely do you see such GUI component to be made available for mozc (and eventually in Google IME)?

The 2nd point is somewhat crucial for me. As you may have seen, I've created a Google Colab page to make this simultaneous feature more widely (and easily) available. But I'm hoping I could someday make this page part of the history.
My personal agenda was to come back to you with this request (to extend the current GUI settings) once this feature becomes known to some people (i.e., there would be obvious amount of people who would benefit from this feature), but if there is already a plan to do that, it'd really be great :)
I haven't tested what you suggested with {c}. Is c in {c} fixed, of it is {c} simply because it's followed by c for "あい" in the 3rd row?
Akira K.

Combining those 2 bullet-point items, do I understand it correctly that the only missing piece in the puzzle (for realizing the simultaneous layout) is the GUI component to specify composing_timeout_threshold_msec?

I hope yes. Please let us know if there are some other missing items.

How likely do you see such GUI component to be made available for mozc (and eventually in Google IME)?

The current implementation is a reuse (and expansion) of the existing supported feature (i.e. toggle input for mobile), and we have not decided supporting this simultaneous input. If we add it to GUI, it means we have finalized the specifications and start supporting the feature for all users with the current implementation.
So at this moment, we would like to keep it as an experimental feature without GUI support.
I appreciate your understanding.
Regarding {c}, {c} can be any special keys (e.g. {x}). However, I noticed there are some unexpected behaviors in the current implementation (2.28.4990). I'm investigating it now.
Thank you,

Related with this topic, we recently added an experimental feature that enables using function keys (e.g. F1, Henkan, etc.) in the romaji customization table.
Here are steps to use Henkan to type あ.

Set Henkan to InsertCharacter in the keymap

Open preference
Select "Customize…" of Keymap style
Export the setting to a file.
Modify the file to append the following rules
Import the file.

Mode
Key
Command

Composition
Henkan
InsertCharacter

DirectInput
Henkan
InsertCharacter

Add the following rules to Romaji table

Input
Output
Next input

{henkan}
あ

With this feature, you can make keymaps of simultaneous input with function keys :-)
Note:

This is an experimental feature. We may not provide a GUI and we may remove this feature.
You might need to restart the Ibus Mozc client to apply the keymap rules for the DirectInput mode.
Some IMF clients might not support the customization of the DirectInput mode.
The names of the function keys: Add GetSpecialKeyString · google/mozc@6f2be66
Test cases as example: Add test cases for simultaneous input. · google/mozc@5fa3558

Hello Hiroyuki-san,
Thank you very much for this update! I myself an not an Oyayubi-typer (and I don't think I'd ever be), but there may be people out there who are interested.
I'll ask around if people may be interested in this functionality, and I may be trying this out myself if/when I'd have enough bandwidth.
If I may say my honest opinion, however, I'm not too convinced if this is something that should be supported by an IME. To me, there should be a clear distinction between function keys and letter-keys (let's call those non-function keys); if one would like to assign a function keys to a letter key, he should be able to do so via other means (key-code hacks, firmware update, etc..). I'm a bit concerned that introducing this functionality could result in some sort of an abuse (although I cannot really pinpoint what exactly such an abuse would be). Nonetheless, I very much appreciate this new functionality (and hope that there would be more people interested in "alternative" layouts :)
Best regards,
Akira K.


Superset install (pip) fails because of a broken dependency (Linux/Ubuntu)

How to reproduce the bug
python3 -m pip install apache-superset
superset db upgrade
export FLASK_APP=superset
superset fab create-admin
superset load_examples
ImportError: cannot import name 'soft_unicode' from 'markupsafe'
It seems latest markupsafe pkg broke backwards compatibility by removing the soft_unicode module.

Came up in other dependence errors conversation.
https://github.com/apache/superset/issues/18723#issuecomment-1045160120

This issue was resolved by disscussion.


Can someone help me about this error?

Use Jumblr to develop an android app.
I got token and secret successfully, and I can get my following blogs' infos.
But the logcat output some error:
E/NativeCrypto: ssl=0x7f58062880 cert_verify_callback x509_store_ctx=0x7f62ebed78 arg=0x0
E/NativeCrypto: ssl=0x7f58062880 cert_verify_callback calling verifyCertificateChain authMethod=ECDHE_RSA

@JosephusChou are you sure that's an error? Looks like its just output from Android's native crypto lib on establishing SSL

@KevinTCoughlin I'm a newbie about SSL and OAUth, so I'm not clear about that, maybe I should learn more about SSL

@JosephusChou I think @KevinTCoughlin is right here, according to this StackOverflow post this is normal for some devices.
If you start getting errors from Jumblr, please reopen a different issue!


feat(directive): add reload-on-resize option

Add a $watch to detect when masonry is resized and call ctrl.reload().
Triggers a reload when the masonry element changes its width, useful when only the parent element is resized (and not the window) and you want the elements to be rearranged. Without this if the parent is resized then some blank space may be left on the sides.

That looks great, thanks for the PR. Do you think you could make the test a bit more specific?

@passy I think this makes the tests more specific (bf280d0095cf15d327936fdcd7903780ab40f32c), what do you think?

Much better! Thanks a lot, Tomer!


Elasticsearch test 'hangs' and gets terminated by timeout

Elasticsearch test seems to get stuck for a long time and subsequently causes the external test build to be terminated by timeout.
We should revisit the set of Elasticsearch tests we are running and either find out why the test is getting stuck or run a separate set of tests which are more stable.
JDK version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-201809230328-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Build link
https://ci.adoptopenjdk.net/view/Test_external/job/openjdk8_hs_externaltest_x86-64_linux/182/console
https://ci.adoptopenjdk.net/view/Test_external/job/openjdk8_hs_externaltest_x86-64_linux/183/console
https://ci.adoptopenjdk.net/view/Test_external/job/openjdk8_hs_externaltest_x86-64_linux/184/console

will take a look

I tried to run with  below OpenJDK  version:-
openjdk version "1.8.0-adoptopenjdk"
OpenJDK Runtime Environment (build 1.8.0-adoptopenjdk-jenkins_2018_08_24_11_23-b00)
OpenJDK 64-Bit Server VM (build 25.71-b00, mixed mode)
Steps :-
git clone https://github.com/elastic/elasticsearch.git
cd elasticsearch
git checkout v6.1.3
gradle assemble
./gradlew test --> Test failed with "java.net.BindException: Address already in use"
This could be a machine issue. I will try with a different machine and the build mentioned by you

I tried with below OpenJDK-Hotspot build and executed the same steps mentioned in the above command manually. Everything worked fine for me.
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-201809250329-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Elasticsearch Build and  test both were successful.

Task :distribution:tools:plugin-cli:test
==> Test Info: seed=B8FCE87DB416FBDC; jvms=4; suites=4
==> Test Summary: 4 suites, 228 tests, 8 ignored (8 assumptions)

BUILD SUCCESSFUL in 39m 56s
307 actionable tasks: 307 executed

The "hang" is not reproducible using grinder: https://ci.adoptopenjdk.net/view/work in progress/job/Grinder/188/consoleFull
Grinder ran on test-packet-ubuntu1604-x64-2
All 3 builds in which the hang was detected ran on test-packet-ubuntu1604-x64-1 .
It could be a machine issue. I'm gonna run the grinder again on test-packet-ubuntu1604-x64-1 to see if I can reproduce the hang there.


modify assetsPublicPath lead to "can not get"

Use the webpack template.
Issue: I modified the assetsPublicPath option in config.js when to built my project, it's run properly.
but when I run dev agian, the page "can not get" appears.
So I find the cause. If the assetsPublicPath change from '/' to others, the dev server can not find the
rootpath when npm run dev. Then I add a option in config.js
dev: {
assetsPublicPath:'/',
port: 8080,
proxyTable: {}
}
and modify dev-server.js
var staticPath = path.posix.join( config.dev.assetsPublicPath, config.build.assetsSubDirectory)
finally modify webpack.base.conf.js
output: {
path: config.build.assetsRoot,
publicPath:  process.env.NODE_ENV === 'production' ? config.build.assetsPublicPath : config.dev.assetsPublicPath ,
filename: '[name].js'
}
So, I do not have to modify assetsPublicPath option after npm run build every time.

@rayzang8 can you explain it a bit more. Also, if you're using one of official vuejs-templates please file an issue on repository you use.

I have file this issue on webpack reposityory

WTF??? loop reference?
I encounter the same problem, how to solve it?


DVX-155: Always include a sort by GUID as the final criteria

It seems like sorting by GUID doesn't work with audit search and search log 🤔. The integration tests fail with a 400 bad request. I think we should exclude sorting by GUID for those search requests.

There won't necessarily be unique record in these other indexes (audit search for example will have many records per asset GUID, one record for each change to the asset) — but sorting by it still shouldn't cause a failure. The issue is more likely because in these different indexes the GUID field has different names. In indexsearch it is __guid, but in audit search it is (iirc) entityId, and in search log it is probably something else still (don't recall what off the top of my head).


Iscroll performance issues on android

Recently I did some iscroll performance analysis on android.  I even wrote my own scroller for comparison. To my surprise, iscroll perform badly when scrolling many dom elements. tested on several native browser on android 4.0 4.1 phones. (chrome for android does not reproduce the issues.)
After some timeline analysis, I found iscroll have this transition: 0ms on the scroll container. With it, browser still fire transition end events and have long recalculate style duration. Even though, it shouldn't. I guess, it's a bug with android browsers triggered by 0 transition.
I removed the transition: 0ms css property, and it actually perform better. I think you need to remove the transition when it's not needed, instead of leaving it at 0ms.
Have a look at this slide, showing some of the timeline I recorded.
https://www.icloud.com/iw/#keynote/BAKqc5x-NqumGyFYKiOBNw7XGVRo4Y1Nm9KF/iscroll_performance
These are test pages I use.
http://static.pyco.me/scroll/iscroll.html
http://static.pyco.me/scroll/xscroller.html

@tasosno1 In my project, which is a page imbedded in a webview, the horizontal scroll is tremulous(maybe called shake). I use Transition to false, it performance well.
Below is my setting:
                            scrollX: true,
                            scrollY: false,
                            tap: true,
                            eventPassthrough: true,
                            useTransform: false,
                            useTransition: false


Replace Google Hire hyperlinks with HireVue Links

DDS has used the https://www.dds.mil/aep redirect to go the Jyn hiring announcement on Google Hire.
Request it be updated to redirect to https://dds.hirevue.com/signup/nCrKGCmnYGjYRFHKvmPAJX/
DDS has also used the "TDY to DDS" button under https://www.dds.mil/#join to link to the Jyn hiring announcement under Google Hire.
Request it be updated to link to
https://dds.hirevue.com/signup/nCrKGCmnYGjYRFHKvmPAJX/

Cannot push, so here is the diff for the two files:
diff --git a/src/_redirects b/src/_redirects
index d2d0298..cc4d87a 100644
--- a/src/_redirects
+++ b/src/_redirects
@@ -1,4 +1,4 @@
 https://dds-mil.netlify.com/* https://dds.mil/:splat 301!

 # AEP is the DDS/ARCYBER Advanced Education Program and is part of the Tatooine effort
-/aep https://dds.hirevue.com/signup/nCrKGCmnYGjYRFHKvmPAJX/
+/aep https://hire.withgoogle.com/public/jobs/ddsmil/view/P_AAAAAAEAAF3ILPyTyec1A7 301

diff --git a/src/index.html b/src/index.html
index 86eb69d..755c78f 100644
--- a/src/index.html
+++ b/src/index.html
@@ -72,8 +72,7 @@ title: Defense Digital Service

Our partners

We work alongside service members and civil servants across the Defense Department. DOD civilians and military members with technical expertise can apply to work with DDS on short-term details.

-      TDY to DDS
+      TDY to DDS


Harbor continuously fails to start after OVA reboot

On an integrated OVA, I was hitting
root@vic-st-h2-132 [ /etc/vmware/harbor ]# docker-compose ps
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
For which I rebooted the OVA after which harbor service fails to start up
mysql.log says:
May 31 06:40:31 <IP_ADDRESS> mysql[368]: 2017-05-31 06:40:31 1 [Note] InnoDB: Starting shutdown...
May 31 06:40:33 <IP_ADDRESS> mysql[368]: 2017-05-31 06:40:33 1 [Note] InnoDB: Shutdown completed; log sequence number 2350137
...
May 31 06:40:33 <IP_ADDRESS> mysql[368]: 2017-05-31 06:40:33 1 [Note] Shutting down plugin 'mysql_native_password'
May 31 06:40:33 <IP_ADDRESS> mysql[368]: 2017-05-31 06:40:33 1 [Note] Shutting down plugin 'binlog'
May 31 06:40:33 <IP_ADDRESS> mysql[368]: 2017-05-31 06:40:33 1 [Note] mysqld: Shutdown complete
Attached the harbor logs from the OVA.
2017-05-31.zip
The service attempts to starts and then exits.
root@vic-st-h2-132 [ /etc/vmware/harbor ]# docker-compose  ps
Name                     Command               State            Ports
harbor-adminserver   /harbor/harbor_adminserver       Up
harbor-db            docker-entrypoint.sh mysqld      Up      3306/tcp
harbor-log           /bin/sh -c crond && rm -f  ...   Up      <IP_ADDRESS>:1514->514/tcp
registry             /entrypoint.sh serve /etc/ ...   Up      5000/tcp
root@vic-st-h2-132 [ /etc/vmware/harbor ]# docker-compose  ps
Name                     Command                State              Ports
harbor-adminserver   /harbor/harbor_adminserver       Exit 2
harbor-db            docker-entrypoint.sh mysqld      Up         3306/tcp
harbor-jobservice    /harbor/harbor_jobservice        Exit 2
harbor-log           /bin/sh -c crond && rm -f  ...   Up         <IP_ADDRESS>:1514->514/tcp
harbor-ui            /harbor/harbor_ui                Exit 2
nginx                nginx -g daemon off;             Exit 128
registry             /entrypoint.sh serve /etc/ ...   Exit 2

vm-2017-05-31.23497.tar.gz

@lgayatri, I cannot reproduce it, can you tell the exactly steps? If possible, can you share your env to have a debug?

@wy65701436

First I hit https://github.com/vmware/harbor/issues/2394
After which I restarted the integrated OVA
Upon its power on checked docker-compose ps which reported the given failure.
Scale : 2500 containers and 50 VCH

can you have your env to debug this issue?

@wy65701436 : I just  deleted all 50 VCHs and 2500 containers. Will provide the setup for you on next tear down window.

@wy65701436 , I could NOT reproduce this issue with build 11340. Will try on the next build as well and conclude.

Thanks @lgayatri for your infor, shall we just close this issue? And you can reopen it if any new clue.

Ok @wy65701436


consolidate analyze and corrected capacitance term

previously analyze was split into two cases: one where the number of total points was an integer multiple of the number of points per CPD and one that was not. this is removed and the extra points (are assumed to be at the end and) are simply left out out of the analysis
capacitance term previously was set to offset in quadratic fit; changed to a term.

What is the benefit of leaving the data off at the end? Just so you have an exact number of cycles or readibility?
Good catch on the capacitance thing!

Unless I'm missing something, it seems like splitting it up into two cases where remainder = 0 and remainder is non-zero is not necessary. So, readability. Also, the extra points aren't lopped off and are still part of self.signal_array, just not processed in the analyze method.

There aren't two cases. It runs the first loop (the for loop). Then, if there are any point that aren't accounted for (remainder), it processes those.
But, this also only ever changes the last point on the CPD trace, so it probably doesn't matter

My bad on this one; I'll add it back next PR for completeness.


Remove most qiskit.org links

Part of https://github.com/qiskit-community/qiskit-machine-learning/issues/760. The release notes will be fixed in a separate PR to simplify cherry-picking.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 92.567%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1868

Relevant Lines:
2018

💛  - Coveralls


[BEAM-403] Get current SDK package from PyPI instead of GitHub

With this change we support (in the DataflowPipelineRunner) getting the SDK package from PyPI where we will release Google packages bundling Apache Beam SDK code. This needs to change since currently we get this package from the old SDK repo (before we moved to Apache Beam).
The CL also has support for computing the proper container version to be used for remote execution and in particular supports the scenario of running a non-Google wrapped Apache Beam SDK which is needed to run E2E tests in the Apache infrastructure. For this latter case the container version is 'latest' which works well for running E2E tests at HEAD.

R: @robertwb and @aaltay can you please take a look?

LGTM. I added a few minor comments.

Addressed your comments @aaltay - Thanks!

Mostly looks good, just a couple of minor comments.

PTAL @robertwb Thanks!

LGTM

Merged at https://github.com/apache/incubator-beam/commit/fbe44ee83574601866fdd2831d44dcbfd764d055

(please manually close PR at your leisure)

I can't :(. Not sure why the "Closes ..." comment didn't do the trick either.

The request was directed @silviulica.
As for why: @aaltay figured this out.
GitHub will close the PR only when merged into master. You can mouse-over the question mark above to see this.
However, if the rebase onto python-sdk is a no-op, then the commit merged into python-sdk when you push will be the exact same commit (including hash) as the one in this PR. In that unique case, GitHub will mark this PR as merged.


x/sys: howto set a autorun registry  in windows

I write the code, but does not work. it cannot create xxx and no error found.
func SetAutoRun(run bool) error {
	k, err := registry.OpenKey(registry.CURRENT_USER,
		`HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Run`,
		registry.QUERY_VALUE|registry.SET_VALUE)
	if err != nil {
		return err
	}
	defer k.Close()
	if run {
		if err := k.SetStringValue("xxx", SysExecPath("xxx.exe")); err != nil {
			return err
		}
	} else {
		k.DeleteValue("xxx")
	}
	return nil
}

another code:
	key, _, err := registry.CreateKey(registry.CURRENT_USER, "HKEY_CURRENT_USER\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run", registry.ALL_ACCESS)
	if err != nil {
		return err
	}
	defer key.Close()

	if run {
		nkey, _, err := registry.CreateKey(key, "xxx", registry.ALL_ACCESS)
		if err != nil {
			return err
		}
		defer nkey.Close()
		err = nkey.SetStringValue("String", SysExecPath("xxx.exe"))
		if err != nil {
			return err
		}
	} else {
		key.DeleteValue("xxx")
	}

I didn't try running the code, but since you already are opening registry.CURRENT_USER, you can drop HKEY_CURRENT_USER from the path:
registry.OpenKey(registry.CURRENT_USER, `SOFTWARE\Microsoft\Windows\CurrentVersion\Run`, ...)

Just posting for the sake of posterity:
The reason I came across this was that I was trying to create a user-mode service that would run on user login every time.
Here's how I did it, which is working for me presently:
https://git.rootprojects.org/root/serviceman/src/commit/f95897cf308c3e19d6fe4b088547d8290907e384/manager/install_windows.go#L60-L109
I also needed to do similar for Mac and Linux, and that project is the cross-platform service manager I came up with, suited to my needs - which are more focused on user-mode services than system-level services.


Port the EvolvedOperatorAnsatz from Nature

Summary
Port the EvolvedOperatorAnsatz from Nature to core.
Related to Qiskit/qiskit-nature#117.
Details and comments
It was initially located in Nature as we needed it for the UCCSD circuits and couldn't wait for a Terra release. However, it should live in Terra as it can be the base for other library circuits, such as QAOA (which is included in this PR).
Two lines in the tests have been modified since they relied on a particular random array but with this PR and the changed code the random array changed. I changed the test criterion from a comparison to the final parameters to the final loss.

Yeah that's a bit annoying (as you can see from the last 4 Sphinx commits 😉). I'm not sure how we can fix this in the future since these are cyclic imports. Maybe Python 3.7s typing system can fix it?

@Cryoris Why do you need to change the parameter order? It causes a unit test failure at qiskit-optimization. See #6424.

In addition, this PR may have forgotten to set the bounds.
In the past, we set bounds here.
This changes the test result.
Before:  x: array([-0.56 ,  0.315])
After: y: array([-0.564, -5.96 ])


MobileDatePicker's Cancel button does not work.

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Current behavior 😯

Changes are not canceled when the CANCEL button is pressed.
Selecting a date will fire onChange, and the date will be updated.

Expected behavior 🤔
Make the firing timing of onChange only when the OK button is pressed.
Steps to reproduce 🕹
Steps:

Access to https://codesandbox.io/s/eloquent-feather-iznb0?file=/src/index.tsx
Press the TextField(For mobile)
Select an appropriate date.( You can see the date change at this time.)
Press the CANCEL button.(You can confirm that the changes have not been canceled. )

Context 🔦
No response
Your environment 🌎

`npx @mui/envinfo`
  Don't forget to mention which browser you used.

    Chrome: 97.0.4692.71
    Firefox: 96.0.1

  Output from `npx @mui/envinfo` goes here.

  System:
    OS: macOS 11.6.1
    CPU: (8) x64 Apple M1
    Memory: 33.54 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.13.1 - ~/.anyenv/envs/nodenv/versions/16.13.1/bin/node
    Yarn: 1.22.17 - /opt/homebrew/bin/yarn
    npm: 8.1.2 - ~/.anyenv/envs/nodenv/versions/16.13.1/bin/npm
  Managers:
    Homebrew: 3.3.9 - /opt/homebrew/bin/brew
    pip3: 21.0.1 - /opt/homebrew/bin/pip3
    RubyGems: 3.0.3 - /usr/bin/gem
  Utilities:
    Make: 3.81 - /usr/bin/make
    GCC: 4.2.1 - /usr/bin/gcc
    Git: 2.30.1 - /opt/homebrew/bin/git
    Clang: 12.0.5 - /usr/bin/clang
    FFmpeg: 4.3.2 - /opt/homebrew/bin/ffmpeg
  Servers:
    Apache: 2.4.48 - /usr/sbin/apachectl
  Virtualization:
    Docker: 20.10.7 - /usr/local/bin/docker
  IDEs:
    Nano: 2.0.6 - /usr/bin/nano
    Vim: 8.2 - /usr/bin/vim
    Xcode: /undefined - /usr/bin/xcodebuild
  Languages:
    Bash: 3.2.57 - /bin/bash
    Perl: 5.30.2 - /usr/bin/perl
    PHP: 7.3.29 - /usr/bin/php
    Python: 2.7.16 - /usr/bin/python
    Python3: 3.9.2 - /opt/homebrew/bin/python3
    Ruby: 2.6.3 - /usr/bin/ruby
  Databases:
    SQLite: 3.32.3 - /usr/bin/sqlite3
  Browsers:
    Chrome: 97.0.4692.71
    Firefox: 96.0.1
    Safari: 15.1

has anyone started working on this issue? Can I pick this one?

Not sure if this helps, but until this gets fixed, you could try creating a "cancel" effect on your end.
You could use onOpen to determine what the value was before changes, and use onAccept and onClose to determine if the OK button was clicked or if the Cancel button was clicked then maintain values accordingly.


Ruby warnings

dropbox-sdk-1.6.5/lib/dropbox_sdk.rb:162: warning: assigned but unused variable - port
dropbox-sdk-1.6.5/lib/dropbox_sdk.rb:951: warning: assigned but unused variable - raw
dropbox-sdk-1.6.5/lib/dropbox_sdk.rb:965: warning: assigned but unused variable - raw
dropbox-sdk-1.6.5/lib/dropbox_sdk.rb:1245: warning: assigned but unused variable - raw
dropbox-sdk-1.6.5/lib/dropbox_sdk.rb:1261: warning: assigned but unused variable - raw
dropbox-sdk-1.6.5/lib/dropbox_sdk.rb:1275: warning: assigned but unused variable - raw

Thanks for the report!

It's not really a "bug" per se, but it clutters our debug output, and also may be an indication of mistakes, like here where you seem to forget to add port to newly built URI.


Bow Support

Looks like the addon currently does not support bows, only crossbows. Would it be possible to add bows to the list? Thank you.

Went to add it myself, see #24


test: add test for dns.rcode

Feature #6621
Redmine ticket: https://redmine.openinfosecfoundation.org/issues/6621
Suricata PR: https://github.com/OISF/suricata/pull/10087
Output from stdout file:
Notice: suricata: This is Suricata version 8.0.0-dev (5cc872fa1 2023-12-19) running in USER mode [LogVersion:suricata.c:1147]
Warning: detect: 1 rule files specified, but no rules were loaded! [SigLoadSignatures:detect-engine-loader.c:359]

This test is missing a pcap. Most of our existing DNS pcaps will have rcode values of 0 so are not that interesting. But ./tests/dns-eve-v2-udp-nxdomain-soa/dns-udp-nxdomain-soa.pcap does have a DNS response with a non-zero error code.

This test is missing a pcap. Most of our existing DNS pcaps will have rcode values of 0 so are not that interesting. But ./tests/dns-eve-v2-udp-nxdomain-soa/dns-udp-nxdomain-soa.pcap does have a DNS response with a non-zero error code.

You should use pcap: in test.yaml to tell which pcap you want to reuse

This test is missing a pcap. Most of our existing DNS pcaps will have rcode values of 0 so are not that interesting. But ./tests/dns-eve-v2-udp-nxdomain-soa/dns-udp-nxdomain-soa.pcap does have a DNS response with a non-zero error code.

On the other hand, maybe we could have two tests, and use one of the pcaps withrcode of zero to test the negated content option? 🤔

New PR: https://github.com/OISF/suricata-verify/pull/1567


Failed to create session

I am getting error saying "Failed to create session" when I start hunting after selecting the data files to hunt. Please find the error screenshot in the URL below
https://drive.google.com/file/d/0B7eJ4gOB47zqb0lJQXFfTzRXY00/view
I am using Windows 8.1 Pro, with 8GB of RAM

@nagarajhubli, is the CMD window still open when you attempt the search? Also, are you using the sample files provided, or other .FITS files?

Yes, the CMD window was still open and the application was running, and I was using the Test Images as provided here.. http://www.topcoder.com/asteroids/asteroiddatahunter/

I found the cause of the issue, after looking at the error in the CMD window, which said...
Caused by:
org.h2.jdbc.JdbcSQLException: The database is read only; SQL statement:|insert into detection_session (id, progress) values (null, ?) [90097-176]
I realised that I had to right click start.bat and run as administrator.

Thanks for the update @nagarajhubli!

I had the same issue and resolved it by running as administrator. I also have an issue during the installation, fixed it the same way.
On Windows, if you install the app in "Program files" directory (that's the most logical directory to install it), you will accounter these problems because this directory is not writable by a common user!
The installation guide should mention it. Or the data should be written in the user's directory (c:\users...).

Thanks @astrochoupe, I'm reopening this ticket until we resolve both of the issues mentioned above.


use kube-batch scheduling batch job

integration kube-batch gang scheduling, queue scheduling, priority scheduling strategy to scheduling batch job.

@ruanxingbaozi
Thanks for this big PR :), will take it a look later.
BTW, have you checked the HivedScheduler (HS) in the Recommended Kubernetes Scheduler Section? :
https://github.com/microsoft/frameworkcontroller#recommended-kubernetes-scheduler
It also supports gang (i.e. take the whole job as into one cell in HS), queue (i.e. cell's virtual cluster in HS) and priority (i.e. cell's priority in HS) scheduling.
Besides, it also supports GPU topology aware scheduling, very very few resource fragmentation and even if fragmentation will never hurt queue capacity.
FrameworkController(FC) can intergate with HS seamlessly, but FC does not really aware HS and have not made any code changes to adapt it.
So, why we have to make FC code changes to adapt kube-batch?
Why not take the same apporach as FC intergate with HS (i.e. a top layer service leverage both FC and HS, and when it submits a framework, it will generate the scheduling spec inside the framework spec)?
Please check FC and HS intergation examples in:
https://github.com/microsoft/frameworkcontroller/blob/master/example/framework/scenario/tensorflow/gpu/tensorflowdistributedtrainingwithhivedscheduledgpu.yaml
and
https://github.com/microsoft/pai/blob/c6c216473a631f22b066d1547ef0f709f4aaaddf/subprojects/GOPATH/src/github.com/microsoft/hivedscheduler/example/request/design/request.yaml#L132

"ignore": "",

Why add so many vendors? Did you use the Gopkg.toml in our project?

Refers to: vendor/vendor.json:3 in ec5ed29. [](commit_id = ec5ed2953d4df906dfe25cedd3f817cf7a109161, deletion_comment = False)

Thanks for your reply, The reason why kube-batch is used is because some extensions have been made on kube-batch, so I hope that FrameworkController will adapt kube-batch.In addition, the HivedScheduler project has done some abstract design, which is closer to some of Hadoop's design, and the migration of the strategy on kube-batch to HivedScheduler will do more work.

"ignore": "",

Why add so many vendors? Did you use the Gopkg.toml in our project?
Refers to: vendor/vendor.json:3 in ec5ed29. [](commit_id = ec5ed29, deletion_comment = False)

sorry,I am using govendor, dep is too slow, I will update it.

FrameworkController(FC) can intergate with HS seamlessly, but FC does not really aware HS and have not made any code changes to adapt it.
So, why we have to make FC code changes to adapt kube-batch?
Why not take the same apporach as FC intergate with HS (i.e. a higher level component leverage both FC and HS, and when it submits a framework, it will auto generate the scheduling spec inside the framework spec)?

FrameworkController(FC) can intergate with HS seamlessly, but FC does not really aware HS and have not made any code changes to adapt it.
So, why we have to make FC code changes to adapt kube-batch?
Why not take the same apporach as FC intergate with HS (i.e. a higher level component leverage both FC and HS, and when it submits a framework, it will auto generate the scheduling spec inside the framework spec)?

Ok, I agree to do these things at a higher level component .I will try to use HS

I just propose the intergation approach with kube-batch, and it is fine to still use kube-batch if you need its dedicated features. Check intergation examples in my first comment.

I just propose the intergation approach with kube-batch, and it is fine to still use kube-batch if you need its dedicated features. Check intergation examples in my first comment.

thanks，I understand what you mean by adapting kube-batch in the rest-server instead of in the frameworkcontroller.

yeap :) The rest server may not have to be the PAI rest server, it can be any your own service or client.


Add cache-size opts

Depends on https://github.com/mafintosh/append-tree/pull/10 and https://github.com/mafintosh/hypercore/pull/124
Adds {metaStorageCacheSize: 65536, contentStorageCacheSize: 65536, treeCacheSize: 65536} opts to the constructor

9.12.0


Cleared UIElement child objects, but still its holding in performance profiler memory

Describe the bug
Cleared UIElement child objects while closing the window but still it holds in performance profiler memory.
In my sample i added 42 CustomCell in row and column manner, when the window is closed i cleared all the child CustomCell from the stack panel, but when i check in performance profiler, CustomCell objects still holds in memory.
Steps to reproduce the bug

Run the sample and click "cells page".
Close the window by clicking "Go to home page" button.
Open performance profiler, follow step 1 and step 2.
Check the memory, it holds CustomCell 42 objetcs.

Expected behavior
All the objects should be cleared.
Screenshots

Sample:
WinUIDesktopCellsDemo.zip
Version Info
WinUI NuGet package version = 3.0.0-preview4.210210.4
OS version = Windows 10 Pro
NuGet package version:
[Microsoft.WinUI 3.0.0-preview4.210210.4]

Windows app type:

UWP
Win32

No
Yes

Windows 10 version
Saw the problem?

Insider Build (xxxxx)

October 2020 Update (19042)
Yes

May 2020 Update (19041)

November 2019 Update (18363)

May 2019 Update (18362)

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Xbox

Surface Hub

IoT

Additional context

Is it just the case that garbage collection hasn't run yet? You can force garbage collection to happen by calling GC.Collect. https://docs.microsoft.com/en-us/dotnet/api/system.gc.collect?view=net-5.0  If you do that do you still see your objects being kept alive?

Hi @StephenLPeters, i have checked by calling GC.Collect, still objects are not cleared.

@Ganeshamoorthy10 probably the same root cause as https://github.com/microsoft/microsoft-ui-xaml/issues/4610?
@Scottj1s FYI

Closing as dupe, if you think its not, let me know :)


conda-lock 1.4.0, remove gitpython, update docs

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Notes:

removes gitpython (has CVEs)
update docs url
clean up jinja vars
add maintainer :wave:

@conda-forge-admin please rerender

@conda-forge-admin please rerender

@conda-forge/conda-lock ready for review :lock: :snake:


Taming polyfills & web components across major browsers

Folks,
We have put a tremendous effort into introducing Web Components as a new standard for content objects in the portal.  As expected, there has been a bit of a "learning curve" in this change.
But the time has come where we need to get a firm handle on what we doing.  We need to be able to deliver working components -- reliably -- across supported browsers;  and we need, furthermore, a component we deliver today still to function properly & look attractive 6, 9, 12 months down the road.  (Spoiler:  we're not there yet.)
Screenshots
Here are some screen captures of the current header content in uPortal-start/data/quickstart across 4 major browsers.  None of them is 100% functional.
Chrome

Firefox

Edge

IE 11

A Good Plan, and Consensus
The most recent plan I am aware of is that (1) uPortal will include polyfills sufficient to take any browser to ES6 support, and (2) uPortal Ecosystem Web Components must "take it from there" if they need further polyfills, CSS applied, etc.
@ChristianMurphy @jgribonvald @cousquer @bjagg @jonathanmtran @leighgordy
Is this a good plan?
Can someone explain what I'm seeing in these screenshots?  (I can make some time to pitch in with fixing.)

Sorry for the late reply, I don't get notified from GitHub anymore, for some odd reason!
I know I have been highly skeptical of web components in the past but with Firefox supporting them and IE currently developing support, I am no longer a skeptic and I can see the value in your approach.
In regards to the plan. We are not developing web components locally as yet but our approach is exactly the same as yours for our AMD modules. We have to transpile to ES5 and use pollyfills to support legacy browsers. You should not think that complexity as a cost of web components, its just how JavaScript projects are built these days.
We have not built any React web components as yet because React events and web components don't mix very well. React Fire promises to fix event handling which means React should hopefully work properly web components in a few months.
I think your plan is a good one!

Heya Drew,
Have a look at https://github.com/cousquer/CardWebComponents , See it in
action: https://cousquer.github.io/CardWebComponents/
For the moment it's a Proof of concept, there is still an issue, Many
Thanks to Christian Murphy for the insights.😉
But This Web Component is i18n in en-US, es-ES, fr-FR, nl-NL, meets AA
level in WCAG 2.1 (yes the new one!)...
would be happy if in the future to push it in uPortal-contrib
-Christian
Le ven. 2 nov. 2018 à 19:39, Leigh Gordon<EMAIL_ADDRESS>a
écrit :

Sorry for the late reply, I don't get notified from GitHub anymore, for
some odd reason!
I know I have been highly skeptical of web components in the past but with
Firefox supporting them and IE currently developing support, I am no longer
a skeptic and I can see the value in your approach.
In regards to the plan. We are not developing web components locally as
yet but our approach is exactly the same as yours for our AMD modules. We
have to transpile to ES5 and use pollyfills to support legacy browsers. You
should not think that complexity as a cost of web components, its just how
JavaScript projects are built these days.
We have not built any React web components as yet because React events and
web components don't mix very well. React Fire
https://github.com/facebook/react/issues/13525 promises to fix event
handling which means React should hopefully work properly in web components
in a few months.
I think your plan is a good one!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Jasig/uPortal/issues/1381#issuecomment-435452557, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABaa1uosoK4isESMy79FowVUVdBhkwZWks5urJFxgaJpZM4X601y
.

These changes have been applied in uPortal 5.3.0 and continued to be enhanced in continued releases.


New / Edit form date fields should use date picker not selects

It's way faster and easier to pick a date out of a date picker then using multiple select menus

+1


Update Posts “query-hierarchal-data-using-tsql”

Automatically generated by Netlify CMS

👷 Deploy Preview for monetaboot processing.

Name
Link

🔨 Latest commit
4ba80ff40515f6e675830401d13e7e9da11f3e84

🔍 Latest deploy log
https://app.netlify.com/sites/monetaboot/deploys/630000f02918ba00094c4132


OpenBLAS on dSPACE realtime hardware

Hello,
I am currently trying to update some code to use OpenBLAS and implement it on a dSPACE 1103 PowerPC board but I am having some issues. The build steps are a little confusing and I will try my best to explain. There are 3 devices in this setup

A laptop where the OpenBLAS is compiled
-Haswell i7-4700HQ
-Ubuntu 18.04 WSL 64 bit

The host for the dSPACE system
-Sandy Bridge-E Xeon E5-1620
-Windows 7 64 bit
-MATLAB 32 bit

Real-time board
-dSPACE 1103
-Power PC 750 GX
-Receives final compiled code from the host

I first compile OpenBLAS on the laptop using
make DYNAMIC_ARCH=1 BINARY=32 HOSTCC=gcc CC=i686-w64-mingw32-gcc FC=i686-w64-mingw32-gfortran CFLAGS='-static-libgcc -static-libstdc++ -static -ggdb' FFLAGS='-static' && mv -f libopenblas.dll.a libopenblas.lib
I then copy over the lib/dll.a/include files to the host PC. In my existing code MYCODE.c I include cblas.h and update my functions to use cblas. I then use MATLAB to compile a mex file by using the following command
mex -v MYCODE.c  libopenblas.lib -g  -I'openblas/include' -lmwlapack
This compiles successfully and I am able to run my mex file and obtain correct results.
Now to upload the code to the real-time board I first create a model in Simulink that uses the mex file and call that using rtwbuild('MYMODEL') which will take my C files and compile them using the PPCTools37 compiler for the real-time board PPC architecture. I then receive the error

COMPILING  "..\MYCODE.c"
(F) C0005; "C:\Users\XXXX\Desktop\openblasproject\openblas\include\common.h", line 87 pos 20; could not open source file "unistd.h"

Any idea why I would be receiving this error?
Thank You

This file is included from common.h when _MSC_VER is not defined (that is, when the compiler is not recognized as MSVC, some kind of Unix-like environment like mingw/msys is assumed). You can try to #define _MSC_VER in your code before it includes any of the OpenBLAS headers, or find out how the PPCTools37 compiler introduces itself in the default preprocessor defines that it adds.

Matlab includes MKL BLAS. You do not need OpenBLAS at that point.
Why would yourcode.c incluse internal OpenBLAS header at all?

Answering @martin-frbg

This file is included from common.h when _MSC_VER is not defined (that is, when the compiler is not recognized as MSVC, some kind of Unix-like environment like mingw/msys is assumed). You can try to #define _MSC_VER in your code before it includes any of the OpenBLAS headers, or find out how the PPCTools37 compiler introduces itself in the default preprocessor defines that it adds.

When using #define _MSC_VER I now get the following when compiling the code for PPC.

(F) C0005; "C:\Users\XXXX\Desktop\openblasproject\openblas\include\common.h", line 117 pos 21; could not open source file "windows.h"

I did some more searching and the full compiler name is "Microtec PowerPC C/C++ Compiler 3.7"
The following compiler mirrors are defined

_MRI Defined as 1.
_MICROTEC Defined as 1.
_TARGET_PPC Defined as 1 if the compiler is being used for a PPC target;
otherwise undefined.
_VERSION The version of the compiler in literal string format.

For @brada4's question

Matlab includes MKL BLAS. You do not need OpenBLAS at that point.
Why would yourcode.c incluse internal OpenBLAS header at all?

From what I know MKL is really only optimized for x86 systems and I did not get much if any performance benefit between reference BLAS and MKL on this PPC system. I ma trying to push the limits of this system and I am hoping OpenBLAS will help me do so.

You build (cross-build linux to win32) on your laptop for use with 5 years old last 32bit matlab release - what does it have to do with some BSP compiler failing in a completely independent build?
Can you get errors and show command lines from that BSP compiler? That build has absolutely nothing to do with matlab or mingw gcc you used before.

It does have something to do with the previous build. The mex file is used to construct a simulink model that the PPC compiler uses in combination with the original source code to create the final compiled software that is sent to the real-time system. This is standard for the system, the only thing I am doing differently than normal is trying to incorporate OpenBLAS. The 5 year old 32 bit matlab release is required (I would love to upgrade if I could).
I have included the output and the makefile that is called when running rtwbuild('MYMODEL')
output.txt
MYMODEL.mk.txt

I see no openblas built in your log nor the error you encounter....

I see no openblas built in your log nor the error you encounter....

Line 73 and 77 in output.txt?

Does it compile when you simply change the #if !defined(_MSC_VER) to #if 0 in line 86 of common.h (and remove the #define _MSC_VER again, as it seems to cause other problems) ? If that does the trick, we could probably change that line to something like #if !defined(_MSC_VER) && !defined(_TARGET_PPC)

USER_LIBS = "C:\Users\XXXX\Desktop\openblasproject\libopenblas.lib" "C:\Program Files (x86)\MATLAB\R2013a\extern\lib\win32\microsoft\libmwlapack
I dont think ppc blas ever link with win32 lapack.
C0005 code is from visual studio compiler, nothing to do with BSP compiler.

@martin-frbg Setting #if 0 ends up causing the same error as defining _MSC_VER

(F) C0005; "C:\Users\Bernstein-Lab\Desktop\rcacspeedup\include\common.h", line 117 pos 21; could not open source file "windows.h"
#include <windows.h>

@brada4 This is an error message from the PPC compiler

C0005 could not open source file “filename”

Also linking with the MATLAB libmwlapack and libmwblas works fine on their own (this is the built in MATLAB MKL). I am going step by step when implementing OpenBLAS by first replacing the BLAS before trying LAPACK.
The error messages I am receiving are before anything from libmwlapack is even loaded.

Could you try cross-building PPC component outside matlab? One single line error for unknown command is not really helpful.
Are you certain microtec compiler emulates MSVC error messages? Is it documented anywhere in the world?

I see now that the #include <windows.h> in line 117 is independent of _MSC_VER, it depends on #ifdef OS_WINDOWS. Probably best to add an #ifndef _TARGET_PPC ... #endif around that line
(or simply comment it out for now) and see where that gets you. (Cannot negate the entire ifdef OS_WINDOWS as its #else branch would again try to include unistd.h)

@martin-frbg
Doing that in combination with setting if !defined(_MSC_VER) to #if 0 I now receive the following
error-5-1.txt
@brada4
I can try but it might take time, it is not documented (and I am not sure if possible) to compile outside of matlab.
I am certain about the error, page 262 of the microtec compiler manual
ccc_ppc_ur_37.pdf

It should be possible to compile using common make , e.g
make CC=ppc-cross.exe HOSTCC=cl.exe TARGET=PPC440 FC=NONE

The "new" errors from common.h appear to be over use of the phrase __inline void somefunction rather than void __inline somefunction (and same with "int"). What is worrying is that it still appears to compile some x86 code (else it should not need to include common_x86.h, and the errors from param.h appear to be in settings for Intel CPUs as well). Wouldn't it need to create PPC code (and access an OpenBLAS compiled for PPC440 target to run on the dSPACE board rather than the x86 cpu of its Windows host ?

The errors say you are building x86 assembly code for supposedly powerpc with a powerpc compiler.
Maybe you start taking things seriously and at least get a compiler matching your target?

@brada4 mind your words please...

Maybe I am confused about what you are trying to compile here with the PPC compiler - is it OpenBLAS itself, or is it your own code ? (I expect you will need to compile both, as you will probably need a PPC version of the library if you want the BLAS calls to be executed on the PPC440 board - the mingw compile on your laptop created the windows-style library for the Xeon cpu only). In your own code, you would probably include openblas_config.h and cblas.h, but not OpenBLAS' internal
headers like common.h

I am not sure if I understand the mex workflow - where you currently have "Reference BLAS", is this the source code or a binary ? (I expect this would have to be either sourcecode or a precompiled ppc binary if you want the BLAS calls to be executed on the dSPACE board)

I am not sure if I understand the mex workflow - where you currently have "Reference BLAS", is this the source code or a binary ? (I expect this would have to be either sourcecode or a precompiled ppc binary if you want the BLAS calls to be executed on the dSPACE board)

It is the source

OK, then you would probably start from the OpenBLAS source as well, and build it for TARGET=PPC440 rather than relying on the autodetection that would only see the Xeon and Windows. But whatever you do you should not include common.h in your own MYCODE.c

In principle at the point OpenBLAS is mixed in in the picture it could be blob produced outside mex just like the "other code" in same static library format. Almost same as with win32 dll you produced earlier.
So mymodel.c would be same as mycode.c (with added matlab function wrappers) then compiled into native PPC executable.

Ok, so I am now trying to make a static library for the powerpc portion of the compilation. Would this be the correct command?
 make TARGET=PPC440 HOSTCC=gcc CC=powerpc-linux-gnu-gcc FC=powerpc-linux-gnu-gfortran CFLAGS='-static-libgcc -static-libstdc++ -static -ggdb' FFLAGS='-static'
I receive the following output. Compilation fails at line 1784 complaining about junk at the end of the line
failedcompile.txt

CFLAGS should be replaced with CCOMMON_OPT and FFLAGS alike, the "standard" names are changed internally e.g. different set for building LAPACK etc.
I think you need to use your dspace compiler (gcc will work if target is really kind of Linux)
OpenBLAS does not use libstdc++ , and it actually builds static library, see Makefile.rule for all possible options (technically "ar" archive on Linux, but check with "file" command so it is same as dspace static components)

The "junk at end of line" messages are from the assembler apparently not understanding register names like r14 (or more likely, misinterpreting the DCBT macro that is defined (as a L1_PREFETCH) in common_power.h. Not sure if this is due to "wrong" compiler (or rather, assembler) or some other problem.

Probably it hits C comments that normally are stripped out by C compiler, but maybe not so everywhere in long dormant codes...
like:
xor r1,r1 // we did something

@mohseninima could you try building with make -i and all your parameters (after make clean)
It will run build through showing all errors, no working library produced, just that it will not stop on first encountered error.

@brada4 no, it is stumbling over instructions like DCBT(A01,PREA) (where "A01" is #defined to be r14 , "PREA" is r24) that expand to dcbt r24, r14 according to common_power.h. No comments anywhere near these. As far as I can tell, PPC440 support in OpenBLAS is inherited from the original GotoBLAS.

I'm hitting this issue trying to build with target PPC970. After looking at this carefully, and running the affected files through the preprocessor (ex: cpp -I src/OpenBLAS-0.3.7 src/OpenBLAS-0.3.7/kernel/power/gemm_kernel_altivec.S), you can see that the DCBT macro is interpreted as something like: dcbt 8, 29, 7
The IBM Documentation claims:

The dcbt instruction serves as both a basic and extended mnemonic. The dcbt mnemonic with three operands is the basic form, and the dcbt with two operands is the extended form. In the extended form, the TH field is omitted and assumed to be 0b0000.

The example assembly code on that page goes on to show them using the extended mnemonic, but does not make use of the basic mnemonic. My assembler (GNU assembler version 2.32 (powerpc64-foxkit-linux-musl) using BFD version (GNU Binutils) 2.32) does not seem to like the "basic mnemonic" syntax with 3 arguments to the dcbt instruction. When I attempt to build OpenBLAS, I get failures in the same files as @mohseninima did above. For instance:
../kernel/power/gemm_kernel_altivec.S: Assembler messages:
../kernel/power/gemm_kernel_altivec.S:348: Error: junk at end of line: `7'
../kernel/power/gemm_kernel_altivec.S:355: Error: junk at end of line: `8'
../kernel/power/gemm_kernel_altivec.S:409: Error: junk at end of line: `7'
make[1]: *** [Makefile.L3:671: strmm_kernel_LN.o] Error 1

I'm not sure which assembler this code was written for or tested on, but it appears that GNU assembler is not one of them.
Additionally, I question the macro definition of DCBT() in this code, which is the chosen form for my hardware based on the defines in the immediately preceding code (I'm building on Linux, not Darwin or FreeBSD). Regardless, it appears that the DCBT() macro reverses the order of the arguments passed to dcbt, which seems to counter the IBM documentation linked above. I'm not sure if it's intentional or not, but it certainly doesn't look right.
I have hardware that I can readily test fixes on. Any help that can be provided in fixing this issue would be greatly appreciated! For reference, my host machine is an IBM Power9, which should be backwards compatible with the PPC970.

I actually have an update! I found this bug in sourceware's tracker. They mention that the 3 argument form of the dcbt instruction is only supported on POWER4 and newer, so you need to pass the -mpower4 flag during assembly.
I've applied some changes to my source tree which seems to cause the package to build successfully at least, but now tests are failing:
./openblas_utest
TEST 1/23 amax:samax [OK]
TEST 2/23 drotmg:drotmg_D1_big_D2_big_flag_zero [OK]
TEST 3/23 drotmg:rotmg_D1eqD2_X1eqX2 [OK]
TEST 4/23 drotmg:rotmg_issue1452 [OK]
TEST 5/23 drotmg:rotmg [OK]
TEST 6/23 axpy:caxpy_inc_0 [OK]
TEST 7/23 axpy:saxpy_inc_0 [OK]
TEST 8/23 axpy:zaxpy_inc_0 [OK]
TEST 9/23 axpy:daxpy_inc_0 [OK]
TEST 10/23 zdotu:zdotu_offset_1 [OK]
TEST 11/23 zdotu:zdotu_n_1 [OK]
TEST 12/23 dsdot:dsdot_n_1 [OK]
TEST 13/23 swap:cswap_inc_0 [OK]
TEST 14/23 swap:sswap_inc_0 [OK]
TEST 15/23 swap:zswap_inc_0 [OK]
TEST 16/23 swap:dswap_inc_0 [OK]
TEST 17/23 rot:csrot_inc_0 [FAIL]
  ERR: test_rot.c:109  expected -2.148e-01, got 3.125e-01 (diff -5.273e-01, tol 1.000e-04)
TEST 18/23 rot:srot_inc_0 [OK]
TEST 19/23 rot:zdrot_inc_0 [FAIL]
  ERR: test_rot.c:71  expected -2.148e-01, got 3.125e-01 (diff -5.273e-01, tol 1.000e-13)
TEST 20/23 rot:drot_inc_0 [OK]
TEST 21/23 potrf:smoketest_trivial [OK]
TEST 22/23 potrf:bug_695 [OK]
TEST 23/23 kernel_regress:skx_avx [OK]
RESULTS: 23 tests (21 ok, 2 failed, 0 skipped) ran in 24 ms

Here is a complete build log: buildlog.txt
I haven't traced csrot or zdrot yet, but I'm still wondering if the argument order of dcbt is an issue; I'll be testing that soon, and attempting to root cause the failure of these tests, but I think I'm nearing the point of needing an expert.

I've determined that the test failure is likely related to issue #1469. I believe that the implementation that is failing on PPC970 is contained in zrot.S (as stated by kernel/Makefile.L1) as the kernel/power/KERNEL.PPC970 file defines no CROTKERNEL or ZROTKERNEL.
I've patched KERNEL.PPC970 with some modifications that are similar to this commit following the rationale from the previously mentioned issue, and it seems to build and pass tests.
It would be fantastic to have the Assembly intrinsics for the older PowerPC chips; considering that the hardware is less powerful than modern chips, getting more performance out of the software would be a nice benefit.
I should probably add that our Power9 build host runs in Big Endian mode. Please let me know how you're going to proceed with fixing the issue with the PPC970 target. As I said before, I'm happy to help out in any way that I can! :)

Seems you are already on the right track with your changes. There is no telling if/how/where the DCBT macro ever worked on PPC970 - git blame confirms that the original choice was inherited from GotoBLAS2 (for which next to no version history was available) and the list of exceptions only got added to whenever somebody hit this problem with a particular operating system.
(As an aside, the current POWER8/POWER9 kernels are Little Endian-only)

@martin-frbg Interestingly enough, the POWER8 kernel seems to build and pass all tests on Big Endian. The utest log is here.
However, the POWER9 kernel does not pass the tests. If you'd like, I could open another issue for the POWER9 test failures on Big Endian. Despite IBM's messaging, there are distributions out there that use and actively develop Big Endian POWER9.
Strangely enough, I seem to run into an intermittent test failure on some builds due to a SIGILL.
Here's a link to the relevant part of the test log. I'll keep an eye on it and try and run a debugger the next time I see the test fail.

If you could re-run failing sample in gdb and disassemble the failing instruction....
Power omits some PowerPC (aka G5) instructions, that are emulated in AIX, but not Linux.

For completeness, here is the full patch that I'm currently using to build and test the PPC970 target.
I seem to have gotten another build where sblat3 failed with SIGILL on the initial test run. Infuriatingly enough, it doesn't seem to be doing it again when the same test is rerun. I've run it over and over again in GDB hoping to catch the issue, but the process is continuously exiting successfully. GDB's environment has OPENBLAS_NUM_THREADS=2 set, and I'm getting output like this:
(gdb) !rm SBLAT3.SUMM
(gdb) run
Starting program: /home/djt/Adelie/Packages/user/openblas/src/OpenBLAS-0.3.7/test/sblat3 < ./sblat3.dat
[New LWP 9814]
[LWP 9814 exited]
[Inferior 1 (process 9813) exited normally]

Here's the contents of SBLAT3.SUMM. It seems strange to me that it is reporting a FATAL ERROR and not exiting with an error code. Is this an expected failure?
I'm not familiar with Fortran enough to understand where or why it could be failing. I assume that Fortran is calling back to PowerPC ASM/C implementations of these methods. I'm also not very experienced at using GDB beyond running programs and setting breakpoints in binaries that have debug symbols. Any help in further debugging this issue would be much appreciated!

This is assembly instruction,not definitely completely illegal like unsupported by cpu, maybe getting unaligned arguments to otherwise legitimate instruction, it does not matter which compiler generated it, it is wronfly permited by -m(arch) flags. Probably test needs to run few more times to crash if it did do once.

I think for POWER9 you would need to try the current develop branch, with 0.3.7 you will be missing at least the fix for TRMM. And maybe I was overly pessimistic about big-endian support - there is the open issue #1997 but the primary obstacle there seemed to be limitations of the AIX assembler.

I seem to have caught the issue in gdb! Logs are here
I'm pretty sure that's what you're looking for, but I've got gdb still open just in case. I realized that I don't actually have debug symbols enabled... I'm going to rebuild another copy with them enabled and see if I can get it to croak in the meantime.
Also of note, considering we're using musl libc here:
<@awilfox> the SIGILL is a red herring
<@awilfox> that's musl's way of forcing a crash quickly when __stack_chk_fail is hit
<@awilfox> __stack_chk_fail purposefully has an invalid opcode to force the program to crash *now* to prevent a buffer attack
<@awilfox> what is actually happening is that the stack canary is being overwritten

ssyr2k_kernel_U is common code (built from driver/level3/syr2k_kernel.c) so if something is trashing the stack it is probably the sgemm kernel.

Try to build with MAX_STACK_ALLOC=0
It could happen that stack allocations fill some values while something else later expects zeroes, and gets confused.

So I've tried a few different things:

Try to build with MAX_STACK_ALLOC=0
It could happen that stack allocations fill some values while something else later expects zeroes, and gets confused.

This didn't have any effect. It still seems to crash in the same place.
musl libc's default stack size is much smaller (128k by default) than glibc. I tried building with -Wl,--stack,4194304 in the linker flags and it seemed to fail at the same place. Regardless, it appears that the OpenBLAS server thread calls pthread_attr_setstacksize with a smaller value. I'm not exactly sure if this alters my linker flag stack size setting in the child threads (where the code actually seems to be failing). This also seems to still fail at around the same place.
Lastly, I disabled all compiler optimizations by removing all optimization flags and adding -O0. Now I'm experiencing a "halting problem". Now it appears that both the sblat3 and the cblat3 tests never exit, and I can't tell if they ever will. I let cblat3 run overnight in gdb, and it never halted or returned.
The un-optimized code appears to be spending most of it's time around here in the callstack:
^C
Thread 1 "sblat3" received signal SIGINT, Interrupt.
0x0000000100034518 in ssyr2k_kernel_U (m=31, n=31, k=1, alpha_r=1, a=0x3ffff5d17a80, b=0x3ffff5d70680, c=0x3ffffffee92c, ldc=32, offset=0, flag=1) at syr2k_kernel.c:179
179                 subbuffer[i + j * nn] + subbuffer[j + i * nn];
(gdb) bt
#0  0x0000000100034518 in ssyr2k_kernel_U (m=31, n=31, k=1, alpha_r=1, a=0x3ffff5d17a80, b=0x3ffff5d70680, c=0x3ffffffee92c, ldc=32, offset=0, flag=1) at syr2k_kernel.c:179
#1  0x00000001000303e4 in ssyr2k_UN (args=0x3ffffffd8e00, range_m=0x0, range_n=0x3ffffffd6130, sa=0x3ffff5d17a80, sb=0x3ffff5d70680, dummy=0) at level3_syr2k.c:205
#2  0x000000010004bf24 in exec_blas (num=1, queue=0x3ffffffd6338) at blas_server.c:826
#3  0x0000000100049734 in syrk_thread (mode=256, arg=0x3ffffffd8e00, range_m=0x0, range_n=0x0, function=0x100030038 <ssyr2k_UN>, sa=0x3ffff5d17a80, sb=0x3ffff5d70680, nthreads=2) at syrk_thread.c:187
#4  0x0000000100019adc in ssyr2k_ (UPLO=0x3ffffffd9518 "U", TRANS=0x3ffffffd9508 "N擄T\327\f\277N", N=0x3ffffffd91ac, K=0x3ffffffd9188, alpha=0x3ffffffd9170, a=0x3ffffffd9f18, ldA=0x3ffffffd9198, b=0x3ffffffe2320, ldB=0x3ffffffd919c, beta=0x3ffffffd9174, c=0x3ffffffee92c, ldC=0x3ffffffd91a0) at syr2k.c:382
#5  0x00000001000037b8 in schk5_ ()
#6  0x0000000100017344 in MAIN__ ()
#7  0x00000001000178e4 in main ()

I found that I can get a full annotated dump of the assembly for these binaries with a command like objdump -lSd cblat3, here's the output from that if it helps.
I'm not exactly sure what my next steps should be.

In first backtrace after 0x0000000100028e94 starts fancy ret with stack check , what is not clear to me - were calls leading there also checking stack or that heavy return happens only when you "exit library"
Call profile is same as x86_64.

Building with TARGET=PPC970 on POWER9 (ppc64le, gcc-9.2.1) does not give me any crashes,
but both SGEMM&friends and the corresponding single-precision complex functions all fail their tests.
(No complaints about the DCBT macro there, but I had to apply the crot/zrot fix)

Changing the SGEMM and CGEMM kernels in KERNEL.PPC970 to the non-altivec versions as used for DGEMM and ZGEMM (with the corresponding blanking of the INCOPY/ITCOPY and -OBJ entries, and changing of the GEMM_UNROLL_M to 4 and 2 respectively) fixed "my" problem, and will probably work for you as well.

(No complaints about the DCBT macro there, but I had to apply the crot/zrot fix)

As noted earlier and in the binutils bug, that macro issue won't happen on LE because binutils as(1) defaults to POWER8 ISA on LE (which is probably wrong and may stop being the default moving forward, since the OpenPOWER FPGAs don't implement ISA 2.07 but are LE).  On BE, it defaults to something much lower (possibly POWER2?) which doesn't support the three-operand dcbt and needs -mpower4 (or later) passed.  Just FYI, that is why that is happening.

@awilfox thanks for the reminder, I had indeed lost track of the context of the dcbt problem. Unfortunately even the old GotoBLAS snapshots do not provide much insight into this choice of macro - GotoBLAS-1.00 from 2006 already had PPC970 support, and DCBT at that time was defined (unconditionally) as
#define DCBT(REGA, REGB, NUM) .long (0x7c00022c | (REGA << 16) | (REGB << 11) | ((NUM) << 21)) (which is still preserved as a comment in the current common_power.h).  This was only used once, #ifdef linux, in the zgemv_t kernel (with the default action a two-operand dcbt).
The DCBT_ARG was introduced with (or before) 1.07, with the "0" value reserved for POWER3 and POWER5 (which were both already supported targets in 1.00). 1.07 had something like a changelog, and a comment there states Some early POWER5 won't accept DCBT extended format. (And the FAQ at that time had an entry for POWER5 owners that they "may modify it" back to the generic DCBT_ARG 8). By then DCBT was already used in axpy,gemv,gemm,trmm and trsm with no distinction on operating system.
Perhaps this was just an ugly workaround for a poorly understood problem all along, and the two-operand form should have been used universally from the start ?

This is what I have come up with so far (excluding the DCBT bug - do not see it on big-endian opensuse so probably recent binutils have changed their baseline to power4. Still probably makes sense to drop the "Darwin or BSD" requirement on the conditional for two-operand dcbt).
ppc970be.diff.txt


pull request for e2e test at 12/29/2016 22:56:07

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


CSS in ds300.github.io is outdated

The CSS file in the bookmarklet is outdated, as it points to a version that still did not have the fixes for Safari.

thanks for the heads-up.


FIX: Do not align origins, align centers of mass

Codecov Report

Merging #253 (e3b8358) into master (c802b4d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #253   +/-   ##
=======================================
  Coverage   83.16%   83.16%
=======================================
  Files          25       25
  Lines        1847     1847
  Branches      212      278   +66
=======================================
  Hits         1536     1536
  Misses        280      280
  Partials       31       31

Flag
Coverage Δ

travis
83.16% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2f01e4...e3b8358. Read the comment docs.