_state is not removed after deleting river

Elasticsearch 1.5.1 & elasticsearch-river-jdbc-<IP_ADDRESS>
I run a one-time river then delete it but the _state persists. Here are all the river logs I captured from each node in my cluster.
.....
[2015-04-17 22:55:02,341][DEBUG][river.jdbc.BulkNodeClient] after bulk [87] [succeeded=129554] [failed=0] [534ms] 0 concurrent requests
[2015-04-17 22:55:06,868][DEBUG][river.jdbc.BulkNodeClient] before bulk [89] [actions=1839] [bytes=10489555] [concurrent requests=2]
[2015-04-17 22:55:07,050][DEBUG][river.jdbc.BulkNodeClient] before bulk [90] [actions=54] [bytes=311707] [concurrent requests=2]
[2015-04-17 22:55:07,094][DEBUG][river.jdbc.BulkNodeClient] after bulk [90] [succeeded=129608] [failed=0] [43ms] 1 concurrent requests
[2015-04-17 22:55:08,394][DEBUG][river.jdbc.BulkNodeClient] after bulk [89] [succeeded=131447] [failed=0] [1525ms] 0 concurrent requests
[2015-04-17 22:55:08,529][DEBUG][river.jdbc.SimpleRiverSource] merged 132513 rows
[2015-04-17 22:55:08,549][DEBUG][river.jdbc.SimpleRiverFlow] after fetch: state posted = {"name":"sales-river-one-time","type":"jdbc","started":"2015-04-17T22:47:58.941Z","last_active_begin":"2015-04-17T22:47:59.004Z","current_active_begin":"2015-04-17T22:47:59.004Z","last_active_end":"2015-04-17T22:55:08.529Z","map":{"lastStartDate":1429310879024,"lastExecutionEndDate":1429311308529,"counter":1,"lastExecutionStartDate":1429310879060}}
[2015-04-17 22:55:08,549][DEBUG][river.jdbc.BulkNodeClient] flushing bulk processor
[2015-04-17 22:55:08,551][DEBUG][river.jdbc.BulkNodeClient] before bulk [91] [actions=589] [bytes=3391746] [concurrent requests=1]
[2015-04-17 22:55:08,710][DEBUG][river.jdbc.BulkNodeClient] after bulk [91] [succeeded=132036] [failed=0] [159ms] 0 concurrent requests
[2015-04-17 22:55:11,947][DEBUG][river.jdbc.BulkNodeClient] closing bulk processor...
[2015-04-17 22:55:11,947][DEBUG][river.jdbc.BulkNodeClient] shutting down...
[2015-04-17 22:55:11,947][DEBUG][river.jdbc.BulkNodeClient] shutting down completed
[2015-04-17 22:55:11,974][DEBUG][river.jdbc.SimpleRiverSource] read connection closed
[2015-04-17 22:55:11,974][DEBUG][river.jdbc.SimpleRiverSource] write connection closed
[2015-04-17 22:55:11,980][INFO ][river.jdbc.RiverMetrics  ] pipeline org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@4bc2fdc5 complete: river jdbc/sales-river-one-time metrics: 132513 rows,<PHONE_NUMBER>3778723 mean,<PHONE_NUMBER>953475<PHONE_NUMBER>743041<PHONE_NUMBER>7826822), ingest metrics: elapsed 7 minutes 12 seconds, 710.42 MB bytes, 5.49 KB avg, 1.641 MB/s
[2015-04-17 22:55:11,980][DEBUG][river.jdbc.RiverPipeline ] releasing river context
[2015-04-17 22:55:11,981][DEBUG][river.jdbc.BulkNodeClient] flushing bulk processor
[2015-04-17 22:55:11,981][DEBUG][river.jdbc.BulkNodeClient] no current ingests
[2015-04-17 22:55:11,981][DEBUG][river.jdbc.SimpleRiverSource] read connection closed
[2015-04-17 22:55:11,981][DEBUG][river.jdbc.SimpleRiverSource] write connection closed
[2015-04-17 22:55:11,982][DEBUG][river.jdbc.RiverThread   ] river flow org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow@131f0575 thread is finished
[2015-04-17 22:58:54,897][DEBUG][river.cluster            ] [Madelyne Pryor] processing [received_state]: execute
[2015-04-17 22:58:54,897][DEBUG][river.cluster            ] [Madelyne Pryor] cluster state updated, version [2], source [received_state]
[2015-04-17 22:58:54,898][DEBUG][river                    ] [Madelyne Pryor] closing river [jdbc][sales-river-one-time]
[2015-04-17 22:58:54,898][DEBUG][river.jdbc.JDBCRiver     ] shutting down river thread scheduler
[2015-04-17 22:58:54,898][DEBUG][river.jdbc.JDBCRiver     ] interrupting river thread
[2015-04-17 22:58:54,898][DEBUG][river.jdbc.JDBCRiver     ] shutting down river flow
[2015-04-17 22:58:54,898][INFO ][river.jdbc.JDBCRiver     ] river closed [jdbc/sales-river-one-time]
.....
[2015-04-17 22:44:12,039][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:44:12,039][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:44:12,057][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:44:12,057][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:44:26,905][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:44:26,905][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:47:57,825][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:47:57,842][DEBUG][river.routing            ] [Kilmer] no river _meta document found retrying in 1000 ms
[2015-04-17 22:47:57,842][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:47:58,843][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:47:58,851][DEBUG][river.routing            ] [Kilmer] _river/sales-river-one-time/_meta document found.
[2015-04-17 22:47:58,854][DEBUG][river.routing            ] [Kilmer] going to allocate river [sales-river-one-time] on node [Madelyne Pryor][233VLIPbRem_zHJEMLqz9A][ip-192-168-211-199][inet[/<IP_ADDRESS>:9300]]{data=false, master=false}
[2015-04-17 22:47:58,854][DEBUG][river.cluster            ] [Kilmer] cluster state updated, version [1], source [reroute_rivers_node_changed]
[2015-04-17 22:47:58,856][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: done applying updated cluster_state
[2015-04-17 22:47:58,956][DEBUG][river.jdbc.RiverStateService] put: first river state [jdbc/sales-river-one-time]
[2015-04-17 22:47:58,979][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:47:58,979][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:47:59,002][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:47:59,002][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:47:59,014][DEBUG][river.jdbc.RiverStateService] post: update river state = {"name":"sales-river-one-time","type":"jdbc","started":"2015-04-17T22:47:58.941Z","last_active_begin":"1970-01-01T00:00:00.000Z","current_active_begin":null,"last_active_end":null,"map":{"aborted":false,"suspended":false,"counter":1}}
[2015-04-17 22:47:59,030][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:47:59,030][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:47:59,057][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:47:59,057][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:55:08,537][DEBUG][river.jdbc.RiverStateService] post: update river state = {"name":"sales-river-one-time","type":"jdbc","started":"2015-04-17T22:47:58.941Z","last_active_begin":"2015-04-17T22:47:59.004Z","current_active_begin":null,"last_active_end":"2015-04-17T22:55:08.529Z","map":{"lastStartDate":1429310879024,"counter":1,"aborted":false,"suspended":false,"lastExecutionEndDate":1429311308529,"lastExecutionStartDate":1429310879060}}
[2015-04-17 22:55:08,555][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:55:08,555][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:55:08,739][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:55:08,739][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-17 22:58:54,903][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: execute
[2015-04-17 22:58:54,904][DEBUG][river.cluster            ] [Kilmer] cluster state updated, version [2], source [reroute_rivers_node_changed]
[2015-04-17 22:58:54,904][DEBUG][river.cluster            ] [Kilmer] processing [reroute_rivers_node_changed]: done applying updated cluster_state
[2015-04-17 22:47:58,858][DEBUG][river.cluster            ] [Beautiful Dreamer] processing [received_state]: execute
[2015-04-17 22:47:58,858][DEBUG][river.cluster            ] [Beautiful Dreamer] cluster state updated, version [1], source [received_state]
[2015-04-17 22:47:58,858][DEBUG][river.cluster            ] [Beautiful Dreamer] processing [received_state]: done applying updated cluster_state
[2015-04-17 22:58:54,904][DEBUG][river.cluster            ] [Beautiful Dreamer] processing [received_state]: execute
[2015-04-17 22:58:54,904][DEBUG][river.cluster            ] [Beautiful Dreamer] cluster state updated, version [2], source [received_state]
[2015-04-17 22:58:54,904][DEBUG][river.cluster            ] [Beautiful Dreamer] processing [received_state]: done applying updated cluster_state

In the logs, there must be a line "river state deleted" after "river closed, see https://github.com/jprante/elasticsearch-jdbc/commit/b3a913cf06dba6fe91f43a16796bdfdf93bc82bc
Please update. I'm not sure why <IP_ADDRESS> does not contain state deletion.

"river state deleted" is for sure not in the logs.
I restarted the cluster and now I see "river state deleted" when I remove rivers and things appear to be working correctly but should I be concerned with the double delete message?
[2015-04-20 17:41:35,589][DEBUG][river.jdbc.JDBCRiver     ] shutting down river thread scheduler
[2015-04-20 17:41:35,589][DEBUG][river.jdbc.JDBCRiver     ] interrupting river thread
[2015-04-20 17:41:35,589][DEBUG][river.jdbc.JDBCRiver     ] shutting down river flow
[2015-04-20 17:41:35,589][INFO ][river.jdbc.JDBCRiver     ] river closed [jdbc/test5]
[2015-04-20 17:41:35,593][DEBUG][river.jdbc.RiverStateService] delete: river state [test5]
[2015-04-20 17:41:35,602][INFO ][river.jdbc.JDBCRiver     ] river state deleted [jdbc/test5]
[2015-04-20 17:41:35,602][DEBUG][river.cluster            ] [Ectokid] processing [reroute_rivers_node_changed]: done applying updated cluster_state
[2015-04-20 17:41:35,602][DEBUG][river.cluster            ] [Ectokid] processing [reroute_rivers_node_changed]: execute
[2015-04-20 17:41:35,602][DEBUG][river.cluster            ] [Ectokid] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-04-20 17:41:35,615][DEBUG][river                    ] [Ectokid] failed to (double) delete river [test5] content
org.elasticsearch.indices.TypeMissingException: [_all] type[[test5]] missing: No index has the type.
at org.elasticsearch.action.admin.indices.mapping.delete.TransportDeleteMappingAction$1.onResponse(TransportDeleteMappingAction.java:138)
at org.elasticsearch.action.admin.indices.mapping.delete.TransportDeleteMappingAction$1.onResponse(TransportDeleteMappingAction.java:117)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim(TransportBroadcastOperationAction.java:252)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation(TransportBroadcastOperationAction.java:217)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$2.handleResponse(TransportBroadcastOperationAction.java:196)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$2.handleResponse(TransportBroadcastOperationAction.java:183)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:161)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:130)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Do not worry, "double delete" message is not harmful.

Thanks @jprante
I still believe there is a scenario where this breaks down but it's intermittent and difficult to duplicate. I'll close the issue for now and re-open it when I have more information.


Bluetooth: Audio: Shell: Audio shell improvements

This commits allows for setting up multiple streams in a single CIG as the unicast client.
It fixes an issue with the unicast server not using the configured QoS when transmitting audio data.
Signed-off-by: Emil Gydesen<EMAIL_ADDRESS>

Just looking at the content of the commit message, and number of lines changed, seems like this should be split up into multiple commute? Or does all of this rea

Just looking at the content of the commit message, and number of lines changed, seems like this should be split up into multiple commute? Or does all of this really hang together and would break bisectability if split up?

To some extent the fixes are quite highly coupled.
There are 3 things fixed by this:

allows for setting up multiple streams in
a single CIG as the unicast client.

It fixes an issue with the unicast server not using
the configured QoS when transmitting audio data.

Improves upon setting the preset for the different
type of streams.

was done as part of fixing 1), but can technically be moved to its own change. 2) was also done as part of fixing 1), and while it technically could be fixed in its own commit after 1), the changes to 1) would basically revert all the changes as everything related to the Codec and QoS was refactored.

I could move 3) to it's own commit before this, but that change won't really do anything until 1) is applied.
I think my conclusion is that the changes could be moved to separate commits and 1) applied last, but the commits separately won't really have any affect and/or most of the changes would be refactored anyways. IMO this is better as a single commit.

Rebased to solve merge conflict

Rebased to solve merge conflicts


[BUG] Forecasting notebook in examples is buggy

Needs to be updated with latest changes from #757, e.g. changing of scitypes
Adjustment of window_lengths in gridsearch, as they are currently too small. I think it is as simple as replacing initial_window with window_length in SlidingWindowSplitter.

To Reproduce
Try running the forecasting notebook for yourself

I am happy to work on this, but I will do it after #757 is completed.

Thanks @Lovkush-A for raising this issue!
Seems like the CI test doesn't fail if it detects an error: https://github.com/alan-turing-institute/sktime/runs/2286754227
I believe we need to add set -e at the top of the build_tools/run_examples.sh file. As this affects all PRs, we shouldn't wait until #757 is merged.
I'd appreciate a separate PR with the fix in build_tools/run_examples.sh and the failing notebooks.


Cannot connect to device

I can search for device but cannot connect. I always get EUNSPECIFED

You need to know the UUID of your device. Example: 00001101-0000-1000-8000-00805f9b34fb.

Hi @douglasjunior
How get the UUID from react-native ?


Benp/jshint violations

@Bachmann1234 mind taking a look?

Ill take a look today or tomorrow!

yo @Bachmann1234 looks like I have broken tests :( I'll address those and let you know when it's GTG for you. Sorry about that...

Coverage decreased (-0.22%) to 99.01% when pulling faa0820bb6e001a561c9027f518c1b90464f6fa7 on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

Coverage decreased (-0.22%) to 99.01% when pulling faa0820bb6e001a561c9027f518c1b90464f6fa7 on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

Coverage decreased (-0.22%) to 99.01% when pulling faa0820bb6e001a561c9027f518c1b90464f6fa7 on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

Coverage decreased (-0.22%) to 99.01% when pulling faa0820bb6e001a561c9027f518c1b90464f6fa7 on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

Coverage decreased (-0.17%) to 99.06% when pulling ec1dfb81411bb3eb709a67eafb1f57f74430c58f on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

Coverage decreased (-0.17%) to 99.06% when pulling ec1dfb81411bb3eb709a67eafb1f57f74430c58f on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

A couple trivial comments. Looks pretty good to me. Can you squash this down?

Coverage increased (+0.03%) to 99.26% when pulling 4eb748fd67cb610ea707a1c559355f86d8c72296 on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

Coverage increased (+0.03%) to 99.26% when pulling 4eb748fd67cb610ea707a1c559355f86d8c72296 on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

Coverage decreased (-0.31%) to 98.92% when pulling 68dc4a458e6d916e31b9301dd5aebd54aa8fd954 on benpatterson:benp/jshint-violations into 8ae1814436448a42745a7346ea0636a275ac3c8d on Bachmann1234:master.

@Bachmann1234 Added tests, fixed up flake8, and squashed. Happy to iterate again if you've got additional comments.

Cool ill give this a last once over soon!

Lets do this! Thanks for the PR

https://pypi.python.org/pypi/diff_cover ITS OUT! :-) Thanks again

Awesome!!!


Custom Build Scripts

This patch provides the ability to provide custom build
scripts for the cross compiler.
Signed-off-by: “Rian<EMAIL_ADDRESS>

@ilovepi once the tests pass I will merge this. You should be able to change the CUSTOM_FETCH_CLANG and CUSTOM_BUILD_CLANG environment variables to suit your needs here. Let me know if you have any issues with this patch

@ilovepi the load-time will need mods to the ELF loader likely. No idea about the run-time


Problems encountered when analyzing BOHB using CAVE

Hi, I am trying to use CAVE to analyze the running results of BOHB, but I have encountered problems that I don't know how to solve. The program seems to be unable to start at all.
I follow the example, using the command :
cave --folders /home/parameters/result --file_format BOHB --output CAVE_BOHB_results

DEBUG:cave.cavefacade.CAVE:Output-dir 'CAVE_BOHB_results' was already created, call ignored
DEBUG:cave.cavefacade.CAVE:Running CAVE version 1.1.6
DEBUG:cave.utils.hpbandster2smac.HpBandSter2SMAC:<itertools.product object at 0x7f73862a86c0>
DEBUG:cave.utils.hpbandster2smac.HpBandSter2SMAC:Sampled 1 interpretations of "/home/parameters/result/configspace.pcs"
DEBUG:cave.utils.hpbandster2smac.HpBandSter2SMAC:[]
DEBUG:cave.utils.hpbandster2smac.HpBandSter2SMAC:Skipped 0 None- and 0 NaN-loss-values in BOHB-result
INFO:cave.utils.hpbandster2smac.HpBandSter2SMAC:Assuming BOHB treats target algorithms as deterministic (and does not re-evaluate)
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_9/scenario.txt.
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_27/scenario.txt.
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_81/scenario.txt.
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_243/scenario.txt.
DEBUG:cave.cavefacade.CAVE:Folders: ('CAVE_BOHB_results/budget_9', 'CAVE_BOHB_results/budget_27', 'CAVE_BOHB_results/budget_81', 'CAVE_BOHB_results/budget_243'); ta-exec-dirs: ['.']
DEBUG:cave.cavefacade.CAVE:Collecting data from CAVE_BOHB_results/budget_9.
DEBUG:cave.ConfiguratorRun.CAVE_BOHB_results/budget_9:Loading from 'CAVE_BOHB_results/budget_9' with ta_exec_dir '.'.
DEBUG:cave.reader:Creating scenario from '.'
DEBUG:smac.utils.io.cmd_reader.CMDReader:Deactivate output directory.
DEBUG:smac.utils.io.cmd_reader.CMDReader:Loading pcs as json from: CAVE_BOHB_results/budget_9/configspace.json
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:cave.ConfiguratorRun.CAVE_BOHB_results/budget_9:RunHistory original runhistory only evaluated on 0/1 train-insts for default in folder CAVE_BOHB_results/budget_9
DEBUG:cave.ConfiguratorRun.CAVE_BOHB_results/budget_9:RunHistory original runhistory only evaluated on 0/1 test-insts for default in folder CAVE_BOHB_results/budget_9
DEBUG:cave.reader.configurator_run.ConfiguratorRun:No rng and no run_id given: using a random value to initialize run_id.
INFO:cave.reader.configurator_run.ConfiguratorRun:Optimizing a deterministic scenario for quality without a tuner timeout - will make SMAC deterministic!
INFO:smac.scenario.scenario.Scenario:No output directory for scenario logging specified -- scenario will not be logged.
INFO:smac.utils.io.traj_logging.TrajLogger:No output directory for trajectory logging specified -- trajectory will not be logged.
In addition, I have some doubts about the input format of BOHB.
The code in the document “Save a ConfigSpace config_space in a directory output_dir using”
does not work here for me
from ConfigSpace.read_and_write import pcs_new
with open(output_dir, 'w') as fh:
    fh.write(pcs_new(config_space))

I am using：
with open(args.shared_directory+"/configspace.pcs", 'w') as fh:
    fh.write(pcs_new.write(w.get_configspace()))

I am not sure if this will change the output of the configspace.pcs file causing the error

Hi. Is it possible for you to try using the json-format? There are some known issues with the pcs-format when it comes to complicated config-spaces. Try:
from ConfigSpace.read_and_write import json
with open(args.shared_directory + "/configspace.json", 'w') as fh:
    fh.write(json.write(w.get_configspace()))

Also, what versions of BOHB, CAVE and ConfigSpace are installed? (you can check with pip freeze, if you're using pip as a version manager)

hello, I am using cave 1.1.6, ConfigSpace 0.4.10, hpbandster 0.7.4. I tried to use the json file to store the configspace. The problem still exists. There aren't too many changes to the log (just changed the read json file)
data https://github.com/87302380/parameters/tree/master/result
Output file: https://github.com/87302380/parameters/tree/master/CAVE_BOHB_results
DEBUG
DEBUG:cave.cavefacade.CAVE:Output-dir 'CAVE_BOHB_results' was already created, call ignored
DEBUG:cave.cavefacade.CAVE:Running CAVE version 1.1.6
DEBUG:cave.utils.hpbandster2smac.HpBandSter2SMAC:Detected and loaded "/home/lchen/parameters/result/configspace.json". No backup-cs necessary
DEBUG:cave.utils.hpbandster2smac.HpBandSter2SMAC:Skipped 0 None- and 0 NaN-loss-values in BOHB-result
INFO:cave.utils.hpbandster2smac.HpBandSter2SMAC:Assuming BOHB treats target algorithms as deterministic (and does not re-evaluate)
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_9/scenario.txt.
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_27/scenario.txt.
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_81/scenario.txt.
INFO:smac.utils.io.cmd_reader.CMDReader:Output to CAVE_BOHB_results
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:smac.scenario.scenario.Scenario:Output directory does not exist! Will be created.
DEBUG:smac.scenario.scenario.Scenario:Writing scenario-file to CAVE_BOHB_results/budget_243/scenario.txt.
DEBUG:cave.cavefacade.CAVE:Folders: ('CAVE_BOHB_results/budget_9', 'CAVE_BOHB_results/budget_27', 'CAVE_BOHB_results/budget_81', 'CAVE_BOHB_results/budget_243'); ta-exec-dirs: ['.']
DEBUG:cave.cavefacade.CAVE:Collecting data from CAVE_BOHB_results/budget_9.
DEBUG:cave.ConfiguratorRun.CAVE_BOHB_results/budget_9:Loading from 'CAVE_BOHB_results/budget_9' with ta_exec_dir '.'.
DEBUG:cave.reader:Creating scenario from '.'
DEBUG:smac.utils.io.cmd_reader.CMDReader:Deactivate output directory.
DEBUG:smac.utils.io.cmd_reader.CMDReader:Loading pcs as json from: CAVE_BOHB_results/budget_9/configspace.json
DEBUG:smac.scenario.scenario.Scenario:SMAC and Scenario Options:
DEBUG:cave.ConfiguratorRun.CAVE_BOHB_results/budget_9:RunHistory original runhistory only evaluated on 0/1 train-insts for default in folder CAVE_BOHB_results/budget_9
DEBUG:cave.ConfiguratorRun.CAVE_BOHB_results/budget_9:RunHistory original runhistory only evaluated on 0/1 test-insts for default in folder CAVE_BOHB_results/budget_9
DEBUG:cave.reader.configurator_run.ConfiguratorRun:No rng and no run_id given: using a random value to initialize run_id.
INFO:cave.reader.configurator_run.ConfiguratorRun:Optimizing a deterministic scenario for quality without a tuner timeout - will make SMAC deterministic!
INFO:smac.scenario.scenario.Scenario:No output directory for scenario logging specified -- scenario will not be logged.
INFO:smac.utils.io.traj_logging.TrajLogger:No output directory for trajectory logging specified -- trajectory will not be logged.
In fact, what should cave output? A web page or document that contains analytical data?

This is really weird, I cannot reproduce the problem. For me, the reports runs through without problems. The weird part of it is that there is no error-message. With CAVE 1.1.6 I get this report. And yes, it's a web-page with plots and analysis-data - checkout report.html.
The only thing I could imagine is missing installs that are silently caught with a sys.exit() in a submodule. Try to install swig, if you haven't yet. Try with:
apt-get install build-essential swig
or, if you use conda, try instead:
conda install gxx_linux-64 gcc_linux-64 swig.
I'll release 1.1.7 today or tomorrow. Please report, if the problem persists.

Hi, I tried installing swig and updated CAVE1.1.7 today. But the problem still exists.
If this can helps you, you can try to reproduce the problem on the Docker Playground.
Docker Playground
(The docker playground provides an online virtual Linux environment)
Excellent if you know docker.
If you don't know, the following commands can help you quickly build the same environment.
After logging in to Docker Playground, click add new instance
docker pull python
docker run -dit python
docker exec -i -t id /bin/bash

for example: If the id "4518ef6b0d77b",it is sufficient to use 451 instead of id here.
docker exec -i -t 451 /bin/bash

cd /home
git clone https://github.com/87302380/parameters.git
cd /home/parameters/whl_data_3.7
pip3 install *.whl

of course, you can also use it directly here.
pip3 install hpbandster

pip3 install cave
cd ..
apt-get update
apt-get install build-essential swig
cave --folders /home/parameters/result --file_format BOHB --output CAVE_BOHB_results

I think this should be able to reproduce the problem.

I found that this problem must come from the environment, but I don’t know where it is.
I tried to repeat the operation on Ubuntu Linux, installed the same package, CAVE runs successfully.
The environments tried before were Fedora Linux and Alpine Linux.

Thanks a lot for reporting that! I wasn't able to reproduce or figure out the problem yet, but it's good to know about environmental issues...

Closing for now - feel free to reopen if there is news on this.


JP Login issue

After updating from commit 8fb7c09... to 2c2809e... I can't login into my JP account anymore. Right after pressing "login" to retrieve banners getting the listed error below.
Before I got the same message over and over trying to use the summon function.
Global login and function still work fine on the other side.
Error
Edit: Same issue with the release / .exe version (crashes there)

https://github.com/shalzuth/BraveHaxvius/commit/ce6e863c01e0fe2f3afcf1e6ab1676e1a1f9413d
Fixed in this release I think.

Hi there, whenever I try to login with my JP accounts to perform summons. The program keeps crashing.

Login seems to be fixed but still issues with pulling / summoning. Tried various banners but it throws same error as before with login.
Error

Hello. I'm trying to enter JP, but i seem to have the same error i got before (i got v. 1.0.28).

I double-checked my pins, internet and downloaded files.


Unable to create a disk due to disk size, how do I fix?

Storage Explorer was unable to create the disk
Service response: {
"name": "Error",
"message": "The upload size in bytes<PHONE_NUMBER>8 - 512 bytes for the VHD footer<PHONE_NUMBER>6 in this case) must be a multiple of MiB.",
"stack": "Error: The upload size in bytes<PHONE_NUMBER>8 - 512 bytes for the VHD footer<PHONE_NUMBER>6 in this case) must be a multiple of MiB.\n    at new RestError (C:\Users\jason\AppData\Local\Programs\Microsoft Azure Storage Explorer\resources\app\node_modules\@azure\ms-rest-js\dist\msRest.node.js:2400:28)\n    at C:\Users\jason\AppData\Local\Programs\Microsoft Azure Storage Explorer\resources\app\node_modules\@azure\ms-rest-js\dist\msRest.node.js:3544:37\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)",
"code": "BadRequest",
"statusCode": 400,
"request": {
"streamResponseBody": false,
"url": "https://management.azure.com/subscriptions/84dcfa23-08cb-48a8-9089-b2009ef27100/resourceGroups/SandA/providers/Microsoft.Compute/disks/quickbooks2export-i-08757318c5c9ce2c0?api-version=2019-03-01",

@the-data-detective the error from the server is pretty clear. The size of VHD file you are using is not compatible. I think you'll need to resize the VHD.

Closing due to lack of response. Please open a new issue if you are still having problems.


Local-to-remote file clipping for xfreerdp

This adds support for file transfer via clipboard redirection to WinPR and teaches xfreerdp how to use it, making some progress with issues #2424, #3302, #3532, FreeRDP/Remmina#323, FreeRDP/Remmina#730.
Only local-to-remote transfers are implemented at the moment. That is, the users can select and copy/cut files in their local file managers (e.g., Nautilus) and then paste them inside the RDP session window. Implementing the remote-to-local transfers is not so trivial, so this will be coming later.
The patchset has been tested by me on Debian 8 and Ubuntu 12.04 with Nautilus and Thunar file managers on the local side and Windows 7 and Windows 10 VMs on the remote side. You can find a rough check list here.
The overall design has been outlined on the mailing list, I won't repeat it here. The detailed comments on the implementation can be found in the commit messages. I tried making the individual commits small enough to ease the code review, but that's 1.6k lines of new code however you look at it. The whole patch sequence can be split into the following parts:

Preparatory refactoring of the common code and some utilities:

winpr: define file attribute flags
channels/cliprdr: CLIPRDR_FILELIST utilities
wClipboard: improve error handling

Parsing and processing the text/uri-list format:

wClipboard: local file subsystem boilerplate
wClipboard/posix: basic file list handling
wClipboard/posix: conversion to FILEDESCRIPTORs
wClipboard/posix: parse text/uri-list format
wClipboard/posix: decode percent-encoding
wClipboard/posix: add local files to file list
wClipboard/posix: add directories to file list

Implementing the interface for retrieving file contents:

wClipboard/posix: basic delegate interface
wClipboard/posix: implement file size retrieval
wClipboard/posix: implement file range retrieval
wClipboard: track sequence numbers of file lists

Adding support of file tranfer to xfreedp:

x11/cliprdr: impove error handling
x11/cliprdr: handle text/uri-list format
x11/cliprdr: provide file content to the server
x11/cliprdr: negotiate file streaming support

Fixing issues that have been found during testing:

channels/cliprdr: fix CLIPRDR_FILECONTENTS_REQUEST
channels/cliprdr: fix CLIPRDR_FILECONTENTS_RESPONSE
channels/cliprdr: handle cliprdr 2 GB size limit
wClipboard: disallow Windows reserved names

I would love to have feedback from @akallabeth (as a person familiar with clipboard redirection) and from @hardening (as a person good at finding stupid bugs early, C is hard).
As for the future, I plan to implement a GVFS subsystem for local files as an example of asynchronous fetching of file content. This should fix support for remote-mapped filesystems in Nautilus and Thunar in GNOME-based environments. After that I am going to take on implementation of remote-to-local file transfers via FUSE. Or instead of that...

@freerdp-bot test

@ilammy Looks promising (and a very welcome addition), but I will need some time to check this one. In the mean time, could you fix the issues preventing a build with vs2010 (that is very old dialect of C)?

On 11 April 2017 at 09:11, akallabeth<EMAIL_ADDRESS>wrote:

@ilammy https://github.com/ilammy Looks promising (and a very welcome
addition), but I will need some time to check this one. In the mean time,
could you fix the issues preventing a build with vs2010 (that is very old
dialect of C)?

Sure. From what I see in the build log it seems that MSVC does not like
when function prototypes include argument names.
I did not check the build on Windows as I though I haven't touched anything
that would be enabled there. But in fact I did. I'll check the Windows
build and a fix.

Well, I'm kinda ashamed to admit it, but I'm still having trouble setting up the development environment on Windows which is able to build FreeRDP (fighting with OpenSSL at the moment). However, I have given another look at the build logs and it seems I have found the issue:
diff --git a/include/freerdp/channels/cliprdr.h b/include/freerdp/channels/cliprdr.h
index ba7fb1b..45fae0c 100644
--- a/include/freerdp/channels/cliprdr.h
+++ b/include/freerdp/channels/cliprdr.h
@@ -23,6 +23,8 @@
 #include <freerdp/api.h>
 #include <freerdp/types.h>

+#include <winpr/shell.h>
+
 #define CLIPRDR_SVC_CHANNEL_NAME	"cliprdr"

diff --git a/include/freerdp/channels/cliprdr.h b/include/freerdp/channels/cliprdr.h
index 45fae0c..2f1d027 100644
--- a/include/freerdp/channels/cliprdr.h
+++ b/include/freerdp/channels/cliprdr.h
@@ -86,6 +86,19 @@ struct _CLIPRDR_MFPICT
 };
 typedef struct _CLIPRDR_MFPICT CLIPRDR_MFPICT;

+#ifdef __cplusplus
+extern "C" {
+#endif
+
+FREERDP_API UINT cliprdr_parse_file_list(const BYTE* format_data, UINT32 format_data_length,
+		FILEDESCRIPTOR** file_descriptor_array, UINT32* file_descriptor_count);
+FREERDP_API UINT cliprdr_serialize_file_list(const FILEDESCRIPTOR* file_descriptor_array,
+		UINT32 file_descriptor_count, BYTE** format_data, UINT32* format_data_length);
+
+#ifdef __cplusplus
+}
+#endif
+
 /* Clipboard Messages */

the compiler seems to treat FILEDESCRIPTOR as an identifier name when it follows const in the declaration of cliprdr_serialize_file_list function. This is because <winpr/shell.h> defines FILEDESCRIPTOR only for non-Windows platforms. I guess it should include appropriate Windows headers when WinPR is built on Windows.
I have added the patch to the pull request which does this, but I did not even build-test it. @freerdp-bot, can you please do this for me? :crossed_fingers:

@freerdp-bot test

Ugh, another blind fix based on scrutilizing the build log. No idea why I have renamed the field in 	6ad05d5 (it used to have the correct name).
@freerdp-bot test again please

@freerdp-bot test

Hi @akallabeth. Sorry, I've been busy with some stuff as well. Finally I've pushed the improvements to the branch. All your suggestions make very much sense. Thank you.
As for adding static to callbacks, there are some other files in X11 client which could have some of their functions marked as static as well, but I've confined the changes only the clipboard-related stuff.

@freerdp-bot test

@hardening could you have a look? Would really like this feature ;)

@frerdp-bot test

@freerdp-bot test

@hardening Do you like the changes? Would like to merge this one.


Update Debian sdk Dockerfiles to CLI 1.0.0-preview3-004050

/cc @dleeapho @MichaelSimons

LGTM, once the Debian tests pass that is good (NanoServer is a known issue).


Check/Decide On License

Chris Topinka was going to look into licensing. The MIT license was discussed as a possibility. License information should either be in the README or included in a second license file that's discussed in the README. Also need to figure out how to handle the copyrighted pieces of code in regards to what that means to the license.

Knowtheory (Ted Han) helped post a copy of a permissive license, with copyright to the University of Missouri System Board of Curators.


docker-compose - invalid string

Issue
When using the docker compose and env files from a git clone of this repo, docker compose up -d returns the following error:
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.arkime.env_file contains {"path": "./env/arkime-compose.env"}, which is an invalid type, it should be a string

Expected Behavior
the docker compose file would read the env file (Noting no customisations have been made)
Workaround
Removing the yaml sections for services.arkime.env_file completely allows the compose to work, but no custom env vars would work. (Unless specified in the compose file)

Found the issue, this version of docker seems to want a full path, could be worth adding to the README

Hello and sorry for the late reply...
What Docker Compose Version did you use?
According to the docs the required attribute requires at least version 2.24.0 (https://docs.docker.com/reference/compose-file/services/#env_file)

All good, thanks for the reminder. Your bang on, I was using a old docker-compose version.


Problem in following Stream Processor Quick Start Guide

Description:
When we started to follow the Quick start Guide for Stream Processor, the step five does not work. the URL is missing. therefore we can not continue further. The description which is given in the link is not accurate
https://docs.wso2.com/display/SP400/Quick+Start+Guide

Updated the Quick Start Guide with a different scenario and tested it. The steps given at present work.


Multiple Keys

Can it encrypt data with multiple public keys ?

I don't understand how to encrypt with multiple pub key.
Is it similar with doing this with a loop?
It is probably not supported


feat: add sparse matrix handling

In order to be able to experiment with sparse inputs, I refactored some base functionality of the library as well as uncertainty_batch_sampling query strategy.
All tests and corresponding example work as before.

Thanks, awesome! I'll review it early next week and try to add also pandas.DataFrame support. I'll let you know here if I add something!

I have reviewed the code, everything checks out perfectly! I especially like the use of sklearn.metrics.pairwise.pairwise_distances, I was unaware of the existence of this function previously, but it makes things much easier.
I have two suggestions regarding the changes, I'll add these soon.

Move the data_vstack function from modAL.utils.combination to a new module called modAL.utils.data. The former one is meant for functions constructing query strategies and new utility measures.
Pass allow_nd=True for check_X_y. The reason for this is when using Keras models for instance, you may want to work with images. In this case, the data array may have 4 dimensions.

There is a slight concern I have with lines like X, y = check_X_y(X, y). As I mentioned, I plan to add Pandas support, however, check_X_y converts pandas.DataFrame objects to NumPy arrays after passing the checks. In practice, the X, y = check_X_y(X, y) solution would work even with DataFrames, but then ActiveLearner instances would store the data internally as a NumPy array. I would like to store the data internally as the user. This would make it easier for the user to use the training data obtained during the active learning process. What do you think about using check_X_y purely as a check and not for conversion?

I will commit changes mentioned in 1 and 2 in a couple of hours.
It is ok to use check_X_y only for checks, not for data conversion (changes will be commtted as well).
Moreover, I found one more bug, that I have already fixed (currently checking the result).

Awesome, thanks! If you think the feature is complete, I'll merge the PR! I will also add unit tests soon.

Yeah, I think this pull request is complete. I checked it on a question classification task
https://github.com/nikolay-bushkov/kaist_internship/blob/master/active_learning_for_question_classification.ipynb
(some gains in comparison with random sampling is observed).
There are some ephemeral thoughts about optimization, but I suggest moving iteratively.

I think we could make here:
if n_jobs is None: pairwise_distances_argmin_min else: pairwise_distances
and change uncertainty_batch_sampling signature correspondingly.
@cosmic-cortex what do you think?

I am fine with this change, if it solves the problem. (I haven't tested it yet.)


Error opening more than 2 GEO attributes, white box

1° Adding Template with the attributes:

GEO -> Dynamic
GEO -> Actuate
GEO -> Static

A publication was made on all attributes with different coordinates, when detailing the device and opening more than two attributes, the third box with the coordinate is blank.
Dynamic:
mosquitto_pub -h <IP_ADDRESS> -p 1883 -t admin:6e18d0/attrs -m '{ "GEO_Dinamico": "48.858196, 2.294510"}'
Actuate:
{
"topic": "/admin/6e18d0/config",
"attrs": {
"GEO_Atuador": "48.850219, 2.294525"
}
}
Static

Version v0.5.0-alpha.3

Possibility of the same error.

https://github.com/dojot/dojot/issues/1153
https://github.com/dojot/dojot/issues/994

Ticket Analysis - It won't be fixed.


types: Drop unsafe/unnecessary memoization in the arguments processing engine

bonus: Adding one more performance regression test to our ASV
the second (and future) op.arguments(...) call in the test, on this branch, takes about 45ms on my workstation.
Ah, the _arg_defaults 's memorization in sparse.py turns out to be unsafe for MatrixSparseFunction

Merged.


[BUG] NOTICE  ssl_verify (ERRNO 1014): can not verify peer from fd#20 with error#20: unable to get local issuer certificate

Execute the command and paste the result below.
Command: uname -a && php -v && composer info | grep hyperf && php --ri swoole
➜  hyperf-reburn git:(master) ✗ uname -a && php -v && composer info | grep hyperf && php --ri swoole

Linux lqllife 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

PHP 7.4.19 (cli) (built: May 15 2021 12:08:02) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Yasd v0.3.7, Our Copyright, by codinghuang

Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]?
hyperf/amqp                             v2.1.6    A amqplib for hyperf.
hyperf/async-queue                      v2.1.0    A async queue component for hyperf.
hyperf/cache                            v2.1.8    A cache component for hyperf.
hyperf/command                          v2.1.0    Command for hyperf
hyperf/config                           v2.1.0    An independent component that provides configuration container.
hyperf/constants                        v2.1.0    A constants component for hyperf.
hyperf/contract                         v2.1.16   The contracts of Hyperf.
hyperf/database                         v2.1.16   A flexible database library.
hyperf/db-connection                    v2.1.13   A hyperf db connection handler for hyperf/database.
hyperf/devtool                          v2.1.6    A Devtool for Hyperf.
hyperf/di                               v2.1.16   A DI for Hyperf.
hyperf/dispatcher                       v2.1.6    A HTTP Server for Hyperf.
hyperf/engine                           v1.1.3
hyperf/event                            v2.1.0    an event manager that implements PSR-14.
hyperf/exception-handler                v2.1.6    Exception handler for hyperf
hyperf/framework                        v2.1.1    A coroutine framework that focuses on hyperspeed and flexible, specifically use for build microservices and middlewares.
hyperf/guzzle                           v2.1.1    Swoole coroutine handler for guzzle
hyperf/http-message                     v2.1.1    microservice framework base on swoole
hyperf/http-server                      v2.1.16   A HTTP Server for Hyperf.
hyperf/logger                           v2.1.0    A logger component for hyperf.
hyperf/memory                           v2.1.0    An independent component that use to operate and manage memory.
hyperf/model-cache                      v2.1.13   A model cache component for hyperf.
hyperf/model-listener                   v2.1.0    A model listener for Hyperf.
hyperf/pool                             v2.1.8    An independent universal connection pool component.
hyperf/process                          v2.1.7    A process component for hyperf.
hyperf/redis                            v2.1.3    A redis component for hyperf.
hyperf/server                           v2.1.17   A base server library for Hyperf.
hyperf/testing                          v2.1.12   Testing for hyperf
hyperf/tracer                           v2.1.5    A open tracing system implemented for Hyperf or other coroutine framework
hyperf/translation                      v2.1.0    An independent translation component, forked by illuminate/translation.
hyperf/utils                            v2.1.14   A tools package that could help developer solved the problem quickly.
hyperf/validation                       v2.1.10   hyperf validation
hyperf/watcher                          v2.1.10

swoole

Swoole => enabled
Author => Swoole Team<EMAIL_ADDRESS>Version => 4.6.4
Built => Mar 17 2021 17:57:05
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.0.2u  20 Dec 2019
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608

Description:
环境是wsl的ubuntu20.04
使用eazywechat的获取授权方实例报错，请求获取授权方的帐号基本信息就没问题。
Steps To Reproduce:

php.ini配置下ca证书

php.ini配置下ca证书

ca证书已经配置过了。
➜  hyperf-reburn git:(master) ✗ php --ri openssl

openssl

OpenSSL support => enabled
OpenSSL Library Version => OpenSSL 1.0.2u  20 Dec 2019
OpenSSL Header Version => OpenSSL 1.0.2u  20 Dec 2019
Openssl default config => /usr/local/openssl/openssl.cnf

Directive => Local Value => Master Value
openssl.cafile => /www/server/cert/cacert.pem => /www/server/cert/cacert.pem
openssl.capath => /www/server/cert => /www/server/cert

➜  hyperf-reburn git:(master) ✗ php --ri curl

curl

cURL support => enabled
cURL Information => 7.70.0
Age => 6
Features
AsynchDNS => Yes
CharConv => No
Debug => No
GSS-Negotiate => No
IDN => No
IPv6 => Yes
krb4 => No
Largefile => Yes
libz => Yes
NTLM => Yes
NTLMWB => Yes
SPNEGO => No
SSL => Yes
SSPI => No
TLS-SRP => Yes
HTTP2 => No
GSSAPI => No
KERBEROS5 => No
UNIX_SOCKETS => Yes
PSL => No
HTTPS_PROXY => Yes
MULTI_SSL => No
BROTLI => No
Protocols => dict, file, ftp, ftps, gopher, http, https, imap, imaps, pop3, pop3s, rtsp, smb, smbs, smtp, smtps, telnet, tftp
Host => x86_64-pc-linux-gnu
SSL Version => OpenSSL/1.0.2u
ZLib Version => 1.2.11

Directive => Local Value => Master Value
curl.cainfo => /www/server/cert/cacert.pem => /www/server/cert/cacert.pem

一直以来都是报这个错误：
[2021-05-20 09:21:35 @3338.0]	NOTICE	ssl_verify (ERRNO 1014): can not verify peer from fd#20 with error#20: unable to get local issuer certificate
[ERROR] cURL error 1014: SSL verify failed (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://api.weixin.qq.com/cgi-bin/component/api_authorizer_token?component_access_token=xxxxxxx[211] in /www/wwwroot/hyperf-reburn/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php

升级下swoole看下

确定了，不是php ca证书的问题，是GuzzleHttp请求https的报错， $client = new \GuzzleHttp\Client(['verify' => 'false]);这样就行了。
我去的github上下载了新的ca文件 https://github.com/guzzle/guzzle/blob/4.2.3/src/cacert.pem ，也不行。:confused:

https://github.com/hyperf/hyperf/issues/3542

#3542

谢谢，我试一下。

#3542

解决了，谢谢。

#3542

解决了，谢谢。

请问下怎么解决的呀？

#3542

解决了，谢谢。

请问下怎么解决的呀？

升级swoole版本到4.6.6以上。


Delete Ingredient

as a user
I want to remove Ingredients from my bar,
so that I can keep my bar organised.
ac

[ ] there is an "remove" button on each saved ingredient
[ ] pressing the button removes the ingredient from my bar

tasks

[ ] add remove button to saved items
[ ] create onClick function to remove items from useStore

quick and clear! thumbs up.


remove test_id 6480,9833 pointing to wrong cases

What this PR does
Before this PR:
Remove test_id 6480,9833 which are pointing to wrong test cases
After this PR:
The test_id are deleted
Special notes for your reviewer
Part of process to map tests under tests/virtctl folder
Release note

NONE

/override  pull-kubevirt-e2e-k8s-1.31-sig-performance
Triggered by status-reconciler when lane was introduced


os: Hostname() truncate hostname more than 15 characters under Windows

Hi Folks,
One of our virtual machine name is pdx-srv2012-wv01. And os.Hostname(..) return hostname as pdx-srv2013-wv0 in this case. The native hostname.exe of windows from sysinternals return the right hostname. It looks like the implementation in GO seem to use GetComputerName(..) instead of GetComputerNameEx(..) to return the NetBIOS name. Is there any reason for it? Can we change this to return the dns hostname as NetBIOS seem to be getting old with later version of windows?
The version of GO I am using is 1.3.
-Karthik

/cc @alexbrainman

os.Hostname() / syscall.ComputerName() call GetComputerName()
https://msdn.microsoft.com/ja-jp/library/windows/desktop/ms724295(v=vs.85).aspx
https://github.com/golang/go/blob/master/src/syscall/ztypes_windows.go#L144
And MAX_COMPUTERNAME_LENGTH is defined as 15 in mingw header.
We should use GetComputerNameEx instead.
https://msdn.microsoft.com/ja-jp/library/windows/desktop/ms724301(v=vs.85).aspx

I have a CL that adds the *Ex equivalent functions to internal/syscall/windows (to not change syscall) but when I modify os.Hostname to use them instead, the bootstrap build of os fails because it can't import internal/syscall/windows for some reason.
The regular build of os, and it's tests seem to run fine if the patch is applied to a previously compiled tree.

You need to add internal/syscall/windows to the list of packages to be
built in cmd/dist/build.go.

@minux That fixed bootstrap, but:

the dependency check in go/build fails as os has a new dependency
the change hard-codes a windows specific package in the list of built pacakges by cmd/dist so might break other GOOS's

I think we can fix the dep test in go/build by adding
internal/syscall/windows to the list (the internal package is effectively
the new syscall package, so its addition should be acceptable)
I don't think adding internal/syscall/windows to the bootstrap package list
will break bootstrap on non-windows systems.

@cookieo9 Did you add internal/... into https://github.com/golang/go/blob/master/src/go/build/deps_test.go#L126 ?

Yup, its in the CL. https://go-review.googlesource.com/#/c/5852/
On Tue, Feb 24, 2015 at 4:52 PM, mattn<EMAIL_ADDRESS>wrote:

@cookieo9 https://github.com/cookieo9 Did you add internal/... into
https://github.com/golang/go/blob/master/src/go/build/deps_test.go#L126 ?
—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/9982#issuecomment-75884144.

@kartlee please check if https://go-review.googlesource.com/#/c/5852 does fixes your issue. Thank you.

@alexbrainman This works great...Thanks for quickly addressing it.
-Karthik

@cookieo9 fix it. Thank him.
Alex

@cookieo9 Thanks for the fix.

@kartlee  I also added new hostname test on windows https://go-review.googlesource.com/6070 Can you, please, update your source and try the test on pdx-srv2012-wv01 ? I am interested if test fail - see what your hostname command says. Thank you.
Alex

@alexbrainman Sorry for not getting back earlier. I will run them today and update the ticket.
BTW, we are still using go 1.2 and doesn't want to upgrade for our current product release, so I just applied the code upstream into our custom package's Hostname() and added a similar nightly build test to  inform when the result between os.Hostname(..) and custom package's  Hostname(..) match for machine name greater than 15 characters.  This will help us to move to your change with go 1.5.

Thank you for confirming.


Disable man support on Windows

Disable man support on Windows.
closes #637

Thanks!
@bors r+

:pushpin: Commit 285e858 has been approved by Diggsey

:hourglass: Testing commit 285e858 with merge a3d5cc8...

:broken_heart: Test failed - status-appveyor

@Diggsey the test failure seems to be unrelated to this PR

@bors retry

:hourglass: Testing commit 285e858 with merge 36936ea...

:broken_heart: Test failed - status-appveyor

ugh @bors retry

:hourglass: Testing commit 285e858 with merge 6019b30...

:sunny: Test successful - status-appveyor, status-travis
Approved by: Diggsey
Pushing 6019b30c627ca781157f31c5e37917f67346d2dd to master...


Add new gulp task to 'gulpfile.js' for running mongo daemon

Applied the mongo daemon startup option to 'gulpfile.ts'.
ISSUE=#503

@llighter Thank you for hard work.
This patch looks good to me. I'll merge this after passing all tests.
BTW, I'll point minor things out.

According to the contribution guide, we don't have to use periods(.) in the title.

https://github.com/lunchclass/absolute/blob/master/docs/contributing.md#commit

If possible, we should use imperative sentense.

For example, you write "I applied something blah blah.." but we had to write "Applied something blah blah".

Thank you very much!


Incorrect implementation?

I have been playing around with this example and I have found something unexpected. I think that the scaling logic never does anything and the thresholding logic does nothing in the provided example.
Take a look at this code:
scale = torch.clamp(
    (1 / torch.abs((noise_pred_text - noise_pred_edit_concept)))
    * edit_guidance_scale_c,
    max=1.0,
)

In my experiments (including reproducing the colab results), each element of noise_pred_text - noise_pred_edit_concept is always less than one. So, 1 / torch.abs((noise_pred_text - noise_pred_edit_concept) is always > 1. Which means that edit_guidance_scale_c affects nothing and scale is always just a matrix of ones.
These ones are then applied to the following step:
edit_concept_scale = torch.where(
    (noise_pred_text - noise_pred_edit_concept)
    <= edit_threshold_c,
    torch.zeros_like(scale),
    scale,
)

Which means the result is always a matrix where each element is either a zero or a one. Additionally, this conditional is almost always met. So, edit_concept_scale is mostly (99% in most cases) an array of ones.
To confirm this, I took the colab example and set the edit_guidance_scale values to 0.001 and 1000000 and the results did not change at all. Changing the edit _threshold to [-0.9, -0.9, -0.9] also resulted in the same generated images. However, changing the edit _threshold to [-0.01, -0.01, -0.01] did change the results.
So, the conclusion I come to from this is that edit_guidance_scale does nothing at any value > 1 and in the example shown, the thresholding seems to be having no effect. The latter is just a consequence of this given example I guess.
Is this the intent of the algorithm? or is the implementation off?

Hi there,
You are absolutely right about your observations. We observed the same flaws in the implementation and fixed them with the latest commits.
Feel free to check out respective paper as well: https://arxiv.org/abs/2301.12247

We also added some additional examples in Jupiter to try out the improved interface.


[Tooltip] Does not open/close on mobile devices (iOS)

Bug report
Current Behavior
The tooltip component does not open/close on iOS Safari. It works in mobile emulation mode in Chrome Dev tools but not on the actual device. Tapping the trigger does not open any tooltip.
Expected behavior
The tooltip should open and close when the trigger is tapped. (Ideally, all primitives would behave on mobile as they do on desktop.)
Reproducible example
The Radix Documentation
https://user-images.githubusercontent.com/2600729/180950707-7f606fd3-4318-449b-89dd-a6c4c102ff52.MOV
Additional context

Your environment
Since this issue can be recreated on your documentation page, i figure you already have environment details.

Software
Name(s)
Version

Radix Package(s)
react-tooltip
1.0

React
n/a

Browser

Assistive tech

Node
n/a

npm/yarn

Operating System
iOS
15.5

Hi @dimamarkus, thanks for raising but this is by design.

@andy-hook Excuse me, would like to ask, how to solve the problem of using and displaying Tootip on mobile terminal.

"Closes when the trigger is activated or when pressing escape." Or can I turn this feature off? @andy-hook

I've found a way to make it work perfectly for my needs, which are:

Keep the same hover functionality on desktop.
Do not enable on-click on desktop.
Enable on-click on mobile.

@matstyler solution worked, but it felt a bit hacky and did not allow me to customize the Trigger like I wanted.
First, create a context which we are going to use all across the Root component tree:

// tooltip-trigger-context.tsx
'use client';

import { createContext } from 'react';

type TooltipTriggerContextType = {
  open: boolean;
  setOpen: React.Dispatch<React.SetStateAction<boolean>>;
};

export const TooltipTriggerContext = createContext<TooltipTriggerContextType>({
  open: false,
  setOpen: () => {}, // eslint-disable-line
});

Then, utilize your context starting from the root:
const TooltipRoot: React.FC<TooltipPrimitive.TooltipProps> = ({ children, ...props }) => {
  const [open, setOpen] = React.useState<boolean>(props.defaultOpen ?? false);

  // we only want to enable the "click to open" functionality on mobile
  const { isMd } = useTwBreakpoint('md');

  return (
    <TooltipPrimitive.Root
      delayDuration={isMd ? props.delayDuration : 0}
      onOpenChange={(e) => {
        setOpen(e);
      }}
      open={open}
    >
      <TooltipTriggerContext.Provider value={{ open, setOpen }}>
        {children}
      </TooltipTriggerContext.Provider>
    </TooltipPrimitive.Root>
  );
};

And update your Trigger component as well:
const TooltipTrigger = React.forwardRef<
  React.ElementRef<typeof TooltipPrimitive.Trigger>,
  React.ComponentPropsWithoutRef<typeof TooltipPrimitive.Trigger>
>(({ children, ...props }, ref) => {
  const { setOpen } = React.useContext(TooltipTriggerContext);

  return (
    <TooltipPrimitive.Trigger
      ref={ref}
      {...props}
      onClick={(e) => {
        e.preventDefault();
        setOpen(true);
      }}
    >
      {children}
    </TooltipPrimitive.Trigger>
  );
});

  const { isMd } = useTwBreakpoint('md');

Just as an improvement suggestion, I don't know what your hook here is referencing but if it is using a media query you should consider using hover instead. Not all devices below an arbitrary breakpoint are touch devices and same goes the other way

Still happening today

FWIW, here's mine that checks for :hover support instead of TW breakpoint:
export default function useHasHover() {
  try {
    return matchMedia('(hover: hover)').matches
  } catch {
    // Assume that if browser too old to support matchMedia it's likely not a touch device
    return true
  }
}

Thank you providing your solution @askorupskyy 🙌


Topic separation for time and depth log

From @vilasdeshmukh on January 23, 2018 17:38
Test out if the spark setup will work for separate topics for time and depth. If not identify and fixes for the spark code if changes are not many.
Copied from original issue: hashmapinc/Timeseries#144

From @vilasdeshmukh on January 23, 2018 19:41
Tested and works fine with minor changes to spark code.


Android: Push notification messages are not received when the application is closed!

[REQUIRED] Please fill in the following fields:

Unity editor version: 2021.3.12f1
Firebase Unity SDK version: 11.0.0
Problematic Firebase Component: com.google.firebase.messaging (11.0.0)
Other Firebase Components in use: com.google.firebase.analytics (11.0.0)
Additional SDKs you are using: com.google.external-dependency-manager (1.2.176)
Platform you are using the Unity editor on: Windows
Platform you are targeting: Android 12 , (TCL30se)
Scripting Runtime: IL2CPP

[REQUIRED] Please describe the issue here:
Notifications are not received when the application is closed.
When the application is open, notifications are received, I get the notification id in OnMessageReceived
When the application is minimized, notifications are received.
I've searched in forums and in issues but in none of them do I see that there is a solution. Reference is made to the documentation but I don't see that the people reporting the problem have resolved it.
Relevant Code:
This is how I register for events after Firebase is initialized.
public void InitializeFirebase()
    {
        Debug.Log("FBG:Start Register Event Messaging");
        Firebase.Messaging.FirebaseMessaging.TokenReceived += OnTokenReceived;
        Firebase.Messaging.FirebaseMessaging.MessageReceived += OnMessageReceived;
        Firebase.Messaging.FirebaseMessaging.SubscribeAsync("TestTopic");
        Debug.Log("FBG:End Register Event Messaging");
    }

    public void OnTokenReceived(object sender, Firebase.Messaging.TokenReceivedEventArgs token)
    {
        Debug.Log("FBG:Received Registration Token: " + token.Token);
    }

    public void OnMessageReceived(object sender, Firebase.Messaging.MessageReceivedEventArgs e)
    {
        Debug.Log("FBG:Received a new message from: " + e.Message.From);
    }

Installing the package and subscribing to the events is not enough to receive messages when the application is closed, so my manifest ends up like this.
<?xml version="1.0" encoding="utf-8"?>
<manifest
    xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.unity3d.player"
    xmlns:tools="http://schemas.android.com/tools">

    <uses-permission android:name="com.google.android.gms.permission.AD_ID" />
    <uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />

    <application>

        <service android:name="com.google.firebase.messaging.MessageForwardingService"
            android:permission="android.permission.BIND_JOB_SERVICE"
            android:exported="false">
            <intent-filter>
                <action android:name="com.google.firebase.MESSAGING_EVENT" />
            </intent-filter>
        </service>

        <activity android:name="com.google.firebase.MessagingUnityPlayerActivity"
            android:theme="@style/UnityThemeSelector">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
            <meta-data android:name="unityplayer.UnityActivity" android:value="true" />
        </activity>
    </application>
</manifest>

In this thread, I see replies stating that Firebase push notifications are not designed to work if the application is closed. Does this make sense?
If so, what is the value of the com.google.firebase.messaging package? Is it intended for another function?
issues:146

Expected behavior in FCM is for the notification to appear in the system tray, but your Activity itself has to create the local notification to pop up a message when the remote notification comes in.
More info: https://firebase.google.com/docs/cloud-messaging/android/receive

I'm adding something important for users who are encountering the same problem. After trying many things, we simply created scheduled messages and suddenly, on some mobiles, notifications started to come through even if the app was closed.
In conclusion, there are mobiles where this simply does not occur, like mine for example :(  , no magic needs to be done, just follow what the documentation says and assume it's correct.
I hope this helps, regards.


factorio: 1.1.110 -> 2.0.7

https://factorio.com/blog/post/factorio-space-age-release
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@ofborg eval

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 350214

x86_64-linux

:x: 2 packages failed to build:

factorio (factorio-experimental)
factorio-space-age (factorio-space-age-experimental)

:white_check_mark: 1 package built:

factorio-headless (factorio-headless-experimental)

can we backport this to 24.05?

I don't fully understand why this caused conflicts but they seem pretty simple to fix. I'm not sure what the etiquette is here (have not done a backport before) but will open a PR unless someone else is already on it


the data in google drive is not available

could you please provide me a new one? thank you.

already updated

Please update the data, thank u!

could you update the data link? thank you!

could you please provide me a new one? thank you.

do you still have the data (rationale alignment) ? if so, could you send me a copy? thank you!
<EMAIL_ADDRESS>


Implement full-text search

I would like to integrate a full-text search for lokijs using a record-level inverted index .
Basic proceed:
For the user:
The user declares some properties at collection creation(new Collection(...), addCollection(...))  to be available for full-text search (similar to indices).
The user can perform a full-text search with find({"<property>": {"$text", "<text query>"}}).
For the db:
With each collection change (insert(...), update(...)) the properties of each document enabled for full-text search get tokenized and an inverted index is created/updated.
If the user performs a query at an full-text search enabled property, the query string gets tokenized and the tokens are getting look up inside the inverted index to find the corresponding documents.
Possible interface:
Adding a new key (fulltext) to options -> new Collection(name, options).
fulltext:
Array of fulltext_options
fulltext_options:

Name
Type
Description

fields
array
array of fields to combine for full-text search

alias opt
string
alias name to search over

tokenizer opt
array
array of functions to tokenize the input/query strings

min_length opt
int
minimum length of a token

max_length opt
int
maximum length of a token

direction opt
string
inverted index direction Query text:  "forward": from left to right"backward:" from right to left"both": in both directions

field:

Name
Type
Description

name
string
property name

boost opt
number
increase rank of search results

Example usage:
options = {
  "fulltext": [{
      "fields": [{"name": "firstName"}, {"name": "surName"; "boost": 10}],
      "token_length": 2,
      "alias": "name",
    }]
};

var users = db.addCollection("users", options);
users.insert([
  {"firstName": "Carly", "surName": "Gross"},
  {"firstName": "Gregory", "surName": "Carlos"}
]);

users.find({"name":      {"$text", "Carl"})   // returns both users, Gregory Carl has higher rang.
users.find({"name":      {"$text", "Gr"})     // returns both users, Carly Gross has higher
users.find({"surName": {"$text", "Gr"})   // returns only Carly Gross
users.find({"name":      {"$text", "Grev"}) // returns none

Implementation:
Tokenizer:
At the beginning, I just want to implement one tokenizer, which splits a text into words at whitespace and lowercase them.
Possible other tokenizer could be:

Split also on punctuation
Language stemmer

Inverted index:
Is a chained js object. Each character is a key with a new object as value - containing the next character, et cetera. Each object contains matching document ids.
// World
{"w": { "doc": [1],
   "o": { "doc": [1],
     "r": { "doc": [1],
       "l": { "doc": [1],
         "d": { "doc": [1]
}}}}}}

My proposal is not complete, so:
Let us discuss about the proposal, interface, names.... Nothing is carved in stone.
I want to start the first implementations next week.

Any suggestions, wishes?
Is it allowed or do you plan to use the new ECMAScript 6 language specification in lokijs?

Reference:

Inverted Index
Lunr
Elasticsearch

Interesting... is this something that can be encapsulated into a class similar to how @techfort implemented 'UniqueIndex'?
I have not run enough javascript filters to do full text search to have an idea of performance gains/cost...  but as long as all constructs are serializable without duplicating object references in serialized output, it sounds like this could be great addition.
Currently our master 1.4x branch is ES5 only (at least for main lokijs.js source file) but we have already begun establishing and working on (well @Cortys has at least) a 2.0.0 branch which can fully utilitize ES6 and you are more than able to participate in that branch if you wish to do so.
A few areas you may want to watch out for... when you update a document you have no way of knowing what the value was before unless we have collection cloning turned on which most people probably don't use.  Also, I imaging deletion of objects will require similar index maintenance to reduce the relevant word hit counts.
Sounds like something @techfort would like as well, maybe he or others can chime in if they have any preferences.

This is interesting indeed! Please do go ahead and experiment, I think it
would greatly improve the existing functionality and make Hel's gods happy.
The inverted index is an awesome idea which might even open up the path to
NLP features.
On Fri 25 Nov 2016, 19:46 obeliskos<EMAIL_ADDRESS>wrote:

Interesting... is this something that can be encapsulated into a class
similar to how @techfort https://github.com/techfort implemented
'UniqueIndex'?
I have not run enough javascript filters to do full text search to have an
idea of performance gains/cost... but as long as all constructs are
serializable without duplicating object references in serialized output, it
sounds like this could be great addition.
Currently our master 1.4x branch is ES5 only (at least for main lokijs.js
source file) but we have already begun establishing and working on (well
@Cortys https://github.com/Cortys has at least) a 2.0.0 branch which
can fully utilitize ES6 and you are more than able to participate in that
branch if you wish to do so.
A few areas you may want to watch out for... when you update a document
you have no way of knowing what the value was before unless we have
collection cloning turned on which most people probably don't use. Also, I
imaging deletion of objects will require similar index maintenance to
reduce the relevant word hit counts.
Sounds like something @techfort https://github.com/techfort would like
as well, maybe he or others can chime in if they have any preferences.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/techfort/LokiJS/issues/513#issuecomment-263015607,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD8Ffaq-2SYic1KmMhsCWzOrNkRjT3oEks5rBzr7gaJpZM4K8QRG
.

@obeliskos
Yes, it will be similar like UniqueIndex. Thank you for the hint. The performance gain will be huge for text search in long text properties.
I will try to implement an acceptable memory usage which is serialisable.
I have to study the insert/update/remove/load process of lokijs to provide a user, memory and performance friendly solution.
I am going to use the ES6 because of maps and sets. So it could be a great feature for V2.0. :)
@techfort
Thank you, I will. Now I know why it is named Loki.  :D

@Viatorus @obeliskos @techfort I am also supporting this request and the idea – indeed, we had other requests in the past asking for more advanced text search options with Loki. And I was considering that same functionality for my use cases as well (as having it could really boost the efficiency for me in some scenarios, if the implementation is done properly for the token-based text search). To note, I also remember that @alexandernst was interested in having similar functionality early this year (though he wanted to rely on Lunr.js and just provide a way to integrate it into Loki – see his issue #333).
@Viatorus Looking forward to your follow-ups on it. Depending on my actual load closer to Christmas / New Year time, I may be able to help you (at least, with code reviews – or even parts of the implementation), though cannot give a solid promise yet. But keep me in the loop anyways.

Hey guys!! Really long time no talk :(  (new job takes all my time, sadly... my GitHub stats are nullified).
Indeed, as @VladimirTechMan recalls, I wanted to join somehow Lunr.js and LokiJS. And IMHO this would be a better solution rather than reinventing the wheel from 0.
Basically, what I was going to do was duplicate all the data from Loki inside Lunr and let it do it's magic, then I'd used a deterministic id or some other sort of identifier and match the query results from Lunr to Loki documents.

Thank for for your participants - I really appreciate it. :)
Yes, at first I wanted also to integrate Lunr.js into LokiJS but now,
I am not quite sure if an own implementation for a cleaner integration
would not be better (as I already started...) - it is not a start from
zero because we can learn from others.
But nevertheless, I want to do it right, so lets see what lunr.js
really does and what we maybe can use or can make better. The
following breakdown should help me for brainstorming and
understanding what is going on.

Create lunr:
At Lunr's construction you define the fields (name and boost) and
the pipeline to convert text to tokens.
var idx = lunr(function () {
    this.field('title', { boost: 10 })
    this.field('body')

    this.pipeline.remove(lunr.stopWordFilter) // Remove one default tokenizer.
    this.pipeline.remove(function(token) { /* custom tokenizer */);
})

Pipeline:
The pipeline is a chain of tokenizer functions which get called for
each token one by one. There are three default tokenizer:

Trimmer to trim a token to bounded by words (e.g. +FooBar, -> FooBar)
English language stemmer (e.g. nothing -> noth)
English stop word filter (e.g. ignore tokens like not, an, or)

You can add/remove tokenizer functions by your own. The pipeline is
serializable (keeping the name of each function) - custom tokenizers
has to be saved/reload separatly for deserialization.
Add document:
An object as document is not fully stored inside lunr, only the tokens
and the reference document id property (here id).
var doc = {
    "title": "Twelfth-Night",
    "body": "If music be the food of love, play on: Give me excess of it…",
    "author": "William Shakespeare",
    "id": 1
}
idx.add(doc)

Split each string into lowercase tokens between a separator (default is whitespace).
Pass each token through the tokenizer pipeline.
Add each token to a document specific SortedSet.
Add each token to a global SortedSet.
Calculate the [term-frequency][tf-idf] for each token.
Add each token to the TokenStore.

TokenStore:
The token store is a tree, starting with the first character of each token as property. Each property is an own branch containing the the next character of each token, after the previous character. The leaf (end of token) contains the real document reference. All other branches contain an empty docs property (why?).
// World
{"docs": {},
 "w": { "docs": {},
   "o": { "docs": {},
     "r": { "docs": {},
       "l": { "docs": {},
         "d": { "docs": {1: {"ref": 1, "tf": 0.5}}
}}}}}}

idx.search("hello world")

[tf-idf]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf

@Viatorus very interesting post - thank you very much for taking the time to work on it.
I believe you're looking for this line with regards to your serialization question:
https://github.com/techfort/LokiJS/blob/master/src/lokijs.js#L973
This allows to specify properties that are not to be serialized to avoid circular references.
PS by the way i have a library https://github.com/techfort/mimir that does BOW and TF-IDF, feel free to pillage if it saves you some work.

@techfort Thank you. I had not yet look closely at Lokijs serialization. But its all there. :)
For the sake of completeness, I analyzed lunrs id-idf/score model to understand how it works. The code below is not the exact implementation. I smarten it up .

Add document
fields = [{"name": "user", "boost": 10}, {"name": "desc", "boost": 2}];
newDoc = {"user": "Carl", "desc": "Well done Carl, well", "id" = 1};

fieldTokens = {"user": ["carl"], "desc": ["well", "done", "carl"] };
documentStore[newDoc.id] = ["carl", "well", "done"];

for (let token of documentStore[newDoc.id])  {
    let tf = 0;
    for (let field of fields {
        let tokenCount = 0;
        for (let fieldToken of fieldTokens[field.name]) {
            if (fieldToken == token) {
                tokenCount++;
            }
        }
        tf += tokenCount / fieldLength * fieldBoost;
    }
    // store token with docRef and tf inside tokenStore
    // ...

    // *1
    let documentFreuqncy = tokenStore.count(token);
    let idf = 1 + log_e(documentStore.length / documentFrequency)
    idfMap.set(token, idf);

    // Find position of similarToken in corpusToken. *2
    let pos = corpusTokens.indexOf(similarToken);
    documentVectors[newDoc.id].insert(pos, tf * idf * similarityBoost)
}

*1 Lazy evaluation, cached.
*2 Lazy evaluation, not cached.
Lunr calculates these values if required at search, for simplification I mention it here.
Search query
query = "Wel Si, wel";
queryTokens = ["wel", "si"];

// fields.length = 2 <- user and desc
// fields.boost = 12 <- users 10 + desc 2
let tf = 1 / queries.length * fields.length * fields.boost;

for (let queryToken of queryTokens) {

    // wel -> well
    // si -> sir
    for (let similarToken of tokenStore.similar(queryToken)) {
        let simularityBoost = 1;
        if (similarToken !== queryToken) {
          let diff = Math.max(3, similarToken.length - queryToken.length);
          similarityBoost = 1 / Math.log(diff);
        }
        // Find position of similarToken in corpusToken.
        let pos = corpusTokens.indexOf(similarToken);
        let idf = idfMap.get(similarToken);
        queryVector.insert(pos, tf * idf * similarityBoost);

        // Find all matching documents.
        docRefs.insert(tokenStore.get(similarToken).docRefs);
    }
}

let score = [];
for (let docRef of docRefs) {
    // Calculate cosine similarity
    score[docRef] = queryVector.dot(documentVectors[docRef]) / (queryVector.magnitude() * documentVectors[docRef].magnitude());
}

return score;

I think there is also a big potential to create a data model which work with tf-idf, Levenshtein distance and naive Bayes classifier.
Okay, now I am going to think about an own implementation concept. See you. ;)

Hey guys,
I have got a first implementation on an inverted index tree. But I still spend a lot of time to analyze different scoring  models.
Some question for me are at the moment:

Split index for different fields/properties?
How to handle not exact matches for scoring?

Some new inputs come from:

Lucence BM25
Lucence TFIDF
Elasticlunr
Lunr V2

Nice, hopefully others can provide suggestions about implementation I am unfamiliar with those algorithms / scoring models.
Do you expect to persist index or rebuild on load?   I wonder what cost/benefit of that is... I know for unique indices we rebuild those on startup but that is mainly the simplest way to remove duplicate document references.  Your fts would have more value in saving meta but worth examining rebuild costs if nominal.
Do you expect algorithms to prefer many documents with small text fields and/or fewer documents with large text fields? Or can it adapt to both?  I mean in theory (as example) in 1 gb heap space we could retrieve numerous txt ebooks from online repository and run searches against them.  Those text properties could be a meg or two text.
I guess you have probably already started thinking about benchmarking and saved database size, so i'm sure you will evaluate an optimal solution for scale.

I am not absolut sure about how to persist the index. It depends - maybe the user have to decide.
If the db contains less data, the index can be recalculate fast, for more data it needs more time. The relative memory overhead would be the similar but for absolute: 1 GB of data needs more storage memory than 1 MB. This should be sure.
Similar applies to removing a document from the inverted index. Keeping track of a token from a document inside the token tree needs some additional memory. Elasticlunr  just tokenize the document again and then remove the founded token in the tree - clever but it needs more calculation time.
Yes, more documents needs more memory/calculation - this should be clear. But also, you gain a faster query for such a big database. With the help of tokens stemmer you can control the inverted index to remove words like and, an in with not...
Yes, I am looking for an optimal implementation. Apache lucene is a very good reference for this. :)

Hey there. :)
Some updates on my side.

What I have done so far:

implementation of a serializable inverted index tree for different document properties

add/update/remove docs without big overhead
deserialization needs some reconstruction because of object references

implementation of BM25 and TF-IDF similarity

Both measure the relevance of a token inside a document but they work a bit different
BM25 is faster and has better scoring results towards TF-IDF (simple summarized)
For me the biggest difference: BM25 does not consider non matching query tokens

implementation of a fuzzy-search with damerau-levenshtein

My extended implementation from kigiri seems to be the fastest out there for js
Can still be optimized to use already calculated rows from similar query tokens

implementation of a wildcard search

you can search for hel?o w?rld
still have no efficiency way to allow the non prefix/suffix star operator: learn*g

For me the most relevant TODOS are:

Combining wildcard, fuzzy search and BM25/TF-IDF similarity to one scoring model

theory is known - implementation is needed ;)

Integrate inverted index into lokijs

Define the new filter query / query parser  (if possible like Elastic)
variable/function naming

Code cleanup and documentation

Other TODOS:

Add a token pipeline (trimmer, stemmer, etc.) with language support (eng, rus, de...)

lunr has this - but I don't really like the implementation
elasticlunr writes wrapper to use lunr language support - which is similar ugly for me - sorry

Test/Benchmark

Currently, there is no up to date working branch - I want to push it soon so you could/can cast an eye at it. ;)

@Viatorus needless to say this is outstanding work and something i will very much look forward to play with. One thing I will ask you is to document this thoroughly as users will find this functionality extremely helpful so we want to be absolutely clear on its usage. Thanks a lot again.

Heyho guys, some update:
I implemented a query builder which can be used to build a (complex) full text search query. While implementing I defined the structure of how a query could be structured.
The structure is similar to elasticsearch but on some points more restricted, a little bit bloated but more precised because I don't want to parse and verify the query over and over again.
The next step would be to interpret the query, do the index search, score and return the resulting documents. :)

Useless example:
var query = new QueryBuilder()
 .bool().boost(2)
  .startFilter().boost(3)
      .wildcard("123", "123")
      .term("how", "are").boost(5)
  .endFilter()
  .startMust().boost(11)
    .match("123", "123")
    .term("with", "todo").boost(5)
  .endMust()
  .startNot()
    .bool()
      .startNot()
        .terms("name", ["carl", "harry", "unknown"])
        .matchPhrasePrefix("123", "123").boost(5)
        .matchPhrase("with", "todo other").slop(5).analyzer("wow")
      .endNot()
  .endNot()
.build();

// Result in:
{
  "query": {
    "type": "bool",
    "boost": 2,
    "filter": {
      "type": "array",
      "value": [
        {
          "type": "wildcard",
          "field": "123",
          "value": "123"
        },
        {
          "type": "term",
          "field": "how",
          "value": "are",
          "boost": 5
        }
      ],
      "boost": 3
    },
    "must": {
      "type": "array",
      "value": [
        {
          "type": "match",
          "field": "123",
          "query": "123"
        },
        {
          "type": "term",
          "field": "with",
          "value": "todo",
          "boost": 5
        }
      ],
      "boost": 11
    },
    "not": {
      "type": "array",
      "value": [
        {
          "type": "bool",
          "not": {
            "type": "array",
            "value": [
              {
                "type": "term",
                "field": "name",
                "value": [
                  "carl",
                  "harry",
                  "unknown"
                ]
              },
              {
                "type": "match_phrase_prefix",
                "field": "123",
                "query": "123",
                "boost": 5
              },
              {
                "type": "match_phrase",
                "field": "with",
                "query": "todo other",
                "slop": 5,
                "analyzer": "wow"
              }
            ]
          }
        }
      ]
    }
  }
}

Hello guys, some updates:
Now there is a working scored search for most query types besides token matches:

Bool, ConstantScore, Term, Terms, Fuzzy, Wildcard, MatchAll

The supported scoring model is BM25 only, which is the current state of art. I dropped TF-IDF similarity because of simplification. If it is really needed, I will implement it but it is mostly out dated by the other libraries (elasticsearch/lucene).
My next steps are:

Scoring with tokens (queries that use trimmer/stemmers).
Better documentation
LokiJS integration

@Viatorus, I apologize for commenting in closed thread, but this seems like an appropriate place. If I understand correctly this (fulltext search) is implemented in lokidb as a plugin? How would one go about using it? I get that documentation is not really written yet - but it seems end usage of lokidb is identical to lokijs (albait a bit faster in my quick tests).

@zipavlin
Yes, LokiDB has now plugins and the full-text search is one of them.
See https://github.com/LokiJS-Forge/LokiDB/blob/master/packages/full-text-search/spec/generic/full_text_search.spec.ts how to use the full-text search with LokiDB.
Yes, the docs are not ready now, but there is a branch at LokiDB#docs with some more information.
See https://github.com/LokiJS-Forge/LokiDB/blob/docs/docs/full-text-search.md

@Viatorus thanks a lot! I tried it and it seems to work great :)

@Viatorus is it possible to search across all fields? Or is it necessary to always specify a field?

You have to specify the field but you can search multiple fields at once.

Sorry to bump this old thread y'all. LokiDB appears to no longer be actively maintained. Did regular LokiJS ever get full-text-search? I know there's regex support but I'd assume that's a slower text comparison with no indexing?


Disable Idle Check

I have encountered a situation where it would be nice to disable the idle interval checking. Is this possible?
For example, when the page becomes idle I display a warning dialog to the user. They have 30 seconds to press a button or their session expires. However, I don't want to track their idle status during those 30 seconds because I don't want the idle state to trigger the dialog to display again.
It is unlikely that my idle timeout will be less than the dialog timeout in practice. But during debugging (with much shorter times) I have noticed this issue. It would be nice to have some way to turn off the idle checking (from within the idle callback), or to at least deregister the idle callback. I can then re-enable the idle checking later when the user clicks the button to maintain their session.
Hope that makes sense.
Thanks for this awesome library btw!

Ugh. Sorry to bother you. I just took 2 seconds to look at the TypeScript definitions and found the on/off methods.


Issue within plugin interaction

Hello, I am having issues with my server currently and there is a possibility the issue might be between the interaction of Quicksell and PermissionsEx. When two or more users are on the server and one interacts with the CommandNPC to trigger Quicksell the server crashes from CPU load. Could someone help me pinpoint if this is a PermissionEx problem or somewhere else?
http://pastebin.com/QNwgp678

Unfortunately PermissionsEx is currently unmaintained and no further updates are planned. Because of this, all pull requests are being closed. If you are looking for an actively maintained permissions plugin, take a look at LuckPerms. It is compatible with major server platforms and can import data from PEX.


Minkowski Sum bug?

Hi,
I'm trying to do the Minkowski Sum between a pentagon and a small linear segment.

Below is the C# code I'm using.
`
double r = 5;
       double twoPi=2 * 3.141592654;

        List<PointD> patternVers= new List<PointD>();

        for (int i = 0; i < 5; i++)
        {
            patternVers.Add(new PointD(r * Math.Cos(2*Math.PI * i / 5 + Math.PI/2), r * Math.Sin(2 * Math.PI * i / 5 + Math.PI / 2)));

        }

        PathD pattern = new PathD(patternVers);

        List<PointD> pathVerts = new List<PointD>() { new PointD(0, 0), new PointD(1, 0) };

        PathD path = new PathD(pathVerts);

        PathD minkSum = Minkowski.Sum(pattern,path, false)[0];`

The result I get does not satisfy me because I expected to obtain a convex polygon with 5 vertices and I don't understand if there is some enum to change inside the MinkowskiSum parameter or if it is a bug

Yep, it looks like a bug. I'll investigate.

Yep, it looks like a bug. I'll investigate.

Or maybe it isn't a bug 😁.

Is that what you were expecting?
If so, then you just needed to close the path 😱.
      PathD pattern = Clipper.MakePath(new double[] { 0, 0, 1, 0 });
      PathD path = Clipper.Ellipse(new PointD(5, 5), 5.0, 5.0, 5);
      PathsD solution = Minkowski.Sum(pattern, path, true); // TRUE == CLOSED PATH


Error message (3)

In a previous version of the JsonPreprocessor, this code
"param1" : "prefix",
"param2" : [1,2,3],
"param3" : {"${param1}_${param2}_suffix" : "value"}

caused:
'Found expression '${param1}_${param2}_suffix' with at least one parameter of composite data type ('${param2}' is of type <class 'list'>). Because of this expression is the name of a parameter, only simple data types are allowed to be substituted inside.'
This is completely fine.
In current version this changed to:
'Found expression '${param2}' with at least one parameter of composite data type ('${param2}' is of type <class 'list'>). Because of this expression is the name of a parameter, only simple data types are allowed to be substituted inside.'
It is a great improvement that within the affected expression now a single parameter that causes the problem, can be detected. But in this case the part "with at least one parameter ..." of the error message can be removed.
Possible alternative:
Found parameter '${param2}' of composite data type ...

Hello Holger,
Because we already blocked the substitution in key names, so the error message in this case now is:
Exception: A substitution in key names is not allowed! Please update the key name "${param1}_${param2}_suffix"
Thank you,
Son

Retest successful. Issue can be closed.

integrated in 0.12.1


Hot reload not working when marko file has no class declared

Version: 2.3.14
Details
Let's say app-main from (https://github.com/marko-js/examples/tree/master/examples/vite-express) is a single index.marko file with the following code:
class {} // THIS NEEDS TO BE HERE FOR HOT RELOAD TO WORK

<div.rect />

style {
    .rect {
        background-color: red;
        height: 100px;
        width: 100px;
    }
};

Without the class declaration, hot reload doesn't work on that component.
Expected Behavior
Hot reload should be happening, but the page needs to be manually reloaded for the changes to happen.
And when I manually reload, the following errors show up in the browser console:
TypeError: error loading dynamically imported module
[hmr] Failed to reload /src/components/app-main/index.marko. This could be due to syntax errors or importing non-existent modules.

Let me know if you need further info. Thank you!

All known HMR issues have been fixed. This appears to be working also!