one cause of frozen scans - using ct_dark_all() plan with fast acquition times - intermittent issue

Versions of DAMA software:
- Bluesky    : v1.6.3
- Ophyd      : v1.5.1
- Databroker : v1.0.6
This problem is old and not consistently occurring.  removed all .value in January and that did not help.  seems to happen when exposure time is short and "network" robustness.
successful run has the setpoint never change (camonitor) but the readback value does change
[xf23id1@xf23id-ws3 ~/Beamline/Commissioning/2020_07]$ camonitor XF:23ID1-ES{FCCD}FastCCD1:CaptureBgnd_RBV
XF:23ID1-ES{FCCD}FastCCD1:CaptureBgnd_RBV 2020-07-31 18:41:04.261963 Off  ### was the immedate after Enter
XF:23ID1-ES{FCCD}FastCCD1:CaptureBgnd_RBV 2020-07-31 19:06:06.197300 On  ### happened with plan start
XF:23ID1-ES{FCCD}FastCCD1:CaptureBgnd_RBV 2020-07-31 19:06:06.272830 Off

waiting for it to fail again to capture camonitor

In [301]: RE(mv(fccd.exposure, (0.02, .1, 20)))
Out[301]: ()

In [302]: RE(bpp.pchain(ct_dark_all(20),rel_scan([fccd],nanop.by,-.1,0.1,201, md={'purpose':'ZP zoom 1/8 cut'})))

Starting procedure to acquire darks 36.4Hz or 0.028s.

	Current number of images = 20.

	Setting to 20 images.

  A 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.
Deferred pause acknowledged. Continuing to checkpoint.
^Ctrying a second time
Pausing...
---------------------------------------------------------------------------
RunEngineInterrupted                      Traceback (most recent call last)
~/Beamline/Commissioning/2020_07/MonoStab.py in <module>
----> 1 RE(bpp.pchain(ct_dark_all(20),rel_scan([fccd],nanop.by,-.1,0.1,201, md={'purpose':'ZP zoom 1/8 cut'})))

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    796
    797         if self._interrupted:
--> 798             raise RunEngineInterrupted(self.pause_msg) from None
    799
    800         return tuple(self._run_start_uids)

RunEngineInterrupted:
Your RunEngine is entering a paused state. These are your options for changing
the state of the RunEngine:

RE.resume()    Resume the plan.
 Mark exit_stats='aborted'.
 Mark exit_status='success'.
RE.halt()      Emergency Stop: Do not perform cleanup --- just stop.

In [303]: RE.abort()
Aborting: running cleanup and marking exit_status as 'abort'...

Returning to intial conditions (pre-count).
        Total images per trigger are NOW:        20
	FCCD FCRIC gain value is NOW:		 auto

Run aborted
Traceback (most recent call last):
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py", line 1353, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1307, in __call__
    return (yield from plan)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1162, in baseline_wrapper
    return (yield from plan_mutator(plan, insert_baseline))
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper
    return (yield from plan2)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper
    return (yield from plan2)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 268, in pchain
    rets.append((yield from p))
  File "/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py", line 190, in ct_dark_all
    yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 258, in mv
    yield Msg('wait', None, group=group)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  [Previous line repeated 2 more times]
bluesky.utils.RequestAbort
Out[303]: ()

In [304]: RE(bpp.pchain(ct_dark_all(20),rel_scan([fccd],nanop.by,-.1,0.1,201, md={'purpose':'ZP zoom 1/8 cut'})))

Starting procedure to acquire darks 36.4Hz or 0.028s.

	Current number of images = 20.

	Setting to 20 images.

Returning to intial conditions (pre-count).
        Total images per trigger are NOW:        20
	FCCD FCRIC gain value is NOW:		 auto

Run aborted
Traceback (most recent call last):
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py", line 1353, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1307, in __call__
    return (yield from plan)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1162, in baseline_wrapper
    return (yield from plan_mutator(plan, insert_baseline))
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper
    return (yield from plan2)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper
    return (yield from plan2)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 268, in pchain
    rets.append((yield from p))
  File "/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py", line 190, in ct_dark_all
    yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 256, in mv
    ret = yield Msg('set', obj, val, group=group, **kwargs)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  [Previous line repeated 2 more times]
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py", line 1413, in _run
    new_response = await coro(msg)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py", line 1909, in _set
    ret = msg.obj.set(*msg.args, **kwargs)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/ophyd/signal.py", line 1670, in set
    return super().set(value, timeout=timeout, settle_time=settle_time)
  File "/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/ophyd/signal.py", line 301, in set
    raise RuntimeError('Another set() call is still in progress '
RuntimeError: Another set() call is still in progress for fccd_fccd1_capture_bgnd
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/Beamline/Commissioning/2020_07/MonoStab.py in <module>
----> 1 RE(bpp.pchain(ct_dark_all(20),rel_scan([fccd],nanop.by,-.1,0.1,201, md={'purpose':'ZP zoom 1/8 cut'})))

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    793             self._task_fut.add_done_callback(set_blocking_event)
    794
--> 795         self._resume_task(init_func=_build_task)
    796
    797         if self._interrupted:

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    917                     if (exc is not None
    918                             and not isinstance(exc, _RunEnginePanic)):
--> 919                         raise exc
    920
    921     def install_suspender(self, suspender):

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1486             exit_reason = str(err)
   1487             self.log.exception("Run aborted")
-> 1488             raise err
   1489         finally:
   1490             if not exit_reason:

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1351                     else:
   1352                         try:
-> 1353                             msg = self._plan_stack[-1].send(resp)
   1354                         # We have exhausted the top generator
   1355                         except StopIteration:

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308
   1309

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163
   1164

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804
    805

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862
    863

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in pchain(*args)
    266     rets = deque()
    267     for p in args:
--> 268         rets.append((yield from p))
    269     return tuple(rets)
    270

/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py in ct_dark_all(numim, detectors)
    188         # SET TO 1 TO ARM FOR NEXT EVENT so that the FastCCD1 is
    189         # already bkg subt
--> 190         yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
    191
    192         # take darks

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/plan_stubs.py in mv(group, *args, **kwargs)
    254     (step,) = utils.merge_cycler(cyl)
    255     for obj, val in step.items():
--> 256         ret = yield Msg('set', obj, val, group=group, **kwargs)
    257         status_objects.append(ret)
    258     yield Msg('wait', None, group=group)

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1411                         # exceptions (coming in via throw) can be
   1412                         # raised
-> 1413                         new_response = await coro(msg)
   1414
   1415                     # special case `CancelledError` and let the outer

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/bluesky/run_engine.py in _set(self, msg)
   1907         group = kwargs.pop('group', None)
   1908         self._movable_objs_touched.add(msg.obj)
-> 1909         ret = msg.obj.set(*msg.args, **kwargs)
   1910         p_event = asyncio.Event(loop=self.loop)
   1911         pardon_failures = self._pardon_failures

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
   1668         '''
   1669         if not self._put_complete:
-> 1670             return super().set(value, timeout=timeout, settle_time=settle_time)
   1671
   1672         # using put completion:

/opt/conda_envs/collection-2020-2.0rc4/lib/python3.7/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
    299
    300         if self._set_thread is not None:
--> 301             raise RuntimeError('Another set() call is still in progress '
    302                                f'for {self.name}')
    303

RuntimeError: Another set() call is still in progress for fccd_fccd1_capture_bgnd
n [305]:

In [305]: fccd.fccd1.capture_bgnd

Out[305]: EpicsSignalWithRBV(read_pv='XF:23ID1-ES{FCCD}FastCCD1:CaptureBgnd_RBV', name='fccd_fccd1_capture_bgnd', parent='fccd_fccd1', value=0, timestamp=1596235264.261963, auto_monitor=False, string=False, write_pv='XF:23ID1-ES{FCCD}FastCCD1:CaptureBgnd', limits=False, put_complete=False)

related to https://github.com/NSLS-II-CSX/xf23id1_profiles/issues/25

https://github.com/bluesky/bluesky/issues/1159 other useful discussion.

It's a bit hard to fully understand the issue without the source code from ~/Beamline/Commissioning/2020_07/MonoStab.py. I think we should develop an approach to keep these files in an accessible place. If there is some private information, there can be a private GitHub repo for these things. Here is a similar repo for the SIX beamline, which make it more convenient to provide support: https://github.com/NSLS-II-SIX/science_commissioning.

@mrakitin I don't understand.  then issue I report is using bpp.pchain.  It doesn't come from a script or ,py file
but you are correct in that this GitHub goodness is missing form commissioning plans.  I've allocated a path on the main workstation but haven't had time to creat the repo.  Are there any guidelines form DSSI?
We want the repo to be independent from the profile

@mrakitin I don't understand. the issue I report is using bpp.pchain. It doesn't come from a script or ,py file

Got it. I was confused by the following lines after a quick look at the error:
RunEngineInterrupted                      Traceback (most recent call last)
~/Beamline/Commissioning/2020_07/MonoStab.py in <module>

but you are correct in that this GitHub goodness is missing form commissioning plans. I've allocated a path on the main workstation but haven't had time to creat the repo. Are there any guidelines form DSSI?
We want the repo to be independent from the profile

I think that can be a repo with the name of your preference. The current Commissioning directory can easily be made a repo with just git init and git add <...> commands. I can assist with that.

Following up.  We discussed at the beamline and @mrakitin suggested we use the facility two-button shutter ophyd device because it is heavily tested and more robust.
our two-button shutter is based off of the same ioc, but the strings associated with the state are "In" and "Out" and "Inserted" and "Not Inserted".   @mrakitin liked other aspects of the ophyd device too but he was not sure that we should try to "fix" the facility two-button shutter since it could potentially break a lot for a lot of people.
The inout device is used to block the beam for dark image collection.  ct_darks_all()
Three options:

integrated facility two-button shutter key points into our two-button shutter
change atleast 4 on CSX from "IN" / "OUT" two-button shutter ioc / eps (CSX has two chassis for itself, so this is why EPS) to match standard facility EPS 2-button shutter
make a new version of a "better" two-button shutter (more generalized) and roll it out with V1 still available.

@ambarb, thanks for documenting more details about the issue!

will update with a scan number and the full traceback as there is more information now.

looks like we should check also exception handling when inout fails.  To be clear, inout is not failing; what happens is bluesky thinks it failed.
generator count_flatfield ['0bb2fe57'] (scan num: 156290)
	Setting to 200 images.

** (2022-07-03 16:13:40) Had to reactuate shutter while Ining
** (2022-07-03 16:13:40) Had to reactuate shutter while Ining

Returning to intial conditions (pre-count).
An exception has occurred, use '%tb verbose' to see the full traceback.
FailedStatus: DeviceStatus(device=inout, done=True, success=False)

During handling of the above exception, another exception occurred:
RuntimeError: trying to set while a set is in progress

See /var/log/bluesky/bluesky.log for the full traceback.

@johnsinsheimer I think this is pretty in control, right?  We can test more by letting making the acquire period as small as possible when running the camera at 0.2s or faster.


[bandcamp] fix failing test. remove subclass hack

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[ ] Improvement
[ ] New extractor
[ ] New feature

Sorry, I didn't realize I broke tests with that previous pull. This removes the hack I added that set Bandcamp album downloads to have numbered filenames by default.

Yeah I have that locally already. I tried using url_result with video_id, video_title but apparently the program will just ignore it because the type is set to url it will doe the whole process again... and keep 4 copies of the dict but never uses it. If type is set to video it will be used. Now I have to change the core a bit and add some option to process or at least hand over the info_dict so that it's usable when the type is url.


Remove performance repo from repo-list

FYI @billwert. This will remove the dotnet/performance repo jobs from Jenkins.

:shipit:
does this mean I can delete the groovy file out of our repo too?

Already put up that PR.


Updateing Model - Latest Version 19.1 Failed

Describe the bug
Error:
Exception occurs while the 'Xpand.ExpressApp.Security.Win.XpandSecurityWinModule' customizes TypesInfo: An error with number 1123 has occurred.
Error message: Custom persistent properties (defined in the Application Model or TypesInfo system directly) are not supported for the 'XpandPermissionPolicyRole' type.
Update failed
To Reproduce
Steps to reproduce the behavior:

Open Visual Studio
Right click on .Module.Win then click Update Module
Then Error coming up at Output

Expected behavior
Updating model successfully
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

Window 10 build

can you post a sample app? whee I can repro the case?

Closing issue for age. Feel free to reopen it at any time.
.Thank you for your contribution.


How to add remote data in DataTable?

The current DataTable and IndexTable provide only server side data feature. I am trying to add remote data using ajax using jquery's datatable plugin. It worked but the DataTable layout looks a bit broken or inconsistent. What would be the right approach for this?

Thanks for the example links @kirillplatonov . I will have a look and see.


fix(Dropdown): your description

Steps to Reproduce

Opening Any Dropdown

Expected
An onOpen Event props func to fire
Result
The function is not fire
Testcase

There is not enough information to go on here.  Please submit a more detailed report.
Here is an example of a Dropdown with a working onOpen and onClose handler:

import React, { Component } from 'react'
import { Dropdown } from 'semantic-ui-react'

const options = [{ text: 'One', value: 1 }]

class DropdownExample extends Component {
  state = {}
  handleOpen = () => this.setState({ open: true })
  handleClose = () => this.setState({ open: false })
  render() {
    return (

It is {this.state.open ? 'open' : 'closed'}.

        <Dropdown
          selection
          onOpen={this.handleOpen}
          onClose={this.handleClose}
          options={options}
        />

    )
  }
}

export default DropdownExample

hi Levi Thanks for your message.
I'm using "version": "0.56.10" and in can't see any onClose props.
~\semantic-ui-react\src\modules\Dropdown\Dropdown.js
I guess this was implement later
On Tue, Nov 29, 2016 at 5:16 PM, Levi Thomason<EMAIL_ADDRESS>wrote:

There is not enough information to go on here. Please submit a more
detailed report.
Here is an example of a Dropdown with a working onOpen and onClose
handler:
[image: http://g.recordit.co/ZzOwDpfZS6.gif]
https://camo.githubusercontent.com/a24ff267b779f91b9468115b38a31a7ee5b0bd87/687474703a2f2f672e7265636f726469742e636f2f5a7a4f774470665a53362e676966
import React, { Component } from 'react'import { Dropdown } from 'semantic-ui-react'
const options = [{ text: 'One', value: 1 }]
class DropdownExample extends Component {
state = {}  handleOpen = () => this.setState({ open: true })  handleClose = () => this.setState({ open: false })
render() {    return (

It is {this.state.open ? 'open' : 'closed'}.

)
}
}
export default DropdownExample
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Semantic-Org/Semantic-UI-React/issues/956#issuecomment-263635229,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMKXoAJjnXx1C4RiqR34VxdloKNJKGS2ks5rDF3sgaJpZM4K_BIW
.

Ah, very possible.  0.56 was sometime ago and we've implemented a lot of new things since then.


Package name should not include -rs

The package name should be sciter, not sciter-rs. Package and crate names should not include a rust- prefix or -rs suffix, because we already know it’s Rust.
(The crate name is already sciter, which is correct; but you should basically never specify lib.name in Cargo.toml; it’s almost always the wrong thing to do.)

Hi.
The intention was to show that this crate is a bindings library. During that time it was somewhat common (and -sys crates were not so popular). Nowadays it would be better to split it to a -sys crate and a library with high level Rust abstractions over the former.
May I ask why it should not include?

-rs has never been popular and was already considered an error when crates.io was first released. Unfortunately, that opinion was not overtly stated anywhere in documentation or warnings inside Cargo or on uploading to crates.io—it was merely held in IRC channels and such—and so some people went and did it. The most obvious explanation at the time seemed to be that people’s repositories often had a rust- prefix or .rs or -rs suffix, which was quite understandable (you do often want to indicate the language in the repository name for bindings), and it grew from there.
The -sys convention did arrive a little later; and yes, splitting this repository into two crates sciter-sys and sciter would be a more idiomatic approach now.
Packages and crates should not include these prefixes and suffixes, because the very fact that you’re using them indicates it’s already Rust. rust-, -rs, _rs—these add nothing of semantic meaning, and only make it harder to find it: it makes sense that the canonical crate for Sciter should be called sciter, right?

I partly agree. But again, what if my library isn't canonical? What if someone will make a better library? He'll deserve the crates.rs/sciter URL (J.)
Anyway, it is indeed something to think about during the future improvements.

Yeah, there has never been a good answer for that question, in any package manager. Sometimes you can pass the name off to someone else, but far too often in such cases the original user of the name is no longer around…


Use official google/protobuf

"But please understand that the official Proto3 support from google/protobuf for PHP will be ready very soon so my Protobuf-PHP fork will be going away soon."
https://github.com/grpc/grpc/issues/7825

Great, that's at least one less dependency to worry about. We'll see how long that will take.
Also see comments/milestone for protobuf 3.1.0 at https://github.com/google/protobuf/issues/1936

My first tests with the official library were successful, although I had to make use of Composer's include-path directive, what felt a little hacky.
I've put my progress in https://github.com/jaspervdm/pogoprotos-php/commit/5f1930b08afe2b89ac49862a3a7d92c4d6bf73ff.
I've successfully decoded buddyWithoutCandy, it has to be noted though that it failed for the message with UK33 in PokemonData, like it does with the current lib.
This is my "test":
$responseEnvelope = new \POGOProtos\Networking\Envelopes\ResponseEnvelope();
$responseEnvelope->decode(file_get_contents('buddyWithoutCandy.txt'));

/** @var \Google\Protobuf\Internal\RepeatedField $inventResp */
$inventResp = $responseEnvelope->getReturns();

$inventoryResponse = new \POGOProtos\Networking\Responses\GetInventoryResponse();
$inventoryResponse->decode($inventResp->offsetGet(0));

/** @var \POGOProtos\Inventory\InventoryDelta $delta */
$delta = $inventoryResponse->getInventoryDelta();

/** @var \Google\Protobuf\Internal\RepeatedField $items */
$items = $delta->getInventoryItems();

foreach ($items as $item) {
    /** @var \POGOProtos\Inventory\InventoryItem $item */
    /** @var \POGOProtos\Inventory\InventoryItemData $data */
    $data = $item->getInventoryItemData();
    var_dump($data->getPokemonData());
}

Resolved.


ApplicationResources does not work once built for the store

I'm having an issue with our own application, switching localization and reloading the view works in dev, but not QA.  The difference being, we create a side-loadable store build.  If you take the current ApplicationResources sample and produce a store build, you can reproduce the same issue across all sample scenarios.

Seems like you have to add the language via the control panel to the machine/user doing the store build.  This seems problematic, and not how it worked in 8.1.   Any way we can get some confirmation that this is working as intended?


re

Describe your problem and how to reproduce it:
Add a Link to the page with the problem:
Tell us about your browser and operating system:

Browser Name:
Browser Version:
Operating System:

If possible, add a screenshot here (you can drag and drop, png, jpg, gif, etc. in this box):

If you need help with a coding challenge, please visit the freeCodeCamp forum.
If you found an issue or a bug with one of the coding challenges or with the freeCodeCamp platform,  please edit your issue to include the relevant required details.


[cassandra-cql] Cassandra PreparedStatements... again

Requesting to pull changes from #929 which include some commits from #98 and addresses #458.
Reviews are welcome.
Client

[X] insert
[X] update (Correct me if I am wrong, insert is basically an update with just one key-value?)
[X] delete
[X] scan (one)
[x] read (one)
[X] scan (many)
[X] read (many)

Unit Tests
Assuming these should also use PreparedStatements

[x] insertRow
[x] testUpdate
[x] testRead

Not too sure about the support for selecting and scanning more than one field.
I started writing this before I realized that

The fields to be selected aren't always sequentially ordered.
I would need to generate a large number of prepared statements. Like nCr(10,1) + nCr(10,2) + nCr(10,3) + ... + nCr(10,10), or 1023 statements, according to WolframAlpha

for (int i = 0; i < fieldCount; i++) {
  ...
  Select.Selection selectManyBuilder = QueryBuilder.select();;
  for (int j = i; j < fieldCount; j++) {
      selectManyBuilder.column(FIELD_PREFIX + j);
  }
  Select selectMany = selectManyBuilder.from(table)
    .where(QueryBuilder.eq(YCSB_KEY, QueryBuilder.bindMarker())).limit(1);
  PreparedStatement preparedSelectManyStatement = session.prepare(selectMany);
}

Therefore, I think keeping the QueryBuilder within the read and scan functions makes sense. Unless someone has a better idea.

Just thinking out loud, can a query be repeated? How much benefit would it be to create the prepared statement on demand (first time) and then try to reuse the prepared statement if the same query comes in again?
That may give the best of both worlds in terms of not generating them all up front, but at least being able to generate them just in time? I don't know the performance of prepared vs nonprepared statements though and how much effort it takes to generate one.

I do think generating thousands of queries ahead of time would be pointless. Similar thoughts for creating those objects without any known future use; maybe lazily preparing them would be the way to go.
Regarding queries repeating - isn't that the point of YCSB? To hit the database with the same query repeatedly?

@risdenk  Ready for review.

@cricket007 any benchmark numbers to share? Just want to confirm this isn't slower than the original implementation?
Another thought - would it make sense to make it an option to use prepared statements or not? A property perhaps?

@risdenk How would you recommend benchmarking it?
I can setup a few node Cassandra cluster in vagrant, but is one YCSB node enough?

@cricket007 It would be good to try to get at least some performance numbers. A single node is better than nothing.

I was going to work on this over the weekend, but seems my previously working Vagrant / Ansible setup is busted. I'll try to see if I can get Docker working.

@risdenk
Preliminary results
Loaded 50000 records
3 Servers
1 Client
35 threads
Workload E for 20 minutes
Before
[OVERALL], Throughput(ops/sec),<PHONE_NUMBER>237526
[INSERT], 95thPercentileLatency(us), 129727.0
[SCAN], 95thPercentileLatency(us), 197375.0

After
[OVERALL], Throughput(ops/sec), 6894.908356323305
[INSERT], 95thPercentileLatency(us), 6595.0
[SCAN], 95thPercentileLatency(us), 0.0

CPU usage ~20% less

@cricket007 that looks good. Only one question, why is [SCAN] after "0.0" for latency in both runs? [SCAN], 95thPercentileLatency(us), 0.0 That seems really weird to me. Did we lose tracking latency on scans?

Yeah, that surprised me too. Something is cached, so reads are really quick? Probably not since it's zero microseconds, though...
I only grep'd those values, but I can give the rest of the run output.

@risdenk
Found the problem. Didn't check the error output
com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not found for requested operation: [bigint <-> java.lang.String]
        at com.datastax.driver.core.CodecRegistry.notFound(CodecRegistry.java:679)
        at com.datastax.driver.core.CodecRegistry.createCodec(CodecRegistry.java:540)
        at com.datastax.driver.core.CodecRegistry.findCodec(CodecRegistry.java:520)
        at com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:470)
        at com.datastax.driver.core.AbstractGettableByIndexData.codecFor(AbstractGettableByIndexData.java:77)
        at com.datastax.driver.core.BoundStatement.bind(BoundStatement.java:196)
        at com.datastax.driver.core.DefaultPreparedStatement.bind(DefaultPreparedStatement.java:126)
        at com.yahoo.ycsb.db.CassandraCQLClient.scan(CassandraCQLClient.java:468)
        at com.yahoo.ycsb.DBWrapper.scan(DBWrapper.java:176)
        at com.yahoo.ycsb.workloads.CoreWorkload.doTransactionScan(CoreWorkload.java:782)
        at com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:640)
        at com.yahoo.ycsb.ClientThread.run(Client.java:458)
        at java.lang.Thread.run(Thread.java:745)

Seems the bind parameters should be the correct types, or something about a CodecRegistry
String startkey, int recordcount
...
final BoundStatement bs = ps.bind(startkey, recordcount);

@cricket007 Did you have to patch anything for that scan latency issue or is it good to go? Thanks!

Ah, hey @manolama.
Unfortunately, no. I actually just tried reaching out to the DataStax slack channel a few days ago and I got a "that's strange", while dismissing it as a driver issue. I was going to try upgrading that and see what happens...
I've had very limited experience with Cassandra, and mostly just going off of the few PreparedStatements pages on the DataStax documentation, so I really don't know what the error is trying to tell me as I would assume a CQL bigint type would have a conversion to a String, and I'm also not sure where that bigint value is at.
I'm only trying to bind
SELECT FROM usertable WHERE token(y_id) >= ? LIMIT ?;

So, the limit must be a number, and I don't think I changed the syntax of the query before these changes, so the while the startkey is a java.lang.String, I don't know why it would be trying to convert into a CQL bigint...  https://github.com/brianfrankcooper/YCSB/blob/master/cassandra/src/main/java/com/yahoo/ycsb/db/CassandraCQLClient.java#L333-L347

omg... months later and I finally see the difference... 😲
The startkey is being tokened and quoted.
  scanStmt.append("token('");
  scanStmt.append(startkey);
  scanStmt.append("')");

I'll start running this again as soon as I can.

Heh, sounds good, thanks!

Oops. Didn't mean to push that commit.
Well, I believe I  fixed the binding problem in the scan, but now I'm getting NullPointerException in about every usage of getting the fieldList or the Hashmaps containing the PreparedStatements.
The unit tests pass fine, so it's probably a multithreading issue

Definitely a threading issue. Moving around the property initialization blocks out of the synchronized blocks has now resolved those issues.
Now getting errors when trying to log out the BoundStatements, but might just leave out the debug feature for those.

anything we can help with @cricket007?

Hey @busbey, I'm not well-versed in Cassandra, so any assistance would be great.
I can't remember what the current errors are, I can collect those later on my other computer.
But if you are able to pull my changes and try things out, I would gladly take suggestions.

Thanks @busbey , I do have a question actually, which might help me.
When I run JUnit in IntelliJ, is that simulating one YCSB instance?  If so, how can I configure IntelliJ so that there would be multiple threads during testing? That way, I can set breakpoints and watch the necessary variables.
I ask because all my unit tests pass, so I believe the root issue is when there is more than one thread.

@cricket007   NullPointException is not caused by multi-threads.
It is because you wrote the following codes for optimizing Cassandra's performance:
        // Select and Scan few statements
        if (!readallfields) {
....
}

Only readallfileds==false, you initialize updatestatements, that is fine.
However, CoreWorkload.READ_ALL_FIELDS_PROPERTY_DEFAULT=true :(
So, you have to add a parameter in your workload file: readallfields=false
I think we need to write it in Readme.md or provide a workload template for Cassandra.
Besides, I have tested the scan operations:
[SCAN], Operations, 408
[SCAN], AverageLatency(us), 3389.5906862745096
[SCAN], MinLatency(us), 391
[SCAN], MaxLatency(us), 18239
[SCAN], 95thPercentileLatency(us), 7087
[SCAN], 99thPercentileLatency(us), 9343
[SCAN], Return=OK, 408

Besides, I think the insert() function should be:
@Override
  public Status insert(String table, String key,
      Map<String, ByteIterator> values) {

    try {
      final int numValues = values.size();
      int i = 0;
      Object[] vals = new Object[numValues + 1];

      // Add key
      vals[i++] = key;
      // Add fieldList
      for (int j=1; j < fieldList.size(); j++) {
        String field = fieldList.get(j);

        if (values.containsKey(field)) {
          vals[i++] = values.get(field).toString();
        }
      }

      // If there is one value, this is actually an update
      BoundStatement bs = (numValues == 1 ?
               updateStatements.get(values.keySet().iterator().next()) :
               insertStatement).bind(vals);

      if (trace) {
        bs.enableTracing();
      }
      if (debug) {
        System.out.println(">> " + getBoundQuery(bs));
      }

      session.execute(bs);

      return Status.OK;
    } catch (Exception e) {
      e.printStackTrace();
    }

    return Status.ERROR;
  }

the original codes are:
      final int numValues = values.size();
      int i = 0;
      Object[] vals = new Object[numValues + 1];
      // Add key
      vals[i++] = key;
      // Add fieldList
      for (; i < fieldList.size(); i++) {
        String field = fieldList.get(i);
        if (values.containsKey(field)) {
          vals[i] = values.get(field).toString();
        }
      }

it is dangerous that ArrayOutOfBound for vals[]

One more thing, it is better to add a dependency in the core project, because core uses snakeyaml version 1.6 (involved by testng) while Cassandra uses version 1.11 which is not compatible with 1.6.
I can find the conflict in Eclipse IDE (but no conlict in a terminal with mvn command).
@cricket007  If you need, I can fork your codes and then submit a new PR.
Or, I can fork your codes and then submit a PR to your branch first. After you merge my modifications, this PR can be updated automatically.

Thanks for reviewing again @jixuan1989 - If you're able to better pinpoint the issue, feel free to fork mine, or if you are free sometime, we can schedule a call to go over what you think needs to be changed to get it fully working.

I have tested the scan operations:

Yes, I believe I was able to make Scan and Insert (Workload E) work with my latest commits. I started getting busy in April and didn't have time to resolve or throughly test Read and Update (Workload A or B)
I claimed it was a threading issue because moving the synchronized blocks around has resolved one issue I was experiencing.

@cricket007  I have reviewed the codes.
I think your codes are correct for multi-threads scenario.
Cassandra Java Driver supports using one cluster instance and session for multi-threads. However, you have to initialize the cluster synchronically.
However, the synchronized keyword in buildStatements  function is useless.
(I am not sure that whether YCSB consider the init() time as a part of time cost for CRUD. If so, it is unfair for current implementation, and we have to try to find another implementation. Otherwise, the current codes are fine.)
I have correct other codes and merge the codes with latest master branch.
I submit a PR to your repository. You can merge it so that they can be pushed here automatically.

@jixuan1989 Can you explain when you say

Only readallfileds==false, you initialize updateStatements, that is fine.

That isn't true - see this section isn't inside of any if statements
    // Insert and Update all statement
    final Insert is = QueryBuilder.insertInto(table);
    is.value(YCSB_KEY, QueryBuilder.bindMarker());

    updateStatements = new ConcurrentHashMap<>(fieldCount);

That is only used when one value is passed to insert(), though.
  @Override
  public Status insert(String table, String key,
      Map<String, ByteIterator> values) {

    try {
      final int numValues = values.size();
      ...
      // If there is one value, this is actually an update
      BoundStatement bs = (numValues == 1 ?
               updateStatements.get(values.keySet().iterator().next()) :
               insertStatement).bind(vals);

It can be moved outside that if statement, if you think that's the solution; I'm not completely sure what field(s) would be passed to the UPDATE query.

@cricket007  Current YCSB workload inserts ten fields for each insertion command, and just update one field for updating commands.
But.. see the Excetion Stack:

java.lang.NullPointerException
at com.yahoo.ycsb.db.CassandraCQLClient.insert(CassandraCQLClient.java:585)
at com.yahoo.ycsb.db.CassandraCQLClient.update(CassandraCQLClient.java:545)
at com.yahoo.ycsb.DBWrapper.update(DBWrapper.java:198)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransactionUpdate(CoreWorkload.java:775)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:608)
at com.yahoo.ycsb.ClientThread.run(Client.java:454)
at java.lang.Thread.run(Thread.java:748)

You call insert() when YCSB calls update(). That's why the exception appears.
I can try to fix it.

@cricket007  I have fixed the update problem as following:
init(){
...
    updateCommands = Double.parseDouble(p.getProperty(CoreWorkload.UPDATE_PROPORTION_PROPERTY,
        CoreWorkload.UPDATE_PROPORTION_PROPERTY_DEFAULT))==0 ? false : true;
}

buildStatements(){
...
if(updateCommands) {
        // Update - single
        Insert updateOneStmt = QueryBuilder.insertInto(table).values(new String[] {YCSB_KEY, field },
            new Object[] {QueryBuilder.bindMarker(), QueryBuilder.bindMarker() });
        if (!updateStatements.containsKey(field)) {
          updateStatements.put(field, prepare(updateOneStmt.getQueryString(), false));
        }
      }
}

but ... I notice that you close the PR of my branch. If so, you can digest the above part of codes manually.

@jixuan1989 I closed the PR, but I did merge the code. I added some changes on top of it in a different branch.
Anyways, I think the latest commit I made was able to run without error with either value of readallfields, however now I see that there are READ-FAILED events, which I do not think were there before.
By the way, ==0 ? false : true; can simply be written as !=0;

@cricket007

however now I see that there are READ-FAILED events, which I do not think were there before.

Can you give me a screenshot or detailed running log? I did not find the event before.
@robertpang  Your codes are more concise. It uses less API of Datastax Java Driver so that it is more easy to be understood.   Do you notice the same problem (READ-FAILED events)?
No matter which PR is accepted finally, I hope we can accelerate the progress... It is really important that using this better way to benchmark Cassandra.

@robertpang Thanks for the review.

that the way statements are pre-prepared during init() requires the same logic to construct the query strings in buildStatements()

Same logic where? init() is calling buildStatements(). I'm not seeing where anything is duplicated.

in the individual read, scan, insert, update and delete() methods and I have concern about this duplication of codes

In the individual methods, you have similar QueryBuilder creation, then map checking. Is that what you are referring to?
@jixuan1989
CLI Opts:
-db com.yahoo.ycsb.db.CassandraCQLClient
-P /opt/ycsb/workloads/benchmark-workloada
-threads 35
-p maxexecutiontime=1200
-p operationcount=900000000
-p hosts=cassandra01,cassandra02,cassandra03
-p cassandra.connecttimeoutmillis=50000
-p cassandra.writeconsistencylevel=QUORUM
-p cassandra.readconsistencylevel=QUORUM
-tYCSB Client 0.14.0-SNAPSHOT

Workload A
recordcount=150000000
operationcount=150000000
maxexecutiontime=1200

workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian

A have a shell script that looks like this - The gist here is that I split apart the stdout and stderr of ycsb run
YCSB_CMD="ycsb run $YCSB_OPTS"
${YCSB_HOME}/bin/${YCSB_CMD} > $OUTPUT_DIR/${OUTPUT_FILE}.log 2> $OUTPUT_DIR/${OUTPUT_FILE}.err

And when I do something like grep FAILED workloada.log, I see
[READ-FAILED], Operations, 1027229
[READ-FAILED], AverageLatency(us), 7635.317831759034
[READ-FAILED], MinLatency(us), 535
[READ-FAILED], MaxLatency(us), 221823
[READ-FAILED], 95thPercentileLatency(us), 18527
[READ-FAILED], 99thPercentileLatency(us), 29007
[READ-FAILED], 535, 1.0
[READ-FAILED], 539, 1.0
[READ-FAILED], 544, 1.0
[READ-FAILED], 545, 1.0
[READ-FAILED], 570, 1.0
...

That actually was a run off of master. I think I didn't notice that before because of the histogram (?) output at the end. Seems to be a new feature.... It prints for all the methods. For example,
[CLEANUP], Operations, 35
[CLEANUP], AverageLatency(us), 64046.08571428571
[CLEANUP], MinLatency(us), 1
[CLEANUP], MaxLatency(us), 2242559
[CLEANUP], 95thPercentileLatency(us), 8
[CLEANUP], 99thPercentileLatency(us), 2242559
[CLEANUP], 1, 31.0
[CLEANUP], 2, 1.0
[CLEANUP], 8, 1.0
[CLEANUP], 36, 1.0
[CLEANUP], 2242559, 1.0

Anyways, a sample run off this branch, also has READ-FAILED, but the READ numbers are .
master
[READ], Operations, 1642900
[READ], AverageLatency(us), 8385.432664191369
[READ], MinLatency(us), 568
[READ], MaxLatency(us), 221311
[READ], 95thPercentileLatency(us), 19759
[READ], 99thPercentileLatency(us), 30863
[READ], Return=OK, 1642900
[READ], Return=NOT_FOUND, 1027227
[READ], Return=ERROR, 2

[READ-FAILED], Operations, 1027229
[READ-FAILED], AverageLatency(us), 7635.317831759034
[READ-FAILED], MinLatency(us), 535
[READ-FAILED], MaxLatency(us), 221823
[READ-FAILED], 95thPercentileLatency(us), 18527
[READ-FAILED], 99thPercentileLatency(us), 29007

cassandra-preparedstatements
[READ], Operations, 2454539
[READ], AverageLatency(us), 6109.3850209754255
[READ], MinLatency(us), 456
[READ], MaxLatency(us), 190207
[READ], 95thPercentileLatency(us), 15463
[READ], 99thPercentileLatency(us), 25567
[READ], Return=OK, 2454539
[READ], Return=NOT_FOUND, 1460773

[READ-FAILED], Operations, 1460773
[READ-FAILED], AverageLatency(us), 4937.38611132599
[READ-FAILED], MinLatency(us), 376
[READ-FAILED], MaxLatency(us), 136575
[READ-FAILED], 95thPercentileLatency(us), 12647
[READ-FAILED], 99thPercentileLatency(us), 20271

So, for these runs, 49% increase in operations (likely due to lower latency), but a 42% increase in READ failures.

@robertpang Your codes are more concise. It uses less API of Datastax Java Driver so that it is more easy to be understood. Do you notice the same problem (READ-FAILED events)?

@jixuan1989 No. I don't encounter READ-FAILED events in my YCSB run with my changes.

@robertpang Thanks for the review.

that the way statements are pre-prepared during init() requires the same logic to construct the query strings in buildStatements()

Same logic where? init() is calling buildStatements(). I'm not seeing where anything is duplicated.

in the individual read, scan, insert, update and delete() methods and I have concern about this duplication of codes

In the individual methods, you have similar QueryBuilder creation, then map checking. Is that what you are referring to?

@cricket007 Yes. I am referring the duplicative QueryBuilder calls that appear in both buildStatements() and in the individual read, scan, insert, update and delete() methods. It requires them to be in sync with each other.

@robertpang Unless I am mistaken, we both have taken a similar approach for only building the queries when needed.
In mine, I only build a subset of queries in buildStatements(), and depending on the value of readallfields. Regarding your comment, I have a map of maps, where the fieldCount acts as the size of "buckets" for queries built later in read, scan, insert/update.
delete is just a one-off query. I don't imagine a significant overhead between preparing in init() or upon delete() invocation.
Also, according to the DataStax CQL documentation, UPDATE and INSERT are identical, so I didn't see the need to prepare those separately.
// Select and Scan many statements
selectManyStatements.put(i, new ConcurrentHashMap<Integer, PreparedStatement>());
scanManyStatements.put(i, new ConcurrentHashMap<Integer, PreparedStatement>());

I then build the query, check if it exists, then add, which is equivalent putIfAbsent.
The only difference I see is that you have a ConcurrentMap<String, PreparedStatement>, making the query the key, and I have a Map<Integer, PreparedStatement> using the hashcode of the query. Hence my comment saying that I hope those are unique enough to prevent collisions. I couldn't think of any immediate alternative.
/* read() */
Map<Integer, PreparedStatement> possibleStatements = selectManyStatements.get(fields.size());

Select.Builder selectBuilder = QueryBuilder.select();
for (String col : fields) {
  ((Select.Selection) selectBuilder).column(col);
}
String statement = selectBuilder.from(table)
    .where(QueryBuilder.eq(YCSB_KEY, QueryBuilder.bindMarker())).getQueryString();

int statementHash = statement.hashCode(); // Hoping these are unique enough between statements
if (!possibleStatements.containsKey(statementHash)) {
  possibleStatements.put(statementHash,  prepare(statement, true));
}
ps = possibleStatements.get(statementHash);

@robertpang

No. I don't encounter READ-FAILED events in my YCSB run with my changes.

Can you provide your property file? Because I pulled your code, ran mvn -pl com.yahoo.ycsb:cassandra-binding clean package -DskipTests=true, overwrote my YCSB directory with that package, did another run with the properties mentioned above, and still am seeing READ-FAILED.
Which again, I am seeing the same when I checkout master branch, so either I have done something wrong on that end (running bad code), there is a property that I have not accounted for, or I should recreate the keyspace and load the data again.

I reproduce the READ-FAILED error.

[READ-FAILED], Operations, 150384
[READ-FAILED], AverageLatency(us),<PHONE_NUMBER>9545696
[READ-FAILED], MinLatency(us), 117
[READ-FAILED], MaxLatency(us), 48415
[READ-FAILED], 95thPercentileLatency(us), 298
[READ-FAILED], 99thPercentileLatency(us), 396
[READ-FAILED], 117, 1.0
[READ-FAILED], 118, 2.0
[READ-FAILED], 120, 2.0
[READ-FAILED], 121, 6.0
[READ-FAILED], 122, 8.0
[READ-FAILED], 123, 5.0
[READ-FAILED], 124, 11.0
[READ-FAILED], 125, 16.0
[READ-FAILED], 126, 22.0

I need more time to find out why..

@cricket007

Also, according to the DataStax CQL documentation, UPDATE and INSERT are identical, so I didn't see the need to prepare those separately.

There are some subtle semantic differences between INSERT and UPDATE not covered in DataStax's documentation. A couple of examples are like when TTL is involved as described here or when a non-key column is set to null (here). A properly written Cassandra application will want to use INSERT vs UPDATE properly and so it is better for YCSB Cassandra binding to reflect how those applications use those statements.

@robertpang Unless I am mistaken, we both have taken a similar approach for only building the queries when needed. In mine, I only build a subset of queries in buildStatements(), and depending on the value of readallfields. Regarding your comment, I have a map of maps, where the fieldCount acts as the size of "buckets" for queries built later in read, scan, insert/update. delete is just a one-off query. I don't imagine a significant overhead between preparing in init() or upon delete() invocation.

@cricket007 Yes, our approaches are similar but I think we can minimize the changes just preparing the statements on-demand and caching them as they are issued from read, scan, ... methods. This way the code changes will be more concise and easy to review and verify for correctness. There may be performance benefits of pre-preparing the statements ahead of time in the init() method but the benefits will be marginal and diminish as YCBS runs for longer and longer period of time.

@cricket007 I don't use any property file but here are the parameters I specify on command-line:
ycsb run cassandra-cql -p hosts= -P workloads/workload... \
-p recordcount=1000000 -p operationcount=10000000 \
-p cassandra.readconsistencylevel=QUORUM -p cassandra.writeconsistencylevel=QUORUM \
-p threadcount=256 -p maxexecutiontime=90
I ran the 6 core workloads.

@robertpang I've ran your branch multiple times now, and even rebuilt my cluster to use the latest DSE version, and I still get READ-FAILED on just workload A. Even using the CLI options you gave.
That being said, my setup is now a 3 node 5.1.5 cluster in VirtualBox, installed by Vagrant, which I've done all the recommended tuning onto.
Any ideas?

@robertpang I've ran your branch multiple times now, and even rebuilt my cluster to use the latest DSE version, and I still get READ-FAILED on just workload A. Even using the CLI options you gave.
That being said, my setup is now a 3 node DSE 5.1.5 (Cassandra <IP_ADDRESS>0) cluster in VirtualBox, installed by Vagrant, which I've done all the recommended tuning onto.
Any ideas?

@cricket007 Since you encounter READ-FAILED with the master branch, I guess you have a general issue with your setup. I am unsure what may have caused this. Can you catch the  exception that causes READ-FAILED and dump further info and see?

@cricket007
I'd say your PR is more focused than #930. You only concern yourself with the core workload value for readallfields=true. You therefore aren't reading fieldcount from the properties, nor do you care about checking fields.size() in any of the methods. Only insert & update use values.size().

Thanks for your review. I think there is a misunderstanding. My PR does not assume any specific workload. That is the reason why it does not depend on any of the core workload properties. On the other hand, your PR #930 depends heavily on those properties and pre-prepares the statements in init() specifically for core workload. For other 3rd-party workloads, say one that updates 2 fields in one UPDATE statement (possible by writing their own Java class instead of using CoreWorkload.java), PR #930 will fall back to preparing the statement on-the-fly. So your PR is essentially special-casing for core workloads.

Here is an example of a non-core time-series workload in YCSB that reads a random set of fields (not all and not one) for your reference.

Closing for #1051


Csh

Fixes #<issue_number_goes_here> (it's a good idea to open an issue first for discussion)

[ ] npm test succeeds
[ ] Pull request has been squashed into 1 commit
[ ] I did NOT manually make changes to anything in apis/
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate JSDoc comments were updated in source code (if applicable)
[ ] Approprate changes to readme/docs are included in PR

Current coverage is 88.56% (diff: 100%)

Merging #653 into master will not change coverage

@@             master       #653   diff @@
==========================================
  Files             8          8
  Lines           271        271
  Methods          41         41
  Messages          0          0
  Branches         59         59
==========================================
  Hits            240        240
  Misses           31         31
  Partials          0          0

Powered by Codecov. Last update 5551fdf...ae18eb0


5/24 pm publish

Docs Build status updates of commit 1ffb029:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-conceptual/azps-7.5.0/azureps-support-lifecycle.md
:white_check_mark:Succeeded
View (azps-7.5.0)

docs-conceptual/azps-8.0.0/azureps-support-lifecycle.md
:white_check_mark:Succeeded
View (azps-8.0.0)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


mBART fails to save checkpoint

🐛 Bug
Hi,
I am running an mBART model to summarize Turkish news on Google Colab. I have mostly followed the instructions at the official example but I have changed source language to news corpus to be summarized and target language to the summarized version of the news. When I begin to train my model it abruptly quits after this line:
2021-01-05 16:49:16 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 1 @ 193 updates
then it prints:
^C.
This is keyboard interrupt however I do not invoke it.
I tried not saving a checkpoint and going on with epoch 2 and there was no problem. I also know the loss is decreasing and the sentencepiece tokenization seems fine when I inspect it.
To Reproduce
Run this notebook. It will take around 15 minutes to load the necessary files and get to the error.
Code sample
This is the training code:
!fairseq-train /content/pre/
--encoder-normalize-before --decoder-normalize-before
--arch mbart_large --layernorm-embedding
--task translation_from_pretrained_bart
--source-lang src --target-lang tgt
--criterion label_smoothed_cross_entropy --label-smoothing 0.2
--optimizer adam --adam-eps 1e-06 --adam-betas '(0.9, 0.98)'
--lr-scheduler polynomial_decay --lr 3e-05 --warmup-updates 25 --total-num-update 400
--dropout 0.3 --attention-dropout 0.1 --weight-decay 0.0
--max-tokens 256 --update-freq 2
--save-interval 1 --save-interval-updates 5000 --keep-interval-updates -1
--seed 222 --log-format tqdm --log-interval 8
--restore-file /content/mbart.cc25.v2/model.pt
--reset-optimizer --reset-meters --reset-dataloader --reset-lr-scheduler
--langs ar_AR,cs_CZ,de_DE,en_XX,es_XX,et_EE,fi_FI,fr_XX,gu_IN,hi_IN,it_IT,ja_XX,kk_KZ,ko_KR,lt_LT,lv_LV,my_MM,ne_NP,nl_XX,ro_RO,ru_RU,si_LK,tr_TR,vi_VN,zh_CN
--ddp-backend no_c10d --memory-efficient-fp16  --checkpoint-suffix den
Expected behavior
It should have saved a checkpoint and carried on with the training.
Environment

fairseq Version : current one
PyTorch Version : 1.7.0+cu101
OS (e.g., Linux): Linux - Google Colab
How you installed fairseq (pip, source): pip from GitHub clone
Build command you used (if compiling from source):
Python version: 3.6.9
CUDA/cuDNN version: 10.1
GPU models and configuration: Tesla T4

This might be because Colab does not have enough memory to save the checkpoint. I was getting a similar error when Colab was low on RAM/disk space.


any audio alternate for android 9 + windows setup

anything will be helpful, Thanks

No.
(#14 is the issue for audio forwarding)


Using relative path?

All the ajax calls in my app's code are using relative paths (since the hostname can change, depending on the user's account).
For example:
$.ajax({
  type: "GET",
  url: "/posts/",
  dataType: "json"
});

Is there a way to make nock catch these calls? I am using karma + mocha as my test runner / library

nock is designed to work on node.
browser is still limited to require('http') in browserify, and still with some problems.
Direct AJAX is not supported.

Just so everyone curious, i'm just using: nock(/.*/).get('/absolute/path') and that seems to work well without a hostname.

Found a workaround for this. Have a _ajax-setup.js in your test folder
import $ from 'jquery'

$(document).ajaxSend((event, jqxhr, settings) => {
  settings.url = 'http://<IP_ADDRESS>' + settings.url;
})

and test with
nock('http://<IP_ADDRESS>')
.get('/posts/')
.reply(200, {data: "hello"})

I modified my code to query:
document.location.host + '/my/path';

and use global.document.location.href = 'http://fake.host/'; in my tests.

Anyone has an idea how to do this when using isomorphic-fetch (a browser fetch implementation) without prefixing every request with the hostname? I'd like not to have to add this in every ajax request I have to do just for the sake of the tests.

While not ideal, as a workaround I turned it around and prefixed the urls when testing.
For this, I check NODE_ENV, which is set to test when testing.
Brief example:
const testing = process.env.NODE_ENV === 'test';
const apiUrl = `${testing ? 'http://localhost' : ''}/api`;

function getStuffFromApi() {
  return fetch(apiUrl, ...)
}

This way I can always use nock like this:
nock('http://localhost')
  .get('/api')
  .reply(200, {data: 'hello'})

I ran into this exact problem and solved it by using the fetch-mock library instead of nock:
https://github.com/wheresrhys/fetch-mock/

You should be able to monkey patch the library. It's not great to pollute your code with testing conditions, so I would much rather use the approach below.
If you are using Jest you could do something like this:
const fetch = require.requireActual('isomorphic-fetch');

module.exports = (url, config) => {
  return fetch('https://localhost' + url, config);
};

and put it into __mocks__\isomorphic-fetch.js Jest then automatically replaces all requires (or imports) for this node module.
In other test frameworks you could use something like rewire

Can node-nock intercept ajax calls? As I read It does not. But from these comments it looks like can. Does it?


Event list item text overflow

What happened?
If the content of an event list item is longer than the available width it will not be truncated.
What type of frontend frameware are you seeing the problem on?
JavaScript
Which version of iX do you use?
2.0.4
Code to produce this issue.
Basic example from iX documentation

Handling of overflow for slots is the responsibilty of the developers and will be documented


feat(Pagination): add Pagination component

References #77

Add Pagination, PaginationItem, PaginationLink
Pagination supports sizing
PaginationItem supports active and disabled states
PaginationLink supports special rendering for next and previous links

Coverage remained the same at 100.0% when pulling 3c906b97ad8f0b5dac6e60d7c3c3468fd51cd5a8 on mking-clari:pagination into 776c8a0c9cfaf2ae623145d7d47a88d1c83d4ba5 on reactstrap:master.

Coverage remained the same at 100.0% when pulling 3c906b97ad8f0b5dac6e60d7c3c3468fd51cd5a8 on mking-clari:pagination into 776c8a0c9cfaf2ae623145d7d47a88d1c83d4ba5 on reactstrap:master.

A few comments, let me know what you think. Great work!

Coverage remained the same at 100.0% when pulling dfbd977be11d5f5491f4030967c7f5ea162cabc3 on mking-clari:pagination into 776c8a0c9cfaf2ae623145d7d47a88d1c83d4ba5 on reactstrap:master.

Coverage remained the same at 100.0% when pulling a34d5cb194edcc2d33c78473cc9db0700682f5be on mking-clari:pagination into 776c8a0c9cfaf2ae623145d7d47a88d1c83d4ba5 on reactstrap:master.

Coverage remained the same at 100.0% when pulling 9850e4f7ce9f3c01294390f5ccbd974894597611 on mking-clari:pagination into 776c8a0c9cfaf2ae623145d7d47a88d1c83d4ba5 on reactstrap:master.

@eddywashere Thanks for the feedback, I updated the code.
Some notes:

If previous or next is set, the children prop replaces just the caret, not all the spans. If the user wants full customization of the spans, they can just not set previous or next.
The aria-label prop is used for both the aria-label attribute and the .sr-only text.

Coverage remained the same at 100.0% when pulling d2570bb000d81eb5cbdfe6afb58518336c75aea5 on mking-clari:pagination into 776c8a0c9cfaf2ae623145d7d47a88d1c83d4ba5 on reactstrap:master.

@eddywashere For the « issue, I replaced the html entity with the unicode character (\u00ab), which should avoid the escaping issue. Also updated the test for consistency.
For the span issue, I added an array (keys required to avoid warning).

Coverage remained the same at 100.0% when pulling 8fa5044a81ce392c4c42dde5ffa28b564df4b45c on mking-clari:pagination into 776c8a0c9cfaf2ae623145d7d47a88d1c83d4ba5 on reactstrap:master.

👍


Consider giving a dockerfile to prevent errors caused by environmental problems？

No. A docker file is way to heavy for these simple scripts.
When something is missing (e.g. gcc) the error message will tell you.


Set Date from JS only

Is it possible to select a date from js directly without using model? Couldn't find any methods to achieve this.

This seems like a Vue question
Sorry if I'm misunderstanding you, but have you tried using value prop?

Yes, tried using value(). It does set date, but it does it on UI only, it doesn't bind to datepicker. Because as soon I open datepicker it gets to default again despite setting from value prop.

@nitin-jotwani you can use this.$refs.picker.setDate(new Date()); for that (where picker is the ref of the datepicker html object)


 This PR addresses a few of those concerns.
Fixes #11558
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Manually for visual changes, and yarn test:unit for unit tests.
Changes
Changes so far:

Faster tooltips for bottom bar quick actions
Branch popover has a backdrop like the modals do.
More than 9 changes are reported as it is
Recent commit shows "less than a minute" ago
BETA tag removed from popover
Faster tooltips for sync branch button in popover
Placeholder for commit message
Relevant placeholder for git remote input
Padding for content in modal is decreased
Notification banner is aligned with SSH key box

https://user-images.githubusercontent.com/1573771/157437661-0d00675f-91de-4400-aa35-d6a1475f07b2.mov
https://user-images.githubusercontent.com/1573771/157438263-904674cd-e0f0-4510-8a9c-7b14ffe08e20.mov

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:red_circle: Total coverage has decreased
// Code coverage diff between base branch:release and head branch: 11558-polish-git-sync

Status
File
% Stmts
% Branch
% Funcs
% Lines

:red_circle:
total
55.87 (-0.01)
37.23 (0)
35.25 (0.01)
56.2 (-0.01)

:green_circle:
app/client/src/ce/constants/messages.ts
76.65 (0.03)
100 (0)
30.75 (0.11)
81.96 (0.04)

:green_circle:
app/client/src/pages/Editor/gitSync/QuickGitActions/index.tsx
65.71 (-0.64)
44 (1.69)
34.78 (0)
67.68 (-0.69)

:red_circle:
app/client/src/pages/Editor/gitSync/components/BranchList.tsx
23.49 (-0.46)
10.13 (0)
2.5 (0)
25.5 (-0.5)

:red_circle:
app/client/src/utils/WorkerUtil.ts
88.98 (-0.78)
70.59 (0)
100 (0)
92.38 (-0.95)

:red_circle:
app/client/src/utils/helpers.tsx
67.47 (-0.47)
34.09 (-0.79)
58.82 (0)
65.64 (-0.58)

:x:
app/client/src/pages/Editor/gitSync/components/BetaTag.tsx
75
100
0
75

/ok-to-test sha=90b6432


AES CTR + HMAC: Never return all data in case HMAC at very end does not match expected HMAC

This patch addresses the issue where the AES decryption has returned
all decrypted data to the client and upon the next Read() from the
cipherstream receives the EOF and 0 bytes but then detects that the HMAC
is not the expected one. In this case the next layer dealing with the
decrypted data would have all the data and could continue. With the
DelayedReader the next layer will run into an error since data were
held back from it.
For this we implement a DelayedReader that allows one to wrap an existing io.Reader
and report any errors that that io.Reader may report (at the end)
early to the client while holding back some data from the client.

Impact on performance is negligible even though a lot more copying is going on now:
time CONTAINERD=/home/stefanb/src/github.com/containerd/containerd/bin/containerd ./script/tests/test_encryption.sh

Before:
real	0m57.130s
user	1m18.231s
sys	0m16.909s

After:
real	0m59.193s
user	1m19.723s
sys	0m17.573s

Pulling of an encrypted image whose layers have already been downloaded before (using same command line as below without --key):
time ./ctr images pull --all-platforms --key ~/src/github.com/containerd/containerd/mykey.pem localhost:5000/ubuntuenc:1

Before:
real	0m36.907s
user	0m0.230s
sys	0m0.115s

After:
real	0m37.517s
user	0m0.217s
sys	0m0.094s

Thanks!


Fix broken link

src/README.md had some outdated links.
I updated the settings-ui link, and removed the settings-web one
How does someone test / validate:
I tried the links and they worked in my repo
Quality Checklist

[ ] Linked issue: #xxx
[ ] Communication: I've discussed this with core contributors in the issue.
[ ] Tests: Added/updated and all pass
[ ] Installer: Added/updated and all pass
[ ] Localization: All end user facing strings can be localized
[X] Docs: Added/ updated
[ ] Binaries: Any new files are added to WXS / YML

[X] No new binaries
[ ] YML for signing for new binaries
[ ] WXS for installer for new binaries

Contributor License Agreement (CLA)
A CLA must be signed. If not, go over here and sign the CLA.
I've signed this.

@MikeTheGreat
Thank you for your contribution. Can you please fill an issue for this change.

@MikeTheGreat @Aaron-Junker this section still mentions html, so there must be more corrected than just the link.

This should have been done with #13155

Jay, agreed but it was missed, still should be tracked as new issue.  can xref the old issue


Example of export (dump) to json file

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

-1 on this one. We want to keep README concise.


Please, consider using github for issues. Current community management is perfectible

Hi there,
I like snowpack, but don't like really the way you handle the community.

I don't want to use yet another platform for issue management.
I won't visit your forum to see notifications to my messages, and don't want email spam neither, so I won't see messages there.

Also,
The first time I used snowpack, I found several bugs. The first bug I reported was https://github.com/pikapkg/snowpack/issues/362, but it was closed after being moved to your discuss forum. Nothing was done nor acknowledged despite very easy reproduction (just try to import any file name 'a.b.c.ts' and see what happens) and a code sample given.
You even disabled feedback on this post redirecting people to discuss which is not great IMO.

So I think I won't be contributing for now.
Snowpack is very interesting, but the current implementation is quite buggy / immature:

lots of hard-coded things, config, paths.
no really usable API to work with custom repository setups
quite poor documentation.

I feel like the lack of issue reporting is related to your issue tracker decision.
I hope you will reconsider your custom issue tracker forum thing to keep everything on github, so more contributors will jump in to help this great project grow and mature.

I also had the same thoughts and posted it in their our discussion channel. https://www.pika.dev/npm/snowpack/discuss/503
I'm looking forward to understanding their motivation for an alternative issue tracker.
I think they are doing more harm than good moving away from GitHub.

I'm sorry that you find it frustrating. If it helps, we're waiting for GitHub Discussions to reach a public beta and will then be switching over to that ASAP. /cc @gr2m do you know anyone at GH who could help us get access? Happy to provide feedback.
In the meantime, please know that our current workflow does offer value over GH alone:

Maintainers: GH issues are 100% prioritized, triaged issues. Easier to manage.
Contributors: Easy to find new issues to work on, vs. repos with hundreds of open issues.
Issue Creators: Discussions are free-flowing, "closing" discussions is unnatural and leads to hurt feelings (for example, I'm going to close this issue now since it's not a prioritized bug/software issue, which I have a feeling you're not going to like :)

I'll be the first one to admit that it's not perfect! But, it works for us for now as a stopgap until we get access to GH Discussions. I appreciate your patience until then.

@gr2m do you know anyone at GH who could help us get access? Happy to provide feedback.

on it

AH that would be AMAZING!

AH that would be amazing! Happy to provide any useful data/info to help.

you should be all set

Amazing, thanks @gr2m!


FedGDO and its variations

New mpl methods:
FedGDO stands for Federated Gradient Double Optimization.
This method is inspired from Federated gradient, but with modification on the local computation of the gradient.
In this version we use a local optimizer (partner-specific) to do several minimization steps of the local-loss
during a minibatch. We use the sum of these weighs-updates as the gradient which is sent to the global optimizer.
The global optimizer aggregates these gradients-like which have been sent by the partners,
and performs a optimization step with this aggregated gradient.
Here three variations of this mpl method are tested.

FedGDO with fresh local optimizer at each minibatch
FedGDO with persistent local optimizer
FedGDO with persistent and informed local optimizer

These methods are tested on this notebook -> https://colab.research.google.com/drive/1CcQpWRpLGldj3iNR7v7Hv2brdwBP3z7D?usp=sharing
Please note that as I am currently working with the notebook, it can change, and be not fully readable
Notebook access is currently limited to substra.org, but don't hesitate to come to me for access.

This draft was behind master from too much commits to be easily rebased. I created a new branch, in a new PR
See PR #345


Incorrect updating of active entity

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request
[ ] Other... Please describe:

Current behavior
When I update the active entity the id property on the state object receives a copy of the new entity state
Expected behavior
The entity should be updated with the values from the partial object.
Minimal reproduction of the problem with instructions
import { Injectable } from "@angular/core";

import { ActiveState, EntityState, EntityStore, ID, StoreConfig, transaction } from "@datorama/akita";
import { NGXLogger } from "ngx-logger";

import { FormStatus, IForm } from "@models";

export interface IFormsState extends EntityState<IForm>, ActiveState { }

@Injectable()
@StoreConfig({ name: "forms" })
export class FormsStore extends EntityStore<IFormsState, IForm> {
  constructor(private logger: NGXLogger) {
    super({ active: null });
  }

  @transaction()
  public startForm(formId: ID) {

    this.setActive(formId);

    this.updateActive({ status: FormStatus.inProgress });
  }
}

Original entity JSON:
{
  "isValid": false,
  "id": 1,
  "status": -1,
  "consentConfirmations": [
    {
      "hasAcknowledged": false,
      "text": "Display text 1"
    },
    {
      "hasAcknowledged": false,
      "text": "Display text 2"
    },
    {
      "hasAcknowledged": false,
      "text": "Display text 3"
    }
  ],
  "displayName": "Electronic Consent",
  "footerText": "Awesome footer text",
  "ordinalPosition": 0,
  "formComponentType": "consent",
  "text": "Display text here"
}

After calling update the selectActive method from the entity query returns this:
{
  "isValid": false,
  "id": {
    "isValid": false,
    "id": 1,
    "status": 1,
    "consentConfirmations": [
      {
        "hasAcknowledged": false,
        "text": "Display text 1"
      },
      {
        "hasAcknowledged": false,
        "text": "Display text 2"
      },
      {
        "hasAcknowledged": false,
        "text": "Display text 3"
      }
    ],
    "displayName": "Electronic Consent",
    "footerText": "Awesome footer text",
    "ordinalPosition": 0,
    "formComponentType": "consent",
    "text": "Display text"
  },
  "status": -1,
  "consentConfirmations": []
}

What is the motivation / use case for changing the behavior?

This appears to be a very basic use case that I'm attempting. I've seen it in a number of examples but for some reason it does not work for me.
Environment

Angular version: 7.2.7
Akita version: 3.2.0

Browser:
- [x] Chrome (desktop) version XX
- [ ] Chrome (Android) version XX
- [ ] Chrome (iOS) version XX
- [ ] Firefox version XX
- [ ] Safari (desktop) version XX
- [ ] Safari (iOS) version XX
- [ ] IE version XX
- [ ] Edge version XX

For Tooling issues:
- Node version: 10.15.3
- Platform:  Windows

What should the status be after the update?

@NetanelBasal I'm not sure if this is a bug or something that needs to be documented better.
I haven't wired up my HTTP service yet so in one of my services I hand back an array of IForm objects where each object is actually an instance of a class. I made the classes as a way to help ease the burden of making a bunch of hardcoded data objects. That way the constructors could take some of the initialization burden off of me. :)
It seems like line 201 is calling the constructor of the object to make the new state object. Is that a correct reading? If that is the case the first parameter of the constructor for my class is the id field. When I saw this it hit me why a copy of the object was getting shoved in there.
I made all the constructors parameter-less in an attempt to solve the issue. This didn't work as well. It just set the state to a default object instance. It then occurred to me that since the code is passing in the new state to the constructor that it is essentially expecting any object to be a copy constructor. Once I did this it appears to work correctly. Is that a correct assumption?
If classes are expected to have a copy constructor that is fine. It would be nice to have this documented or called out.

When working with classes, you must use one object parameter in the constructor. I also not recommend using classes. Read the section about it in the docs:
https://netbasal.gitbook.io/akita/general/class-support

I have added a paragraph about the class constructor. Thanks.

@NetanelBasal, did that page get added before or after I logged this ticket? I closed all the tabs with the docs that I was looking at when working the issue so I don't know if I had v3.0.0 selected or v3.2.0. I feel silly if that was there the entire time...
Thanks again!

It was in earlier versions, and now I added it back. Don't feel silly :)


Bump ecto version to 3.0

Upgrading to ecto 3.0 doesn't seem to break anything.
Also, after merging we need to publish the new version to hex.

It would be great to see this merged and pushed to hex 😉


progress not coming.

@Satish-Eruvaka  Can you describe it in more detail? Give me some more information.

You can open it anytime if you have any questions


Rewrote C code with C++ extensions

Rewrote the C backend with our new C++ extensions. Much less config, much less less dependencies (-cffi), much less code, much more happy.
Test plan:

Ran existing tests,
Loaded an mp3 and wav file, and re-saved them. Listened to them and verified they're the same.

@soumith

Will add more tests. Still WIP.


Implement contentsOfDirectoryAtPath

Implementing NSFileManager::contentsOfDirectoryAtPath() and NSFileManager::subpathsOfDirectoryAtPath().
Error messages aren't currently an exact match for the standard Foundation library, which throws:
Fatal error: Error raised at top level: Error Domain=NSCocoaErrorDomain Code=260 "The folder “notexist” doesn’t exist." UserInfo={NSFilePath=/Users/euan/Desktop/notexist, NSUserStringVariant=(
    Folder
), NSUnderlyingError=0x1007019f0 {Error Domain=NSOSStatusErrorDomain Code=-43 "fnfErr: File not found"}}: file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-7<IP_ADDRESS>/src/swift/stdlib/public/core/ErrorType.swift, line 56
(lldb)

Whereas this commit throws:
fatal error: Error raised at top level: Error Domain=NSCocoaErrorDomain Code=260 "The file notexist couldn’t be opened because there is no such file." UserInfo={NSFilePath=/Users/euan/Desktop/notexist}: file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-7<IP_ADDRESS>/src/swift/stdlib/public/core/ErrorType.swift, line 56

However, I lack the knowledge to know what I'm missing with the NSError.
Otherwise, the test cases pass and this should hopefully work.

Overall this looks pretty good, with the minor comment on ._, which is worth testing.
Can you rebase this on top of master and into a single commit?

Not a problem, I'll do so tonight once I've tested the ._ issue.

Rebased and squashed, hopefully that's all fixed! Your comments about the ._ seem to have been lost, but basically the docs here seem to be wrong. The comments state that This method also does not return URLs for the current directory (“.”), parent directory (“..”), or resource forks (files that begin with “._”) when in actual fact files starting with ._ are indeed returned. This PR copies that behaviour, but removes that comment.

Fixed the concerns regarding .destroy(). Any other possible blockers?

looks pretty reasonable overall to me; just need to run tests before merging

Fixed. I haven't tested on Linux, I'm setting up a VM to do so now.

yep, seems to be good now.

:+1:


Connect with GCPE to access licensed image service

Describe the issue
GCPE Government Communications and Public Engagement has a licensed image service
Which Sprint Goal is this issue related to?

Note: 'Milestone' is a ZenHub term that we use synonymously with 'Sprint'.

The 'Milestone' description (created in ZenHub) should clearly list Sprint Goals. This section should indicate which 'goal' this issue is related to.
Additional context
Add any other context, attachments or screenshots
Definition of done

replace slide deck material

Reached out to GCPE awaiting answer

No longer relevant


Form validation

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
In the user registration form, numeric values are also accepted however it should accept only characters and spaces.
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a SWOC24 contributor
[X] I want to work on this issue

assign me this issue

Assigned this issue to @Sanjana787 , kindly drop a star on this repo
HAPPY CODING !

@dakshsinghrathore, if this task hasn't been completed yet, kindly assign it to me.


Remove running checkpoint experiment workspace race condition code from plots

3/5 main <- #2877 <-  #2888 <- this <- #2889 <- #2890
Housekeeping follow-up. This PR removes code that stopped the plots webview workspace entry from getting ahead of the most recently fetched checkpoint revision when a checkpoint experiment was running in the workspace.

The number on the plots icon in the experiments table changing is normal right (seen in the UI tests)?


Player moves slower when the mouse is inside the circle

Add me to contributors also

@giongto35 Could you please solve the current merge conflicts?

Closing it as merged on #54. Thanks for the PR @giongto35 :)


Switch to generator-jhipster v5.8.1 branch

For the next release, we should stick to generator-jhipster v5.8.1
We should start a new branch 'develop' or 'spring-boot_2.1.0' branch

Please make sure the below checklist is followed for Pull Requests.

[ ] Travis tests are green

[ ] Tests are added where necessary

[ ] Documentation is added/updated where necessary

[ ] Coding Rules & Commit Guidelines as per our CONTRIBUTING.md document are followed

I'm merging this.
Hope you're OK, @sahbi-ktifa. We can discuss about it this morning :-)


NTR: database extract, transform, and load process

Please could someone consider whether the term IAO term
'database extract, transform, and load process' IAO:0000650
could be imported into OBI  as a child of 'planned process' OBI:0000011
Many thanks,
Varsha

OBI is currently using an outdated version of IAO. We plan to update this in our next OBI release, at which point it should be automatically appear under 'planned process'.

Great, thanks

This term was added to OBI in commit b76068195d619f0da32d653389c2bc0b70c1adf2 on 2017-07-24.


Fix blockquote line height on iOS

Details
This PR fixes line height inside blockquote on iOS.
When lineHeight style is set on MarkdownTextInput, React Native adds NSParagraphStyle with minimumLineHeight and maximumLineHeight to the attributed string.
In Live Markdown, blockquote indent is set using another NSParagraphStyle with firstLineHeadIndent and headIndent.
Currently, we create NSParagraphStyle using [NSParagraphStyle new], ignoring the NSParagraphStyle created by React Native.
This PR uses the NSParagraphStyle set by React Native when creating NSParagraphStyle for blockquotes.

Before
After

Related Issues

GH_LINK
Manual Tests
When testing, please set lineHeight: 30 inside styles.input in App.tsx.
Linked PRs

Waiting for @289Adam289's review

BUG example app iOS:
Toggle text font size button does not actually change size.
https://github.com/user-attachments/assets/fa6ee113-5e9d-41b3-9a91-900c029c112e

@289Adam289 As for https://github.com/Expensify/react-native-live-markdown/pull/587#issuecomment-2545794089, this is not a regression since it also happens on current main (8b1789a):
https://github.com/user-attachments/assets/0a9bdc2d-3e58-43ef-9b5e-bc85220492fb

@289Adam289 I've narrowed down the root cause of this issue to those lines:
https://github.com/Expensify/react-native-live-markdown/blob/f165434a62852973f1e46b6271a49c8928d73cbe/apple/RCTTextInputComponentView%2BMarkdown.mm#L56-L63

I've checked https://github.com/Expensify/react-native-live-markdown/pull/587#issuecomment-2545794089 on main and indeed it is not a regression. I will test this pr on E/App and if everything goes well I think we can merge it.

I've tested this pr on E/App and I could not find any regressions. We can safely merge.

Fixed baseline offset jumps when adding heading in fc1d1e9

Before
After

Some video recordings from E/App with visible improvements:

Before
After


ImportError: cannot import name 'ArangoClient'

When i import ArangoClient from arango, It raise
from arango import ArangoClient
Traceback (most recent call last):

  File "<ipython-input-30-fb5a35ea2f64>", line 1, in <module>
    from arango import ArangoClient

ImportError: cannot import name 'ArangoClient'

then dir(arango)
dir(arango)
Out[29]:
['Connection',
 '__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 'aql',
 'c',
 'clients',
 'collection',
 'core',
 'create',
 'cursor',
 'db',
 'document',
 'edge',
 'exceptions',
 'http_clients',
 'index',
 'mixins',
 'response',
 'utils']

Hi yiyisan,
I believe you've installed the wrong package. I know... there are quite a few out there and it's rather confusing. Make sure you've uninstalled all other arangodb drivers and install python-arango:
pip install python-arango

Thanks,
Joohwan

@joowani  Thank you for replying!
I'm  surely that installed correct packages
Requirement already satisfied (use --upgrade to upgrade): python-arango in ./anaconda3/lib/python3.5/site-packages Requirement already satisfied (use --upgrade to upgrade): six in ./.local/lib/python3.5/site-packages (from python-arango) Requirement already satisfied (use --upgrade to upgrade): requests in ./anaconda3/lib/python3.5/site-packages (from python-arango)

thei reason is nstalled the  version  from GitHub:  after uninstalling ,it is ok now!


Update Switch state data conditions

Signed-off-by: Tihomir Surdilovic<EMAIL_ADDRESS>Many thanks for submitting your Pull Request :heart:!
Please specify parts this PR updates:

[x] Specification
[x] Examples
[x] Schema
[ ] Roadmap
[ ] Use Cases
[ ] Community
[ ] TCK
[ ] Other

What this PR does / why we need it:
Updates switch state data conditions to use a jsonpath condition expression rather than having to specify "value" and "operator" properties.
Special notes for reviewers:
Additional information (if needed):

@ricardozanini thanks for review - fixed all suggestions. I will do a separate pr to change all ' to ` as you have it.


Adding build script for windows along with tests in meta yaml

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [20, 21, 24, 25, 28, 29, 33]

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender