[FEATURE] Salvage volumes when there is no disk space

Is your feature request related to a problem? Please describe
Our disks are full and we can't salvage volumes to delete some data to reclaim disk space.
Describe the solution you'd like
It would be perfect if Longhorn would be able to salvage volumes even when there is no available disk space (maybe use /tmp directory for the salvaging process)
Describe alternatives you've considered
Another manual solution would be to create a new PV then mount the existing volume using Docker and then copy all the data manually.

@iosifnicolae2 what version do you use?

@innobead v1.3.1

Also, it would be awesome if we can delete snapshots to trigger the compaction process so we free up actual disk space (this would make it much easier to fix incidents caused by high write workloads like databases or Kafka)

are there some workarounds to save the volume now?

Any updates? I met exactly same issue and would like to know a way to recover the data and free up disk space.


Remove unused permissions

If spark CI needs these again someday, they can be added by spark CI.

Trying a strict run here: https://teamcity.mesosphere.io/viewQueued.html?itemId=831604&tab=queuedBuildOverviewTab


RuntimeError: Layer data_mul_2528825292/copy_const not found in network

Hello,
When implementing this interesting application, I get the follwoing runtime error:
RuntimeError: Layer data_mul_2528825292/copy_const not found in network.

Details are given below.
Any help is greatly appreciated.
Environment:
OS: Ubuntu 18.04.1 LTS
Language: Python 3.6.9
OpenVINO toolkit: 2020 R1 release
Implementation steps
(venv) vax:~$ unzip pneumonia-classification-master.zip
(venv) vax:~$ cd pneumonia-classification-master/
(venv) vax:~/pneumonia-classification-master$ chmod +x setup.sh
(venv) vax:~/pneumonia-classification-master$ ./setup.sh
...
[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file: ~/pneumonia-classification-master/resources/FP32/model.xml
[ SUCCESS ] BIN file:     ~/pneumonia-classification-master/resources/FP32/model.bin
[ SUCCESS ] Total execution time: 68.01 seconds.
[ SUCCESS ] Memory consumed: 445 MB.
...
[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file:     ~/pneumonia-classification-master/resources/FP16/model.xml
[ SUCCESS ] BIN file: ~/pneumonia-classification-master/resources/FP16/model.bin
[ SUCCESS ] Total execution time: 66.36 seconds.
[ SUCCESS ] Memory consumed: 441 MB.
(venv) vax:~/pneumonia-classification-master$ source /opt/intel/openvino/bin/setupvars.sh
[setupvars.sh] OpenVINO environment initialized

Running the application
(venv) vax:~/pneumonia-classification-master$ cd application/
(venv) vax:~/pneumonia-classification-master/application$ python3 pneumonia_classification.py -m ../resources/FP32/model.xml
[ INFO ] Loading network files:
        ../resources/FP32/model.xml
        ../resources/FP32/model.bin
[ INFO ] Initializing plugin for CPU device...
[ INFO ] Reading IR...
[ INFO ] Loading IR to the plugin...
Traceback (most recent call last):
  File "pneumonia_classification.py", line 376, in <module>
    sys.exit(main() or 0)
  File "pneumonia_classification.py", line 338, in main
    infer_network.load_model_for_activation_map(bn, 0, device)
  File "~/pneumonia-classification-master/application/inference.py", line 189, in load_model_for_activation_map
    self.net.add_outputs(bn)
  File "ie_api.pyx", line 1109, in openvino.inference_engine.ie_api.IENetwork.add_outputs
RuntimeError: Layer data_mul_2528825292/copy_const not found in network

Checking that the missing Layer is present in the model.xml description
(venv) vax:~/pneumonia-classification-master/application$ grep data_mul_2528825292/copy_const ../resources/FP32/model.xml
            <layer id="365" name="data_mul_2528825292/copy_const" type="Const" version="opset1">

@edhgit
This application has been tested on OpenVINO 2019 R3 and is yet to be updated for OpenVINO 2020.


Sail run fails when host isn't specified

Running sail run scsmithr/test.com  fails with
Cloning into '/home/sean/Projects/scsmithr/test.com'...
ssh: Could not resolve hostname scsmithr: No address associated with hostname
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
2019-05-22 13:50:25 FATAL       failed to clone<EMAIL_ADDRESS>to '/home/sean/Projects/scsmithr/test.com': exit status 128

Running sail run https://github.com/scsmithr/test.com works correctly.

Had an outdated sail.toml


k_busy_wait() never returns when called - litex vexriscv soc and cpu on xilinx ac701 board

Describe the bug
k_busy_wait() never returns when called.
Hardware details
Board - Xilinx AC701
SoC - Litex
CPU - Vexriscv
To Reproduce
Steps to reproduce the behavior:

Build litex: python3 ./xilinx_ac701.py --build --csr-csv test/csr.csv --csr-json test/csr.json

Generate dts overlay: litex/litex/tools/litex_json2dts_zephyr.py --dts overlay.dts --config overlay.config csr.json

Build sample code: west build -b litex_vexriscv samples/synchronization --pristine -DDTC_OVERLAY_FILE=/Users/user/project/litex/litex-boards/litex_boards/targets/test/overlay.dts -DCONFIG_UART_LITEUART=y -DCONFIG_LITEX_TIMER=y -DCONFIG_ETH_LITEETH=n -DCONFIG_SPI_LITESPI=n -DCONFIG_I2C_LITEX=n

Load kernel onto platform: sudo lxterm /dev/ttyUSB2 --kernel zephyr_multi_thread.bin --kernel-adr 0x40000000

Load bitstream: sudo openFPGALoader -c digilent xilinx_ac701.bit

Expected behavior
Hello world printed alternatively on to console by thread a and thread b
Impact
High
Logs and console output
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2022 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs
Running on AC701 FPGA Development Board
Added Feature: Ethernet, SDCard

 BIOS CRC passed (ebbfb352)

 Migen git sha1: ac70301
 LiteX git sha1: dff33885

--=============== SoC ==================--
CPU:            VexRiscv @ 100MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            8-bit data
ROM:            62KiB
SRAM:           8KiB
L2:             8KiB
SDRAM:          1048576KiB 16-bit @ 800MT/s (CL-7 CWL-5)

--========== Initialization ============--
Ethernet init...
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Read leveling:
  m0, b00: |00000000000000000000000000000000| delays: -
  m0, b01: |00000000000000000000000000000000| delays: -
  m0, b02: |00000000000000000000000000000000| delays: -
  m0, b03: |00111111111111100000000000000000| delays: 08+-06
  m0, b04: |00000000000000000111111111111100| delays: 23+-06
  m0, b05: |00000000000000000000000000000000| delays: -
  m0, b06: |00000000000000000000000000000000| delays: -
  m0, b07: |00000000000000000000000000000000| delays: -
  best: m0, b04 delays: 23+-06
  m1, b00: |00000000000000000000000000000000| delays: -
  m1, b01: |00000000000000000000000000000000| delays: -
  m1, b02: |00000000000000000000000000000000| delays: -
  m1, b03: |01111111111111100000000000000000| delays: 07+-07
  m1, b04: |00000000000000000111111111111110| delays: 24+-07
  m1, b05: |00000000000000000000000000000000| delays: -
  m1, b06: |00000000000000000000000000000000| delays: -
  m1, b07: |00000000000000000000000000000000| delays: -
  best: m1, b04 delays: 24+-07
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB
   Read: 0x40000000-0x40200000 2.0MiB
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
  Write speed: 27.2MiB/s
   Read speed: 27.3MiB/s

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading zephyr_multi_thread.bin to 0x40000000 (114868 bytes)...
[LXTERM] Upload calibration... (inter-frame: 10.00us, length: 64)
[LXTERM] Upload complete (9.9KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
Executing booted program at 0x40000000

--============= Liftoff! ===============--
*** Booting Zephyr OS build v2.7.99-2698-gf92542ecf3e4  ***
thread_a: Hello World from cpu 0 on litex_vexriscv!

Environment (please complete the following information):

OS: Linux
Toolchain zephyr

Additional context
comment out the line number 67 k_busy_wait(1000); in the main.c file of zephyr/samples/synchronization/src/
rebuild the code and run the bin file on hardware again. The application works as expected.
Logs
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2022 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs
Running on AC701 FPGA Development Board
Added Feature: Ethernet, SDCard

 BIOS CRC passed (ebbfb352)

 Migen git sha1: ac70301
 LiteX git sha1: dff33885

--=============== SoC ==================--
CPU:            VexRiscv @ 100MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            8-bit data
ROM:            62KiB
SRAM:           8KiB
L2:             8KiB
SDRAM:          1048576KiB 16-bit @ 800MT/s (CL-7 CWL-5)

--========== Initialization ============--
Ethernet init...
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Read leveling:
  m0, b00: |00000000000000000000000000000000| delays: -
  m0, b01: |00000000000000000000000000000000| delays: -
  m0, b02: |00000000000000000000000000000000| delays: -
  m0, b03: |00111111111111100000000000000000| delays: 08+-06
  m0, b04: |00000000000000000111111111111100| delays: 23+-06
  m0, b05: |00000000000000000000000000000000| delays: -
  m0, b06: |00000000000000000000000000000000| delays: -
  m0, b07: |00000000000000000000000000000000| delays: -
  best: m0, b04 delays: 23+-06
  m1, b00: |00000000000000000000000000000000| delays: -
  m1, b01: |00000000000000000000000000000000| delays: -
  m1, b02: |00000000000000000000000000000000| delays: -
  m1, b03: |01111111111111100000000000000000| delays: 08+-07
  m1, b04: |00000000000000000111111111111110| delays: 24+-07
  m1, b05: |00000000000000000000000000000000| delays: -
  m1, b06: |00000000000000000000000000000000| delays: -
  m1, b07: |00000000000000000000000000000000| delays: -
  best: m1, b04 delays: 24+-07
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB
   Read: 0x40000000-0x40200000 2.0MiB
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
  Write speed: 27.2MiB/s
   Read speed: 27.3MiB/s

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading zephyr_multi_thread.bin to 0x40000000 (114756 bytes)...
[LXTERM] Upload calibration... (inter-frame: 10.00us, length: 64)
[LXTERM] Upload complete (9.9KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
Executing booted program at 0x40000000

--============= Liftoff! ===============--
*** Booting Zephyr OS build v2.7.99-2698-gf92542ecf3e4  ***
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!
thread_b: Hello World from cpu 0 on litex_vexriscv!
thread_a: Hello World from cpu 0 on litex_vexriscv!

https://github.com/zephyrproject-rtos/zephyr/issues/23622

Hi! Thanks for reporting this issue, we're looking into it.

Closing this one as #45196 and #45198 which fix this issue had been merged.


[SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp support to ArrowConverters for toPandas() Conversion

What changes were proposed in this pull request?
WIP started with DateType
How was this patch tested?

Test build #79693 has started for PR 18664 at commit 69e1e21.

Test build #79693 has finished for PR 18664 at commit 69e1e21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79693/
Test FAILed.

Merged build finished. Test FAILed.

Test build #79781 has started for PR 18664 at commit b709d78.

@ueshin @holdenk I think I'm seeing an issue with transferring timestamp data to Pandas with Arrow, so I'll try to explain.  Spark will assume the timestamp is in local time, so when converting to internal data, it will adjust with an offset to UTC from local timezone.  Currently, the internal data is converted to Arrow without a timezone, which Arrow takes as timezone unaware.  When a Pandas DataFrame is created from that data, it does not adjust to local time, so a different timestamp is shown.  For my case below, using PST as local time, it will add 8 hours.

In [2]: dt = datetime.datetime(1970, 1, 1, 0, 0, 1)

In [5]: TimestampType().toInternal(dt)
Out[5]:<PHONE_NUMBER>0

In [8]: df = spark.createDataFrame([(dt,)], schema=StructType([StructField("ts", TimestampType(), True)]))

In [7]: df.show()
+-------------------+
|                 ts|
+-------------------+
|1970-01-01 00:00:01|
+-------------------+

In [9]: spark.conf.set("spark.sql.execution.arrow.enable", "true")

In [10]: df.toPandas()
Out[10]:
                   ts
0 1970-01-01 08:00:01

In [11]: spark.conf.set("spark.sql.execution.arrow.enable", "false")

In [12]: df.toPandas()
Out[12]:
          ts
0 1970-01-01 00:00:01

It wasn't a problem before Arrow because the data gets converted before going into Pandas.  I believe there are a few different ways to handle this

Adjust the Spark internal data to represent local time, not UTC time, and create an Arrow field without specifying the timezone.

Give the Arrow field the timezone from DateTimeUtils.defaultTimeZone() and adjust the internal data to represent local time, not UTC time.

Give the Arrow field a "UTC" timezone, then no adjustments need to be done to the internal data but I think Pandas will still display as UTC and it would be up to the user to change timezone.

I'm not sure what the best solution is because there could be issues with them all, any thoughts?

Test build #79781 has finished for PR 18664 at commit b709d78.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79781/
Test FAILed.

Merged build finished. Test FAILed.

In my opinion, we should definitely specify the timezone to keep the correct timestamp.
I'm not sure which is the suitable one yet, but the candidates would be:

"UTC"
Spark SQL has timestamp value as the number of micros since 1970-01-01 00:00:00.0 UTC internally.
SQLConf.SESSION_LOCAL_TIMEZONE
Spark SQL represents and calculates in timezone related operations using this timezone. If there isn't the config value, the value will fallback to DateTimeUtils.defaultTimeZone().
DateTimeUtils.defaultTimeZone()
The system timezone.

Hopefully we might specify the timezone when spark.conf.set("spark.sql.execution.arrow.enable", "false"), too, but it would affect backward-compatibility?

Excited to see this being worked on.

SQLConf.SESSION_LOCAL_TIMEZONE
I like this the best. This presents timestamp in local time which is compatible with the existing toPandas() and collect().  If we really want to have the result exactly the same as the non-arrow version of toPandas, we can do sth like df[col] = df[col].dt.tz_localize(None) after getting the pandas.DataFrame from arrow.

Test build #79922 has started for PR 18664 at commit 719e77c.

Thanks @ueshin and @icexelloss.  I like SQLConf.SESSION_LOCAL_TIMEZONE also, but it ended up being pretty messy trying to fit the SQLConf into these static functions - if there is an easier way let me know.  So I just used DateTimeUtils.defaultTimeZone() for now, maybe it could be changed at a later point if needed.

, but it would affect backward-compatibility?

It does somewhat since without Arrow has no timezone, and with Arrow has the local timezone.  In order to pass the tests, I needed to do something like @icexelloss suggested to remove the timezone before comparing the pandas.DataFrames, which seems ok to me.

Test build #79922 has finished for PR 18664 at commit 719e77c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79922/
Test FAILed.

Test build #79939 has started for PR 18664 at commit 3585520.

Test build #79939 has finished for PR 18664 at commit 3585520.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79939/
Test FAILed.

Merged build finished. Test FAILed.

jenkins retest this please

Test build #79945 has started for PR 18664 at commit 3585520.

Test build #79945 has finished for PR 18664 at commit 3585520.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79945/
Test FAILed.

I'm not sure why this is failing, ping @holdenk @wesm .  This is after introducing a timezone in the Arrow data.  The error is
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests.py", line 3058, in test_pandas_round_trip
    pdf_arrow = df.toPandas()
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/dataframe.py", line 1734, in toPandas
    return table.to_pandas()
  File "pyarrow/table.pxi", line 766, in pyarrow.lib.Table.to_pandas (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:30571)
  File "/home/anaconda/envs/py3k/lib/python3.4/site-packages/pyarrow/pandas_compat.py", line 157, in table_to_blockmanager
    dtype = DatetimeTZDtype('ns', tz=item['timezone'])
TypeError: object() takes no parameters

And I think it's orginating from here but I can't reproduce this locally, even when I try to reproduce the env on Jenkins (although Jenkins seems to be using python 3.4, numpy 1.11.3, pandas 0.19.2, and pyarrow 0.4.1 - but I can only get these versions installed with python 3.5)
Any ideas?

cc @jreback, he may know more quickly. Possible it's a bug in 0.19.x, but it should be fixable

I cannot repro this; can you show what item['timezone'] is?

Test build #79972 has started for PR 18664 at commit f977d0b.

Thanks @jreback and @wesm for looking. I'm going to try one more fix and if that doesn't work then I can add some printouts to see what is going on.

Test build #79972 has finished for PR 18664 at commit f977d0b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79972/
Test FAILed.

Merged build finished. Test FAILed.

Test build #79975 has finished for PR 18664 at commit a0f912f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79975/
Test FAILed.

Merged build finished. Test FAILed.

So this is currently failing due to this issue:
ERROR: test_bbb_fail (pyspark.sql.tests.ArrowTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests.py", line 3055, in test_bbb_fail
    from pandas.types.dtypes import DatetimeTZDtype
ImportError: No module named 'pandas.types'

Which is strange because there is a 'pandas.types' module in Pandas 0.19.2.  @holdenk or @shaneknapp  is it possible to verify the version installed on these?  It failed on both worker7 and worker8, thanks!

Test build #79982 has started for PR 18664 at commit 709427d.

i'm now thoroughly confused as pandas is now showing 0.18.0 on workers 2-8,
and 0.19.2 on worker 1.
latest is 0.20.3...  should i just bump everything to that?
also, this will give me an opportunity to watch for strange version bumps.
and finally:  i'm on the road until tomorrow night helping a friend move
from CA to TX and will only be able to check intermittently.
On Wed, Jul 26, 2017 at 4:53 PM, Apache Spark QA<EMAIL_ADDRESS>wrote:

Test build #79982 has started
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79982/testReport
for PR 18664 at commit 709427d
https://github.com/apache/spark/commit/709427debdee3d92d166d5d575f6f5a947626e9a
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/18664#issuecomment-318216391, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABiDrBxeI451Vxgghzvz2a4zbch6iAb_ks5sR9FpgaJpZM4Oateu
.

nvm, i just force-installed pandas on the workers to be 0.19.2
let's see if this helps.
On Wed, Jul 26, 2017 at 6:11 PM, shane knapp ☠<EMAIL_ADDRESS>wrote:

i'm now thoroughly confused as pandas is now showing 0.18.0 on workers
2-8, and 0.19.2 on worker 1.
latest is 0.20.3...  should i just bump everything to that?
also, this will give me an opportunity to watch for strange version bumps.
and finally:  i'm on the road until tomorrow night helping a friend move
from CA to TX and will only be able to check intermittently.
On Wed, Jul 26, 2017 at 4:53 PM, Apache Spark QA<EMAIL_ADDRESS>
wrote:

Test build #79982 has started
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79982/testReport
for PR 18664 at commit 709427d
https://github.com/apache/spark/commit/709427debdee3d92d166d5d575f6f5a947626e9a
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/18664#issuecomment-318216391, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABiDrBxeI451Vxgghzvz2a4zbch6iAb_ks5sR9FpgaJpZM4Oateu
.

great, thanks @shaneknapp !  that should be fine

Test build #79982 has finished for PR 18664 at commit 709427d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79982/
Test FAILed.

Jenkins retest this please
On Jul 26, 2017 6:54 PM, "UCB AMPLab"<EMAIL_ADDRESS>wrote:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79982/
Test FAILed.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/18664#issuecomment-318235381, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEUwdXGKPKe2-xsehC_xK4bGOmYKqK8Oks5sR-3PgaJpZM4Oateu
.

jenkins retest this please

Test build #79987 has started for PR 18664 at commit 709427d.

Test build #79987 has finished for PR 18664 at commit 709427d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79987/
Test FAILed.

Test build #80008 has started for PR 18664 at commit 3b83d7a.

Test build #80008 has finished for PR 18664 at commit 3b83d7a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ArrowWriter(val root: VectorSchemaRoot, fields: Array[ArrowFieldWriter])

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80008/
Test FAILed.

@ueshin @icexelloss I'm not sure SESSION_LOCAL_TIMEZONE really makes things better for this.  The issue I see is that it does nothing when importing/exporting data from Spark DataFrames, it only pertains to timestamp data while in the context of Spark SQL.  Let me show an example, where my local timezone is "America/Los_Angeles" and I set the session tz to "New_York".
In [1]: import datetime
   ...: from pyspark.sql.types import *
   ...: spark.conf.set("spark.sql.session.timeZone", "America/New_York")
   ...: spark.conf.set("spark.sql.execution.arrow.enable", "false")
   ...: dt = datetime.datetime(1970, 1, 1, 0, 0, 1)
   ...: TimestampType().toInternal(dt)
   ...:
Out[1]:<PHONE_NUMBER>0  # this is still offset to "Los_Angeles"

In [2]: df = spark.createDataFrame([(dt,)], schema=StructType([StructField("ts", TimestampType(), True)]))
   ...: df.show()
   ...:
+-------------------+
|                 ts|
+-------------------+
|1970-01-01 03:00:01|
+-------------------+
#  this displays correctly

In [3]: df.collect()
Out[3]: [Row(ts=datetime.datetime(1970, 1, 1, 0, 0, 1))]
# Spark does not pass the tz info on collect

In [4]: df.toPandas()
Out[4]:
                   ts
0 1970-01-01 00:00:01
# if this had used arrow with session tz, it would display 1970-01-01 03:00:01

When importing the timestamp, it still assumes that it is from my local timezone.  When calling show() it displays with the session timezone, but with collect() or toPandas() it is ignored and just converts back to my local timezone.  So I think this doesn't really improve much and still leads to inconsistent behavior.  Do you guys agree?

It looks like "SESSION_LOCAL_TIMEZONE" is not respected in most of the pyspark functionality.
I think df.collect() and df.toPandas can be fixed to respect SESSION_LOCAL_TIMEZONE.
TimestampType().toInternal(dt) is tricky because it doesn't have a reference to the SQLConf object, we should maybe deprecate this method?
I think we can also create separate Jira to address these, this PR can fix just the Arrow path. @BryanCutler and @ueshin do you think?

I agree with @icexelloss that we should use SQLConf.SESSION_LOCAL_TIMEZONE so I sent a pr to use it to @BryanCutler.
And hopefully we should use it for df.collect() and df.toPandas as well.
I'm wondering it would affect backward-compatibility as I mentioned before, though.

@ueshin I amd +1 for fixing df.collect() and df.toPandas(), I don't think it is much of a backward-compatibility issue because the current behavior of df.collect() and df.toPandas() is broken when SESSION_LOCAL_TIMEZONE is set. The fact that no pyspark users complain that it doesn't work as expected is probably because it's not used widely in pyspark. IMHO we should fix this as soon as possible.

The root of the problem is that SESSION_LOCAL_TIMEZONE is not used in toInternal/fromInternal in the timestamp type here.  It's the same issue in Scala with the Timestamp encoder that uses DateTimeUtils.fromJavaTimestamp/toJavaTimestamp.  Fixing these is well beyond the scope of this PR, is it planned to be fixed elsewhere?
Given that this will cause problems with importing/exporting Arrow data, and that is a big part of the use case for Arrow, what exactly is using SESSION_LOCAL_TIMEZONE going to help with until the above is fixed?  I'm fine with putting in the PR from @ueshin if that is what everyone prefers, but I just want to understand a little more about the intent.

I don't think Scala/Java Timestamp encoder has the same issue because java.sql.Timestamp always has the timestamp value from 1970-01-01 00:00:00.0 UTC regardless of timezone as the same as Spark SQL. We can safely roundtrip the values.
Btw, maybe there are 2 issues here.
The one is what the Arrow data should be and the other is how PySpark should handle the Arrow data.
As for issue 1, what the Arrow data should be, I believe we should use session local timezone for the timezone of ArrowType.Timestamp type because Spark SQL is handling the value as if in the timezone.
As for issue 2, how PySpark should handle the Arrow data, as we get the timestamp value as timezone-aware, we can handle it any ways such as converting the timezone to system timezone, localizing the timezone to None as the tests are doing, or leaving it as is.

I don't think Scala/Java Timestamp encoder has the same issue

Scala and Python handle Timestamps the same way, they both store internally as time from 1970-01-01 00:00:00.0 UTC and conversion to/from internal is done with local time, even with SESSION_LOCAL_TIMEZONE set.  here is an example, create internally as EST timestamp, but when collect, it is PST.
scala> spark.conf.set("spark.sql.session.timeZone", "America/New_York")

scala> val ds = spark.range(3).withColumn("ts", current_timestamp())
ds: org.apache.spark.sql.DataFrame = [id: bigint, ts: timestamp]

scala> ds.show(truncate=false)
+---+-----------------------+
|id |ts                     |
+---+-----------------------+
|0  |2017-08-01 14:21:31.386|
|1  |2017-08-01 14:21:31.386|
|2  |2017-08-01 14:21:31.386|
+---+---------------------+

scala> dss.select("ts").collect()
res6: Array[org.apache.spark.sql.Row] = Array([2017-08-01 11:22:23.2], [2017-08-01 11:22:23.2], [2017-08-01 11:22:23.2])

For the 2 issues you mentioned

I'm fine with Arrow data stored with session local timezone.  I think it makes things a little more confusing for the user since it is not used when import/exporting data, but it doesn't cause any big problem.  The issues I'm bringing up are really with how Spark handles timestamps in general and not really related to what we do with Arrow data.  So hopefully they can be addressed later.

Once Python receives Arrow data, I think it's best to leave as is for performance reasons so that pyarrow can just read the buffers without any further conversions.

For item 2, in Arrow-land if the data is time zone aware, then it must be internally normalized to UTC. Conversions are therefore metadata-only operations and do not require any computation. The problem arises if you have values which are not UTC.
To make this concrete, let's look at pandas:
In [4]: import pandas as pd

In [5]: val = pd.Timestamp(0)

In [6]: val
Out[6]: Timestamp('1970-01-01 00:00:00')

In [7]: val.value
Out[7]: 0

In [8]: val_utc = val.tz_localize('utc')

In [9]: val_utc.value
Out[9]: 0

Here, if you make the tz-naive timestamp 0 timezone-aware by localizing to UTC, the value is unaltered.
Converting to Eastern time zone does not alter the integer value:
In [10]: val_nyc = val_utc.tz_convert('America/New_York')

In [11]: val_nyc.value
Out[11]: 0

In [12]: val_nyc
Out[12]: Timestamp('1969-12-31 19:00:00-0500', tz='America/New_York')

If you have NAIVE timestamps, but localize to something other than UTC, then the values are altered. Suppose we have now the value 0 again and localize to NYC:
In [13]: val = pd.Timestamp(0)

In [14]: val
Out[14]: Timestamp('1970-01-01 00:00:00')

In [15]: val_nyc = val.tz_localize('America/New_York')

In [16]: val_nyc
Out[16]: Timestamp('1970-01-01 00:00:00-0500', tz='America/New_York')

In [17]: val_nyc.value
Out[17]:<PHONE_NUMBER>0000

It is fine to send session-local time zone aware data to Python via Arrow. We should definitely avoid a conflation between tz-naive values (which Arrow and Python will display as though they were UTC) and tz-aware (but session local time zone)

To Wes's concern, I think we are only dealing with values in UTC here, both Spark and Arrow internally represents timestamp as microseconds since epoch.
To the two issues Bryan and Ueshin brought up:
Issue 1:
I agree with Ueshin we should stick to SESSION_LOCAL_TIMEZONE.
Bryan brought up a good point there in pyspark df.toPandas(), df.collect() and the python udf (through Timestamp.fromInternal) doesn't respect SESSION_LOCAL_TIMEZONE and therefore is confusing and inconsistent with Spark SQL behavior such as df.show(). Since it's going to be either inconsistent with Spark SQL (df.show()) or inconsistent with PySpark (i.e., the default df.toPandas()), I'd rather we do the right thing (by using SESSION_LOCAL_TIMEZONE) and fix other PySpark behavior separately.
Issue 2:
I agree with Bryan that we leave the timezone as is.
I don't think there is performance issue because like Wes mentioned, it's just metadata operation. I think converting it back to system timezone defeat the purpose of using session timezone and throwing away the tzinfo seems unnecessary.

Test build #80134 has started for PR 18664 at commit fee197b.

I merged your changes @ueshin , but having timezone as an Option this way makes me a little nervous.  It will be easy for people to omit it and in doing so won't cause an immediate failure, but lead to small discrepancies that might not be obvious.  For instance, I think ArrowWriter.create(schema: StructType) probably needs an Option here but it wouldn't cause any failures, it would just use DateTimeUtils.defaultTimeZone() instead.
I think if we are going to use SESSION_LOCAL_TIMEZONE in Arrow timestamp data, then it should be more heavily enforced to not allow anything else.  I have an alternative to your changes that will throw an exception if a TimestampType is used for Arrow conversions without first attaching the session local time zone.  It goes about it a little differently, so please take a look and if you still prefer using Options, then I can revert it, thanks.

Test build #80136 has started for PR 18664 at commit a5faa75.

Test build #80134 has finished for PR 18664 at commit fee197b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80134/
Test PASSed.

Test build #80136 has finished for PR 18664 at commit a5faa75.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80136/
Test FAILed.

Test build #80167 has started for PR 18664 at commit 57d388a.

Sorry @ueshin, I forgot to push the changes described in my last comment, please take a look when you can.

Test build #80168 has started for PR 18664 at commit d8278a7.

Test build #80169 has started for PR 18664 at commit c808020.

Test build #80169 has finished for PR 18664 at commit c808020.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80169/
Test FAILed.

Test build #80167 has finished for PR 18664 at commit 57d388a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #80168 has finished for PR 18664 at commit d8278a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80167/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80168/
Test PASSed.

Test build #80180 has started for PR 18664 at commit aff4abf.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80180/
Test FAILed.

Hi @ueshin , do you have an idea on how to proceed here?

@BryanCutler I'm sorry for the delay.
I think it's too strict as an API to use SparkSession to apply timezone. How about throwing an exception instead of using DateTimeUtils.defaultTimeZone() when timeZoneId is None? In that case, we should use String without default parameter instead of Option[String] = None for timeZoneId?

I'm ok with that @ueshin , I'll revert back to the PR you made then remove the default value and throw exception if there is a TimestampType and timeZoneId is `None.

Test build #80646 has started for PR 18664 at commit 7dbdb1f.

Test build #80646 has finished for PR 18664 at commit 7dbdb1f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80646/
Test PASSed.

@ueshin @HyukjinKwon , I think it would be critical for users to have timestamps working for Arrow. Just to recap, the remaining issue here was that toPandas() without Arrow does not have timestamps with a timezone. Is it possible we can document that difference and not hold this up for fixing the case without Arrow?  Arrow is still disabled by default, so the default behavior of toPandas() does not change.

I agree with Bryan. I think we might want to rethink the assumption that
toPandas result with arrow / without arrow should be 100% the same.
For instance, non-Arrow doesn't respect session local timezone, and if for
compatible reasons that we cannot fix this in the non-Arrow version, then
let's fix it in the Arrow version and document the difference. IMHO,
keeping a new feature bug compatible of the existing feature is not
necessarily, fixing non ideal behavior in the new feature it provides a
migration path off the buggy behavior of existing feature.
On Thu, Oct 5, 2017 at 7:34 PM Bryan Cutler<EMAIL_ADDRESS>wrote:

@ueshin https://github.com/ueshin @HyukjinKwon
https://github.com/hyukjinkwon , I think it would be critical for users
to have timestamps working for Arrow. Just to recap, the remaining issue
here was that toPandas() without Arrow does not have timestamps with a
timezone. Is it possible we can document that difference and not hold this
up for fixing the case without Arrow? Arrow is still disabled by default,
so the default behavior of toPandas() does not change.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/18664#issuecomment-334620292, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAwbrGUKC2aM-9gfFvLlw_SN2SUwrFQQks5spWehgaJpZM4Oateu
.

I think all of the involved reviewers agree this is a pretty serious design issue. We are unable to change the behavior after we officially release it. Thus, we have to be very very careful about the design.
Can anyone first write user facing documents to explain the proposal and the potential impact?
Let us list a few typical scenarios
Given the users/applications containing the Timestamp in their Dataset and their processing algorithms have the corresponding logics:

For the new users/applications, they first enabled Arrow and later hit an Arrow bug? Can they simply turn off spark.sql.execution.arrow.enable? If not, what should they do?

For the existing users/applications, they want to utilize Arrow for better performance. Can they just turn on spark.sql.execution.arrow.enable? What should they do?

Thanks @gatorsmile for the constructive feedback!
I don't want to make this more complicated but I also want to make sure we are aware that there is also difference between Arrow/non-Arrow version when treating array and sstruct type:
Array:
non-Arrow:
In [47]: type(df2.toPandas().array[0])
Out[47]: list

Arrow:
In [45]: type(df2.toPandas().array[0])
Out[45]: numpy.ndarray

Struct:
Arrow:
In [35]: type(df.toPandas().struct[0])
Out[35]: pyspark.sql.types.Row

non-Arrow:
In [37]: type(df.toPandas().struct[0])
Out[37]: dict

I think there should be a high level doc capturing all differences between Arrow/non-Arrow version.
Unfortunately I cannot commit much time until Nov but I am happy for help with review and discussion.

cc @ueshin

The baseline should be (as said above): Internal optimisation should not introduce any behaviour change, and we are discouraged to change the previous behaviour unless it has bugs in general.
So, the issue is the previous behaviour without-Arrow does not respect SESSION_LOCAL_TIMEZONE but we should respect; however, this is a behaviour change so we should discuss further and possibly introduce a configuration to control this to prevent the behaviour change. Also, to be clear, I assume we admit not respecting SESSION_LOCAL_TIMEZONE is not a bug but a behaviour change to be fixed as an improvement, right?
How about matching the behaviour with-Arrow to the previous behaviour without-Arrow (not respecting SESSION_LOCAL_TIMEZONE) first and then separately make a PR to introduce a configuration to respect SESSION_LOCAL_TIMEZONE for both?
I actually think I a little bit doubt about respecting SESSION_LOCAL_TIMEZONE when we collect() or toPandas(), and want to separate this discussion into another PR, JIRA or mailing list if possible.

The baseline should be (as said above): Internal optimisation should not introduce any behaviour change, and we are discouraged to change the previous behaviour unless it has bugs in general.

I am not sure if I totally agree with this.
Take the struct  for instance, in the non-Arrow version, struct is turned to pyspark.sql.Row object in toPandas(). I wouldn't call this bug because it's design choice. However, this is maybe not the best design choice because if the user pickle the pandas.DataFrame to a file and send it to someone, the receiver won't be able to deserialize this pandas.DataFrame without having the pyspark library dependency.
Now I am not trying to argue we should or should not turn struct column to pyspark.sql.Row, my point is that there might be some design choices in the non-Arrow versions that are not ideal and maybe that's not a hard requirement to make behavior 100% the same between non-Arrow and Arrow version.

Yup, I admit there could be some exceptions (there have been actually) but that should still be the baseline we should basically pursue. Probably, we could treat this Arrow optimisation as an evolving configuration option for now due to those differences but the final goal should be the matched behaviour.

I agree. I think some high level document describing these differences so we can discuss it. I think we should be more careful about Arrow-version behavior before releasing support for timestamp and nested types.

If we all agree on the necessity of a design doc first, I can create a Jira and we can make progress there.
What do you all think? @BryanCutler  @gatorsmile @HyukjinKwon

I am okay with proceeding separately for dealing with timezone, and matching the behaviour with Arrow to the existing behaviour without Arrow here with respect to timezone.
Less sure yet if we need a design doc for dealing with timezone but will follow if other committers feel so.

Thanks all for the discussion. I think there are a lot of subtleties at play here, and what may or may not be considered a bug can depend on the users intent. Regardless, I agree that there needs to be user facing documentation that will address these details, such as the questions posed by @gatorsmile above. I can create a JIRA for this if @icexelloss hasn't already.

I am okay with proceeding separately for dealing with timezone, and matching the behaviour with Arrow to the existing behaviour without Arrow here with respect to timezone.

@HyukjinKwon so you are suggesting to not use a timezone for Arrow timestamps?  We discussed that earlier in this PR, but maybe that is the best solution for now

Made SPARK-22221 for user doc, once we decide what to do with timestampes it can be completed

Yup, that's what I suggested. To me, it sounds few issues are convoluted here and want to proceed what we are clear for now separately.

I'm sorry for the delay.
I agree with @HyukjinKwon's suggestion to keep the behavior of current toPandas without Arrow for now.

@HyukjinKwon and @ueshin so with Arrow, the Pandas DataFrame from toPandas() timestamp columns will not have a timezone - are we going to do the same thing for pandas_udf Series?  I was planning on broadening the scope of this PR to include pandas_udfs now that they are there.

Yup, I think we already don't have timezone in udf too? I think we are fine as long as it keeps the existing behaviour. Let's don't forget to handle all those cases when we deal with timezone in a separate PR.

Ok sounds good.  Could I get some opinions on the best way to convert internal Spark timestamps since they are stored as UTC time?  I think we have the following options:

Write Arrow data with SESSION_LOCAL timestamp (as is currently in this PR), then convert to local timezone without timestamp in Python after the data is loaded into Pandas.  This would be at the end of toPandas() or just before the user function is called in pandas_udfs, and convert back to UTC again just after.

Convert Spark internal data to local timezone without timestamp in Scala and write to Arrow data as timezone naive.

With (1) it's easy to do the conversion with Pandas, but we have to make sure it gets done at multiple places.  With (2), it's just in one spot but I'm not sure if it's possible to end up doing the conversion more than once

I think I prefer 1. Do you maybe have a preference @ueshin? I believe you are more insightful in this.

@BryanCutler, BTW, do you think it is possible to de-duplicate timezone handling within Python side if we go for 1.?

Write Arrow data with SESSION_LOCAL timestamp (as is currently in this PR)

BTW, could we just use DateTimeUtils.defaultTimeZone() instead of SQLConf.SESSION_LOCAL_TIMEZONE if you really meant it? I think the previous behaviour without Arrow does not count SQLConf.SESSION_LOCAL_TIMEZONE?

I'd say I prefer 1, too. I'm just wondering what if we use timestamp in nested types. Currently we don't support nested types but in the future?

BTW, do you think it is possible to easily de-duplicate timezone handling for both with-Arrow and without-Arrow within Python side if we go for 1. in the separate PR?

@HyukjinKwon , are you referring to how we might handle timestamps with timezones in a future PR?  I think doing this for non-arrow would be a bit more complicated than just the Arrow cases, and I'm not sure the best way to handle it unless we just make a new TimestampWithTimezone data type..

I'm just wondering what if we use timestamp in nested types. Currently we don't support nested types but in the future?

I'll try to take this into account, or at least add a note for timestamps in nested types

BTW, could we just use DateTimeUtils.defaultTimeZone() instead of SQLConf.SESSION_LOCAL_TIMEZONE if you really meant it? I think the previous behaviour without Arrow does not count SQLConf.SESSION_LOCAL_TIMEZONE?

@ueshin do you agree to just use DateTimeUtils.defaultTimeZone()? Most places where Arrow is used this would be the case.  The only scenario I can think of that would not use it is if the user generated timestamps with SparkSQL and then called toPandas() or ran a pandas_udf.

I'll work on doing (1) to have conversions in Python for Arrow to match Non-Arrow and we can see how that turns out.

Test build #82611 has started for PR 18664 at commit d8bae0b.

Test build #82611 has finished for PR 18664 at commit d8bae0b.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82611/
Test FAILed.

Test build #82612 has started for PR 18664 at commit 36f58b1.

@HyukjinKwon @ueshin , please take a look.  This should handle timestamps with Arrow the same as without Arrow.  I still need to add some tests for timestamps with pandas_udfs.

Test build #82612 has finished for PR 18664 at commit 36f58b1.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
          s\"$name is not a subclass of $
case class FlatMapGroupsInPandas(
    .doc(\"List of class names implementing QueryExecutionListener that will be automatically \" +
case class FlatMapGroupsInPandasExec(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82612/
Test FAILed.

Test build #82613 has started for PR 18664 at commit c4fd5ae.

Test build #82613 has finished for PR 18664 at commit c4fd5ae.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82613/
Test FAILed.

Merged build finished. Test FAILed.

I disagree with using DateTimeUtils.defaultTimeZone() for the timezone.
If DateTimeUtils.defaultTimeZone() is different from system timezone in Python, the return values are different between with-Arrow and without-Arrow.
Current without-Arrow seems to respect system timezone in Python, so we should respect it and keep the timezone for Arrow type what it should be.
Let me show the example (similar to https://github.com/apache/spark/pull/18664#issuecomment-318791052):
I'm in Japan, so DateTimeUtils.defaultTimeZone() would be "Asia/Tokyo", and modify os.environ['TZ'] to "America/Los_Angeles" and session timezone to "America/New_York".
>>> import datetime
>>> import time
>>> from pyspark.sql.types import *
>>>
>>> os.environ['TZ'] = "America/Los_Angeles"
>>> time.tzset()
>>>
>>> dt = datetime.datetime(1970, 1, 1, 0, 0, 1)
>>> dt
datetime.datetime(1970, 1, 1, 0, 0, 1)
>>>
>>> spark.conf.set("spark.sql.session.timeZone", "America/New_York")
>>> df = spark.createDataFrame([(dt,)], schema=StructType([StructField("ts", TimestampType(), True)]))
>>> df.show()
+-------------------+
|                 ts|
+-------------------+
|1970-01-01 03:00:01|
+-------------------+

>>> spark.conf.set("spark.sql.execution.arrow.enabled", "false")
>>> df.toPandas()
                   ts
0 1970-01-01 00:00:01

Current HEAD (c4fd5ae1e61d118877151d713e8ea0e53a7afaf4) + fix compilation error:

>>> spark.conf.set("spark.sql.execution.arrow.enabled", "true")
>>> df.toPandas()
                   ts
0 1970-01-01 03:00:01

Use DateTimeUtils.defaultTimeZone() as the timezone:

>>> spark.conf.set("spark.sql.execution.arrow.enabled", "true")
>>> df.toPandas()
                   ts
0 1970-01-01 17:00:01

Adjust using system timezone in Python:

>>> spark.conf.set("spark.sql.execution.arrow.enabled", "true")
>>> df.toPandas()
                   ts
0 1970-01-01 00:00:01

A patch I used here is:
 def _localize_series_timestamps(s):
     """ Convert a tz-aware timestamp to local tz-naive
     """
-    return s.dt.tz_localize(None)
+    return s.dt.tz_convert('tzlocal()').dt.tz_localize(None)

That's a great explanation. I think you are right. Using SQLConf.SESSION_LOCAL_TIMEZONE makes much more sense to me now.

Test build #82641 has started for PR 18664 at commit d1617fd.

Thanks @ueshin , I agree it is better to convert the timezone to Python system local first and then localize to make tz-naive in case the Python system local tz is different that DateTimeUtils.defaultTimeZone().  I'll apply your patch.

Test build #82641 has finished for PR 18664 at commit d1617fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82641/
Test PASSed.

Test build #82650 has started for PR 18664 at commit efe3e27.

I think I sort of have things working now the way we discussed. Working with timestamps in toPandas() was pretty straightforward, but there are some differences with them in pandas_udf and there are still a couple issues which I'll point out.

Test build #82650 has finished for PR 18664 at commit efe3e27.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82650/
Test FAILed.

Test build #82826 has started for PR 18664 at commit 9894519.

Test build #82826 has finished for PR 18664 at commit 9894519.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82826/
Test FAILed.

Merged build finished. Test FAILed.

Test build #82832 has started for PR 18664 at commit a3ba4ac.

Test build #82832 has finished for PR 18664 at commit a3ba4ac.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82832/
Test FAILed.

Jenkins, retest this please.

Test build #82836 has started for PR 18664 at commit a3ba4ac.

Test build #82836 has finished for PR 18664 at commit a3ba4ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82836/
Test PASSed.

Test build #82862 has started for PR 18664 at commit e428cbe.

Test build #82862 has finished for PR 18664 at commit e428cbe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82862/
Test PASSed.

Merged build finished. Test PASSed.

LGTM too

I'd cc @gatorsmile, @cloud-fan for final check.

Will take another look tomorrow. Thanks for your work! @BryanCutler

Sorry for joining this discussion so late. My 2 cents:
Spark SQL only has timestamp without timezone, and internally we use a java long to store the value of a timestamp. This is OK as timestamp is the milliseconds since Unix epoch which is timezone independent.
However the timezone information is needed when doing some calculation like adding a day to a timestamp or converting a timestamp to a string. At this time, Spark SQL will pick SESSION_LOCAL_TIMEZONE instead of java local timezone(TimeZone.getDefault()). BTW if Spark supports timestamp with timezone, then the timezone carried by the timestamp value should be picked.
One special case is Dataset.collect, Spark converts internal row to external row, and thus converts the long value of timestamp column to java.sql.Timestamp. At this point, timezone information is not needed as java.sql.Timestamp is also timezone independent. Ideally users should use SimpleDateFormat to print java.sql.Timestamp with a specific timezone, but if they just call java.sql.Timestamp.toString, the java local timezone is picked and they will get inconsistent result if the java local timezone is different from spark SESSION_LOCAL_TIMEZONE.
For pyspark DataFrame.collect, we have a similar issue. When we convert the long value of timestamp column to python timestamp, we use datetime.datetime.fromtimestamp which respects the python local timezone. So users will also get inconsistent result if the python local timezone is different from spark SESSION_LOCAL_TIMEZONE.
DataFrame.toPandas is a little different. I think we are able to set the timezone of pandas DataFrame to spark SESSION_LOCAL_TIMEZONE instead of sticking with python local timezone. We can create a new JIRA to resolve it.
Looking back to this PR, I think we should follow the existing rule: we should write tz-naive timestamp(the long value in Spark SQL) to arrow. Then at the python side we create pandas dataframe with this tz-naive timestamp. This behavior is exactly same with non-arrow-optimized toPandas.

Thanks @cloud-fan. To sum up the current state of this PR: timestamps are written to Arrow as values from unix epoch, in UTC (same long value as Spark SQL internal storage) and given a timezone from SESSION_LOCAL_TIMEZONE.  When Arrow data is converted to Pandas for user interaction (in toPandas and pandas_udf) timesatmps are converted to be tz-naive in local time as defined by the Python system.  This matches the behavior for these cases without using Arrow.

we should write tz-naive timestamp(the long value in Spark SQL) to arrow. Then at the python side we create pandas dataframe with this tz-naive timestamp
Are you saying we should create pandas with UTC time, or write Arrow data as local time?  The former would cause the user to see UTC time instead of local, and the latter would require a conversion in Scala and might not match the Python system timezone.

BTW if Spark supports timestamp with timezone, then the timezone carried by the timestamp value should be picked

+1 for Spark supporting this, I think it would make things easier code-wise and for the user to understand.

@cloud-fan Unfortunately, tz-naive Arrow and the current toPandas() without Arrow make different result.
For example, let's say we use "America/Los_Angeles" as session timezone and have a timestamp value<PHONE_NUMBER>0 which means "1970-01-01 00:00:01" in the timezone. Btw, I'm in Japan so python local timezone would be "Asia/Tokyo".
The timestamp value from current toPandas() without Arrow is created from the following value:
>>> ts =<PHONE_NUMBER>0

>>> dt = datetime.fromtimestamp(ts // 1000000).replace(microsecond=ts % 1000000)
>>> pd.DataFrame({'ts': [dt]})['ts']
0   1970-01-01 17:00:01
Name: ts, dtype: datetime64[ns]

As you can see, the value becomes "1970-01-01 17:00:01" because it respects python timezone.
On the other hand, the timestamp value from tz-naive Arrow is created from something like the following Arrow table:
>>> naive = pa.Table.from_arrays([pa.Array.from_pandas(pd.Series(pd.Timestamp((ts -<PHONE_NUMBER>0) * 1000)))], ['ts'])
>>> naive.to_pandas()['ts']
0   1970-01-01 00:00:01
Name: ts, dtype: datetime64[ns]

Here, we need to modify timestamp value by the offset<PHONE_NUMBER>0 in this case) in Spark side.
The value became "1970-01-01 00:00:01" this time.
Btw, our current implementation in this pr can make the same result as current toPandas() without Arrow:
>>> aware = pa.Table.from_arrays([pa.Array.from_pandas(pd.Series(pd.Timestamp(28801000000000, tz='America/Los_Angeles')))], ['ts'])
>>> aware.to_pandas()['ts'].dt.tz_convert('tzlocal()').dt.tz_localize(None)
0   1970-01-01 17:00:01
Name: ts, dtype: datetime64[ns]

Now I have to say that I'm not yet exactly sure the behavior if the timestamp values appear in nested structure.

DataFrame.toPandas is a little different. I think we are able to set the timezone of pandas DataFrame to spark SESSION_LOCAL_TIMEZONE instead of sticking with python local timezone. We can create a new JIRA to resolve it.

Do you think the behavior of current toPandas() without Arrow is a bug and should be fixed?
There is a similar trial #18933 where we try to add a config for toPandas() without Arrow to convert python timezone values to session local timezone values and retain it as tz-aware value.
I guess we can fix the bug(?) by removing the config from the pr and forcing to convert and drop timezone info at last, and the result will be the same as tz-naive way.
Or if you respect the current behavior, we should use the current implementation in this pr.
What do you think?

Do you think the behavior of current toPandas() without Arrow is a bug and should be fixed?

I personally consider it a bug. Having the same code return different results depending on your system locale is going to be consistent source of issues for users, or, worse, a silent cause of incorrect data analysis.
In the absence of a time zone, for Python users it's hard to see any other reasonable interpretation of such data (WITHOUT TIME ZONE) than tz-naive. To localize using the system time zone is to apply tz-aware semantics (and forcing this detail on the Python user) to data that never had any.

LGTM, my only concern is, the timezoneId parameter should not be a Option[String] but just a String.

Thanks @cloud-fan , so is everyone in agreement with the current behavior of this PR?  It sounds like we will probably need a follow up discussion to address some details, but with Arrow enabled matching non-Arrow, hopefully we can move forward on this.

LGTM, my only concern is, the timezoneId parameter should not be a Option[String] but just a String.

This is fine. It's not an optional parameter but I can see how it's confusing - @icexelloss brought up the same point earlier, so I'll change it to be clear

Test build #82915 has started for PR 18664 at commit f512deb.

Test build #82915 has finished for PR 18664 at commit f512deb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82915/
Test PASSed.

Before we merging this PR, could anybody submit a PR for documenting this issue? Then, we can get more feedbacks from the others.

BTW, we might need to resolve the comments in https://github.com/apache/spark/pull/18933 and merge that PR first.

Test build #82997 has started for PR 18664 at commit 79bb93f.

I cleaned up some of the timestamp conversion code and added a test for a pandas_udf that returns a DateType which is currently causing an error.  see https://github.com/apache/spark/pull/18664#pullrequestreview-71358383

Test build #82997 has finished for PR 18664 at commit 79bb93f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82997/
Test FAILed.

@ueshin I have the requested changes done, just wondering if you could clarify https://github.com/apache/spark/pull/18664#discussion_r146472494 before I push them, thanks!

Test build #83050 has started for PR 18664 at commit 4d40893.

Before we merging this PR, could anybody submit a PR for documenting this issue? Then, we can get more feedbacks from the others.

@gatorsmile I started a wip at #19575 for adding documentation, nothing there yet but I'll work on it

Test build #83050 has finished for PR 18664 at commit 4d40893.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83050/
Test PASSed.

Test build #83087 has started for PR 18664 at commit addd35f.

Test build #83087 has finished for PR 18664 at commit addd35f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83087/
Test PASSed.

@ueshin do you think this is ready to merge? cc @gatorsmile @cloud-fan for another look

LGTM

@BryanCutler Thank you for all your hard work. We really appreciate it!
Merged to master.

No problem, thanks @gatorsmile !


convert famlogger to bmlogger with nop

Add benchmark logger (bmlogger) to make transition to new logger.
Changes

Thanks to @samiwilf, add no op logger instead of conditionals.
Change name of fb5logger to bmlogger
Squashed in a stacked PR #56 for adding XLMR warmup. I really should have merged this PR and #56 in when they were both approved separately. Please skip over xlmr.py changes in this PR since those were already approved.

Could you please rebase this, I think some of these changes are already in main

Please make the same name changes to DLRM and RNN-T, I only see XLMR here.

@aaronenyeshi This is on purpose. The repo will have two loggers temporarily. fb5logger (the old one) and bmlogger (the new one). I'd like to make this conversion first as an intermediate step.


Drupal LAMP server illustration for Ubuntu 16.04, the text says 14.04

The app versions shown in the illustration appear to match a 16.04 installation, but the text says 14.04. I'm guessing this is just a spot that was missed when refreshing the book at some point.

Good catch, and I'm glad you're enjoying the book!
I just updated that line in the manuscript, and will update the version on LeanPub shortly. It will be rolled out to the other book editions soon.
I'll also make sure to add a mention crediting you for the fix in the changelog :)

Good catch, and I'm glad you're enjoying the book!

Thank you, and very much so.

I just updated that line in the manuscript, and will update the version on LeanPub shortly. It will be rolled out to the other book editions soon.

Nice. I saw on leanpub.com where they have a new version available for download. I'm still reading through the September 2018 edition and have been highlighting it extensively (Google Play Books). I'm curious about the changes in the new edition: is there a changelog for the book updates that you provide?
Thanks.


"Stuck Key" Error when launching from base

If you hold boost as you dock when you relaunch your booster will be firing until you press and release the booster button to stop it. Similar issues can be seen with normal thrusters.
Imported from trac ticket #210,  created by Xynth on 08-05-2010 at 21:48, last modified: 08-16-2010 at 04:28

If anyone knows how to consistently dupe this, please let me know.  I have a few theories but can't really test them.
Trac comment by Xynth on 08-06-2010 at 04:37

r974
This happens all the time in R5, but haven't duped in R6 yet (maybe that is because there is never a game played in R6.) Maybe this change helps, maybe some other changes fixed it.  Either way, I'll close it until proven otherwise.
Trac comment by Xynth on 08-16-2010 at 04:28


Issue-fix controller

Automated Review Comments
Issues in backend/controllers/adminController.js:

(9, 'Hardcoded string found')
Issues in backend/controllers/clientController.js:
(3, 'Hardcoded string found')
(5, 'Hardcoded string found')
Issues in backend/controllers/crudController/crudMethods.js:
(9, 'Hardcoded string found')
Issues in backend/controllers/leadController.js:
(3, 'Hardcoded string found')
(5, 'Hardcoded string found')
Issues in backend/controllers/productController.js:
(3, 'Hardcoded string found')
(5, 'Hardcoded string found')
Issues in frontend/src/components/SidePanel/index.jsx:
(13, 'Hardcoded string found')

I got it.


How close the session

How i can close the session in the webterminal?, with the letter "h" close the conection but when I update my page with "F5" the session is  connecting again.

Hello!
Web Terminal does not save it's state when you press F5: all local variables are removed.
But if you mean HTTP session, it is fully controlled by CSP gateway, and sessions are alive until you clear any cookies and browser's cache. What's the purpose of session reset?

Do you think Web Terminal should implement "session reset" somehow?
For example, I suggest adding /logout command to change the user.
Is this what you meant?
P.S. Currently I think Chrome incognito mode would allow you to do so.

do %session.Logout(1) should do the trick

Edward, do you think we need to make a terminal shortcut for this, like
/logout?
On Sun, Mar 13, 2016, 7:12 PM Eduard<EMAIL_ADDRESS>wrote:

do %session.Logout(1) should do the trick
—
Reply to this email directly or view it on GitHub
https://github.com/intersystems-ru/webterminal/issues/60#issuecomment-196003282
.

Maybe automatically send it to a server on tab close/page reload?

I think it shouldn't be send automatically. The only user I use is SYSTEM,
but I reload the tab frequently. Here I see logout as a useful thing, do
you see this also useful?
On Mon, Mar 14, 2016, 4:09 PM Eduard<EMAIL_ADDRESS>wrote:

Maybe automatically send it to a server on tab close/page reload?
—
Reply to this email directly or view it on GitHub
https://github.com/intersystems-ru/webterminal/issues/60#issuecomment-196323375
.

It's just I also see no particular intention to use it - what for?
One use case I can see is closing webterminal session on unsecured computer but that's a bad idea in the first place. @jjzamoram any thought on why we need this feature?

Closed due to discussion inactivity.
@jjzamoram, please, reopen the issue if you think we need to implement something more than just do %session.Logout(1). Thanks!

Getting back to this question, @eduard93, %session.Logout(1) doesn't seem to work. Even when I made a test REST URL handler:
ClassMethod LogOut() As %Status
{
    do %session.Logout(1)
    write "OK"
    return $$$OK
}

It responds "OK", but indeed doesn't ask me to log in after I reload the page and continues to work under the user I logged in first time. (Cache for Windows (x86-64) 2016.2 (Build 636U)) Do you know any ways to fix this?

That depends upon your security setup.

@eduard93, so is there a way to log out on minimal security installations?

After you log out, you'll still be an UnknownUser with access to a terminal, so no. You can remove roles from UnknownUser I suppose, but that might be not a good idea.

@eduard93,
Not this case:
ClassMethod LogOut() As %Status
{
    write $username
    do %session.Logout(1)
    write " -> ", $username
    return $$$OK
}

Results with _SYSTEM -> _SYSTEM

Does next request return  _SYSTEM?

Yes.

@jjzamoram, now WebTerminal v4.0.0-beta.8 supports /logout command, you can update to the latest version or download WebTerminal here.

It it solved on server or client side?
Can you please point me to  the specific commits.

Hi @eduard93,
It was a client problem, with browser's basic authentication cache. See my community question for more details.

@ZitRos thank you!


Allow optional driver for LoggingParameters

The driver field for logging parameters is optional in practice and according to the services spec

Thanks!

I don't think this should have been released in a patch release (0.7.2). This is a public field of a public type, so it's breaking downstream code written against docker-compose-types 0.7.1, such as https://github.com/k9withabone/podlet/issues/69.

@dtolnay @k9withabone thanks for this heads-up. My enthusiasm to be responsive to PR's let a couple of quality issues slip through, I do this on a very small budget of free time. Going to use cargo-semver-checks going forward, but also be more careful with my versioning.

It's a good lesson, I could have called out this would be a breaking change too. My apologies for the headache although I also appreciate the responsiveness of getting this merged.


accessible_by not returning the correct results when multiple rules are present

I have a complex data structure with a large number of abilities. I have recently added two new abilities to add permissions for two new scenarios.
Model files:
class Asset < ActiveRecord::Base
  has_and_belongs_to_many :clients
  has_and_belongs_to_many :applications
end

class Client < ActiveRecord::Base
  belongs_to :firm
  belongs_to :owner, class_name: 'User'

  has_and_belongs_to_many :assets
  has_and_belongs_to_many :borrower_applications, class_name: 'Application', join_table: :borrowers
  has_and_belongs_to_many :guarantor_applications, class_name: 'Application', join_table: :guarantors
end

class Application < ActiveRecord::Base
  has_and_belongs_to_many :assets
  has_and_belongs_to_many :borrowers, class_name: 'Client', join_table: :borrowers
  has_and_belongs_to_many :guarantors, class_name: 'Client', join_table: :guarantors
end

So my assests can be linked to either a Client or an Application, and I want my permissions to be driven by client.firm_id, so a user in the right firm should have permissions on all Assets related to the firms -> clients, or the firms -> clients -> applications where the client is either a borrower or a guarantor.
So I have set up abilities like so:
firm_id = user.current_firm.try(:id)
can :manage, Asset, applications: { guarantors: { firm_id: firm_id } }
can :manage, Asset, applications: { borrowers: { firm_id: firm_id } }
can :manage, Asset, clients: { firm_id: firm_id }

So to test this I set up a client as a borrower on an application, and linked an asset to both the client and the application (as a borrower, not a guarantor)
Results:
firm_id = User.last.current_firm.id

Asset.last.clients.first.firm_id == firm_id #=> true
Asset.last.applications.first.borrowers.first.firm_id == firm_id #=> true
Asset.last.applications.first.guarantors.first.try(:firm_id) == firm_id #=> false

a = Ability.new(User.last)

a.can?(:read, Asset.last) #=> true
Asset.accessible_by(a, :read) #=> #<ActiveRecord::Relation []>

a.can?(:destroy, Asset.last) #=> true
Asset.accessible_by(a, :destroy) #=> #<ActiveRecord::Relation []>

Manually testing each of the rules suggest that two pass and one fails, but according to the ability precedence wiki page Adding can rules do not override prior rules, but instead are logically or'ed. so I would expect the user to have permissions on the asset.
While it is very possible that I am doing something wrong I dont know why I am getting different results from the can? and accessible_by methods.
For the record it works when I comment out the first two abilities.

Looks like the accessible_by method is backwards for some queries which is quite concerning.
I have another ability defined as:
can :manage, Application, borrowers: { firm_id: firm_id }

I have now set up some rspec tests to make sure that my permissions are working as expected.
This works:
context 'when client is a borrower' do
  let(:application) { create :application, borrowers: [client] }

  it 'user should be able to manage the application' do
    expect(ability.can?(:manage, application)).to eq true
  end

  it 'accessible_by should include loan_application' do
    expect(Application.accessible_by(ability)).to include application
  end
end

This does not:
context 'when client is a guarantor' do
  let(:application) { create :application, guarantors: [client] }

  # This test passes with flying colours
  it 'user should not be able to manage the application' do
    expect(ability.can?(:read, application)).to_not eq true
  end

  # This crashes and burns
  it 'accessible_by should not include application' do
    expect(Application.accessible_by(ability, :read)).to_not include application
  end
end

I'm seeing this problem as well, but only in a pretty specific case. Given the following model:

class User < ActiveRecord::Base
   has_many :alphas
end

class Alpha < ActiveRecord::Base
   belongs_to :user
   has_one :gamma
   has_many :betas
end

class Beta < ActiveRecord::Base
   belongs_to :alpha
   has_one :gamma
end

class Gamma < ActiveRecord::Base
   belongs_to :alpha
   belongs_to :beta
end

and a permission model like so:
class Ability
  include CanCan::Ability

  def initialize(user)
    can [:read], Gamma, alpha: { user_id: user.id }
    can [:read], Gamma, beta: { alpha: { user_id: user.id } }
  end
end

when I run
ability = Ability.new(User.first)
Gamma.accessible_by(ability).count.to_sql # count is just there to silence all the 'AS' clauses in SQL

what comes out is something like (paraphrased, since I'm rewriting from my own private model):
SELECT COUNT(DISTINCT "gammas"."id") FROM "gamma" LEFT OUTER JOIN "alphas" ON "alphas"."id" = "gammas"."alpha_id" LEFT OUTER JOIN "betas" ON "betas"."id" = "gammas"."beta_id" LEFT OUTER JOIN "alphas" "alphas_betas" ON "alphas_betas"."id" = "betas"."alpha_id" WHERE (("alphas"."user_id" = 1) OR ("alphas"."user_id" = 1))

Basically, it looks like the second clause's JOIN is correctly getting a distinct name (alphas_betas), but the condition on the far side of that join isn't being rewritten with that distinct name, and is thus being evaluated in the context of the direct relation and not the join through beta. In the specific case above, I believe the WHERE clause of the query should be  WHERE (("alphas"."user_id" = 1) OR ("alphas_betas"."user_id" = 1)).

Seeing this as well. +1

I'm working up a PR for this against develop, but it's a pretty large change. To wit:
The root of the problem is fundamental to the way ActiveRecordAdapter is architected. Essentially, it builds up the conditions and the joins for a given accessible_by call (realized by a database_records call into ActiveRecordAdapter) as separate steps, after aggregating together all the relevant rules. As part of building up the conditions, the tableize_conditions method bakes the table's name right into the conditions (and then into the hand-rolled SQL that the module subsequently produces. On the join side, the list of tables to be joined against gets passed directly into Active Record's include and references methods, which don't even allow for join aliases to be defined explicitly (see this rails/rails issue and this SO question for details).
All of this ends up being more or less impossible to graft a decent solution on to. I've explored a few solutions from ideas in the above SO question, trying to divine the join alias names that rails ends up producing fairly late in the query generation process. This kinda-sorta works, but it's pretty evident that it will be flaky, hard to test, and impossible to generalize. Which brings me to my current working plan:
Basically, I'm working on an approach like the following:

Build up joins & conditions for each rule separately
Using arel, extract out the joins needed by the rule, and explicitly alias them
Again using arel, rewrite the conditions on the rule to use the aliased name
Within a master SelectManager, use arel's .or / .and / .not chaining to bind together the rules' conditions, and chain all the rules' joins together there as well.
Use that SelectManager as the source of the database_records query.

IMO, it's a much more elegant and DRY approach to the overall architecture of the adapter, but it's obviously also a pretty fundamental rewrite and will likely fix / cause a number of bugs in the project. On the plus side, there are a number of long-term benefits from this approach:

Much more testable since we're working with individual rules in the adapter.
More DRY since we lean on arel to do the boolean ops between separate rules
More general & testable since our responses are based on arel, and not on hand rolled SQL
Closer to Active Record's golden path than is hand rolled SQL.

but given the scope of this change I'd really like to seek counsel (or at least consensus) on the approach (and maybe some help! :)
Feedback / other suggestions are welcome!


$ bash start_hyperenclave.sh error

$ bash start_hyperenclave.sh
sysctl: cannot stat /proc/sys/dev/hyper_enclave/enabled: No such file or directory
rmmod: ERROR: Module hyper_enclave is not currently loaded
insmod: ERROR: could not load module ../../hyperenclave-driver/driver/hyper_enclave.ko: No such file or directory
sysctl: cannot stat /proc/sys/dev/hyper_enclave/enabled: No such file or directory
$ uname -a
Linux ubuntu 5.15.0-82-generic #91~20.04.1-Ubuntu SMP Fri Aug 18 16:24:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/issue
Ubuntu 20.04.6 LTS \n \l
$  cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
6  Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz

Please refer to supported-linux-kernel-version and install Linux kernel 5.4.


DOWN result in error thrown [ERROR] TypeError: destination.substring is not a function

We are running db-migrate-0.10.0-beta.16. When running a down migration we get an error thrown: [ERROR] TypeError: destination.substring is not a function     at isIncludedInDown (/Users/ian/Code/resolver-dev-all/resolver-object-service/node_modules/db-migrate-shared/util.js:18:37)
Debugging and Stack tracing results in destination having a number value which is passed down as count but it's expecting a string.

Probably related to an issue from this morning #432
Please try updating to the latest version and report back if this does not fix the problem for you. If it does please close this issue.

We opted to use beta12 since upgrading t beta17 or up will have more impact on our implementation. Thanks

You mean from beta16 on?
I think you refer to #417, if so, this is fine but you should be able to go to anything below beta16 until you're ready to upgrade though, you may need to invest a bit of time to upgrade if you are still using a very old migration template.

Yes from beta16 to beta12. Yes we will upgrade in the future for sure. Thanks


Merge all changes from release/2.0.0 back to dev

I'm not sure https://github.com/aspnet/Identity/pull/1485 made it back into dev. I tried merge the branches and there were a bunch of conflicts. We should look into this closer.

I'll double check that everything we wanted from #1485 made it back, but most of that change shouldn't come back to dev as it was a quirks mode fix

Ok good to know. Also, this change shouldn't come back to dev either. It was a 2.0.x infrastructure-only change. https://github.com/aspnet/Identity/commit/becf1df9c5a3b908e28ec3c1272b072d8801e3dc


Detect newer versions of MSBuild

Currently, the native-compile task is hard-coded to use MSBuild 12.0. On dev14-only machines this fails since the version there is 14.0. We should detect the latest installed version, and use it.

Closing because we are not planning to make changes to this or it is no longer needed.


Update versions

With ruby 2.7 we receive many different deprecation warnings, and indeed recent version of ruby were broken in CI.
I have:

converted URI.encode to URI.encode_www_form since it has been deprecated.
updated all of the dependent gems, including switching from fakeweb to webmock for mocking the server in tests.
updated the versions of ruby which travis tests against.

Could you please review (and merge) this Merge Request?

Could you please review (and merge) this Merge Request?


Include Disaster Recovery Plan

Write a dock on how to fully restore my system.
Include in here fonts, slack themes, Chrome, things that can't be saved in Dotfiles.

Including installing docker for mac bc there is no easy way to do this programatically.... Or automated it


Best practices for using globalize and Node.js with express

The documentation isn't super clear on this, so I was hoping to get a little clarification/confirmation on the recommended best practices for scenarios using globalize within a node.js app powered by express.
Consider the scenario where an app supports multiple locales, and needs to assign the correct one to each incoming request based on the user's chosen preference. For efficiency, it seems like the best approach is to simply create one Globalize object for each supported locale, and then attach that locale-specific instance to the req object for the request, so it's available for use throughout the various codepaths handling the request. I'm trying to avoid the overhead of having to load data for each new request, utilizing too much memory loading duplicate data (e.g., in multiple instances) and to minimize the amount of work that needs to be done when a request comes in.
Is the right way to do this something like the following?
var Globalize = require('globalize');

// Load the correct i18n content
Globalize.load(
    require('cldr/main/en/currencies.json'),
    require('cldr/main/en/ca-gregorian.json'),
    require('cldr/main/en/dateFields.json'),
    require('cldr/main/en/numbers.json'),
    require('cldr/main/en/timeZoneNames.json'),

    require('cldr/main/fr/currencies.json'),
    require('cldr/main/fr/ca-gregorian.json'),
    require('cldr/main/fr/dateFields.json'),
    require('cldr/main/fr/numbers.json'),
    require('cldr/main/fr/timeZoneNames.json'),

    // Repeat above blocks for each locale we want to support
    // ...

    require('cldr/supplemental/currencyData.json'),
    require('cldr/supplemental/likelySubtags.json'),
    require('cldr/supplemental/numberingSystems.json'),
    require('cldr/supplemental/ordinals.json'),
    require('cldr/supplemental/plurals.json'),
    require('cldr/supplemental/timeData.json'),
    require('cldr/supplemental/weekData.json')
);

var en = new Globalize("en");
var fr = new Globalize("fr");
// ... repeat for each locale we want to support

Or, is it better/possible to create the Globalize objects for each locale, and then load the locale-specific CLDR files into just that instance?
Is it necessary to call Globalize.locale() in this scenario to set a global default locale, or is that only required if one is using the global object, vs individual instances?
All of the examples I've seen only seem to show loading a single locale at a time, so perhaps expanding the docs to discuss this scenario would be helpful for others as well.

And to add to this question, what's the right way to map from a list of supported "locales" to their proper CLDR JSON files? e.g., if my app supports "en-US", "zh-TW", "fr-FR" -- what's the recommended (automatic) way to map those to the correct CLDR files to pass to load()?

Hi @dpolivy, although we've been discussing this in other places (e.g., https://github.com/rxaviers/cldrjs/issues/30), I haven't replied to this issue directly... Improving our documentation with best practices and better examples is of our interest. We're working towards it and all help is very much appreciated.

For efficiency, it seems like the best approach is to simply create one Globalize object for each supported locale

For optimal performance, you should cache your formatters and parsers per locale.

I'm trying to avoid the overhead of having to load data for each new request,

utilizing too much memory loading duplicate data (e.g., in multiple instances)

The CLDR data you load via Globalize.load() is never duplicated. The JSON is extended with unique data, but never duplicated. The formatters and parsers access this shared data when they are created (e.g., var formatter = Globalize("en").numberFormatter()).
The generated formatters and parsers keep its own copy of processed CLDR data (properties). So, var formatter1 = Globalize("en").numberFormatter() and var formatter2 = Globalize("en").numberFormatter() will obviously waste memory with duplicate properties. So, a rule of thumb is to avoid creating duplicate formatters or parsers and instead to cache them (for example, by re-using formatter created above).

Or, is it better/possible to create the Globalize objects for each locale, and then load the locale-specific CLDR files into just that instance?

The data loaded by Globalize.load() is globally accessible and unique for all instances. When you create an instance, it's assumed that any required CLDR data for such instance has been already loaded.

Is it necessary to call Globalize.locale() in this scenario to set a global default locale, or is that only required if one is using the global object, vs individual instances?

You can either use the instances or the static methods (that requires setting the global locale using Globalize.locale()). So, nope. You can use either one.

And to add to this question, what's the right way to map from a list of supported "locales" to their proper CLDR JSON files? e.g., if my app supports "en-US", "zh-TW", "fr-FR" -- what's the recommended (automatic) way to map those to the correct CLDR files to pass to load()?

Globalize should have improved docs and examples to answer to this question. But for now, anyone can find details in the conversation we're having at https://github.com/rxaviers/cldrjs/issues/30#issuecomment-121461720.
Please, just let me know if you still have any questions.
Do you have any suggestion on how we could improve our docs? Thanks

@dpolivy please could you review #435 solution provided by @devangnegandhi? (note https://github.com/devangnegandhi/globalize-express/issues/1)

+1
thanks for the library and effort you guys put into it but documentation is poor
2 or 3 tutorials with simple and slighlty more complicated workflows/use-cases would work for many/most people

Thank you @siddo420. It would be awesome if you could help us to improve documentation. Would you like to submit a PR with the tutorials? I can help in any questions with the library for that. Thanks.

Due to time constraints, I've started looking at FormatJS now.
If I am not satisfied with FormatJS, I'll come back and look at the details here again and ask any questions for tutorials.
Thanks rxaviers

Take your time and you're welcome.


delete useless "pos_sp_triplet" in commander

Delete useless uorb topic "position_setpoint_triplet" in commander.cpp

Thanks @oneWayOut. I believe I also deleted it here. https://github.com/PX4/Firmware/pull/7517

That's great, I didn't see. Thank you! @dagar


List the PackageIdentifier of the package to be added to automation
NirSoft.NirCmd
Details about the application

https://github.com/microsoft/winget-pkgs/pull/94607

Website: https://www.nirsoft.net/utils/nircmd.html

Can it detect versioning from portable applications?

I don't know. If it is the case, I will use web scraping to detect updates.

https://github.com/microsoft/winget-pkgs/discussions/116639


⚠ Bofrost.at has degraded performance

In 458751c, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1433 ms

Resolved: Bofrost.at performance has improved in 5ca6c30.


Missing pygments Styles on RTD Hosted Docs

Problem
Since the first week of May 2018 (coinciding with the release sphinx_rtd_theme 0.3.1), some projects on http://readthedocs.org no longer display syntax highlighting. While the live site no longer highlights code, the HTML and PDF downloads seem to be fine.
The HTML files do include the proper Pygments class attributes...

            html_theme = "sphinx_rtd_theme"
            html_theme_path = ["_themes", ]

...but builds on http://readthedocs.org no longer include any CSS styles for these classes.
These styles used to be included in theme.css but they were removed: https://github.com/rtfd/sphinx_rtd_theme/commit/c43d473cb576ac2d10007d4baf45a4cdf960c38d#diff-8b5d5b0b20fffbc206fe1081243fb125L4. Locally, my builds include pygments.css but this file is not being included on the live site.
Reproducible Project
I first noticed this with my own docs but I see that this is also happening in the sphinx-rtd-theme docs too:

http://datatest.readthedocs.io/en/stable/tutorial/querying-data.html#loading-data
https://sphinx-rtd-theme.readthedocs.io/en/latest/installing.html#via-git-or-download

Error Logs/Results
My builds are passing so I don't think this has anything to do with a change in my own conf.py:

https://readthedocs.org/projects/datatest/builds/

Environment Info
Since this occurs on http://readthedocs.org/, I don't have access to the environment details but it does appear to be using sphinx-rtd-theme 0.3.1.

@agjohnson seems like an error on RTD.

Maybe is the same issue as https://github.com/rtfd/readthedocs.org/issues/4050?

Your latest docs do have syntax highlighting, but your stable does not. This looks like it was fixed by pinning to a newer version of the theme.

@jessetan, yes I pinned the new version (as mentioned in the link @stsewd provided) and changed my pygments_style property.
Here are the specific changes I made:

In the documentation's conf.py file, I changed pygments_style = 'sphinx' to pygments_style = 'default'.
In the project's requirements.txt, I changed sphinx_rtd_theme to sphinx_rtd_theme>=0.3.1.

This seems to have worked for me. I'll leave closing this issue up to your discretion--I wouldn't have opened a new issue, myself, if I had seen the closed one in the readthedocs.org repo so maybe it's good to keep this open until the updated theme is used by default on Read the Docs.

This should be fixed 👍


Mobile nav items dont expand on click

I can reproduce this on desktop and ios devices running chrome. The gios-asu.github.io pages don't have this bug so its a new issue

Can reproduce, I'll look into a fix.

Not an issue with the Web Standards Wordpress Theme.  It was in issue with an out of date build for the School of Sustainability Theme.

I will email Chris Boone that it has been fixed in a bit, just want to make sure the file is not cached.

The nav works for me on the live site without clearing my cache on my phone and desktop so I think its safe to email him back.


train from checkpoint

Hi,
is there any option to continue training from a checkpoint?
I cannot find such option.

args.restrore default=True